Manage FHIR data using data integration toolkit
The data integration toolkit helps you sync patients' PHI (protected health information) between EHR systems and Dataverse. This integration gives care teams and patients fast and secure access to their data in the Microsoft Cloud for Healthcare environment.
Your FHIR system stays separate, but its data is quickly available via Dataverse. The Dataverse healthcare APIs provide entry points to ingest and retrieve FHIR data into the Microsoft Cloud for Healthcare Dataverse database. Care providers can focus on patient needs and admins can focus on managing people, places, and clinical resources.
Features that simplify working with FHIR data
The following table describes how the data integration toolkit can simplify working with protected patient data.
| Feature | What the feature does | How it helps you manage |
|---|---|---|
| Entity maps | Entity maps associate FHIR resources with Dataverse entities. For example, the FHIR resource Patient corresponds to the Dataverse entity Contact. They're both about the same thing - patients. When you want to read or change data, the map shows the data's source, and helps the Dataverse healthcare APIs transform data between Dataverse entity records and FHIR resources. |
Instead of creating every entity map yourself, you can enable the maps you need. If a map needs a small change, adjust it, or copy it and add a tag. If you don't want to see a map anymore, you can archive it. And for special use cases, you can create a new map. |
| Attribute maps | Attribute maps associate FHIR resource elements with Dataverse attributes. For ingestion, these maps contain details on how to locate the data element and its corresponding data type in Dataverse. For retrieval, the maps describe how to transform the Dataverse data element into a FHIR resource node. These maps are enabled by default, but don't affect synchronization unless you enable their parent entity map. |
The default attribute maps provide a baseline set of details driven by the HL7 specification for the parent FHIR resource. Often, you only need to enable the parent entity map, but you can configure attribute maps much like entity maps. You can disable the maps if the data isn't relevant for the entity. You can also change from where the data is retrieved if your system differs from the FHIR specification. Attribute maps inherit their parent entity maps' tag to help you tell them apart. |
| Expansion maps | Expansion maps provide the rules for transforming JSON document based hierarchical data into Dataverse's relational data model. | Expansion maps let admins map complex JSON attributes to multiple, related Dataverse attributes. With expansion maps, you can specify parent link attributes to determine the relationship between parent and child records. |
| Logs | Data integration toolkit captures details on the Dataverse healthcare API activities and lets you view, sort, and query the transaction logs. The log entries don't contain EMR data; they only indicate what the API calls tried to do and the result. Each entry has a description and more details to enable troubleshooting for admins and developers. |
Logs are how you validate your maps, monitor the Dataverse healthcare API operations, and troubleshoot issues. For example, a log entry with the description "Missing Required Fields" indicates that the data change tried to write to a FHIR resource instance, but didn't include a value for a required element. You might see this entry if the data input form either lacks a field for entering the value or has a field that allows submission even when left empty. The log entry also includes a list of missing fields, so you can quickly find and fix the problem. |
| Azure FHIR resources | The Azure FHIR resource record defines the currently supported FHIR resources in the Microsoft Cloud for Healthcare Dataverse solution. This information is also used by the Dataverse healthcare APIs when transforming data between the two systems. | As the Dataverse data model expands, this list is updated to indicate which FHIR resources are available for integration. |
| Integration settings | The data integration toolkit includes component-specific tools for managing configuration options. These settings tell Dataverse how to recognize FHIR data changes, and where to send FHIR data transactions. They also let you toggle synchronization on or off, and set the logging level for each component. | Allows managing components individually for both new installations and existing running instances. |
| Healthcare data | When patients provide consent to share their healthcare data, admins can see it in the data integration toolkit. | Visualizing the data helps you manage the data flows, better understand the log items, and validate your data maps. |
Entity maps
Entity maps are at the highest level in the mapping process. They map FHIR resources to their corresponding Dataverse entities (also known as Dataverse tables). You can map one entity concept to multiple FHIR resources. Entity maps help the Dataverse healthcare APIs and virtual health data tables transform data between the FHIR messages and Dataverse entities. They also help Dataverse post the correct FHIR resources back to remote FHIR endpoints.
To see a list of all the entity maps, in the data integration toolkit application, select the Agent Admin navigation in the Change area section, and then select Entity Maps under Map Setup.
Entity maps included with data integration toolkit
The data integration toolkit includes many built-in entity maps, attribute maps, and FHIR element maps for standard FHIR resources that you might probably need. Because everyone's system is different, the entity maps aren't enabled during deployment.
- To enable the required entity maps, you only need to change a specific setting.
- To modify the maps to fit your specific EMR systems, the data integration toolkit maps are highly configurable.
- If you don't find a specific map to suit your needs, you can easily create your own map.
For more information on how to configure entity maps, go to Configure entity maps.
The following table lists the entity maps included with data integration toolkit. This table includes the root level FHIR resources, but not the expansion maps covered in detail in Configure expansion maps.
| Azure FHIR Resource | Dataverse Entity | Description |
|---|---|---|
| Organization | account | A formally or informally recognized grouping of people or organizations formed for achieving some form of collective action. This grouping includes groups such as companies, institutions, corporations, departments, community groups, and healthcare practice groups. |
| Patient | contact | Demographics and other administrative information about a person or animal receiving care or other health-related services. |
| Practitioner | contact | A person who is directly or indirectly involved in the healthcare provisioning. |
| AllergyIntolerance | msemr_allergyintolerance | Risk of harmful or undesirable physiological response, which is unique to an individual and associated with exposure to a substance. |
| Appointment | msemr_appointmentemr | A scheduled healthcare event for a patient and/or practitioners where a service might take place at a specific date or time. |
| CarePlan | msemr_careplan | Describes the intention of how one or more practitioners intend to deliver care for a particular patient for a time period, possibly limited to care for a specific condition or set of conditions. |
| CareTeam | msemr_careteam | Includes all the people and organizations who plan to participate in the coordination and delivery of care. |
| Claim | msemr_claim | A provider issued list of professional services and products provided (or to be provided) to a patient, which is sent to an insurer for reimbursement. |
| ClaimResponse | msemr_claimresponse | Provides the adjudication details from the processing of a claim resource. |
| Condition | msemr_condition | Used to record detailed information about conditions, problems, or diagnoses recognized by a clinician. |
| Device | msemr_device | This resource identifies an instance of a manufactured thing used in the provision of healthcare without being substantially changed through that activity. The device might be a machine, an insert, a computer, or an application. It includes durable (reusable) medical equipment and disposable equipment used for diagnostic, treatment, and research for healthcare and public health. |
| DiagnosticReport | msemr_diagnosticreport | The findings and interpretation of diagnostic tests performed on patients, groups of patients, devices, and locations, and the specimens derived from them. The report includes clinical context such as requester and provider information. It also includes a mix of atomic results, images, textual and coded interpretation, and formatted representation of diagnostic reports. |
| Encounter | msemr_encounter | An interaction between a patient and healthcare providers for providing healthcare services or assessing the health status of a patient. |
| EpisodeOfCare | msemr_episodeofcare | An association between a patient and an organization or healthcare providers, during which encounters might occur. The managing organization assumes a level of responsibility for the patient during this time. |
| Goal | msemr_goal | Describes the intended objectives of the care. |
| Group | msemr_group | Represents a defined collection of entities that might be discussed or acted upon collectively. These entities aren't expected to act collectively, and aren't formally or legally recognized. |
| Location | msemr_location | Details and position information for a physical place where services are provided and resources and participants may be stored, found, contained, or accommodated. |
| Medication | msemr_medication | Used to identify and define medication, but also covers ingredients and packaging. |
| MedicationRequest | msemr_medicationrequest | An order or request for both supply of medication and instructions for administration of the medication to a patient. The resource is called "MedicationRequest" rather than "MedicationPrescription" or "MedicationOrder". We use this term to generalize the use across inpatient and outpatient settings, including care plans, and to harmonize with workflow patterns. |
| Observation | msemr_observation | Measurements and simple assertions made about a patient, device, or other subject. |
| Procedure | msemr_procedure | An action performed on a patient. This action can be a physical activity like an operation, or less invasive like counseling or hypnotherapy. |
| RelatedPerson | msemr_relatedperson | Information about a person involved in the care for a patient. This person isn't the target of healthcare, nor has a formal responsibility in the care process. |
| RiskAssessment | msemr_riskassessment | An assessment of the likely outcomes for a patient or other subject and the likelihood of each outcome. |
| Schedule | msemr_schedule | A container for slots of time that may be available for booking appointments. |
| Slot | msemr_slot | A slot of time on a schedule that may be available for booking appointments. |
Attribute maps
Entity maps have one or more related attribute maps that map the individual elements in the FHIR resource.
To see a list of all the attribute maps, in the data integration toolkit application, select the Agent Admin navigation in the Change area section, and then select Attribute Maps under Map Setup.
These related maps provide field-by-field specific detail on which FHIR values map to which Dataverse column values. Each attribute map includes configuration values for data type mapping, and custom JSONPath snippets for selecting the correct value from the inbound FHIR JSON messages.
For more information on how to configure attribute maps, go to Configure attribute maps.
Manage map records
Map records are "solution-aware" components, which means that a component of a solution knows that it's part of that solution. Our maps know that they're part of the data integration toolkit. Because they're in Dataverse, you can use Application Lifecycle Management (ALM) to safely manage the changes you make to your mapping tables. This feature allows you to update your maps in a development environment, where you can test and validate your map changes without risking your production EHR/EMR data systems.
In the past, to transfer map changes between environments, you had to copy the data manually. Tools such as the Configuration Migration tool could help, but it was possible to make manual errors. Now, after you ensure that your updates are ready, you can safely deploy them to your production environment as a single package. ALM saves you time and helps you protect your business as it evolves and grows. For more information, go to ALM solution concepts.
You can delete any maps that you create from scratch. However, you can't delete some of the Microsoft Cloud for Healthcare default maps and keep others. You can remove all the default maps by removing the managed solution that gets installed.
Mapping records have a hierarchy with entity maps at the top level (Entity maps > Attribute maps > Attribute value maps). Dataverse doesn't allow deleting a record that has child records (related records in lower tiers). To delete an entity map, first find and delete all of its child maps, and then delete the entity map.
If you don't wish to use a default map, you can hide the map in your environment by archiving it. To archive an entity map, select the record from the grid and then select the Archive button in the command bar. When you archive a map, it automatically deactivates and disables the entity map. It also archives the entity's attribute maps and attribute value maps.
You can access archived maps by changing the view to Archived Update Service Entity Maps.
You can restore archived maps using the Restore command. Select the archived entity map from the grid, and then select Restore on the action pane.
You may undo changes to the default maps by removing the active layer of changes on the maps solution. The default maps ship as a managed solution. When you make changes to those maps, the changes apply at the active layer (unmanaged). You can use the solution layers to review the different layers and remove the active layer at any point. Removing the active layer restores the map to the last managed layer for that component. For more information, go to ALM solution layers. For steps to view the solution layers, go to View solution layers.
The latest release provides entity map tags to help you uniquely identify the maps you add to solutions. You can see the tag at the beginning of the entity map name, and the names of its attribute maps and attribute value maps. To add a tag, edit the Tag attribute in the entity map header.
Each map can only have one tag assigned at a time.
Logs
Select Logs to view the transactions occurring inside Dataverse. The logs help you understand how the data flows, why something is sent to the Dataverse healthcare API, and why something isn't sent. They provide a view into the data integration toolkit components' activities.
The logs section contains information about each service, including the custom API.
Azure FHIR resources
The Azure FHIR Resources area defines the universe of FHIR resources. Because you can't visualize the FHIR entities directly to see all the resources, the data integration toolkit provides the list of resources mapped to a Dataverse entity. The FHIR resources are primarily used in entity maps.
Integration settings
The Integration Settings area gives you access to the Dataverse environment variables that define and control the integration of Azure FHIR and Dataverse. You must set these variables to initialize the data integration toolkit, or make adjustments if something in your Microsoft Cloud for Healthcare environment changes.
For more information about the integration settings, go to Configure integration settings for Dataverse healthcare APIs and Configure integration settings for virtual health data tables.
Healthcare data
The Healthcare Data module allows map administrators to view and interact with user data flowing into the system. This feature helps the map administrators to understand and troubleshoot the data.
Map administrators can use the healthcare data module to:
- View the data.
- Understand what's occurring as they build the maps.
- Ensure that the data is coming over correctly and into the right field.
For example, to see patient information:
Select Healthcare Data.
Under Administration, select People.
Change the view to Patients FHIR View.
The map administrator can see all the patient data flowing into Microsoft Cloud for Healthcare, such as the FHIR ID of a patient, the last sync of the patient, whether sync is enabled for that record, and the Azure FHIR version of the record.
Example 1: Contact not synced
This example shows how you might have a Contact entity that doesn't sync when you try to sync a patient record that isn't enabled for sync.
Select Healthcare Data.
In the Patients FHIR View, find a patient who doesn't have an Azure FHIR ID.
Select the patient record and select Edit. Notice that the Azure FHIR Sync Enabled value is set to No.
Change the phone number for the patient and select Save.
In the Logs section:
- For the Description column for that record, select Write Back Event Handler Successfully processed.
- For the Source, select Write-Back.
- For Entity Type, select contact.
Notice that the record doesn't have an ID for FHIR Resource Id.
Example 2: Contact missing Azure FHIR details
For another example, try to sync an existing patient who doesn't have a FHIR ID.
Select Healthcare Data.
In the Patients FHIR View, find a patient who doesn't have an Azure FHIR ID.
Select the patient record and select Edit. Notice that the Azure FHIR Sync Enabled value is set to No.
Update the Azure FHIR Sync Enabled value to set it to Yes.
Change the phone number for the patient and select Save.
In the Logs section:
- For the Description column for that record, select Write Back Process Failed.
- For the Source, select Write-Back.
- For Entity Type, select contact.
The message states that the contact can't be synced to the FHIR endpoint. This behavior is because the Azure FHIR ID or other FHIR version information is missing for an existing record that's being updated. A null Azure FHIR ID indicates the contact didn't originate from the FHIR server, which is an unsupported scenario for the writeback process.
Example 3: Writeback log entries
When a contact record does have a FHIR ID and you change the record, you can see two log messages. The first message indicates that the writeback process is initiated. The record titled Write Back Event Handler Successfully processed indicates that the changes are queued for sending back to the FHIR endpoint.
When the update succeeds in posting to the FHIR endpoint, you can see the record titled Write Back Process Succeeded. If you select this description, you see the message in the log record. The message specifies the changed attribute and indicates what it sent to the FHIR endpoint.
A final log message titled Write Back FHIR Provenance indicates a final audit message posted to the FHIR endpoint about the recently posted updates.
This combination of messages indicates a successful update to the FHIR endpoint via the writeback process.
Example 4: Consent flow
In healthcare data, you might find a patient record with a FHIR ID and the Azure FHIR ID Sync Enabled value set to No. The Dataverse healthcare APIs have a consent flow. The consent flow between FHIR and Dataverse ensures that the whole universe of FHIR doesn't end up in Dataverse. Vast amount of patient data resides in FHIR and you only want this data in Dataverse when you're working with that patient.
When the FHIR sync is enabled, the Dataverse healthcare API syncs the universe of patients to Dataverse. However, it doesn't sync the entire patient data to Dataverse unless the Azure FHIR Sync Enabled setting is set to Yes. This feature is called consent flow because, typically, a patient must consent that they want their information brought into a system. When the patient consents to have their information brought into the system, the Azure FHIR Sync Enabled value is set to Yes. Then, that patient's data starts flowing into the system.
If the Azure FHIR Sync Enabled value is set to No, and you make a change to the patient's record, you see a log entry with a message that states The data cannot be sent to the Azure FHIR Server, indicating that the contact entity updates can't be pushed to the FHIR endpoint. Setting Azure FHIR Sync Enabled to No prevents the data from flowing to FHIR. It also prevents FHIR from pushing data back to Dataverse for this patient.
Consent flow is unique only to Patients as contacts. No other record entity in Dataverse has this consent flow built into it.
Expansion maps
Expansion maps in the data integration toolkit allow admins to transform FHIR data into Dataverse tables by expanding complex JSON attributes into multiple, related Dataverse records.
For example, you can use expansion maps to bring patient identifiers and patient links into Dataverse. For patient links, expansion maps allow admins to integrate patient merge requests with the following link types:
- See also
- Replaces
- Replaced by
For more information on these links types, go to FHIR link type. With expansion maps, you can specify parent link attributes to determine the relationship between parent and child records.
For more information on expansion maps, go to Configure expansion maps.