Share via

Virtual Entities

  • Article

Interact with data from external systems using the new virtual entities

Starting with the July 2017 Update for Dynamics 365 (online), virtual entities enable the integration of data residing in external systems by seamlessly representing that data as entities in Dynamics 365, without replication of data and often without custom coding.

Virtual entities replace previous client-side and server-side approaches to integrating external data, which required customized code and suffered from numerous limitations, including imperfect integration, data duplication, or extensive commitment of development resources. In addition, for administrators and system customizers, the use of virtual entities greatly simplifies administration and configuration

A virtual entity is a definition of an entity in the Dynamics 365 platform metadata without the associated physical tables for entity instances created in the Dynamics 365 database. Instead during runtime, when an entity instance is required, its state is dynamically retrieved from the associated external system. Each virtual entity type is associated with a virtual entity data provider and (optionally) some configuration information from an associated virtual entity data source.

Records based on virtual entities are available from all Dynamics 365 (online) clients, including custom applications developed using the Dynamics 365 SDK.

Virtual entities provide these benefits:

  • End users work with the records created by the virtual entity to view the data in fields, grids, search results, and Fetch XML-based reports and dashboards.
  • System customizers can configure the data source record and create virtual entities without writing any code.
  • Developers can implement plugins to read external data using the Dynamics 365 SDK and Dynamics 365 (online) Plug-in Registration tool.

Considerations

In this release, virtual entities have a couple of restrictions:

  • Data is read-only
  • Only organization-owned entities are supported
  • Field-level security is not supported
  • It must be possible to model the external data as a Dynamics 365 entity. This means:
    • All entities in the external data source must have an associated GUID primary key
    • All entity properties must be represented as Dynamics 365 attributes - you can use simple types representing text, numbers, optionsets, dates, images, and lookups
    • You must be able to model any entity relationships in Dynamics 365

Example

In this blog post I'll walk you through a simple way of creating a virtual entity.

What you need to test this out:

  1. Access to a service exposing data in OData v4
  2. An entity in this external data source with an associated GUID primary key
  3. Access to the latest preview of Dynamics 365

On the odata.org site you will find a service you can use for testing. Just type services.odata.org/V4/OData/OData.svc/$metadata in your browser to see a list of entities

Fig. 1

If you expand a given entity branch, eg Product, you will need to check if the entity has an associated key of type GUID (requirement #2 above). Please note that this is NOT the fact for the Product entity (ref picture below). So this entity is not one we can bring in.

Fig. 2

However expanding the entity Advertisement branch, we see that this entity HAS got an associated key of type GUID, so we will use that. Note the ID is spelled "ID" (all capital letters), and also note the Name and AirDate properties - we will get back to those

Fig. 3

To see which records this data set returns we will need the collection (ie plural) name of the entity - for that we type services.odata.org/V4/OData/OData.svc in the browser, and see the collection name is Advertisements

Fig. 4

Using this information its a matter of typing services.odata.org/V4/OData/OData.svc/Advertisements in the brower to learn that the data set holds two records (these are the two external records we will surface in Dynamics 365 using the new Virtual Entity capability).

Note: You can use this OData API Explorer for a more visual representation of the data set (in its v3 edition though)

Fig. 5

 

Create a Virtual Entity Data Source

With the data source identified we can proceed to the next step - creating a Virtual Entity Data Source. In Dynamics 365 click Settings -> Administration (1) and then Virtual Entity Data Sources (2) to open the Data Sources grid.

Fig. 6

In the Data Sources grid click NEW (1) to open the Select Data Provider dialog

Fig. 7

In the Select Data Provider dialog select OData v4 Data Provider (the only option) and then click OK to open the New OData v4 Data Source dialog

Fig. 8

Fig. 9

In the New OData v4 Data Source dialog fill out the three to fields in the General section

  1. In the Name text box type a name of your choice for the data source (I'll use "Public Service")
  2. In the URL text box type or paste the URL from above (fig 4) services.odata.org/V4/OData/OData.svc
  3. In the Timeout text box (optional) type the number of seconds to wait for a response from the web service before quitting a data operation

And then click OK (4) to return to the Data Sources grid

Fig. 10

Fig. 11

 

Create a Virtual Entity

Last thing is to create a Virtual Entity to bring in data from the OData source. Click Settings -> Administration -> Customize the System 

Fig. 12

Create a virtual entity like any custom entity, and then select the Virtual Entity check box (1) - see below

Selecting the check box displays additional information requirements for the data source (2), as well as the External Name and External Collection Name values (3) for the entity definition.

So in this example the

  1. name of the new Virtual Entity is called "Advertisement" (a name I typed)
  2. name of the Data Source is "Public Service" (as per fig. 10 above)
  3. External Name = "Advertisement" and External Collection Name = "Advertisements" (as per fig. 3 and 4 above)

Fig. 13

Once the new Virtual Entity is created a couple of important things to ensure/edit in order for the mapping to work.

The system automatically creates two fields, one for the id (1) and one for the name (2). You will need to map those to the external data source names.

Fig. 14

So for the ID field (the primary key) ensure that the name in the External Name text box (1) matches the property name in the data source (2) as per figure 3 above. Observe case sensitivity.

Fig. 15

For the Name field ensure that the name in the External Name text box (1) and the Data Type (2) matches the property name and type in the data source (3)  as per figure 3 above. Observe case sensitivity and Data Type.

Fig. 16

Optionally you can create a third field to bring in the AirDate field from the data source (Date type)  as per figure 3 above. Observe case sensitivity and Data Type. If the property has Nullable = "false" like seen for the AirDate property (3), then you must set Field Requirement = Business Required (4)

Fig. 17

Create a form with the desired fields

Fig. 18

Create a view with the desired columns

Fig. 19

Publish customizations

Fig. 20

 

You now have a list with the two records from fig. 5 above

Fig. 21

Opening a record will display the form you defined

Fig. 22

I hope you will enjoy this new elegant way of surfacing data from external data sources in Dynamics 365 without the data residing in Dynamics 365. Data are mastered outside of Dynamics 365, yet available to work with in Dynamics 365 like any other entity.

Enjoy.

See Also

  • Virtual Entities Demo Solutions (download) - link
  • Virtual Entities - Relationships  - link

Comments

  • Anonymous
    September 10, 2017
    Superb article! Thanks a lot.
    • Anonymous
      September 10, 2017
      For some reason, this is failing for me in the Entity. Generic Error. The one that is embedded in the main window and not the one we get to Download the Log File from. I did all steps correctly. (Just that I have a publisher and hence a different prefix to my entity. I don't think this should be a problem)Any ideas?
      • Anonymous
        September 10, 2017
        Yes Even I am facing the same issue as Priyesh, Any idea, why this would happen ? I followed the exact same steps as the blog, but it is failing.
      • Anonymous
        September 11, 2017
        We are experiencing same as Priyesh and Somesh - followed steps to the letter but still when try to access Entity we get a generic error.We noticed that on the screenshot in the above article it wasn't clear regarding which option to select for "Pagination Mode" and it also wasn't a mandatory field. Also "Return Inline Count" wasn't shown in the screenshot.Not sure if either of the above have anything to do with the error we are seeing?Any help would be greatly appreciated!
    • Anonymous
      September 11, 2017
      The comment has been removed
      • Anonymous
        September 11, 2017
        The comment has been removed
  • Anonymous
    September 11, 2017
    Hi Jesper - what build were you using to do this? Our instances were updated at the end of last to build .1054 then further new instances with build .1054 provisioned over the weekend. We can't get these to work with Virtual Entities by following your instructions, nor the ones in the Quick Start Guide on Connect, nor with the OData endpoint listed here:https://connect.microsoft.com/DynamicsCustomerEngagement/feedback/details/3137528/bug-question-as-a-system-customizer-developer-i-am-unable-to-implement-virtual-entitiesThe lack of any meaningful error messages with Virtual Entities at the moment makes it all really difficult to figure out what is going on.
  • Anonymous
    September 11, 2017
    Great Article! However, we too get "An Error has occurred".
  • Anonymous
    September 11, 2017
    Thanks for all the comments. Between writing this post and the time of the comments, a new preview build was deployed. An issue with the OData provider will be fixed with the next build as I understand it. Please be patient.
  • Anonymous
    September 11, 2017
    Hi, thank you for the article. Wondering if it's going to be possible to define relationships between the virtual entities and "native" entities? For example.. if I wanted to surface some additional contact information in the contact records.. Would I be able to somehow display "quick view" form of the virtual entity on the contact forms?
  • Anonymous
    September 12, 2017
    The TechNet article says:"All entities in the external data source must have an associated GUID primary key". Does it mean if the external data source wont have a Guid column, the D365 platform will not read external data into Virtual Entities?
    • Anonymous
      September 14, 2017
      I believe so Khadmin yes but because of the problems with the latest build I haven't been able to verify. This seems to mean that, (for the moment at least), the use of Virtual Entities can only be extremely limited as many OData sources that you might want to connect to are unlikely to use GUIDs. as the primary key.
      • Anonymous
        October 11, 2017
        Since the GUID's won't really be important long term (since nothing gets stored in the DB), they could just be dynamically generated with a wrapper. The GUID's are just important for CRM to 'treat' it like a regular record. We have done 'virtual' records for a while via custom code on retrievemultiple plugins that talks to external data - this just makes it more accessible without needing to write plugin code. Adding a GUID attribute, or wrapping an OData service that you don't control, to generate a GUID attribute should be by far less painful than the plugin code.
  • Anonymous
    September 17, 2017
    The issue has now been fixed. One thing to notice; if a property is listed with Nullable = False then the corresponding Field Requirement must be marked as Business Required.
    • Anonymous
      October 12, 2017
      Still not working. I have followed all given steps. Pagination mode set as Client side..Any idea..?
  • Anonymous
    October 09, 2017
    Thank you for the great article.Which Authentications methods are supported for Virtual Entities? In the ver. 1612 (Build 1103), I don't see an option to specify/set up/configure authentication for the OData service.
    • Anonymous
      October 11, 2017
      I presume based on the interface for configuring data sources, your only authentication options will be to provide request parameters like client keys in the data source configuration. As long as only your system administrator can access the data source records this /may/ be enough depending on your security requirements. If your data does not have a GUID record id, you could add one, or create a wrapper for that data that does.
  • Anonymous
    October 11, 2017
    Thanks for the detailed write-up! This certainly helps with understanding how this all works. Is there a published roadmap on future plans for this capability? This thing has a lot of promise once the CRM data structure and oData constraints go away. Hopefully, that's where this is headed?
  • Anonymous
    October 11, 2017
    Is this currently working for anyone? I just get OData errors even for this simple example...on version 1612 ( 9.0.0.2072 )
    • Anonymous
      October 18, 2017
      Works for me without any problem. Just be careful about the Requirement level on the DateTime field - otherwise you'll get an following error - Type mismatch of the external attribute of type ‘System.DateTimeOffset’ expected a 'System.Nullable`1[[System.DateTimeOffset, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]]' value. Attribute Name: new_airdate
      • Anonymous
        November 02, 2017
        I followed the blog step by step and I am getting an error "Not implemented"
        • Anonymous
          November 29, 2017
          I ve got also the feedback "Not Implemented" CRM Online Version 1612 (9.0.0.2180)
  • Anonymous
    October 12, 2017
    Hi JesperI get the following error: Record is UnavailableThe requested record was not found or you do not have sufficient permissions to view it.Using Dynamics 365 v9.0.0.2072
    • Anonymous
      May 30, 2018
      Hi Jason,I am also getting - "Record is Unavailable"Adv. Find for the Virtual entity returns all data correctly, but when I click on the record, I get that error.My Dynamics 365 is at v9.0.2.193Did you find a resolution to this issue ?ThanksNeel
  • Anonymous
    October 28, 2017
    I got this output. I make it work this example
  • Anonymous
    October 30, 2017
    No luck. Not able to get the records though followed all steps. I am using Dynamics 365 online trial to check this. Is it causing the error?
    • Anonymous
      November 04, 2017
      Hi All,I can able to configure the Virtual entity using above steps. But now i want to know how do we configure the virtual entity using the Credentials? See the above WebAPI url do not need the authentication so, now if i want to connect to my another CRM instance so how can i connect it using virtual entity. Thanks!Vikrant
  • Anonymous
    November 07, 2017
    Hi there,I'm trying to see how we can register a custom data provider also, besides just a data source. The article at https://docs.microsoft.com/en-us/dynamics365/customer-engagement/developer/virtual-entities/custom-ve-data-providers#steps-to-use-a-custom-data-provider only says "Create a data provider solution" in CRM. There are no steps on how to go about this process. Any pointers? Thanks,Shahnaz
  • Anonymous
    November 07, 2017
    Hi Jesper,Could you please guide us how we can map Lookup type field from OData endpoint to Virtual entity field ?ThanksIndika
  • Anonymous
    December 05, 2017
    Hello,Is there a way to dynamically pass Request parameters? I basically have to pass few attribute values to the WebService as parameter. I tried accessing entitydatasources through WebApi in Javascript, so that I can set value for connectiondefinitionsecrets. But unfortunately I am getting Internal Server error as response. Kindly share your thoughts.Thank you.
  • Anonymous
    December 08, 2017
    Such a clean and sorted article. Thank you! Cheers.
  • Anonymous
    January 29, 2019
    Hithanks for the article, it was very nice and right to the point.I really appreciate it.However, although I followed all the steps (a few times) and I couldn't get it to work.I even try different environment (Production, sandbox, trial) but I always get the below error which looks like a generic error.0x80050263 ErrorI hope you don't mind my question: I wonder maybe you have a suggestion for me? or maybe simply this article is too old? (or I'm too late)
  • Anonymous
    May 03, 2019
    Thanks for sharing this. This is a mix of OData v3 and v4. In fact it is something that helped me to start exploring the behavior which was misleading at the outset, but the article made it very clear as to how the D365 OData v4 should be created. Many thanks for the insider information!