Edit

Share via

How to enable External Data Sources (EDS) Preview?

  • Article

External Data Sources (EDS) is a capability in OSDU® that allows data from an OSDU® compliant external data source to be shared with an Azure Data Manager for Energy resource. EDS is designed to pull specified data (metadata) from OSDU-compliant data sources via scheduled jobs while leaving associated dataset files (LAS, SEG-Y, etc.) stored at the external source for retrieval on demand.

For more information about External Data Sources (EDS), see The OSDU Forum 2Q 2022 Newsletter - EDS.

Note

EDS M23 preview is now available on Azure Data Manager for Energy in Developer tier only.

Important

Limit your Identity Provider (IdP) token to read operations only.

Protect your search and delivery wrapper APIs from unauthorized access. As a best practice, ensure only Azure Data Manager for Energy can access data through your wrapper APIs. This helps with data discovery and delivery for EDS Preview.

Prerequisites

  1. Create a new or use an existing key vault to store secrets managed by OSDU® secret service. To learn how to create a key vault with the Azure portal, see Quickstart: Create a key vault using the Azure portal.

    Important

    Your key vault must exist in the same tenant and subscription as your Azure Data Manager for Energy resource.

    Tip

    When you create the key vault, select Enable purge protection (enforce a mandatory retention period for deleted vaults and vault objects).

  2. In the Access configuration tab, under Permission model select Vault access policy. Screenshot of create a key vault.

  3. Give permission to Azure Data Manager for Energy Service Principal (SPN) on key vault (existing or new). How to give access to Azure Data Manager for Energy SPN - Assign an Azure Key Vault access policy (CLI) | Microsoft Learn.

    1. Under Access Policies, select Create to create an access policy:
      1. In the Permissions tab, select the permissions. Screenshot of select permissions.
      2. Under the Principal selection pane, enter AzureEnergyRpaaSAppProd. Screenshot of create an access policy.
      3. Review + Create the access policy.

To enable EDS Preview, create an Azure support request

To enable External Data Sources Preview on your Azure Data Manager for Energy, create an Azure support ticket with the following information:

  • Subscription ID
  • Azure Data Manager for Energy developer tier resource name
  • Data partition name (the data partition in which EDS needs to be enabled for automated triggering of EDS-Scheduler)
  • Key Vault name (created in Prerequisites)

Note

Support for multiple data partitions is currently enabled for manual triggering of the EDS Ingest DAG, but this feature has not yet been implemented for the EDS Scheduler.

We notify you once EDS preview is enabled in your Azure Data Manager for Energy resource.

Known issues

  • Below issues are specific to OSDU® M23 release:
    • EDS ingest DAG results in failures when the data supplier’s wrapper Search service is unavailable.
    • EDS Dataset service response provides an empty response when data supplier’s Dataset wrapper service is unavailable.
    • Secret service responds with 5xx HTTP response code instead of 4xx in some cases. For example,
      • When an application tries to recover a deleted secret, which isn't deleted.
      • When an application tries to get an invalid deleted secret.

Limitations

The Naturalization DAG workflow won't be included in the M23 release.

FAQ

See External data sources FAQ.

How to register an external data source with Azure Data Manager for Energy?