Rediger

Del via

Troubleshoot connectivity to the data processing service and telemetry endpoints

  • Artikel

Applies to: SQL Server

In addition to the usual endpoints, the Azure Arc extension for SQL Server connects to two other endpoints:

  • Data processing service (DPS) endpoint

    The collected inventory information about SQL Server instances, databases, availability groups, and usage data for billing purposes is sent to this endpoint.

  • Telemetry endpoint

    The Azure Connected Machine agent logs, the Azure extension for SQL Server logs, and the Dynamic Management Views (DMV) data is sent to this endpoint.

Communication to these endpoints uses HTTPS with SSL/TLS and port TCP/443 for encrypted secure connections. The agent initiates communication to send the data to Azure. Azure never initiates communication. Connectivity to these endpoints is therefore only one way.

When communication to these endpoints is blocked, the service has the following symptoms:

Azure extension current state

You can view the current state of the Azure extension for SQL Server in the portal. The status is refreshed every 15 minutes.

Healthy state:

Screenshot of portal for Azure extension for SQL Server in a healthy state.

Unhealthy state:

Screenshot of portal for Azure extension for SQL Server in an unhealthy state.

Check if you have a problem connecting to the DPS or telemetry endpoints

There are two ways to check if you have connectivity problems to the DPS or telemetry endpoints.

Check the Azure Extension for SQL Server status in the Azure portal

If it's connected to Azure in general, the Azure Extension for SQL Server reports its status in the Azure portal.

  • Navigate to the Machines - Azure Arc view in the Azure portal and locate the machine by name and select it.
  • Select Extensions.
  • Select WindowsAgent.SqlServer or LinuxAgent.SqlServer to bring up the details.
  • Look at the Status message and the uploadStatus value. If it's anything other than OK, there's a problem with connecting to the DPS. If it's 0, it's likely that there's a firewall blocking the communication to the DPS endpoint. There could be more details in the status message or the uploadStatus error code that can provide insights into the connectivity problem.

Check the Azure Extension for SQL Server logs

The extension log file is at:

C:\ProgramData\GuestConfig\extension_logs\Microsoft.AzureData.WindowsAgent.SqlServer\

The log file name depends on the version Azure Extension for SQL Server. For the latest version of Azure Extension for SQL Server, the log file is:

unifiedagent.log

For extension version 1.1.24724.69 and earlier, the log file is:

ExtensionLog_0.log

Check for log entries that indicate a problem connecting to the DPS or telemetry endpoints.

Probe web server endpoints

You can use various tools to probe the web server endpoints for DPS and telemetry. For example, Invoke-WebRequest or curl.

The following example uses Invoke-Webrequest:

Invoke-WebRequest telemetry.<region>.arcdataservices.com

A possible response status code is:

Invoke-WebRequest: Response status code does not indicate success: 401 (Unauthorized).

401 is expected because there is no unauthenticated route on the telemetry endpoint.

For DPS:

Invoke-WebRequest san-af-<region>-prod.azurewebsites.net

A possible response status code is:

StatusCode        : 200

StatusDescription : OK

This one should return a 200 as there is an unauthenticated route.

Probe connectivity to all regions

You can probe connectivity to all regions with the test-connectivity.ps1 PowerShell script.

#This script repeatedly probes all regions for connectivity to the Azure Arc data services/Arc-enabled SQL Server endpoints for telemetry and the data processing service.
#The script will output the status of the connectivity to the console.
#The script will run indefinitely until stopped by the user.
#The script will iterate through all regions in the $regions array.
#The list of regions are updated as of June 7,2024 to reflect all publicly available, supported Azure regions for Arc-enabled SQL Server.

$regions = @(
    "East US",
    "East US 2",
    "West US 2",
    "West US 3",
    "Central US",
    "North Central US",
    "South Central US",
    "West Central US",
    "Canada Central",
    "Canada East",
    "UK South",
    "UK West",
    "France Central",
    "West Europe",
    "North Europe",
    "Switzerland North",
    "Central India",
    "Brazil South",
    "South Africa North",
    "UAE North",
    "Japan East",
    "Korea Central",
    "Southeast Asia",
    "Australia East",
    "Sweden Central",
    "Norway East"
)

$regions = $regions | ForEach-Object { $_.Replace(" ", "") }

do{
    $regions | ForEach-Object {
        $dps_url =  "dataprocessingservice.$_.arcdataservices.com"
        $ti_url =  "telemetry.$_.arcdataservices.com"
        try{
            $dps_response_time = Measure-Command { $response = Invoke-WebRequest -Uri $dps_url -Method Get }
            $dps_result = ($response).StatusCode
        }catch{
            $dps_result = $_.Exception.Message
        }
        try{
            $ti_response_time = Measure-Command { $response = Invoke-WebRequest -Uri $ti_url -Method Get -SkipHttpErrorCheck }
        }catch{
            if($_.Exception.Message -like "*401*"){
                $ti_result = "Expected"
            }
            else {
                $ti_result = $_.Exception.Message
            }
        }
        if ($ti_response_time.TotalSeconds -gt 3 -or $dps_response_time.TotalSeconds -gt 3 -or $dps_result -ne 200 -or $ti_result -ne "Expected") {
            Write-Host $dps_result "($dps_response_time) " $ti_result " ($ti_response_time) :: $_" -ForegroundColor Red
        }
        elseif ($ti_response_time.TotalSeconds -gt 1 -or $dps_response_time.TotalSeconds -gt 1) {
            Write-Host $dps_result "($dps_response_time) " $ti_result " ($ti_response_time) :: $_" -ForegroundColor Yellow
        }
        else
        {
            Write-Host $dps_result "($dps_response_time) " $ti_result " ($ti_response_time) :: $_"
        }
    }
    Write-Host "====================================================================="
} while($true)

Endpoint reference

Beginning with March, 12 2024, the Azure Extension for SQL Server uses the following endpoints:

  • DPS: dataprocessingservice.<region>.arcdataservices.com
  • Telemetry telemetry.<region>.arcdataservices.com

Replace <region> with the short name of the Azure region where the Arc machine resource is located. The short name is derived from the Azure region name without spaces and all lower case.

For example, if your Arc machine resource is located in East US 2 the short name of the region is eastus2 and the telemetry endpoint is:

telemetry.eastus2.arcdataservices.com

Up to and including the February 13, 2024, The specific endpoints were:

  • DPS: san-af-<region>-prod.azurewebsites.net.
  • Telemetry telemetry.<region>.arcdataservices.com.

Your extension continues to use these services until it is updated.

Use an HTTPS proxy server for outbound connectivity

If your network requires using an HTTPS proxy server for outbound connectivity, you can read more about configuring that at Update or remove proxy settings.

Query Azure Resource Graph for telemetry upload stats

Use Azure Resource Graph to query the upload status for your environment.

resources
    | where type =~ 'microsoft.hybridcompute/machines/extensions'
    | where properties.type in ('WindowsAgent.SqlServer','LinuxAgent.SqlServer')
    | parse id with * '/providers/Microsoft.HybridCompute/machines/' machineName '/extensions/' *
    | parse properties with * 'uploadStatus : ' uploadStatus ';' *
    | project uploadStatus, subscriptionId, resourceGroup, machineName
    | where uploadStatus !in ('OK') //comment this out to see all upload stats
    | order by uploadStatus desc

Find SQL extensions that have not connected to DPS in a long time

Query Azure Resource Graph to find extensions that have not connected to DPS recently.

resources
    | where type =~ 'microsoft.hybridcompute/machines/extensions'
    | where properties.type in ('WindowsAgent.SqlServer','LinuxAgent.SqlServer')
    | parse id with * '/providers/Microsoft.HybridCompute/machines/' machineName '/extensions/' *
    | parse properties with * 'timestampUTC : ' timestampUTC ';' *
    | project timestampUTC, subscriptionId, resourceGroup, machineName
    | order by timestampUTC desc

Error codes

The following table shows some of the common DPS upload status values and what you can do to troubleshoot further.

DPS upload status value HTTP error code Troubleshooting suggestions
0 Likely cause: a firewall is blocking the transmission of the data to the DPS. Open the firewall to the DNS endpoint for the DPS (TCP, port: 443).
OK 200 The connection is working as expected.
Bad request 400 Possible cause: The resource name (SQL Server instance or database name) doesn't conform to Azure resource naming conventions. For example, if the database name is a reserved word.
Unauthorized 401 Likely cause: the extension is configured to send data through an HTTP proxy that requires authentication. Using an HTTP proxy that requires authentication is not currently supported. Use an unauthenticated HTTP proxy or no proxy.
Forbidden 403 If the Azure Connected Machine agent is otherwise working as expected and this error doesn't resolve itself after a reboot, create a support case with Microsoft Support through the Azure portal.
NotFound 404 The endpoint that the extension is trying to connect to doesn't exist.

To check which endpoint it is trying to connect to, search the logs for dataprocessingservice (or before March, 2024 san-af). This condition can happen if the Azure Connected Machine agent was deployed and connected to an Azure region in which the Microsoft.AzureArcData resource provider is not yet available. Redeploy the Azure Connected Machine agent in a region that the Microsoft.AzureArcData resource provider for aSQL Server enabled by Azure Arc is available. See also Region availability.

It is possible that the DNS resolver cache is not refreshed for your machine. To refresh:
- On Windows run: ipconfig /flushdns
- On Linux (if systemd is being used) run: sudo resolvectl flush-caches
Conflict 409 Likely cause: temporary error happening inside of the DPS. If this does not resolve itself, create a support case with Microsoft Support through the Azure portal.
InternalServerError 500 This is an error that is happening inside of the DPS. Create a support case with Microsoft Support through the Azure portal.

Related content