403 Client Error forbidden for url:https://eastus.monitoring.azure.com//subscriptions/<my-subscription>/resourceGroups/<my-resource-group>/providers/Microsoft.OperationalInsights/workspaces/<my_workspace_name>/metrics

Question

I am working off of my account and I am the owner of all the resources created under it. I have a resource group and a log analytics workspace created under it - both in eastus.
Both of them have the Monitoring Metrics Publisher role assigned to it for my user account.
I also have additional roles like Owner, Log Analytics Contributor, Monitoring Contributor as well.

However I keep getting the error:
Error uploading logs: 403 Client Error: Forbidden for url: https://eastus.monitoring.azure.com//subscriptions/<my_subscription>/resourceGroups/<my_resource_group>/providers/Microsoft.OperationalInsights/workspaces/<my_workspace_name>/metrics

Response content: b'{"error":{"code":"AuthorizationFailed","message":"CheckAccess API did not respond with HTTP 200. User Tenant Id: <user_tenant_id> (possibly invalid). Response Content: {\"error\":{\"code\":\"InvalidAuthenticationToken\",\"message\":\"The received access token is not valid: at least one of the claims 'puid' or 'altsecid' or 'oid' should be present. If you are accessing as application please make sure service principal is properly created in the tenant.\"}}"}}

After decoding the token at jwt.ms I see that I have all of the claims that they ask for
"aud": "https://monitoring.azure.com",
"altsecid": "1:live.com:<some_id>",
"puid": "<some_puid>",
"oid": <some_oid_value>,
"idp": "live.com",
"idtyp": "user",
"scp": "user_impersonation"

Steps to recreate:

1. Create azure account
2. Create a resource group under east us
3. Create a Log Analytics Workspace under resource group created in 1
4. Under IAM section of resource group add Monitoring Metrics Publisher role to the user account created in 1
5. Do az login to login to the correct account using terminal
6. Fill in the correct values in the script and run the script

import requests
import datetime
from azure.identity import DefaultAzureCredential

resource_name = 'my_resource_name'
resource_type = 'Microsoft.OperationalInsights/workspaces'
resource_group = 'my_resource_group'
subscription_id = 'my_subscription_id'
resource_id = f'/subscriptions/{subscription_id}/resourceGroups/{resource_group}/providers/{resource_type}/{resource_name}'


credential = DefaultAzureCredential()
token = credential.get_token('https://monitoring.azure.com/.default').token
url = f'https://eastus.monitoring.azure.com/{resource_id}/metrics'


date = datetime.datetime.now(datetime.timezone.utc).isoformat()

metric_data_1 = {
    'time': date,
    'data': {
        'baseData': {
            'metric': 'test_metric',
            'namespace': 'test_namespace',
            'dimNames': ['test_id'],
            'series': [
                {
                    'dimValues': ['123'],
                    'min': 0,
                    'max': 100,
                    'sum': 100,
                    'count': 1
                }
            ]
        }
    }
}

headers = {
    'Authorization': f'Bearer {token}',
    'Content-Type': 'application/json'
}


print(" \n metric data: \n", metric_data_1)
try:
    response = requests.post(url, headers=headers, data=metric_data_1)
    response.raise_for_status()
    print('Uploaded Logs!')
except requests.exceptions.HTTPError as err:
    print(f'Error uploading logs: {err}')
    print(f'Response content: {response.content}')

Answer 1

Hi ar
The 403 Client Error: Forbidden usually indicates an issue with authentication or authorization. Below are the troubleshooting steps:

Check Token Audience: Ensure the aud claim in your token is set to https://monitoring.azure.com.

Validate Token Scope: The scp claim should include user_impersonation. Ensure the scope you requested is correct for the operation.

Verify IAM Roles: Confirm that roles like Monitoring Metrics Publisher, Owner, Log Analytics Contributor, or Monitoring Contributor are correctly assigned, with no delays in propagation.

Format Payload Correctly: Make sure the payload sent with requests.post is properly formatted as JSON. Convert the data to a JSON string if needed.

Confirm URL: Double-check the endpoint URL for accuracy, avoiding typos or extra slashe

import requests
import datetime
from azure.identity import DefaultAzureCredential
import json

resource_name = 'my_workspace_name'
resource_type = 'Microsoft.OperationalInsights/workspaces'
resource_group = 'my_resource_group'
subscription_id = 'my_subscription_id'
resource_id = f'/subscriptions/{subscription_id}/resourceGroups/{resource_group}/providers/{resource_type}/{resource_name}'

credential = DefaultAzureCredential()
token = credential.get_token('https://monitoring.azure.com/.default').token
url = f'https://eastus.monitoring.azure.com/{resource_id}/metrics'

date = datetime.datetime.now(datetime.timezone.utc).isoformat()

metric_data_1 = {
    'time': date,
    'data': {
        'baseData': {
            'metric': 'test_metric',
            'namespace': 'test_namespace',
            'dimNames': ['test_id'],
            'series': [
                {
                    'dimValues': ['123'],
                    'min': 0,
                    'max': 100,
                    'sum': 100,
                    'count': 1
                }
            ]
        }
    }
}

headers = {
    'Authorization': f'Bearer {token}',
    'Content-Type': 'application/json'
}

print(" \n metric data: \n", json.dumps(metric_data_1, indent=2))
try:
    response = requests.post(url, headers=headers, json=metric_data_1)
    response.raise_for_status()
    print('Uploaded Logs!')
except requests.exceptions.HTTPError as err:
    print(f'Error uploading logs: {err}')
    print(f'Response content: {response.content}')

I hope this has been helpful! Your feedback is important so please take a moment to accept answers. If you still have questions, please let us know what is needed in the comments so the question can be answered. Thank you for helping to improve Microsoft Q&A!

Answer 2

Hi ar,

Apologies for the delayed response. Below are the steps we followed to replicate the issue using PowerShell.

We have recreated the same as you mentioned and tried with Azure PowerShell

Where we have removed the Variables and tested using direct resource id.

go to vm>>properties>>Resource id

We create a VM and Enabled the Appinsights. And also have additional roles like Log Analytics Contributor, Monitoring Contributor as well.

And we tried to get the details, The Access token is generated successfully and Valid but how ever Its throwing an error

Microsoft.Insights/Metrics/write was notallowed, Microsoft.Insights/Telemetry/write was notallowed. Warning: Principal will be blacklisted if the service principal is not granted proper access while it hits the GIG endpoint continuously."}}

In your case the error is {"error":{"code":"InvalidAuthenticationToken","message":"The received access token is not valid: at least one of the claims 'puid' or 'altsecid' or 'oid' should be present. If you are accessing as application please make sure service principal is properly created in the tenant."}}"}}

Can you please try the Below Script and see if this can resolve the issue.

# Resource ID for the specific virtual machine
$ResourceId = "/subscriptions/cbc41000-8b50-41e9-919a-419d9609f7e8/resourceGroups/AI-POC-RG-01/providers/Microsoft.Compute/virtualMachines/AIWUS02WIN10-Latest"
# Authenticate and acquire the token
# Import the required modules
Import-Module Az.Accounts
Import-Module Az.Monitor
$subscptionid= “Yoursubscptionid”
connect-azaccount
Select-azsubscption -subscption $subscptionid
$Token = (Get-AzAccessToken -ResourceUrl "https://monitoring.azure.com").Token
# Set the metrics endpoint for West US 2 region
$Url = "https://westus2.monitoring.azure.com$ResourceId/metrics"
# Get the current date and time in UTC
$Date = (Get-Date).ToUniversalTime().ToString("o")
# Construct the metric data
$MetricData = @{
    time = $Date
    data = @{
        baseData = @{
            metric = "test_metric"
            namespace = "test_namespace"
            dimNames = @("test_id")
            series = @(
                @{
                    dimValues = @("123")
                    min = 0
                    max = 100
                    sum = 100
                    count = 1
                }
            )
        }
    }
}
# Convert the metric data to JSON
$MetricDataJson = $MetricData | ConvertTo-Json -Depth 3
# Define the headers for the HTTP request
$Headers = @{
    Authorization = "Bearer $Token"
    "Content-Type" = "application/json"
}
# Print the metric data (for debugging purposes)
Write-Output "Metric Data to be sent:"
Write-Output $MetricDataJson
# Make the POST request to upload the metric data
try {
    $Response = Invoke-RestMethod -Uri $Url -Method POST -Headers $Headers -Body $MetricDataJson
    Write-Output "Uploaded Metrics Successfully!"
} catch {
    Write-Output "Error uploading metrics: $_"
    if ($_.Exception.Response -ne $null) {
        $ErrorResponse = $_.Exception.Response.GetResponseStream()
        $ErrorReader = New-Object System.IO.StreamReader($ErrorResponse)
        $ErrorContent = $ErrorReader.ReadToEnd()
        Write-Output "Response content: $ErrorContent"
    }
}