Azure Stream Analytics - Can't Access Metadata

Question

I am using stream analytics to consume event from EventHub and aggregate them into a Data Lake Storage account. I am trying to access the "applicationProperties" of the event to add it to the data.

I have tried using but the main issue is that it tries to parse the value as a String when my data is a binary Array. So the output of this command is the sting "System.Byte[]". So i cannot use a Javascript UDF to decode the binary string.

GetMetadataPropertyValue([input], '[User].[metadata_key]') AS metadata

I tried to use the GetRecordProperties([User]) to try and get the "applicationProperties" object directly but whatever i set as prameters the output is always null.

If anyone knows any way of doing this it would be a life saviour

Answer 1

@Quentin Levasseur

Welcome to the microsoft Q&A community.

It sounds like you're encountering a common issue with Azure Stream Analytics when trying to access binary data as a string. Here are a few suggestions that might help:

1.Convert Binary Data to String: You can use a JavaScript UDF (User-Defined Function) to convert the binary data to a string before processing it. Here's an example of how you might do this:

function ConvertBinaryToString(binaryData) {
    return binaryData.toString('base64');
}

2. Use GetRecordProperties: Ensure that you're using the correct syntax for GetRecordProperties. Here's an example:

SELECT GetRecordProperties([User]) AS applicationProperties
FROM [input]

3. Check Input Configuration: Make sure that your Event Hub input is configured to include the applicationProperties in the metadata. You can verify this in the Azure portal or through the Azure CLI.

4.Test with ASA Extension for Visual Studio Code: If you're having trouble testing your queries in the Azure portal, consider using the ASA extension for Visual Studio Code. This allows you to test your queries with live data and might help you debug the issue more effectively.

See:

• https://learn.microsoft.com/en-us/stream-analytics-query/getmetadatapropertyvalue
• https://learn.microsoft.com/en-us/azure/stream-analytics/stream-analytics-stream-analytics-query-patterns

Answer 2

Hi @Quentin Levasseur ,

welcome to this moderated Azure community forum.

It must be clear the message body is not the same as application properties. See also the class definition.

That said, the message body can be passed on as a byte array. The property values are an 'Object' likely a string. Technically this can also be a byte array although I have never working that situation.

Check out this blog post about working with application properties.

Notice that the Azure portal experience regarding editing and testing the query is not recommended. It's slow, it's not under version control and even more important, the application properties are not visualized.

you can make a recording of the input:

My fellow MVP @Chiugo Okpala pointed out how to work with byte arrays in UDFs.

So, I recommend testing your query with the Visual Studio Code extension for Stream Analytics and work with a recording (or even with live data to confirm your tests).

---

If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.