Azure function eventHub typescript trigger is converting message to UTF-8 string and cause data missing

Question

I am using Event Hub to store protobuf serialized message, Everything is fine, If I am using SDK library (python or JS library) to access Event hub.

But when I use Azure function Eventhub trigger (when there is new message on event hub, trigger a function app to do the processing)

the message passed into the handler is actually a UTF-8 converted "string", and it contains fallback characters. Because there are missing information, There is no way to convert it back to original binary (to do further protobuf parsing).

Please let me know if there are any configurations that I can make the "message" param of handler to be binary (like Buffer).

Thanks.

app.eventHub(`eventhub_trigger`, {
  eventHubName: 'notification',
  connection: 'CONN_STR',
  cardinality: 'one',
  handler: async (message: unknown, context: InvocationContext): Promise<void> => {
    const value = message as string
    console.log("------------------------------")
    console.log(typeof message)
    console.log(value.length)
    console.log(Buffer.from(value, "latin1").toString("base64"))
    console.log(Array.from(value, char => char.charCodeAt(0)))
    // await processNotification(value)
  }
})

Answer 1

Hi ,

Thanks for reaching out to Microsoft Q&A.

You can try configuring the azfunction to receive the raw message payload as a binary Buffer instead of a string.

Here's what you need to know and do:

Problem Details

By default, Azure Functions try to deserialize eventhub messages into a JSON object or string, depending on the content type. This behavior can cause issues with binary data (ex: Protobuf?), as it might lose precision or introduce unwanted transformations.

Solution

You need to explicitly configure the eventhub trigger binding to receive raw binary data. Here's how to achieve that:

Step 1: Update the Event Hub Trigger Binding

Make sure to set the dataType property of the Event Hub trigger binding in your function.json file to binary. This ensures the message is passed as a Buffer instead of being converted to a UTF-8 string.

Example function.json:

{ "bindings": [ { "type": "eventHubTrigger", "name": "eventHubMessages", "direction": "in", "eventHubName": "notification", "connection": "CONN_STR", "cardinality": "one", "dataType": "binary" } ] }

Step 2: Update the Azure Function Code

With dataType: "binary", the message parameter will now be of type Buffer. You can directly process it as a binary payload. Here's an updated example of your TypeScript handler:

app.eventHub("eventhub_trigger", { eventHubName: "notification", connection: "CONN_STR", cardinality: "one", handler: async (message: Buffer, context: InvocationContext): Promise<void> => { console.log("------------------------------"); console.log(Message type: ${typeof message}); console.log(Message length: ${message.length}); console.log(Base64 Encoded Message: ${message.toString("base64")}); console.log(Message as Array: ${Array.from(message)}); // Process the raw binary message as needed // For example, parse it with Protobuf: // const parsedMessage = ProtobufMessage.decode(message); }, });

Step 3: Test and Verify

1. Deploy your updated function.
2. Send a Protobuf encoded message to the eventhub.
3. Verify that the message parameter in the func isnow a Buffer object containing the raw binary data.

Additional Tips

• Protobuf Decoding: Ensure you have the correct .proto schema file and use a Protobuf library (protobufjs or google-protobuf) to decode the message.
• Logging: Use extensive logging during development to ensure you're receiving the message correctly.

This setup will preserve the integrity of your protobuf serialized messages and avoid data corruption caused by UTF-8 string conversion.

Please feel free to click the 'Upvote' (Thumbs-up) button and 'Accept as Answer'. This helps the community by allowing others with similar queries to easily find the solution.