Kafka REST proxy endpoint is corrupting and wrongly parsing data

AwesomePaul 20

Hello Team,

I recently activated REST proxy endpoint for HdInsight kafka. The endpoint is used to produce data to a topic, and upon consuming from the topic I observed that the data is wrongly encoded and parsed. If I try to produce data from the client using kafka console producer, the data is parsed and consumed with no issues.

My Kafka cluster version is 5.1, and below are the screenshots from the producer and consumer.

Case 1 ( using Content-type as "application/json")

Producer User's image

Consumer

Case 2 ( using Content-type as "application/vnd.kafka.json.v2+json")

Producer

User's image

Consumer

Case 3 (Using Kafka console producer from terminal)

Producer

User's image

Consumer

User's image

Please advise on possible solution.

Thank you.

PRADEEPCHEEKATLA-MSFT 90,226 Reputation points Microsoft Employee

2024-07-12T08:36:11.2666667+00:00

@AwesomePaul - Thanks for the question and using MS Q&A platform.

It seems like you are facing an issue with the Kafka REST proxy endpoint for HDInsight Kafka. Based on the information you provided, it looks like the data is wrongly encoded and parsed when you consume it from the topic using the REST proxy endpoint. However, when you produce data using the Kafka console producer, the data is parsed and consumed with no issues.

To better understand the issue, could you please provide more information on how you are producing and consuming data using the REST proxy endpoint?
AwesomePaul 20 Reputation points

2024-07-15T12:26:47.5866667+00:00

Thanks for your response @PRADEEPCHEEKATLA-MSFT

I am producing using the REST proxy endpoint from my local PC, and I consume from a kafka console producer running on the header node of the cluster.

In summary, i'm working on building a pipeline that uses kafka as a message broker, and i'll be streaming data from a various sources (could be an sdk, iOT, website events etc) , which are outside of Azure environment. Hence, the reason i'm using the REST proxy to connect to kafka.

I also have another test consumer setup using spark streaming on databricks, and this consumer is also consuming data in a wrongly encoded format. Same issue with the console consumer i mentioned above.

Not sure why the data is wrongly encoded, but i'm basically pushing JSON, and unable to consume the JSON, even after trying to decode.

Kindly see attached screenshots below, thanks.
PRADEEPCHEEKATLA-MSFT 90,226 Reputation points Microsoft Employee

2024-07-16T04:30:59.88+00:00

@AwesomePaul - I agree that this issue looks strange and I wasn't able to reproduce this issue. If you have a support plan could you please file a support ticket for deeper investigation and do share the SR# with us?
PRADEEPCHEEKATLA-MSFT 90,226 Reputation points Microsoft Employee

2024-07-18T05:29:32.4966667+00:00

@AwesomePaul - Did you get a chance to open a support ticket?
AwesomePaul 20 Reputation points

2024-07-19T07:46:41.32+00:00

Hi @PRADEEPCHEEKATLA-MSFT,

Thanks for your feedback.

I'm yet to create the ticket which required payment, had this error while creating the ticket "You already have technical support with your Enterprise Agreement. Contact your Large Account Reseller if you would like to change your support plan". I did a followup with the account manager, and they still were not helpful. Not a good experience.

Its a bit frustrating, i've spent several days on this issue, created multiple kafka cluster of version 5.1 and the same encoding issue continued on all.

My team was blocked for this, and couldn't wait anymore, so i setup Kafka Confluent instead and progressed with the build.
PRADEEPCHEEKATLA-MSFT 90,226 Reputation points Microsoft Employee

2024-07-22T05:02:13.68+00:00

@AwesomePaul - Thanks for the update.

Please do continue to use MS Q&A platform for any question related to Azure!

Share via

Kafka REST proxy endpoint is corrupting and wrongly parsing data

Your answer