Stream Analytics does not refresh the input reference blob data when it is updated on the blob storage account

zarcap 0 Reputation points
2024-06-12T13:59:16.8033333+00:00

Hi Colleagues; I'm dealing with a problem with Stream Analytics jobs and their corresponding input reference blob data.

Basically Stream Analytics is not refreshing the input reference snapshot when a new blob file has been created within the storage account. The blob files pattern is as follows: "reference/xxx-xxxx/{date}/{time}/xxxx-xxxxxx.json" so we have a new json file comming when the reference data has to be refreshed. ( with a greater date-time than the previous one )

Sometimes I find in the log that Stream Analytics correctly re-create the snapshop:

"Reference Data full snapshot "reference/xxx-xxxx/2024/06/11/16/03/xxxx-xxxxxx.json" covering 06/11/2024 16:03:45 == 638537186250633356 to 06/11/2024 16:04:40 == 638537186800245033 with size 643578 bytes is being loaded into memory."

BUT in many other cases there is no reference Data full snapshot refresh at all in the log, then, then, reference data for the running jobs remains outdated.

for the previous example we have a new file:

reference/xxx-xxxx/2024/06/11/16/20/xxxx-xxxxxx.json

but Stream Analytics has never detect it.

Some of you had to deal with this problem?; don't known whether I'm doing something wrong...

Many thanks in advance.

Azure Stream Analytics
Azure Stream Analytics
An Azure real-time analytics service designed for mission-critical workloads.
360 questions
{count} votes

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.