Missing Columns in ADLS from Storage Blob Logs

Question

After changing the storage blob logs destination from Log Analytics to ADLS, the logs table StorageBlobLogs contains 52 columns, but certain columns for read, delete, and write actions are missing in the ADLS logs file.

What could be the reason for this discrepancy?

Answer 1

Hello Siddharth Choudhary

Adding to your replied comment the missing columns in the ADLS JSON logs, despite being present in the Log Analytics table, suggest an issue with the export mechanism or data serialization Certain columns that are found in the Log Analytics StorageBlobLogs table, which may contain empty values, are absent from the corresponding JSON log files stored in ADLS. This occurrence is unexpected. It is important to note that even if these columns have empty values in Log Analytics, they should still be included in the JSON logs that are exported to ADLS. The built-in Log Analytics export mechanism might have some undocumented behavior or limitations regarding the export of empty columns. It might be filtering out columns with null or empty values during the export process. The export process might be inferring the schema based on the non-empty values in the logs. If certain columns are always empty during the export period, they might not be included in the inferred schema.

The serialization of log data into JSON format may result in the exclusion of empty columns. Certain JSON serializers are configured to automatically omit null or empty values. Additionally, there may be particular settings or configurations within ADLS that influence the storage or formatting of the logs, which could contribute to the absence of empty columns.

Steps

• I request you to temporarily create storage blob logs containing non-empty values for the columns that are currently missing. This approach will assist in identifying whether the problem is directly associated with the absence of values and verify whether the columns are present in the ADLS JSON logs when they contain non-empty values.
• Examine whether there are any configurations in ADLS that could influence the log format or the presence of empty columns. Utilize Azure Storage Explorer to examine the JSON log files along with their associated properties.

References:

1. https://learn.microsoft.com/en-us/azure/azure-monitor/logs/manage-logs-tables?tabs=azure-portal
2. https://stackoverflow.com/questions/60398189/azure-log-query-show-null-values
3. https://learn.microsoft.com/en-us/azure/application-gateway/application-gateway-diagnostics

Hope the above answer helps! Please let us know do you have any further queries.

---

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members. 

Answer 2

Hi Siddharth Choudhary

Thanks for reaching out over Q&A Forum. Are you stating the fields are present but the values are empty or the values are present in the output but they are empty?

Also, what is the destination that you have configured for the logs? Is it LA or Storage Account? Are you observing any difference if you select different destination?

For Re-hydration Priority - This will only come into picture if rehydration is being performed.

For SASExpiryStatus - The fields is associated to SAS expiration policy and depend on the configuration.
https://learn.microsoft.com/en-us/azure/storage/common/sas-expiration-policy?tabs=azure-portal#about-sas-expiration-policies

For OperationCount - The number of each logged operation that is involved in the request. Most requests perform only one operation so it will ideally be 0.

For RequestMD5 - The value can also be empty