Summary rules - Limit on total aggregated size

Question

Folks,

I'm trying to use summary rules to aggregate firewall logs. There's a hard size limit from MS per result of 100 MB which I think is not up to the mark for firewall logs. While summarizing I'm creating two sets and grouping by 7 other fields (I need for alarms). The summary rule works 60% of the time, 40% of the time it fails because of a spike in logs that leads to the size exceeding 100 MB. I can't think of a way to workaround this. Anybody figured it out yet?

Probably can create a limit in kql to check for size and ignore anything above 100 MB but I can't think of a way I can do this in kql.

Answer 1

Hello @Khanna, Keshav,

Thank you for posting your query on Microsoft Q&A.

Yes, you are correct—there is a limitation on summary rules where the maximum result set size is 100MB.

To address this limitation, you might consider adding a time filter to your query. The time range applied in the query will be the intersection of the filter and the bin size. You can refer to the following documentation for more details: Create or update a summary rule

Instead of processing all logs at once, partition your data based on smaller time intervals. For instance, if you're currently summarizing logs for an entire day, try breaking it down into shorter intervals, such as an hour or 30 minutes. This approach will reduce the amount of data processed in each run and help you stay within the size limit.

Please test this approach and let me know the results.

Please "Accept the answer" if the information helped you. This will help us and others in the community as well.

Thanks,
Raja Pothuraju.