Guidance on Filtering AppTraces Logs to Optimize Sentinel Workspace Usage

Question

Guidance on Filtering AppTraces Logs to Optimize Sentinel Workspace Usage

Someiah C S 100

Hi Community,

I'm seeking advice on how to filter out AppTraces logs from being ingested into our Sentinel workspace. These logs are consuming significant storage space and, being categorized under Analytics logs, are contributing to increased costs. Since we're not utilizing them for our security monitoring purposes, I'd like to exclude them from ingestion.

I understand that implementing data collection rules (DCRs) can help manage log ingestion. However, I'm uncertain about the specific steps to configure these rules to filter out AppTraces logs effectively. Additionally, I'm aware that certain tables, including AppTraces, can be configured for Basic Logs, which might offer a more cost-effective solution.

Could anyone provide detailed guidance or share best practices on setting up these configurations? Any insights or resources would be greatly appreciated.

Thank you in advance for your assistance.

Sakshi Devkante 1,335 Reputation points Microsoft External Staff

2025-02-27T11:09:27.95+00:00

Hello Someiah,

Thank you for posting your query on Microsoft Q&A.

I understand that you want to filter out AppTraces logs from being ingested into your Azure Sentinel workspace. in this case you can use Data Collection Rules (DCRs) to exclude specific logs or configure Basic Logs for cost-effective storage.

Find out which data source is providing your Sentinel workspace with AppTraces logs. This might be a custom application or an Azure resource like an App Service.

A Data Collection Rule (DCR) should be created.
Navigate to the Azure portal's -> Azure Monitor service. Go to the Settings section and select -> Data Collection Rules. To create a new DCR, click Create.

Indicate which resources (such as particular App Services) you wish to exclude AppTraces logs from.

Add a Data Source and choose the relevant log type (such as Custom Logs or Azure Diagnostics) in the DCR. - Transformation is used to remove AppTraces logs.

For instance: Only logs with a Category that does not contain "AppTraces" are ingested thanks to this KQL (Kusto Query Language) adjustment. To use the filtering rule, connect the DCR to your Sentinel workspace.

Make sure that AppTraces logs are no longer being ingested by keeping an eye on your Sentinel workspace after applying the DCR.

AppTraces logs can be set up as "Basic Logs" if you still need to keep them but wish to cut expenses. Although basic logs are less expensive to keep, they have certain drawbacks, like shorter retention periods and delayed query availability. In your Sentinel workspace, identify the table where AppTraces logs are stored (e.g., AppTraces). Configure Basic Logs:

Go to your Log Analytics workspace in the Azure portal.

Navigate to ->Tables under the -> Settings section.

Locate the "AppTrace" table and click on it.

Change the table's configuration to "Basic Logs".

Ensure that the table is now set to Basic Logs, which will reduce storage costs.

Refer similar threads:
1.https://learn.microsoft.com/en-us/answers/questions/1851367/data-collection-rule-transformation-not-filtering

2.https://learn.microsoft.com/en-us/answers/questions/2086637/applying-custom-data-collection-rules-for-ama-to-r

3.https://learn.microsoft.com/en-us/answers/questions/2182885/issue-with-log-filtering-in-containerlogv2-using-d

Refer documents:
https://learn.microsoft.com/en-us/azure/sentinel/connect-data-sources?tabs=azure-portal
https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/data-collection-rule-overview
https://learn.microsoft.com/en-us/azure/sentinel/best-practices-data
https://learn.microsoft.com/en-us/azure/azure-monitor/logs/logs-table-plans?tabs=portal-1

I hope this clarifies things.
Someiah C S 100 Reputation points

2025-02-27T12:29:07.7533333+00:00

Hi @Sakshi Devkante

Thanks for your response. The AppTraces logs are coming from Application Insights, so I’m aware of the source. However, when I tried to create a DCR, I wasn’t able to select Application Insights from the available resource types. The options listed below are selectable, but not Application Insights.

Is this due to a permission issue, or is it not possible to select Application Insights as a resource when configuring a DCR?
Ashok Gandhi Kotnana 4,310 Reputation points Microsoft External Staff

2025-02-27T15:56:40.3+00:00

Hi @Someiah C S ,

Adaptive Sampling automatically adjusts how much telemetry data is collected based on your app’s traffic. This helps reduce the amount of data sent to Application Insights, which can also lower the data sent to Sentinel. https://learn.microsoft.com/en-us/previous-versions/azure/azure-monitor/app/sampling

If possible, you can add custom logic to your application to log only important telemetry data. This allows you to filter out unnecessary logs, like App Traces, reducing the amount of data sent to Application Insights. https://learn.microsoft.com/en-us/azure/azure-monitor/reference/tables/apptraces

In the Log Analytics workspace, you can use Kusto Query Language (KQL) to filter or modify data when running queries. While this helps improve query performance and can lower analysis costs, it doesn’t reduce ingestion costs since the data is still stored in the workspace. https://learn.microsoft.com/en-us/azure/azure-monitor/logs/queries?tabs=groupby

If you want to keep some telemetry data but reduce costs, you can set up Basic Logs for the relevant tables in your Log Analytics workspace. Basic Logs are cheaper to store, but they may offer less detail and slower query performance.

https://learn.microsoft.com/en-us/azure/azure-monitor/logs/cost-logs

If you found this answer helpful, please consider "upvote" it.
Ashok Gandhi Kotnana 4,310 Reputation points Microsoft External Staff

2025-02-28T14:22:53.5833333+00:00

Hi @Someiah C S ,

Just want to check if the above Comment worked for you or else please let us know if any help, we are always here to help whenever you need us.

If the comment is helpful, please click "Upvote it"
Vinod Pittala 735 Reputation points Microsoft External Staff

2025-03-03T01:12:01.6533333+00:00

Hello Someiah C S ,

I would like to check if the issue has been resolved or if you need any assistance. Please feel free to let us know, as we are always here to help.

Additionally, if you have a moment, please review the previous comment.

Please do not forget to “upvote it” wherever the information provided helps you, this can be beneficial to other community members.it would be greatly appreciated and helpful to others.

Thanks.
Someiah C S 100 Reputation points

2025-03-04T06:28:25.35+00:00

Hi All,

Thanks for checking.

For AppTraces, the only way to filter logs from ingestion is through DCR. However, since AppTraces originates from the App Functions (Auth0 solution) we deployed, it is hardcoded to send Trace-level logs by default. Based on my understanding, DCR can only control logs from Azure Diagnostics within Azure resources, making direct filtering not possible.

Workaround: To optimize costs, I have moved AppTraces logs from Analytics Logs to Basic Logs and set the retention period to 30 days, as they are not being used for search queries. These logs will still be accessible within the App Function itself.

Accepted answer

0 additional answers

Your answer

Someiah C S 100 Reputation points

2025-02-27T12:29:07.7533333+00:00

Hi @Sakshi Devkante

Thanks for your response. The AppTraces logs are coming from Application Insights, so I’m aware of the source. However, when I tried to create a DCR, I wasn’t able to select Application Insights from the available resource types. The options listed below are selectable, but not Application Insights.

Is this due to a permission issue, or is it not possible to select Application Insights as a resource when configuring a DCR?
Ashok Gandhi Kotnana 4,310 Reputation points Microsoft External Staff

2025-02-27T15:56:40.3+00:00

Hi @Someiah C S ,

Adaptive Sampling automatically adjusts how much telemetry data is collected based on your app’s traffic. This helps reduce the amount of data sent to Application Insights, which can also lower the data sent to Sentinel. https://learn.microsoft.com/en-us/previous-versions/azure/azure-monitor/app/sampling

If possible, you can add custom logic to your application to log only important telemetry data. This allows you to filter out unnecessary logs, like App Traces, reducing the amount of data sent to Application Insights. https://learn.microsoft.com/en-us/azure/azure-monitor/reference/tables/apptraces

In the Log Analytics workspace, you can use Kusto Query Language (KQL) to filter or modify data when running queries. While this helps improve query performance and can lower analysis costs, it doesn’t reduce ingestion costs since the data is still stored in the workspace. https://learn.microsoft.com/en-us/azure/azure-monitor/logs/queries?tabs=groupby

If you want to keep some telemetry data but reduce costs, you can set up Basic Logs for the relevant tables in your Log Analytics workspace. Basic Logs are cheaper to store, but they may offer less detail and slower query performance.

https://learn.microsoft.com/en-us/azure/azure-monitor/logs/cost-logs

If you found this answer helpful, please consider "upvote" it.
Ashok Gandhi Kotnana 4,310 Reputation points Microsoft External Staff

2025-02-28T14:22:53.5833333+00:00

Hi @Someiah C S ,

Just want to check if the above Comment worked for you or else please let us know if any help, we are always here to help whenever you need us.

If the comment is helpful, please click "Upvote it"
Vinod Pittala 735 Reputation points Microsoft External Staff

2025-03-03T01:12:01.6533333+00:00

Hello Someiah C S ,

I would like to check if the issue has been resolved or if you need any assistance. Please feel free to let us know, as we are always here to help.

Additionally, if you have a moment, please review the previous comment.

Please do not forget to “upvote it” wherever the information provided helps you, this can be beneficial to other community members.it would be greatly appreciated and helpful to others.

Thanks.
Someiah C S 100 Reputation points

2025-03-04T06:28:25.35+00:00

Hi All,

Thanks for checking.

For AppTraces, the only way to filter logs from ingestion is through DCR. However, since AppTraces originates from the App Functions (Auth0 solution) we deployed, it is hardcoded to send Trace-level logs by default. Based on my understanding, DCR can only control logs from Azure Diagnostics within Azure resources, making direct filtering not possible.

Workaround: To optimize costs, I have moved AppTraces logs from Analytics Logs to Basic Logs and set the retention period to 30 days, as they are not being used for search queries. These logs will still be accessible within the App Function itself.

Answer 1

Hello Someiah C S ,

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to "Accept " the answer.

Issue:

Guidance on Filtering App Traces Logs to Optimize Sentinel Workspace Usage?

Solution:

To optimize costs, I have moved AppTraces logs from Analytics Logs to Basic Logs and set the retention period to 30 days, as they are not being used for search queries. These logs will still be accessible within the App Function itself.

If you have any other questions or are still running into more issues, please let me know. Thank you again for your time and patience throughout this issue.

Please remember to "Accept Answer" if any answer/reply helped, so that others in the community facing similar issues can easily find the solution.

https://learn.microsoft.com/en-us/answers/support/accept-answer#why-only-one-accepted-answer

Share via

Guidance on Filtering AppTraces Logs to Optimize Sentinel Workspace Usage

0 additional answers

Your answer