Issue with Log Filtering in ContainerLogV2 using Data Collection Rule

Question

Hello,

I am experiencing an issue with the following configuration of my Data Collection Rule (DCR):

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "dataCollectionRules_name": {
            "defaultValue": "dcr-integration",
            "type": "String"
        },
        "logAnalyticsWorkspaceId": {
            "defaultValue": "/subscriptions/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX/resourceGroups/my-resource-group/providers/microsoft.operationalinsights/workspaces/my-log-workspace",
            "type": "String"
        }
    },
    "resources": [
        {
            "type": "Microsoft.Insights/dataCollectionRules",
            "apiVersion": "2023-03-11",
            "name": "[parameters('dataCollectionRules_name')]",
            "location": "francecentral",
            "kind": "WorkspaceTransforms",
            "properties": {
                "dataSources": {},
                "destinations": {
                    "logAnalytics": [
                        {
                            "workspaceResourceId": "[parameters('logAnalyticsWorkspaceId')]",
                            "name": "log-destination"
                        }
                    ]
                },
                "dataFlows": [
                    {
                        "streams": [
                            "Microsoft-Table-ContainerLogV2"
                        ],
                        "destinations": [
                            "log-destination"
                        ],
                        "transformKql": "source\n| where tostring(LogMessage) !contains \"Picked up _JAVA_OPTIONS\"\n| where tostring(LogMessage) !contains \"Spring Boot\"\n"
                    }
                ]
            }
        }
    ]
}

With this configuration, I expect logs containing "Spring Boot" or "_JAVA_OPTIONS" to be filtered out and not appear in the ContainerLogV2 table. However, these logs are still being ingested.

I also tried modifying my DCR to use the Microsoft-ContainerLogV2 table instead, but this did not change anything.

My issue is simple: logs from my AKS cluster are being sent to ContainerLogV2, and I want to filter out specific messages so they are no longer collected.

Could you help me understand why these filters are not working and how I can correctly apply them?

Thank you in advance for your support.

Best regards, Melvin

Answer 1

Hello Masdieu, Melvin,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

I understand that your Data Collection Rule (DCR) is not filtering out specific log messages from ContainerLogV2 as expected.

Logs are not filtered because transformKql applies after ingestion and the correct approach is to apply filtering at the dataSources level.

To resolve this, and to ensures logs containing Spring Boot or _JAVA_OPTIONS are never collected you can do the followings:

• Verify the correct log stream name. If the logs originate from ContainerInsights, confirm that the stream in the DCR should be Microsoft-ContainerLogV2 instead of Microsoft-Table-ContainerLogV2. - https://learn.microsoft.com/en-us/azure/azure-monitor/containers/container-insights-log-query something like this:

    ContainerLogV2
    | summarize count() by SourceSystem
  

• Test the filter in Log Analytics before modifying the DCR.

    ContainerLogV2
    | where tostring(LogMessage) !contains "Picked up _JAVA_OPTIONS"
    | where tostring(LogMessage) !contains "Spring Boot"
  

  If logs still appear, check if LogMessage is a string field. If not, cast it explicitly:

    | extend LogMessage = tostring(LogMessage)
  

  https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/)
• Move filtering logic to dataSources in the DCR. Instead of using transformKql inside dataFlows, apply log filtering at the dataSources level using streamDeclarations. Update your DCR to:

    "dataSources": {
        "extensions": [
            {
                "name": "container-logs",
                "streams": ["Microsoft-ContainerLogV2"],
                "extensionName": "ContainerInsights",
                "extensionParameters": {
                    "filtering": {
                        "filter": "tostring(LogMessage) !contains \"Picked up _JAVA_OPTIONS\" and tostring(LogMessage) !contains \"Spring Boot\""
                    }
                }
            }
        ]
    }
  

• Redeploy and verify log ingestion. - https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/data-collection-rule-azure-cli

I hope this is helpful! Do not hesitate to let me know if you have any other questions or clarifications.

---

Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful.