Size of storage account (tables) for Service Fabric Diagnostics

Question

Has any guidance been published on best practices around collecting diagnostic logs for Service Fabric. Our Diagnostic storage accounts are currently over 4TB, the majority of which seems to be taken up by the WADServiceFabricSystemEventTable and WADPatchOrchestrationApplicationTable tables?

Only information I can find is here service-fabric-best-practices-monitoring.md, but this doesn't talk about retention.

Here is the diagnostics snippet from the ARM template we use to create the scalesets:

"publisher": "Microsoft.Azure.Diagnostics",
                  "settings": {
                    "WadCfg": {
                      "DiagnosticMonitorConfiguration": {
                        "overallQuotaInMB": "50000",
                        "EtwProviders": {
                          "EtwEventSourceProviderConfiguration": [
                            {
                              "provider": "Microsoft-ServiceFabric-Actors",
                              "scheduledTransferKeywordFilter": "1",
                              "scheduledTransferPeriod": "PT5M",
                              "DefaultEvents": {
                                "eventDestination": "ServiceFabricReliableActorEventTable"
                              }
                            },
                            {
                              "provider": "Microsoft-ServiceFabric-Services",
                              "scheduledTransferPeriod": "PT5M",
                              "DefaultEvents": {
                                "eventDestination": "ServiceFabricReliableServiceEventTable"
                              }
                            },
                            {
                              "provider": "e39b723c-590c-4090-abb0-11e3e6616346",
                              "scheduledTransferPeriod": "PT5M",
                              "DefaultEvents": {
                                "eventDestination": "PatchOrchestrationApplicationTable"
                              }
                            },
                            {
                              "provider": "fc0028ff-bfdc-499f-80dc-ed922c52c5e9",
                              "scheduledTransferPeriod": "PT5M",
                              "DefaultEvents": {
                                "eventDestination": "PatchOrchestrationApplicationTable"
                              }
                            },
                            {
                              "provider": "24afa313-0d3b-4c7c-b485-1047fd964b60",
                              "scheduledTransferPeriod": "PT5M",
                              "DefaultEvents": {
                                "eventDestination": "PatchOrchestrationApplicationTable"
                              }
                            },
                            {
                              "provider": "05dc046c-60e9-4ef7-965e-91660adffa68",
                              "scheduledTransferPeriod": "PT5M",
                              "DefaultEvents": {
                                "eventDestination": "PatchOrchestrationApplicationTable"
                              }
                            },
                            {
                              "provider": "Asos.Finance.EtlLogs",
                              "scheduledTransferPeriod": "PT5M",
                              "DefaultEvents": {
                                "eventDestination": "AsosFinanceEtlLogs"
                              }
                            }
                          ],
                          "EtwManifestProviderConfiguration": [
                            {
                              "provider": "cbd93bc2-71e5-4566-b3a7-595d8eeca6e8",
                              "scheduledTransferLogLevelFilter": "Information",
                              "scheduledTransferKeywordFilter": "4611686018427387912",
                              "scheduledTransferPeriod": "PT5M",
                              "DefaultEvents": {
                                "eventDestination": "ServiceFabricSystemEventTable"
                              }
                            }
                          ]
                        }
                      }
                    }

Answer 1

I'd missed this section from the guidance:

[!NOTE] There is currently no way to filter or groom the events that are sent to the table. If you don't implement a process to remove events from the table, the table will continue to grow. Currently, there is an example of a data grooming service running in the Watchdog sample, and it is recommended that you write one for yourself as well, unless there is a good reason for you to store logs beyond a 30 or 90 day timeframe.

Looks like we'll have to add to our Watchdog