Azure Data Explorer monitoring data reference
This article contains all the monitoring reference information for this service.
See Monitor Azure Data Explorer for details on the data you can collect for Azure Data Explorer and how to use it.
Metrics
This section lists all the automatically collected platform metrics for this service. These metrics are also part of the global list of all platform metrics supported in Azure Monitor.
For information on metric retention, see Azure Monitor Metrics overview.
Supported metrics for Microsoft.Kusto/clusters
The following table lists the metrics available for the Microsoft.Kusto/clusters resource type.
- All columns might not be present in every table.
- Some columns might be beyond the viewing area of the page. Select Expand table to view all available columns.
Table headings
- Category - The metrics group or classification.
- Metric - The metric display name as it appears in the Azure portal.
- Name in REST API - The metric name as referred to in the REST API.
- Unit - Unit of measure.
- Aggregation - The default aggregation type. Valid values: Average (Avg), Minimum (Min), Maximum (Max), Total (Sum), Count.
- Dimensions - Dimensions available for the metric.
- Time Grains - Intervals at which the metric is sampled. For example,
PT1Mindicates that the metric is sampled every minute,PT30Mevery 30 minutes,PT1Hevery hour, and so on. - DS Export- Whether the metric is exportable to Azure Monitor Logs via diagnostic settings. For information on exporting metrics, see Create diagnostic settings in Azure Monitor.
| Category | Metric | Name in REST API | Unit | Aggregation | Dimensions | Time Grains | DS Export |
|---|---|---|---|---|---|---|---|
| Ingestion health and performance | Batch Blob Count Number of data sources in an aggregated batch for ingestion. |
BatchBlobCount |
Count | Average, Maximum, Minimum | Database |
PT1M | Yes |
| Ingestion health and performance | Batch Duration The duration of the aggregation phase in the ingestion flow. |
BatchDuration |
Seconds | Average, Maximum, Minimum | Database |
PT1M | Yes |
| Ingestion health and performance | Batches Processed Number of batches aggregated for ingestion. Batching Type: whether the batch reached batching time, data size or number of files limit set by batching policy |
BatchesProcessed |
Count | Total (Sum), Average, Maximum, Minimum | Database, SealReason |
PT1M | Yes |
| Ingestion health and performance | Batch Size Uncompressed expected data size in an aggregated batch for ingestion. |
BatchSize |
Bytes | Average, Maximum, Minimum | Database |
PT1M | Yes |
| Ingestion health and performance | Blobs Dropped Number of blobs permanently rejected by a component. |
BlobsDropped |
Count | Total (Sum), Average, Minimum, Maximum | Database, ComponentType, ComponentName |
PT1M | Yes |
| Ingestion health and performance | Blobs Processed Number of blobs processed by a component. |
BlobsProcessed |
Count | Total (Sum), Average, Minimum, Maximum | Database, ComponentType, ComponentName |
PT1M | Yes |
| Ingestion health and performance | Blobs Received Number of blobs received from input stream by a component. |
BlobsReceived |
Count | Total (Sum), Average, Minimum, Maximum | Database, ComponentType, ComponentName |
PT1M | Yes |
| Cluster health | Cache utilization (deprecated) Utilization level in the cluster scope. The metric is deprecated and presented for backward compatibility only, you should use the 'Cache utilization factor' metric instead. |
CacheUtilization |
Percent | Average, Maximum, Minimum | <none> | PT1M | Yes |
| Cluster health | Cache utilization factor Percentage of utilized disk space dedicated for hot cache in the cluster. 100% means that the disk space assigned to hot data is optimally utilized. No action is needed in terms of the cache size. More than 100% means that the cluster's disk space is not large enough to accommodate the hot data, as defined by your caching policies. To ensure that sufficient space is available for all the hot data, the amount of hot data needs to be reduced or the cluster needs to be scaled out. Enabling auto scale is recommended. |
CacheUtilizationFactor |
Percent | Average, Maximum, Minimum | <none> | PT1M | Yes |
| Export health and performance | Continuous Export Max Lateness The lateness (in minutes) reported by the continuous export jobs in the cluster |
ContinuousExportMaxLatenessMinutes |
Count | Maximum | <none> | PT1M | Yes |
| Export health and performance | Continuous export - num of exported records Number of records exported, fired for every storage artifact written during the export operation |
ContinuousExportNumOfRecordsExported |
Count | Total (Sum) | ContinuousExportName, Database |
PT1M | Yes |
| Export health and performance | Continuous Export Pending Count The number of pending continuous export jobs ready for execution |
ContinuousExportPendingCount |
Count | Maximum | <none> | PT1M | Yes |
| Export health and performance | Continuous Export Result Indicates whether Continuous Export succeeded or failed |
ContinuousExportResult |
Count | Count | ContinuousExportName, Result, Database |
PT1M | Yes |
| Cluster health | CPU CPU utilization level |
CPU |
Percent | Average, Maximum, Minimum | <none> | PT1M | Yes |
| Ingestion health and performance | Discovery Latency Reported by data connections (if exist). Time in seconds from when a message is enqueued or event is created until it is discovered by data connection. This time is not included in the Azure Data Explorer total ingestion duration. |
DiscoveryLatency |
Seconds | Average | ComponentType, ComponentName |
PT1M | Yes |
| Ingestion health and performance | Events Dropped Number of events dropped permanently by data connection. An Ingestion result metric with a failure reason will be sent. |
EventsDropped |
Count | Total (Sum), Average, Minimum, Maximum | ComponentType, ComponentName |
PT1M | Yes |
| Ingestion health and performance | Events Processed Number of events processed by the cluster |
EventsProcessed |
Count | Total (Sum), Average, Minimum, Maximum | ComponentType, ComponentName |
PT1M | Yes |
| Ingestion health and performance | Events Received Number of events received by data connection. |
EventsReceived |
Count | Total (Sum), Average, Minimum, Maximum | ComponentType, ComponentName |
PT1M | Yes |
| Export health and performance | Export Utilization Export utilization |
ExportUtilization |
Percent | Maximum | <none> | PT1M | Yes |
| Cluster health | FollowerLatency The follower databases synchronize changes in the leader databases. Because of the synchronization, there's a data lag of a few seconds to a few minutes in data availability.This metric measures the length of the time lag. The time lag depends on the overall size of the leader database metadata.This is a cluster level metrics: the followers catch metadata of all databases that are followed. This metric represents the latency of the process. |
FollowerLatency |
MilliSeconds | Average, Maximum, Minimum | State, RoleInstance |
PT1M | Yes |
| Ingestion health and performance | Ingestion Latency Latency of data ingested, from the time the data was received in the cluster until it's ready for query. The ingestion latency period depends on the ingestion scenario. |
IngestionLatencyInSeconds |
Seconds | Average, Maximum, Minimum | IngestionKind |
PT1M | Yes |
| Ingestion health and performance | Ingestion result Total number of sources that either failed or succeeded to be ingested. Splitting the metric by status, you can get detailed information about the status of the ingestion operations. |
IngestionResult |
Count | Total (Sum) | IngestionResultDetails, FailureKind |
PT1M | Yes |
| Cluster health | Ingestion utilization Ratio of used ingestion slots in the cluster |
IngestionUtilization |
Percent | Average, Maximum, Minimum | <none> | PT1M | Yes |
| Ingestion health and performance | Ingestion Volume Overall volume of ingested data to the cluster |
IngestionVolumeInMB |
Bytes | Total (Sum), Maximum | Database |
PT1M | Yes |
| Cluster health | Instance Count Total instance count |
InstanceCount |
Count | Average, Maximum, Minimum, Count, Total (Sum) | <none> | PT1M | Yes |
| Cluster health | Keep alive Sanity check indicates the cluster responds to queries |
KeepAlive |
Count | Average | <none> | PT1M | Yes |
| Materialized View health and performance | Materialized View Age The materialized view age in minutes |
MaterializedViewAgeMinutes |
Count | Average | Database, MaterializedViewName |
PT1M | Yes |
| Materialized View health and performance | Materialized View Age The materialized view age in seconds |
MaterializedViewAgeSeconds |
Seconds | Average, Minimum, Maximum | Database, MaterializedViewName |
PT1M | Yes |
| Materialized View health and performance | Materialized View Data Loss Indicates potential data loss in materialized view |
MaterializedViewDataLoss |
Count | Maximum | Database, MaterializedViewName, Kind |
PT1M | Yes |
| Materialized View health and performance | Materialized View Extents Rebuild Number of extents rebuild |
MaterializedViewExtentsRebuild |
Count | Average | Database, MaterializedViewName |
PT1M | Yes |
| Materialized View health and performance | Materialized View Health The health of the materialized view (1 for healthy, 0 for non-healthy) |
MaterializedViewHealth |
Count | Average | Database, MaterializedViewName |
PT1M | Yes |
| Materialized View health and performance | Materialized View Records In Delta The number of records in the non-materialized part of the view |
MaterializedViewRecordsInDelta |
Count | Average | Database, MaterializedViewName |
PT1M | Yes |
| Materialized View health and performance | Materialized View Result The result of the materialization process |
MaterializedViewResult |
Count | Average | Database, MaterializedViewName, Result |
PT1M | Yes |
| Partitioning | Partitioning Percentage Percentage of records partitioned versus total number of records. |
PartitioningPercentage |
Percent | Average, Maximum, Minimum | Database, Table |
PT1M | Yes |
| Partitioning | Partitioning Percentage Hot Percentage of records partitioned versus total number of records (in hot / cached extents only). |
PartitioningPercentageHot |
Percent | Average, Maximum, Minimum | Database, Table |
PT1M | Yes |
| Partitioning | Processed Partitioned Records Number of records partitioned in measured time window. |
ProcessedPartitionedRecords |
Count | Average, Maximum, Minimum | Database, Table |
PT1M | Yes |
| Query performance | Query duration Queries duration in seconds |
QueryDuration |
MilliSeconds | Average, Maximum, Minimum, Total (Sum) | QueryStatus |
PT1M | Yes |
| Query performance | Query Result Total number of queries. |
QueryResult |
Count | Count | QueryStatus |
PT1M | No |
| Ingestion health and performance | Queue Length Number of pending messages in a component's queue. |
QueueLength |
Count | Average | ComponentType |
PT1M | Yes |
| Ingestion health and performance | Queue Oldest Message Time in seconds from when the oldest message in queue was inserted. |
QueueOldestMessage |
Count | Average | ComponentType |
PT1M | Yes |
| Ingestion health and performance | Received Data Size Bytes Size of data received by data connection. This is the size of the data stream, or of raw data size if provided. |
ReceivedDataSizeBytes |
Bytes | Average, Total (Sum) | ComponentType, ComponentName |
PT1M | Yes |
| Ingestion health and performance | Stage Latency Cumulative time from when a message is discovered until it is received by the reporting component for processing (discovery time is set when message is enqueued for ingestion queue, or when discovered by data connection). |
StageLatency |
Seconds | Average | Database, ComponentType |
PT1M | Yes |
| Streaming Ingest | Streaming Ingest Data Rate Streaming ingest data rate |
StreamingIngestDataRate |
Bytes | Average, Minimum, Maximum | <none> | PT1M | Yes |
| Streaming Ingest | Streaming Ingest Duration Streaming ingest duration in milliseconds |
StreamingIngestDuration |
MilliSeconds | Average, Minimum, Maximum | <none> | PT1M | Yes |
| Streaming Ingest | Streaming Ingest Result Streaming ingest result |
StreamingIngestResults |
Count | Count | Result |
PT1M | Yes |
| Streaming Ingest | Streaming Ingest Utilization Streaming Ingest Utilization is the percentage of actual concurrent streaming ingestion requests performed, compared to the maximum number of concurrent streaming ingestion requests. |
StreamingIngestUtilization |
Percent | Average, Maximum, Minimum | <none> | PT1M | Yes |
| Query performance | Total number of concurrent queries Total number of concurrent queries |
TotalNumberOfConcurrentQueries |
Count | Average, Maximum, Minimum | <none> | PT1M | Yes |
| Cluster health | Total number of extents Total number of data extents |
TotalNumberOfExtents |
Count | Average, Maximum, Minimum | <none> | PT1M | Yes |
| Cluster health | Total number of throttled commands Total number of throttled commands |
TotalNumberOfThrottledCommands |
Count | Average, Maximum, Minimum, Total (Sum) | CommandType |
PT1M | Yes |
| Query performance | Total number of throttled queries Total number of throttled queries |
TotalNumberOfThrottledQueries |
Count | Average, Maximum, Minimum, Total (Sum) | <none> | PT1M | Yes |
| Query performance | Weak consistency latency The max latency between the previous metadata sync and the next one (in DB/node scope) |
WeakConsistencyLatency |
Seconds | Average, Maximum, Minimum | Database, RoleInstance |
PT1M | Yes |
Metric dimensions
For information about what metric dimensions are, see Multi-dimensional metrics.
This service has the following dimensions associated with its metrics.
- CommandType
- ComponentName
- ComponentType
- ContinuousExportName
- Database
- FailureKind
- IngestionKind
- IngestionResultDetails
- Kind
- MaterializedViewName
- QueryStatus
- Result
- RoleInstance
- SealReason
- State
- Table
Resource logs
This section lists the types of resource logs you can collect for this service. The section pulls from the list of all resource logs category types supported in Azure Monitor.
Supported resource logs for Microsoft.Kusto/clusters
| Category | Category display name | Log table | Supports basic log plan | Supports ingestion-time transformation | Example queries | Costs to export |
|---|---|---|---|---|---|---|
Command |
Command | ADXCommand Azure Data Explorer command execution summary. |
No | Yes | No | |
DataOperation |
Data operation | ADXDataOperation Azure Data Explorer data operation summary. |
No | No | Yes | |
FailedIngestion |
Failed ingestion | FailedIngestion Failed ingestion operations logs provide detailed information about failed ingest operations. Logs include data source details, as well as error code and failure status (transient or permanent), that can be used for tracking the process of data source ingestion. Users can identify usage errors (permanent bad requests) and handle retries of transient failures. Ingestion logs are supported for queued ingestion to the ingestion endpoint using SDKs, data connections, and connectors. |
No | Yes | Queries | No |
IngestionBatching |
Ingestion batching | ADXIngestionBatching Azure Data Explorer ingestion batching operations. These logs have detailed statistics of batches ready for ingestion (duration, batch size and blobs count). |
No | No | Queries | No |
Journal |
Journal | ADXJournal Azure Data Explorer journal (metadata operations). |
No | Yes | Yes | |
Query |
Query | ADXQuery Azure Data Explorer query execution summary. |
No | Yes | No | |
SucceededIngestion |
Succeeded ingestion | SucceededIngestion Succeeded ingestion operations logs provide information about successfully completed ingest operations. Logs include data source details that together with |
No | Yes | Queries | No |
TableDetails |
Table details | ADXTableDetails Azure Data Explorer table details. |
No | Yes | No | |
TableUsageStatistics |
Table usage statistics | ADXTableUsageStatistics Azure Data Explorer table usage statistics. |
No | Yes | Queries | No |
Azure Monitor Logs tables
This section lists the Azure Monitor Logs tables relevant to this service, which are available for query by Log Analytics using Kusto queries. The tables contain resource log data and possibly more depending on what is collected and routed to them.
Azure Data Explorer Microsoft.Kusto/Clusters
-
For information about error codes, see Ingestion error codes.
-
For information about batching types, see Batching policy.
Activity log
The linked table lists the operations that can be recorded in the activity log for this service. These operations are a subset of all the possible resource provider operations in the activity log.
For more information on the schema of activity log entries, see Activity Log schema.
Related content
- See Monitor Azure Data Explorer for a description of monitoring Azure Data Explorer.
- See Monitor Azure resources with Azure Monitor for details on monitoring Azure resources.