How to debug if my DCR isn't working

Caday, Belmark June 0

I have an on prem Linux server that was onboarded successfully through Azure Agent. I verified the server's status through KQL -> Heartbeat

My purpose for this is to collect Custom application logs from this server and feed it to Log Analyitcs Workspace through DCR. I have followed everything here:

https://learn.microsoft.com/en-us/azure/azure-monitor/agents/data-sources-custom-logs

I have successfully created the Log Analytics Workspace -> Custom table under this workspace

Created the DCR as well, connect the on prem server as resource for this DCR, add the file patterns and file location in the source as well.

But I can't see the rawdata when I try to query it in LogAnalytics Workspace -> Logs using

MyCustomTable_CL | project RawData

So how to debug if the Data Collection Rule is really working? or where is the issue in this?

Here is my Source config:

User's image

And here is what the log data looks like in my Linux server:

2025-01-28 08:04:44(PID = 79773) (HOST = 98yd98sadsaddsa.org) INFO - DB CONNECT 'dsdsada'

This is inside .log file

Naveena Patlolla 5 Reputation points Microsoft Vendor

2025-01-29T06:21:18.2166667+00:00

Hi Caday, Belmark June,

Thanks for posting your query and using Microsoft Q&A Forum,

We are checking with our internal team to get more information related to your query and will get back to you as soon as once we have an update.
Naveena Patlolla 5 Reputation points Microsoft Vendor

2025-01-29T13:11:05.7266667+00:00
Hi Caday, Belmark June,

Since you're not seeing the raw data when querying, can you please try the below troubleshooting steps:

Check Data Collection Status: Ensure that the data collection is active and not hitting any limits. You can run the following query in KQL to check the status:

Operation | where OperationCategory == 'Data Collection Status'

Look for OperationStatus values of Succeeded or Warning

Check Logs for Errors: Look at the DCR error logs for any ingestion errors. You can query the DCRLogErrors table to see if there are any logged errors related to your DCR. Azure Monitor Agent logs are stored in:

var/opt/microsoft/azuremonitoragent/log/mdsd.log

Verify the File Path in DCR

Ensure the file path matches exactly what’s on the server.

The path should be absolute and not relative.

var/log/mycustomapp*.log

If you specified a folder, ensure all files inside match the pattern. Manually Test the DCR

Please refer the below MS document: https://learn.microsoft.com/en-us/azure/azure-monitor/agents/data-sources-custom-logs

let us know if you have any further queries. I’m happy to assist you further.
Naveena Patlolla 5 Reputation points Microsoft Vendor

2025-01-30T05:11:58.5066667+00:00

Hi Caday, Belmark June,

Just want to check if the above Comment worked for you or else please let us know if any help, we are always here to help whenever you need us.

If the comment is helpful, please click "Upvote it"

Thankyou
Caday, Belmark June 0 Reputation points

2025-01-30T05:32:53.9466667+00:00

Hi @Naveena Patlolla So what does it mean if :

Operation | where OperationCategory == 'Data Collection Status'

and

DCRLogErrors

doesn't return any result? even if I adjusted the timeframe. Using

Heartbeat

I can still see result proving that the resource (on-perm server that was on-boarded) is still connected via Azure Arc.
Madugula Jahnavi 0 Reputation points Microsoft Vendor

2025-01-30T09:29:13.2466667+00:00

Hello Caday, Belmark June, can you try checking the DCR is correctly associated to the Linux server that you are using! Also, verify if the agent is sending the logs to the server.

Check the logs with command: sudo journalctl -u azuremonitor-agent
Caday, Belmark June 0 Reputation points

2025-01-30T10:18:16.7066667+00:00

@Madugula Jahnavi "verify if the agent is sending the logs to the server" How?
Madugula Jahnavi 0 Reputation points Microsoft Vendor

2025-01-30T10:21:54.5066667+00:00

Visit your Linux server and try to execute the command I provided. Caday, Belmark June
Caday, Belmark June 0 Reputation points

2025-01-30T11:03:55.0833333+00:00

The command doesn't return anything. Also I checked this log file mdsd.err, it has an interesting entry which i am unable to make sense:

Mcs Callback didn't result in a configuration download. Abort mdsd./etc/opt/microsoft/azuremonitoragent/config-cache/etc/opt/microsoft/azuremonitoragent/config-cache/mcsconfig.lkg.json/etc/opt/microsoft/azuremonitoragent/config-cache/mcsconfig.latest.json
Madugula Jahnavi 0 Reputation points Microsoft Vendor

2025-01-30T11:49:01.5233333+00:00

This is strange error. Have you referred this https://learn.microsoft.com/en-us/azure/azure-monitor/agents/azure-monitor-agent-troubleshoot-linux-vm#issues-collecting-syslog for troubleshooting it! Caday, Belmark June
Caday, Belmark June 0 Reputation points

2025-01-30T11:52:56.4566667+00:00

Yes, that's what I was reading before I checked the logs in my Linux server. That's how I learned to check the mdsd.* logs

So what does it mean? is it perhaps the cause why the agent is not collecting logs?
Madugula Jahnavi 0 Reputation points Microsoft Vendor

2025-01-30T11:57:40.5666667+00:00

Caday, Belmark June Ok understood. And also, if you still do not able to see the logs emitting in msd files it might be related to authentication issue. Have you added the syslog user to the group of himds as given clearly here in https://learn.microsoft.com/en-us/azure/azure-monitor/agents/azure-monitor-agent-troubleshoot-linux-vm#troubleshooting-issues-on-arc-enabled-server.
Caday, Belmark June 0 Reputation points

2025-01-30T12:01:19.5366667+00:00
I don't see any authentication related error in mdsd.err log file.

Yes, syslog is already part of group himds, verified by executing the following command: id syslog
Madugula Jahnavi 0 Reputation points Microsoft Vendor

2025-01-30T12:37:56.72+00:00

Have you checked the Linux server network settings whether they are blocked by any inbound or outbound internet connection, which is making the agent cannot collecting the custom logs. Caday, Belmark June
Caday, Belmark June 0 Reputation points

2025-01-30T12:51:52.6766667+00:00

I successfully on-boarded the Linux machine Azure Arc, that means it has connectivity to Azure. Ping www.google.com is good too, so it's not internet connection issue. And again, I verified that the resource (my Linux machine) is connected to Azure, how? Via Heartbeat in KQL.

Share via

How to debug if my DCR isn't working

Your answer