How do you stop duplicate CEF and Syslog entries with the new Azure Monitor Agent

Question

Hi there, I have the new Azure Monitor Agent for Linux installed and have created and run the new Data Collection Rule set without issue. I now have CEF and Syslog coming through but want to filter out CEF from Syslog.

In /etc/rsyslog.d I created a new file called 5-cef.conf with the line:

if ($rawmsg contains "CEF:") or ($rawmsg contains "%ASA-") then @@127.0.0.1:25226

& stop

...but I still keep getting the duplicates. The docs have data for the old OMS agent but I cant see if I am doing something wrong with the new AMA to make this filter work. I tried the Sentinel Ninja training but it is still focused on the old OMS agent.

Any help appreciated.

Thx

Answer 1

Hi Lloyd Carnie,

To avoid having data ingestion duplication when using the same facility for both Syslog and CEF, please try one of these methods:

• If the source device enables configuration of the target facility: On each source machine that sends logs to the log forwarder in CEF format, edit the Syslog configuration file to remove the facilities used to send CEF messages. This way, the facilities sent in CEF won't also be sent in Syslog. Make sure that each DCR you configure in the next steps uses the relevant facility for CEF or Syslog respectively.
• If changing the facility for the source appliance isn't applicable: Use an ingest time transformation to filter out CEF messages from the Syslog stream to avoid duplication. The data will be sent twice from the collector machine to the workspace:

source |
where ProcessName !contains \"CEF\"

Here are some updated documents you might find useful:
https://learn.microsoft.com/en-us/azure/sentinel/connect-cef-ama
https://learn.microsoft.com/en-us/azure/sentinel/connect-cef-syslog

If you're still having trouble after trying the above, please let us know and we can dive deeper!

Best,
Jessica

---

Please "Accept the answer" (Yes), and share your feedback if the suggestion answers you’re your query. This will help us and others in the community as well.