Freigeben über

Syslog collection in Operations Management Suite

  • Artikel

Summary: Learn how to configure and use the OMS Agent for Linux for syslog collection.

Good morning everyone, Anurag Gupta from the OMS team here, and today I want to talk about syslog collection with Operations Management Suite.

One of the core features with Linux management in Operations Management Suite (OMS) is collection of Syslog events. Linux syslog events can range from kernel logs to application data and audit logs. In addition to Linux servers, many network devices, firewalls, and applications allow native syslog forwarding.

With the OMS Agent for Linux, you can bring all of these forwarded syslog messages into the powerful OMS Log Analytics platform to quickly search through meaningful logs or alert on certain syslog events.

This post gives an overview of syslog collection, steps to enable high-volume syslog collection, and useful scenarios with syslogs.

Overview

Illustration that shows how a Linux computer forwards a syslog to the OMS Agent for Linux and then to OMS.

The OMS Agent for Linux, which is based on Fluentd, includes native support for syslog protocol messages. When the OMS Agent for Linux is installed, it automatically configures other syslog applications (rsyslog and syslog-ng) to write all warning and above events to its listener on port 25224.

If you are already forwarding events to rsyslog or syslog-ng on a single Linux server by simply installing the OMS Agent for Linux, all events that are warning or higher are automatically routed to the OMS service. If you want to enable additional log levels or facilitate collection, refer to Configuring syslog collection from the OMS portal .

Configuring the OMS Agent for Linux for direct forwarding

Additionally, instead of forwarding syslog messages to rsyslog or syslog-ng, syslog messages can be routed directly to the OMS Agent for Linux.

To enable direct collection, edit the configuration file located at /etc/opt/microsoft/omsagent/conf/omsagent.conf.

1. Add a new source in the configuration file to ingest the incoming syslog data.

a. Replace <TAGNAME> with a custom name for this new incoming data stream e.g. netdevice.

b. Replace the <UDP or TCP> with the protocol_type that’s required.

c. Change port settings and bind address as needed.

<source>

type syslog

port 25226

bind 0.0.0.0

tag oms.<TAGNAME>

protocol_type <UDP OR TCP>

</source>

2. Add a new filter in the configuration file to format the incoming syslog data.

a. Replace <TAGNAME> with the same custom name that you used in Step 1.

<filter oms.<TAGNAME>.**>

type filter_syslog

</filter>

3. Restart omsagent for new settings to take effect.

service omsagent restart or systemctl restart omsagent

Configuring the OMS Agent for Linux for high-volume syslog collection

By default, the OMS Agent for Linux can handle a throughput of 500 messages/second without change in the configuration. The following configuration changes throughput to collect 2000 messages/second.

Note: With high-volume syslog collection, we recommend the TCP protocol to avoid message loss.

1. Enable new OMS Syslog endpoint by using the previous instructions.

2. Edit the OMS Agent for Linux configuration file.

a. Add more output threads

b. Change buffer size

c. Reduce time for retries

<match oms.** docker.**>

type out_oms

log_level info

num_threads 10

buffer_chunk_limit 5m

buffer_type file

buffer_path /var/opt/microsoft/omsagent/state/out_oms*.buffer

buffer_queue_limit 30

flush_interval 20s

retry_limit 10

# Do not wait more than 5 minutes before sending again

max_retry_wait 300s

retry_wait 2s

</match>

Syslog scenarios

Alert on firewall blocks

Because many firewall systems allow native syslog forwarding, it’s simple to send these events to OMS through the OMS Agent for Linux. After these logs are in OMS, we can use the power of OMS Log Analytic alerts to mark when specific firewall events are detected and route these events to required owners via email or a Webhook.

As a sample case, if we are using the Universal Firewall on systems that are based on Debian, we can alert every time we receive a “UFW BLOCK” event. We can quickly search for interesting events in Log Analytics and then select the Alert option on the top bar.

Screenshot of that shows events in Log Analytics that are related to the UFW-BLOCK event. Screenshot of an alert rule for the UFW Block event.

For a full list of OMS Log Analytic alert features, check out the Enable Microsoft Operations Management Suite email alerts blog post.

Using Minify to detect patterns

Because syslog contains a multitude of types, the ability to read and gain insight about those logs in a scalable manner becomes a challenge. Minify, which is one of the latest features of OMS Log Analytics, can help you quickly find meaningful messages and cluster syslog events.

In Search, click the Minify tab to start to bundle messages and give you the ability to quickly see commonality among events.

Screenshot of Minify as it collects results from the previous seven days.

For a detailed view of Minify functionality, see the Introducing Minify! blog post.

That is all I have for you today.

Get a free Microsoft Operations Management Suite (#MSOMS) subscription so that you can test the new alerting features. You can also get a free subscription for Microsoft Azure.

I invite you to follow me on Twitter and the Microsoft OMS Facebook site. If you want to learn more about Windows PowerShell, visit the Hey, Scripting Guy Blog. If you have any questions, send email to me at scripter@microsoft.com. I wish you a wonderful day, and I’ll see you tomorrow.

Anurag Gupta
Microsoft Operations Management Team

Comments

  • Anonymous
    May 11, 2016
    Thanks to share this post! A very good guide especially for a beginner like me. Thanks for your information, thank you for making it easy and very detailed. I’ll questions for some here, if I need help, hope that’s okay.
  • Anonymous
    May 12, 2016
    Thanks to share this post! ;)
  • Anonymous
    August 17, 2016
    The comment has been removed
    • Anonymous
      August 17, 2016
      I send syslog messages to the omsagent on 9514 but I only ever see local Linux message from the Linux host in the OMS Dashboard.I don't see any exysyslog messages.
      • Anonymous
        October 04, 2016
        This is the same issue I'm having. Did you ever figure it out?
  • Anonymous
    August 31, 2016
    The comment has been removed
  • Anonymous
    October 04, 2016
    So I have followed the instructions but I still can't get it to work. I'm wondering if it's because not all of the facilities are supported by OMS. While OMS supposedly works off the RFC 3164 spec, it doesn't seem to support all facility codes, namely 13 and 14, log audit and log alert. Since that's what my devices use, those messages are never getting to OMS. They are making it to the server which is talking to OMS, but OMS is not picking them up. I know that the keyword/shortname for log audit and log alert vary between application, but it is a fixed facility number. It would be great if OMS would let you enter the Facility by number.
  • Anonymous
    August 14, 2017
    We have a hardened Linux-based appliance with the capability to forward syslog data but no way to install the OMS agent, so this solution doesn't work in this case.