Syslog-ng configuration: Building a log aggregation server

Tip

Syslog-ng configuration: Building a log aggregation server

The system logging facility syslog has a great deal of power largely untapped by many enterprises. Syslog originated on BSD Unix

Continue Reading This Article

Enjoy this article as well as all of our content, including E-Guides, news, tips and more.

and is now available on all modern Unix-like systems, including GNU Linux  Every systems administrator knows how to log in to a remote system and seek clues in the local log files, but some may not realize that syslog messages can be sent over the network and stored centrally for easy perusal. The syslog daemons included with modern systems allow you to receive log messages from the network and write them either to disk files or back onto the network. However, these features are often undeveloped or lack extensive documentation.

The syslog-ng framework is an alternative implementation that acts as a drop-in replacement for the default syslog daemon on most systems and offers outstanding flexibility for receiving, filtering and writing log messages. The maintainers of syslog-ng, BalaBit IT Security, offer the framework in an open source edition -- free software licensed under the GNU GPLv2 -- as well as a premium edition with some enhanced capabilities.

Here, we’ll walk through installation and syslog-ng configuration of OSE version 2.1 on a Red Hat Enterprise Linux (RHEL) 5.5 or CentOS 5.5 server so that log messages it receives are written in an organized structure of disk files. Fancier setups are possible with additional configuration, including writing log messages into a relational database with full-text search functionality.

Down to business: Installing EPEL and syslog-ng
We'll assume that you're starting with a fresh install of RHEL or CentOS on a 64-bit Intel server. Since the official repositories for these operating systems don’t provide syslog-ng, we'll need to install the Extra Packages for Enterprise Linux (EPEL) repository. From the command line and as the root user, run the following command:

rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm

Once the EPEL repository package is installed, the YUM package manager can install the syslog-ng package from EPEL:

yum install syslog-ng

YUM will calculate the full list of packages needed to install and will prompt you to confirm before proceeding. It also may ask if you want to import the GPG key for the EPEL repository. If everything looks correct, answer affirmatively to both prompts.

With syslog-ng now installed, there's no need to run the default syslog daemon. Execute the following commands as root to stop the default syslog daemon and to disable its automatic startup on subsequent reboots:

/sbin/service syslog stop

/sbin/chkconfig syslog off

Then start the syslog-ng daemon and set it to start automatically on each boot:

/sbin/service syslog-ng start

/sbin/chkconfig syslog-ng on

Traffic cop: Configuring the firewall
Since we'll receive syslog messages via the network, we'll need to configure the system's host firewall to allow appropriate traffic. The default network transport for syslog is UDP port 514. You can modify the firewall directly by editing the file /etc/sysconfig/iptables and adding the following line just above the REJECT entry at the bottom:

-A RH-Firewall-1-INPUT -m state --state NEW -m udp -p udp --dport 514 -j ACCEPT

Then restart the firewall by issuing this command as root:

/sbin/service iptables restart

Alternatively, you may use the GUI Security Level and Firewall tool, found under the SystemAdministration menu on the desktop (Figure 1).

(click image for a larger picture) 
Figure 1
: Configuring the firewall via the GUI to allow syslog traffic from the network

Making it so: Syslog-ng configuration
By default, syslog-ng chains together the host names of all systems that a given log message passes through. This is useful if you're relaying log messages through several syslog servers, but it's less confusing to have only the originating host logged. Another default behavior that must be changed for this example is the automatic creation of log file destination directories. It's disabled by default, but our dynamic hierarchical layout requires its enablement. Both options are configured globally in the options section near the top of /etc/syslog-ng/syslog-ng.conf. The additions needed appear below in bold:

options {

  sync (0);

  time_reopen (10);

  log_fifo_size (1000);

  long_hostnames (off);

  use_dns (no);

  use_fqdn (no);

  create_dirs (no);

  keep_hostname (yes);

  chain_hostnames (no);

  create_dirs (yes);

};

The default message source, s_sys, accepts log messages only from syslog-ng, the kernel (via the /proc/kmsg interface) and local processes (via the /dev/klog special device node). You also want to read messages from a UDP network socket. It's nice to differentiate those network messages from local ones, so you can define a separate source by adding the following lines after the close of the default s_sys definition in syslog-ng.conf:

source s_net {

  udp(ip(0.0.0.0) port(514));

};

You also must define a destination that points into the hierarchical directory structure. Syslog-ng will use this destination to write the messages it receives from hosts on the network. For the following example, we’ll organize the files into directories according to the year, month and day, and then into one file per remote host. This definition will be called d_hier, which fits the convention of the existing destinations defined in the file by default. Add the following definition after the existing d_mlal destination in syslog-ng.conf:

destination d_hier {

  file("/var/log/net-daily/$YEAR/$MONTH/$DAY/$HOST.log");

};

A different directory structure may, however, better meet your needs; you should experiment to find a layout that suits you. A reference of the various macro names is available for use in destination definitions on the syslog-ng server in the file /usr/share/doc/syslog-ng/syslog-ng.txt.

Finally, we need to tie together our network message source (s_net) with a message filter. Use the existing f_default -- which approximates the familiar set of messages normally found on Linux systems in /var/log/messages -- and your destination (d_hier) by defining a log action. Add the following definition near the bottom of syslog-ng.conf, below the last existing log clause:

log { source(s_net); filter(f_default); destination(d_hier); };

Save your changes to syslog-ng.conf and issue the following command as root to reload the configuration:

/sbin/service syslog-ng reload

Next, configure a remote system on your network to send syslog messages to your new log aggregation server. This operation differs from one operating system to the next, but it’s often simply changing the /etc/hosts file so that the name loghost resolves to the IP address of the syslog-ng server. Sometimes it's necessary to explore /etc/syslog.conf or a similar file. If you're unsure how to proceed on a given system, the operating system's manual page for syslog often makes a good starting point. After you've made the necessary changes, you'll see files appear on the syslog-ng server under the directory structure designed earlier. For example, the path name for log messages from my laptop is /var/log/net-daily/2011/02/07/flyingfox.log.

Here’s some food for thought: Servers aren't the only devices that support syslog. Most switches and routers -- and even some printers and other network-connected devices -- can send syslog messages over the network. Check the Web-based management console or CLI manual of your devices for details.

This tip demonstrated how a syslog-ng configuration can be the basis of an enterprise-wide log aggregation server and greatly simplify interactions with your systems' log messages. The next tip in this series will show you how to extend the syslog-ng configuration to turn messages into actionable notifications and alarms in the OpenNMS network management system.

This was first published in February 2011

Disclaimer: Our Tips Exchange is a forum for you to share technical advice and expertise with your peers and to learn from other enterprise IT professionals. TechTarget provides the infrastructure to facilitate this sharing of information. However, we cannot guarantee the accuracy or validity of the material submitted. You agree that your use of the Ask The Expert services and your reliance on any questions, answers, information or other materials received through this Web site is at your own risk.