Centralized rsyslog server

---

Description

We have a centralized rsyslog server that receives all syslog messages from the servers and networking devices from our infrastructure.

We’ll not discuss the redundancy part of the architecture so I’ll consider having a single rsyslog server with all necessary resources for storing and processing logs.

On centralized rsyslog server, part of the messages are stored locally and ones that are relevant to security will be sent to a graylog server/architecture  (https://www.graylog.org/) for further processing.

Explanation

So, on all our servers from our infrastructure we’ll have a simple sentence that will say how to export the logs through syslog protocol to the centralized place. For exemple, if the address of the centralized rsyslog server is 192.168.100.1, than we have to add to rsyslog.conf following sentence on all servers that will export logs:

@192.168.100.1:514

Communication between centralized rsyslog server and graylog server is through port 10514 UDP.

On centralized server we are applying custom filters to the logs.

First we are creating a custom log format that satisfies our needs as follows:

%$year%-%$month%-%$day% %$hour%:%$minute% %FROMHOST% (%HOSTNAME%): %syslogtag% %msg%\n

After that rules are created:

– if messages are coming from host ip addresses 1.1.1.1/1.1.1.2 which are ldap servers we want to store logs locally and we don’t send them to graylog. This is final. Please see the special directive that ends the statement & ~.

– if messages are from syslog facilities (0, 1, through 7) we are storing messages locally and also we are sending them to graylog server for further processing.

–  if messages are from syslog facilities like auth or cron we want to store messages locally and send them to graylog for further processing.

Some observations

–  We can use filters based on other parameters but syslogfacility-text and fromhost-ip are most common.

– In order to take care of disk space on rsyslog we can use logrotate.

–  Graylog is a very powerful tool and can give you a good image of what is happening in your infrastructure. I was using it for example to count the number of ssh authentication failure on different servers. This one in conjunction with fail2ban will give a good sleep to a SysAdmin.

Implementation

# rsyslog v5 configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html

# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)

$ModLoad imklog # provides kernel logging support (previously done by rklogd)

#$ModLoad immark # provides –MARK– message capability

# Provides UDP syslog reception

$ModLoad imudp

$UDPServerRun 514

# Provides TCP syslog reception

$ModLoad imtcp

$InputTCPServerRun 514

#### GLOBAL DIRECTIVES ####

# Use default timestamp format

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,

# not useful and an extreme performance hit

#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/

$IncludeConfig /etc/rsyslog.d/*.conf

#### RULES ####

# Log all kernel messages to the console.

# Logging much else clutters up the screen.

#kern.* /dev/console

# Log anything (except mail) of level info or higher.

# Don’t log private authentication messages!

*.info;mail.none;authpriv.none;cron.none;local*.none /var/log/messages

#*.info;mail.none;authpriv.none;cron.none;local0.none;local1.none;local2.none;local3.none;local4.none;local5.none;local6.none /var/log/messages

# The authpriv file has restricted access.

authpriv.* /var/log/secure

# Log all the mail messages in one place.

mail.* -/var/log/maillog

# Log cron stuff

cron.* /var/log/cron

# Everybody gets emergency messages

*.emerg *

# Save news errors of level crit and higher in a special file.

uucp,news.crit /var/log/spooler

# Save boot messages also to boot.log

#local7.* /var/log/boot.log

# ### begin forwarding rule ###

# The statement between the begin … end define a SINGLE forwarding

# rule. They belong together, do NOT split them. If you create multiple

# forwarding rules, duplicate the whole block!

$umask 0000

$FileCreateMode 0640

$DirCreateMode 0750

$FileOwner root

$FileGroup netadmin

$template customFormat,”%$year%-%$month%-%$day% %$hour%:%$minute% %FROMHOST% (%HOSTNAME%): %syslogtag% %msg%\n”

if $fromhost-ip==’1.1.1.1′ then /logs/ldap-01.log;customFormat

& ~

if $fromhost-ip==’1.1.1.2′ then /logs/ldap-02.log;customFormat

&~

if $fromhost-ip==’10.2.2.2′ then /logs/server1.log;customFormat

& @graylog-01:10514

& ~

if $fromhost-ip==’10.2.2.4′ then /logs/server2.log;customFormat

& @graylog-01:10514

& ~

if $syslogfacility-text == ‘local0’ then /logs/local0.log;customFormat

& @graylog-01:10514

& ~

if $syslogfacility-text == ‘local1’ then /logs/local1.log;customFormat

& @graylog-01:10514

& ~

if $syslogfacility-text == ‘local2’ then /logs/local2.log;customFormat

& @graylog-01:10514

& ~

if $syslogfacility-text == ‘local3’ then /logs/local3.log;customFormat

& @graylog-01:10514

& ~

if $syslogfacility-text == ‘local4’ then /logs/local4.log;customFormat

& @graylog-01:10514

& ~

if $syslogfacility-text == ‘local5’ then /logs/local5.log;customFormat

& @graylog-01:10514

& ~

if $syslogfacility-text == ‘local6’ then /logs/local6.log;customFormat

& @graylog-01:10514

& ~

if $syslogfacility-text == ‘local7’ then /logs/local7.log;customFormat

& @graylog-01:10514

& ~

if $syslogfacility-text == ‘authpriv’ or $syslogfacility == ‘auth’ then /logs/authpriv.log;customFormat

& @graylog-01:10514

& ~

if $syslogfacility-text == ‘cron’ then /logs/cron.log;customFormat

& @graylog-01:10514

& ~