Stealth Syslog-NG
Remote logging through a one-way network cable,
to a centralized location,
separated by machine, log-type and date.
Concepts and design by Steve Kolars and Mark Mayfield.
Testing, documentation and implementation by Mark Mayfield.
Abstract
Configure the Systems
Configure Syslog-NG
Loghost syslog-ng.conf Example
Logsource syslog-ng.conf Example
Fedora Service Script Example
Bibliography
The purpose of this paper is to inform and explain different aspects of creating a stealth logging
server with Linux and Syslog-ng. The approach discussed was the best fit for our situation.
For background information a logsource refers to a machine that is
configured to send its logs through the network to another machine. The loghost is the machine
receiving logs from one or more logsources. Networking throughout this paper will be with IPv4. The
Linux versions used in the fieldwork for this project are Fedora6 (logsource) and Fedora12 (loghost).
Some variations of BSD with syslog-ng may also be logsources.
This project was designed to take system logs from multiple machines and allow them to be
collected and centralized on one machine, the loghost. In addition, the intention is to be able to achieve
this without giving away the location of the loghost or allowing it to become compromised. The latter
two objectives are meant to be accomplished through the use of a one-way networking cable. The oneway
cable enables the loghost only to receive UDP broadcasts. No TCP connections can be established.
Sending logs to the broadcast address prevents the machine from having packets addressed to its unique
IP address. Although other methods for providing centralized logging security exist, they cannot be
achieved with a one-way cable because two way communications are required. Establishing
connections through reverse ssh tunnels is one such method (“Remote Logging with SSH and Syslog-
NG,” n.d.).
Although this set-up meets our goals, it also introduces several problems.
Problems:
Solutions:
By using machines that have multiple network adapters we can create a staticallyaddressed
network whose sole purpose is to provide an infrastructure for logging. The network will not
have access to any networks outside of its own. The only machines that will be able to see the network
are the logsources and the loghost. All machines connect to a hub and have private static addresses. The
loghost connects to the hub with a one-way cable and is only capable of receiving packets(logs).
Logsources send their logs to the broadcast address and the loghost listens to the broadcast address.
With this method broadcasts only get sent to the logging network and do not interfere or interact with
the production network.
In the example configuration files “broadcast address” refers to the dotted decimal address that
must be used in an in place configuration file.
All machines need to have network adapters which can be dedicated for the logging network
with static addresses. Logsources connect to a hub with regular straight through cables. The loghost
connects to the hub with a customized one-way cable. Schematics and instructions for building a oneway
cable are located at sentinalsecurity.net (“OneWayCable.pdf,” n.d.).
An addressing scheme needs to be chosen with a private address range. It is recommended that a
range is chosen with a subnet mask that allows addresses for all machines in the logging network but
not much more. Fedora uses configuration files in /etc/sysconfig/network-scripts to control network
interfaces. These files have the prefix of ifcfg- followed by the interface name, for example “ifcfg-eth0”.
The contents of these files are shown below.
#
DEVICE=
HWADDR=
ONBOOT=
BOOTPROTO=
IPADDR=
NETMASK=
TYPE=
Following the comment on the first line (#) should be a description of the interface
(manufacturer and model). The second line specifies the device name, example “eth0”. “HWADDR=”
takes the MAC address of the network card. The fourth line determines whether the interface is
activated at boot time and takes either a yes or no. Boot Protocol, is usually dhcp or static, in this case it
should be static. An IP address is specified after “IPADDR=”. “NETMASK=” is the setting for the
network mask and like the IP address field takes a four octet dotted decimal address. The “TYPE=”
field will usually be “Ethernet”.
Example:
# Intel Corporation 82801CAM (ICH3) PRO/100 VE (LOM) Ethernet Controller
DEVICE=eth1
HWADDR=00:08:02:6c:13:f2
ONBOOT=yes
BOOTPROTO=static
IPADDR= 192.168.0.2
NETMASK= 255.255.255.0
TYPE=Ethernet
Syslog-ng is available in the software repositories of many Linux distributions. Using Fedora to
install syslog-ng with yum while rsyslog is installed results in a software conflict error. Unfortunately
yum will remove a large part of the system as dependencies when “yum remove rsyslog” is run. To
resolve this problem a little finesse and a little force is required. At a root prompt “rpm -e --nodeps
rsyslog” or if the user is a sudoer “sudo rpm -e –nodeps rsyslog” will remove rsyslog without taking
half the system with it. After rsyslog is removed “yum install syslog-ng” can be run without any
dependency or software conflict issues.
In some cases syslog-ng is not available in repositories or the version is too old to provide the
needed options (at least version 2 is required for sending to a broadcast address). If the needed version
is not available it msut be compiled from source. Eventlog will need to be compiled first as it is a
dependency for syslog-ng. There are other options syslog-ng can provide that require other
dependencies, but they are not needed in this situation.
Detailed instruction on the basic compilation and dependencies for features we are not using are
located at balabit.com (“Open source logging system | Syslog-ng,” n.d.)Before compiling eventlog the
PKG_CONFIG_PATH variable needs to be adjusted by issuing the command “export
PKG_CONFIG_PATH=/usr/local/lib/pkgconfig:$PKG_CONFIG_PATH”. Henceforth the process
should complete for eventlog and syslog-ng with the standard ./configure, make and make install.
Rsyslog is run as a service in Fedora and if syslog-ng is installed from the repositories it does
too. However when syslog-ng is compiled from source it is not set up as a service. A service script can
be set up manually to achieve the same results. The script should be set up to start early in the boot
process and end late. It should provide the three standard options of start, stop and restart. When
completed it can be placed in /etc/init.d/ and registered with “chkconfig --add syslog-ng”. Check status
by issuing “chkconfig --list syslog-ng” as root or “sudo chkconfig --list syslog-ng” as a sudoer. If
necessary runlevels are set with the command “chkconfig --levels 2345 syslog-ng on”.
Both rsyslog and syslog-ng are derived from the original BSD syslog protocol. For more
information about the BSD syslog protocol review the RFC at tools.ietf.org (“RFC 3164 - The BSD
Syslog Protocol,” n.d.). .
The syslog-ng configuration file by default is located at
/etc/syslog-ng/syslog-ng.conf. The configuration file consists of five elements.
The examples below use the syntax for syslog-ng.conf from syslog-ng version 2. Version 3 contains some syntax changes in its configuration file. At the end of the loghost and source configuration examples there is a section detailing some of the syntax differences between versions 2 and 3.
The loghost must be configured to receive logs through the proper protocol, address and port.
You do not have to include the address but if you have multiple interfaces and you listen on all of them,
you may be leaving a window open that someone will notice. Our loghost's source configuration is
shown below.
source s_sys {
file ("/proc/kmsg" log_prefix("kernel: "));
unix-stream ("/dev/log");
internal();
udp(ip("broadcast address") port(514));
};
The destination entries must also be edited to include remote machines. This can be done either
by editing existing destinations to use variables such as $HOST and $FACILITY or new destinations
can be created and customized for each source.
Syslog-ng comes with sample configuration which can be modified. In this case the sample
configuration file was edited to suit our purposes. The standard destination directories were changed
from “/var/log/” to “/var/log/machines/$HOST/$FACILITY/$MONTH-$DAY-$YEAR/”. This
configuration separates the logs in /var/log/machines into separate folders for each machine. It further
separates the logs by type and then by date.
The source machines must be configured to send their logs to the loghost.
In order to send logs to a machine connected with a receive only cable the source machines must
send the logs without checking if the logs are received or checking whether there is an available
loghost. Syslog-ng supports unix-stream and unix-dgram. Unix-stream checks for the loghost and
requests confirmation of received logs so it will not work. Unix-dgram sends the logs using UDP
without concern for whether anyone is receiving them or not, which is exactly what this situation
requires. The source section also needs to be configured to use UDP locally.
Our source section for a machine sending logs to a loghost through a one-way cable looks like
this.
source s_sys {
file ("/proc/kmsg" log_prefix("kernel: "));
unix-dgram ("/dev/log");
internal();
};
Next, we configure a destination statement. One remote destination entry can be used to send all
logs to the loghost. The loghost can filter the logs when it receives them. Destinations need to specify
the protocol being used (UDP) and the IP address and port number the logs are being sent to. In our
case, because we are using a one-way cable, we are sending to the broadcast address. To enable syslogng
to send messages to a broadcast address the option “so_broadcast(yes)” must be included in the
destination statement. Additional information about so_broadcast and other socket options can be
found at linux.die.net (“ip(7): IPv4 protocol implementation - Linux man page,” n.d.). Our destination
entry looks like this;
destination loghost {
udp(“broadcast address” port(514) so_broadcast(yes));
};
Note: The so_broadcast option was added to syslog-ng in version 2. Versions of syslog-ng prior to 2.0
do not support sending logs to a broadcast address.
Last, we write a log statement that uses our local source and specifies our remote destination.
log {
source(s_sys);
destination(loghost):
};
Syntax differences between syslog-ng versions 2 and 3:
There are other changes but these are the main ones I noticed when updating a configuration file from 1.6 to 3.0.1. When you start syslog-ng you should get warning messages if your configuration file is using outdated syntax.
# syslog-ng configuration file.
#
# This should behave pretty much like the original syslog on RedHat. But
# it could be configured a lot smarter.
#
# See syslog-ng(8) and syslog-ng.conf(5) for more information.
#
options {
sync (0);
time_reopen (10);
log_fifo_size (1000);
long_hostnames (off);
use_dns (no);
use_fqdn (no);
create_dirs (yes);
keep_hostname (yes);
dir_perm(0755);
perm(0600);
};
source s_sys {
file ("/proc/kmsg" log_prefix("kernel: "));
unix-stream ("/dev/log");
internal();
udp(ip("broadcast address") port(514));
};
destination d_cons { file("/dev/console"); };
destination d_mesg { file("/var/log/machines/$HOST/messages"); };
destination d_auth { file("/var/log/machines/$HOST/$FACILITY/$MONTH-$DAY-
$YEAR/secure"); };
destination d_daemon { file("/var/log/machines/$HOST/$FACILITY/$MONTH-$DAY-
$YEAR/daemon.log"); };
destination d_mail { file("/var/log/machines/$HOST/$FACILITY/$MONTH-$DAY-$YEAR/maillog"
sync(10)); };
destination d_spool { file("/var/log/machines/$HOST/spool/$MONTH-$DAY-$YEAR/spooler"); };
destination d_boot { file("/var/log/machines/$HOST/boot/$MONTH-$DAY-$YEAR/boot.log"); };
destination d_cron { file("/var/log/machines/$HOST/$FACILITY/$MONTH-$DAY-$YEAR/cron"); };
destination d_kern { file("/var/log/machines/$HOST/$FACILITY/$MONTH-$DAY-$YEAR/kern"); };
destination d_netfilter{ file("/var/log/machines/$HOST/netfilter/$MONTH-$DAY-
$YEAR/netfilter.log"); };
destination d_mlal { usertty("*"); file("/var/log/machines/$HOST/$FACILITY/$MONTH-$DAY-
$YEAR/emergency.log"); };
destination d_user { file("/var/log/machines/$HOST/$FACILITY/$MONTH-$DAY-
$YEAR/user.log"); };
filter f_kernel { facility(kern); };
filter f_default { level(info..emerg) and
not (facility(mail)
or facility(authpriv)
or facility(cron)); };
filter f_auth { facility(authpriv); };
filter f_daemon { facility(daemon); };
filter f_mail { facility(mail); };
filter f_emergency { level(emerg); };
filter f_news { facility(uucp) or
(facility(news) and level(crit..emerg)); };
filter f_boot { facility(local7); };
filter f_cron { facility(cron); };
filter f_netfilter { facility(kern) and level(info); };
filter f_user { facility(user); };
log { source(s_sys); filter(f_kernel); destination(d_kern); };
log { source(s_sys); filter(f_default); destination(d_mesg); };
log { source(s_sys); filter(f_auth); destination(d_auth); };
log { source(s_sys); filter(f_mail); destination(d_mail); };
log { source(s_sys); filter(f_emergency); destination(d_mlal); };
log { source(s_sys); filter(f_news); destination(d_spool); };
log { source(s_sys); filter(f_boot); destination(d_boot); };
log { source(s_sys); filter(f_cron); destination(d_cron); };
log { source(s_sys); filter(f_daemon); destination(d_daemon); };
log { source(s_sys); filter(f_user); destination(d_user); };
log { source(s_sys); filter(f_netfilter); destination(d_netfilter); };
# syslog-ng configuration file.
# this configuration does not store any logs locally
# it only sends them to the network
options {
sync (0);
time_reopen (10);
log_fifo_size (1000);
long_hostnames (off);
use_dns (no);
use_fqdn (no);
keep_hostname (yes);
};
source s_sys {
file ("/proc/kmsg" log_prefix("kernel: "));
unix-dgram ("/dev/log");
internal();
};
destination loghost {
udp("broadcast address" port(514) so_broadcast(yes));
};
log {
source (s_sys);
destination (loghost);
};
#!/bin/sh
#
# chkconfig: 2345 60 88
# description: Starts and stops the syslog-ng logging system
#
# config: /etc/syslog-ng/syslog-ng.conf
# processname: syslog-ng
# Source function library
. /etc/rc.d/init.d/functions
BASE=syslog-ng
CONF="/etc/syslog-ng/syslog-ng.conf"
# Check that $BASE exists.
[ -f /usr/local/sbin/$BASE ] || exit 0
RETVAL=0
# See how we were called.
case "$1" in
start)
if [ -n "`/sbin/pidof $BASE`" ]; then
echo -n $"$BASE: already running"
echo ""
exit $RETVAL
fi
echo -n "Starting syslog-ng service: "
/usr/local/sbin/$BASE -f $CONF
sleep 1
action "" /sbin/pidof $BASE
RETVAL=$?
[ $RETVAL -eq 0 ] && touch /var/lock/subsys/syslog-ng
;;
stop)
echo -n "Shutting down syslog-ng service: "
killproc $BASE
RETVAL=$?
echo
[ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/syslog-ng
;;
restart|reload)
$0 stop
$0 start
RETVAL=$?
;;
status)
status $BASE
RETVAL=$?
;;
*)
echo "Usage: syslog {start|stop|restart|reload|status}"
exit 1
esac
exit $RETVAL
ip(7): IPv4 protocol implementation - Linux man page. (n.d.). . Retrieved May 1, 2010, from
http://linux.die.net/man/7/ip
OneWayCable.pdf. (n.d.). . Retrieved May 1, 2010, from
http://www.sentinelsecurity.net/whitepapers/OneWayCable.pdf
Open source logging system | Syslog-ng. (n.d.). . Retrieved May 1, 2010, from
http://www.balabit.com/network-security/syslog-ng/opensource-logging-system/support/compiling
OneWayCable.pdf. (n.d.). . Retrieved May 1, 2010, from
http://www.sentinelsecurity.net/whitepapers/OneWayCable.pdf
Remote Logging with SSH and Syslog-NG. (n.d.). . Retrieved May 1, 2010, from
http://www.deer-run.com/~hal/sysadmin/SSH-SyslogNG.html
RFC 3164 - The BSD Syslog Protocol. (n.d.). . Retrieved May 1, 2010, from
http://tools.ietf.org/html/rfc3164
This document can be reposted/redistributed as long as credit is given to the original site and author. Mark Mayfield at http://mark.kandm-solutions.com/mmayfield