Part I - Syslog

1.1) Facility Levels

1.2) Severity levels
1.3) Common Linux log files name and usage
1.) syslog variants
Part II - syslog-ng
2.1) Features!
2.1.1) Client Side Features!
2.1.2) Server Side Features!
2.2) Installation!
2.3) "vervie#
2.) $odes of "%eration!
2..1) Client mode
2..2) &elay mode
2..3) Server mode
2.') (xam%le configuration file
2.)) t*e configuration syntax in detail
2.).1) +lo,al o%tions
2.).2) Syslog-ng modules
2.).2.1) Loading modules
2.).3) $anaging com%lex syslog-ng configurations
2.).3.1) including configuration files!
2.).) Sources
2.)..1) Collecting %rocess accounting logs on Linux!
2.)..2) &eceiving messages from external a%%lications
2.).') -estinations
2.).)) Creating Filters for $essages
2.)..) Log Pat*s
2.)./) $anaging incoming and outgoing messages #it* flo#-control
2.).0) 1ave syslog-ng reload t*e configuration file
2.).12) log to anot*er file
2..) Configuring as a log*ost
2./) Client side configurations
2.0) Im%rove Performance
2.0.1) 3rite every so often
2.0.2) 4void redundant %rocessing and dis5 s%ace
2.12) PostgreS6L -estination
2.11) IS" /)21 timestam%s
2.12) Log Levels
2.13) $acros and 7aria,les
2.1) 8rou,les*ooting syslog-ng!
2.1') Possi,le causes of losing log messages!
2.1)) Procedure 9 Creating syslog-ng core files
2.1.) Collecting de,ugging information #it* strace: truss: or tusc
2.1/) 4%ac*e and $yS6L Logging #it* Syslog-ng
2.10) ,uild a Secure Logging Server #it* syslog-ng using SSL;8LS
2.22) syslog-ng < logrotate
2.21) &eference
Part III - rsyslog
3.1) Features!
3.2) Installation
3.3) Configuration Structure
3.3.1) $odules
3.3.2) Configuration -irectives
3.3.3) &ule line
3.3.) 4 Sam%le Configuration
3.3.') 8em%lates
3.) Central Logging Server
3.') (ncry%ting Syslog 8raffic #it* 8LS =SSL)
3.)) >nrelia,ility of %lain tc% syslog
3..) 8LS secured syslog via &(LP
3./) &eferences!
Part I7. 3(? +>IS to $4@4+( Logging in Linux
Part I - Syslog
Syslog is a standard for computer data logging. It separates the software that generates messages from the system
that stores them and the software that reports and analyzes them. There will be a syslog-compatible service running
on your server that implements the standard. Some applications, such as the apache web server, do not use the
syslog service and instead manage their own logging. If applications implement their own logging their
configuration files will say where to find the log files.
Syslog can be used for computer system management and security auditing as well as generalized informational,
analysis, and debugging messages. It is supported by a wide variety of devices (like printers and routers and
receivers across multiple platforms. !ecause of this, syslog can be used to integrate log data from many different
types of systems into a central repository.
"essages are labeled with a facility code (auth, authpriv, daemon, cron, ftp, lpr, kern, mail, news, syslog, user,
uucp, local# to local$ indicating the type of software that generated the messages, and are assigned a severity
(%mergency, &lert, 'ritical, %rror, (arning, )otice, Info, *ebug.
Implementations are available for many operating systems. Specific configuration may permit directing messages
to various devices (console, files (+var+log+ or remote syslog servers. "ost implementations also provide a
command line utility, often called logger, that can send messages to the syslog. Some implementations permit the
filtering and display of syslog messages.
Syslog is now standardized within the Syslog working group of the I%T,.
Syslog was developed in the -./#s by %ric &llman as part of the Sendmail pro0ect, and was initially used solely for
Sendmail. It proved so valuable that other applications began using it as well. Syslog has since become the
standard logging solution on 1ni2 and 1ni2-like systems3 there have also been a variety of syslog implementations
on other operating systems and is commonly found in network devices such as routers.
1.1) Facility Levels
& facility level is used to specify what type of program is logging the message. This lets the configuration file
specify that messages from different facilities will be handled differently.
AB
The list of facilities available4
(defined by &FC 31)
,acility )umber 5eyword ,acility *escription
# kern kernel messages
- user user-level messages
6 mail mail system
7 daemon system daemons
8 auth security+authorization messages
9 syslog messages generated internally by syslogd
: lpr line printer subsystem
$ news network news subsystem
/ uucp 11'; subsystem
. clock daemon
-# authpriv security+authorization messages
-- ftp ,T; daemon
-6 - )T; subsystem
-7 - log audit
-8 - log alert
-9 cron clock daemon
-: local# local use # (local#
-$ local- local use - (local-
-/ local6 local use 6 (local6
-. local7 local use 7 (local7
6# local8 local use 8 (local8
6- local9 local use 9 (local9
66 local: local use : (local:
67 local$ local use $ (local$
1.2) Severity levels
&FC '2 defines eight severity levels4
'ode Severity 5eyword *escription <eneral *escription
# %mergency
emerg
(panic
System is unusable.
& =panic= condition usually affecting multiple
apps+servers+sites. &t this level it would usually notify all
tech staff on call.
- &lert alert
&ction must be
taken immediately.
Should be corrected immediately, therefore notify staff who
can fi2 the problem. &n e2ample would be the loss of a
primary IS; connection.
6 'ritical crit 'ritical conditions.
Should be corrected immediately, but indicates failure in a
secondary system, an e2ample is a loss of a backup IS;
connection.
7 %rror err (error %rror conditions.
)on-urgent failures, these should be relayed to developers or
admins3 each item must be resolved within a given time.
8 (arning
warning
(warn
(arning conditions.
(arning messages, not an error, but indication that an error
will occur if action is not taken, e.g. file system /9> full -
each item must be resolved within a given time.
9 )otice notice
)ormal but
significant
condition.
%vents that are unusual but not error conditions - might be
summarized in an email to developers or admins to spot
potential problems - no immediate action re?uired.
: Informational info
Informational
messages.
)ormal operational messages - may be harvested for
reporting, measuring throughput, etc. - no action re?uired.
$ *ebug debug
*ebug-level
messages.
Info useful to developers for debugging the application, not
useful during operations.
& common mnemonic used to remember the syslog levels from bottom to top is4 =*o I )otice (hen %venings
'ome around %arly=.
In @inu2, &lmost all logfiles are located under +var+log directory (and subdirectory. Aou can change to this
directory using cd command but you need to be the root user. Aou can use less, more, cat or tail command
to see the logs.
1.3) Common Linux log files name and usage
+var+log+messages4 <eneral message and system related stuff
+var+log+auth.log4 &uthenication logs
+var+log+kern.log4 5ernel logs
+var+log+btmp 4 Invalid @ogin &ttempts
+var+log+cron.log4 'rond logs (cron 0ob
+var+log+dmesg 4 5ernel boot messages
+var+log+maillog4 "ail server logs
+var+log+?mail+ 4 Bmail log directory (more files inside this directory
+var+log+httpd+4 &pache access and error logs directory
+var+log+lastlog4 user last login information
+var+log+boot.log 4 System boot log
+var+log+mys?ld.log4 "ySB@ database server log file
+var+log+secure4 &uthentication log
+var+log+utmp or +var+log+wtmp 4 @ogin records file
+var+log+yum.log4 Aum log files
,or more info refer4 http4++www.thegeekstuff.com+6#--+#/+linu2-var-log-files+
In short +var+log is the location where you should find all @inu2 logs file. Cowever some applications such as httpd
have a directory within +var+log+ for their own log files. Aou can rotate log file using logrotate software and
monitor logs files using logwatch software.
1.) Syslog variants
There are several implementations of the syslog standard, including syslog-ng, sys5logd and rsyslogd. These
applications are e2amples of the message logging software used by @inu2, and their configuration files can be
found in the ;etc directory.
"any systems use sysklogd to provide syslog functionality. Its configuration file can be found at ;etc;syslog.conf.
Some other distributions use rsyslogd. *epending on the version, its configuration files will be in either
;etc;rsyslog.conf or ;etc;rsyslog.d; (the default configuration file in that directory is /etc/rsyslog.d/50-
default.conf.
Still other distributions might use syslog-ng, which stores its config file at ;etc;syslog-ng;syslog-ng.conf. Dnce
you find your syslog config file you can search for the facility entry to see where those entries are logged.
Part II - syslog-ng
syslog-ng is an open source implementation of the Syslog protocol for 1ni2 and 1ni2-like systems. It e2tends the
original syslogd model with content-based filtering, rich filtering capabilities, fle2ible configuration options and
adds important features to syslog, like using T'; for transport. &s of today syslog-ng is developed by !alabit IT
Security @td. It has two editions with a common codebase. The first is called syslog-ng Dpen Source %dition
(DS% with the license @<;@. The second is called ;remium %dition (;% and has additional plugins (modules
under proprietary license.
2.1) Features!
Syslog-ng provides a number of features other than transporting syslog messages and storing them to plain te2t log
files4
2.1.1) Client Side Features!
Secure logging using SSL;8LS
@og messages may contain sensitive information that should not be accessed by third parties. Therefore, syslog-ng
uses the Transport @ayer Security (T@S protocol to encrypt the communication. T@S also allows the mutual
authentication of the host and the server using E.9#. certificates.
&elia,le log transfer
The syslog-ng application enables you to send the log messages of your hosts to remote servers. The logs of
different servers can be collected and stored centrally on dedicated log servers. Transferring log messages using
the T'; protocol ensures that no messages are lost.
Su%%ort for standard syslog %rotocols
The syslog-ng application can send and receive traditional syslog messages (as described in F,'7-:8 and also
messages conforming to the latest I%T,-syslog protocol standard (as described in F,'9868 using the 1*;, T';,
or T@S protocol over I;v8 and I;v: networks.
Collecting local messages
The syslog-ng application can collect messages of local applications and processes using 1)IE domain sockets,
named pipes, and plain te2t files. "essages can be received also from the standard output (stdout.
2.1.2) Server Side Features!
Flexi,le message filtering and sorting
The syslog-ng application can sort the incoming log messages based on their content and various parameters like
the source host, application, and priority. *irectories, files, and database tables can be created dynamically using
macros. 'omple2 filtering using regular e2pressions and boolean operators offers almost unlimited fle2ibility to
forward only the important log messages to the selected destinations.
$essage %arsing and re#riting
The syslog-ng application can segment log messages to named fields or columns. These fields can be used like
macros in filenames and database tables. It is also possible to modify the values of these fields, for e2ample, to add
missing values.
$essage classification
The syslog-ng application can compare the contents of the log messages to a database of predefined message
patterns. !y comparing the messages to the known patterns, syslog-ng is able to identify the e2act type of the
messages, and sort them into message classes. The message classes can be used to classify the type of the event
described in the log message. The message classes can be customized, and for e2ample can label the messages as
user login, application crash, file transfer, etc. events.
1andle extreme load
The syslog-ng application is optimized for performance, and can handle enormous amount of messages.
*epending on its e2act configuration, it has been known to process over 9##,### messages per second real-time,
and over 68 <! raw logs per hour on average server hardware.
-irect data,ase access
Storing your log messages in a database allows you to easily search and ?uery the messages and interoperate with
log analyzing applications. The syslog-ng application supports the following databases4 "ySB@, "icrosoft SB@
("SSB@, Dracle, ;ostgreSB@, and SB@ite.
IPv and IPv) su%%ort
The syslog-ng application can operate in both I;v8 and I;v: network environments3 it can receive messages from
and send messages to both types of networks.
$ongo-? su%%ort
1sing "ongo*! instead of SB@ is faster and allows better representation of log data.
4$6P destination
The destination allows one to publish log messages to an &"B;-capable messaging server.
2.2) Installation!
See list of Su%%orted Platforms in ,elo# lin5!
*tt%!;;###.,ala,it.com;net#or5-security;syslog-ng;central-syslog-server;s%ecifications;
a) Com%iling ; installing from source code!
To compile syslog-ng Dpen Source %dition (DS% from the source code, complete the following steps.
-. *ownload the latest version of syslog-ng DS% from https4++www.balabit.com+downloads+filesG
pathH+syslog-ng+sources+. The source code is available as a tar.gz archive file.
6. *ownload the latest version of the %vent@og library available at
https4++www.balabit.com+downloads+files+eventlog+#.6+.
7. Install the following packages that are re?uired to compile syslog-ng. These packages are available for
most 1)IE+@inu2 systems. &lternatively, you can also download the sources and compile them.
o the gcc ' compiler (at least version 6.$.6,
o the GNU flex le2ical analyser generator, available at http4++fle2.sourceforge.net+3
o the bison parser generator, available at http4++ftp.gnu.org+gnu+bison+3
o and the development files of the glib library, available at http4++freshmeat.net+pro0ects+glib+.
8. If you want to use the spoof-source function of syslog-ng, install the development files of the libnet
library, available at http4++libnet.sourceforge.net.
9. If you want to use the /etc/hosts.deny and /etc/hosts.allow for T'; access, install the development files
of the libwrap (also called T';-wrappers library, available at
ftp4++ftp.porcupine.org+pub+security+inde2.html.
:. 1ncompress the eventlog archive using the
$. I tar 2vfz eventlog-2.2.2.2.tar.gz
Dr the
I gunzip -c eventlog-2.2.2.2.tar.gz J tar 2vf -
'ommand. & new directory containing the source code of eventlog will be created.
/. !y default, eventlog creates a file used by the syslog-ng configure script in the /usr/local/lib/pkgconfig
directory. Issue the following command to add this directory to your ;5<K'D),I<K;&TC4
.. ;5<K'D),I<K;&TCH+usr+local+lib+pkgconfig4I;5<K'D),I<K;&TC
-#. %nter the new directory and issue the following commands4
--. I .+configure
-6. I make
-7. I make install
-8. 1ncompress the syslog-ng archive using the
-9. tar 2vfz syslog-ng-2.22.tar.gz
or the
unzip -c syslog-ng-2.22.tar.gz J tar 2vf -
command. & new directory containing the source code of syslog-ng will be created.
-:. %nter the new directory and issue the following commands4
-$. I .+configure
-/. I make
-.. I make install
These commands will build syslog-ng using its default options.
6#. If needed, use the following options to change how syslog-ng is compiled using the following command
synta24
6-. I .+configure --compile-time-option-name
,or an up-to-date list of options run4
I .+configure --help
,) Installing from ?inaries! For list of 7rd party binaries of syslog-ng, refer below link
*tt%!;;###.,ala,it.com;net#or5-security;syslog-ng;o%ensource-logging-system;do#nloads;3rd-%arty
For Cent"S or &1(L: you *ave to install e%el re%o and t*en do Cyum install sysog-ngD. 8*atDs it.
2.3) "vervie#
Syslog-ng takes incoming log messages from defined LsourcesL and forwards them to the appropriate destinations,
based on powerful filter directives. The syslog-ng application can receive messages from files, remote hosts, and
other sources.
@og messages enter syslog-ng in one of the defined sources, and are sent to one or more destinations.
Sources and destinations are independent ob0ects3 log paths define what syslog-ng does with a message,
connecting the sources to the destinations. & log path consists of one or more sources and one or more
destinations3 messages arriving from a source are sent to every destination listed in the log path. & log path defined
in syslog-ng is called a log statement.
Dptionally, log paths can include filters. ,ilters are rules that select only certain messages, for e2ample, selecting
only messages sent by a specific application. If a log path includes filters, syslog-ng sends only the messages
satisfying the filter rules to the destinations set in the log path.
Dther optional elements that can appear in log statements are parsers and rewriting rules. ;arsers segment
messages into different fields to help processing the messages, while rewrite rules modify the messages by adding,
replacing, or removing parts of the messages.
In a typical simple set-up, syslog-ng will read messages from three sources4
-. the default +dev+log device, where most logs are sent
6. syslog-ng =internal= log messages
7. +proc+kmsg kernel messages
Sources are defined using the =source= directive. These incoming messages are then filtered according to defined
filters (=filter= keyword, i.e. according to originating program or log level, and sent to the appropriate
=destination=. *estinations include log files (e.g. +var+log+messages.log, printing messages on a console and
remote servers. The pivotal function is log. This function defines which filters should be applied to a certain
source, and where the resulting messages should be sent to.
2.) $odes of "%eration!
The syslog-ng Dpen Source %dition application has three typical operation scenarios4 lient, !er"er, and #elay.
2..1) Client mode
Figure 2..1. Client-mode o%eration
In client mode, syslog-ng collects the local logs generated by the host and forwards them through a network
connection to the central syslog-ng server or to a relay. 'lients often also log the messages locally into files.
2..2) &elay mode
Figure 2..2. &elay-mode o%eration
In relay mode, syslog-ng receives logs through the network from syslog-ng clients and forwards them to the
central syslog-ng server using a network connection. Felays also log the messages from the relay host into a local
file, or forward these messages to the central syslog-ng server.
2..3) Server mode
Figure 2..3. Server-mode o%eration
In server mode, syslog-ng acts as a central log-collecting server. It receives messages from syslog-ng clients and
relays over the network, and stores them locally in files, or passes them to other applications, for e2ample log
analyzers.
2.') (xam%le configuration file
,or a ?uick start, here there is a classic configuration file. This e2ample includes logging to a remote server
destination.
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
N <lobal Dptions
NOversion47.6
options P
N sync (#3
flushKlines (#3
timeKreopen (-#3
dnsKcacheKhosts(+etc+hosts3
dnsKcacheKe2pire(/$:##3
dnsKcacheKhosts(+etc+hosts3
N chainKhostnames(#3
N longKhostnames (off3
useKdns (persistKonly3
useKf?dn (no3
createKdirs(yes3
owner(root3
group(root3
perm(#:8#3
dirKowner(root3
dirKgroup(root3
dirKperm(#$8#3
keepKhostname (yes3
Nstats (7:##3
statsKfre?(#3
timeKreap(7:#3
tsKformat(iso3
Q3
N*efine network log sources
Nsource sKnet P syslog(ip(-6$.#.#.- port(-### transport(tcp3 Q3
Nsource sKnet P syslog(ip(-6$.#.#.- port(9-8 transport(udp3 Q3
Nsource sKnet P udp(ip(-.6.-:/.6.. port(9-83Q3
source sKnet P tcp(ip(-.6.-:/.6.. port(-9-83 Q3
N*efine @ocal log sources
source sKsys P
file (=+proc+kmsg= logKprefi2(=kernel4 =3
uni2-stream (=+dev+log=3
internal(3
N udp(ip(#.#.#.# port(9-83
Q3
N source sKsyslog P syslog(ip(-6$.#.#.- port(-... transport(=tcp=3 Q3
N source sKpipe P pipe(=+dev+pipe= padKsize(6#8/3 Q3
N *efine destinations for local logs
destination dKcons P file(=+dev+console=3 Q3
destination dKmesg P file(=+var+log+messages=3 Q3
destination dKauth P file(=+var+log+secure=3 Q3
destination dKmail P file(=+var+log+maillog= flushKlines(-#3 Q3
destination dKspol P file(=+var+log+spooler=3 Q3
destination dKboot P file(=+var+log+boot.log=3 Q3
destination dKcron P file(=+var+log+cron=3 Q3
destination dKkern P file(=+var+log+kern=3 Q3
destination dKmlal P usertty(=M=3 Q3
N *efine destinations for remote logs
destination dKremoteKalllogs P file ( =+var+log+syslog+remote+I,1@@CDST+IA%&F+I"D)TC+I*&A+
I,1@@CDST-IA%&F-I"D)TC-I*&A.log= owner(root group(root perm(#:## dirKperm(#$##
createKdirs(yes template(=I*&T% I,1@@CDST I;FD<F&" IT&< RI,&'I@ITA.I@%S%@T
I"%SS&<%Un=3 Q3
Ndestination dKnetKeverything P file(=+var+log+syslog+remote+IA%&F.I"D)TC.I*&A+everything.log=
template(=ICD1F4I"I)4IS%' RI@%S%@T RI,&'I@ITAT RI;FD<F&"T I"S<Un= templateKescape(no3 Q3
destination dKnetKauth P file(=+var+log+syslog+remote+ICDST+auth.log=3 Q3
destination dKnetKcron P file(=+var+log+syslog+remote+ICDST+cron.log=3 Q3
destination dKnetKdaemon P file(=+var+log+syslog+remote+ICDST+daemon.log=3 Q3
destination dKnetKkern P file(=+var+log+syslog+remote+ICDST+kern.log=3 Q3
destination dKnetKlpr P file(=+var+log+syslog+remote+ICDST+lpr.log=3 Q3
destination dKnetKmail P file(=+var+log+syslog+remote+ICDST+mail.log=3 Q3
destination dKnetKsyslog P file(=+var+log+syslog+remote+ICDST+syslog.log=3 Q3
destination dKnetKuser P file(=+var+log+syslog+remote+ICDST+user.log=3 Q3
destination dKnetKuucp P file(=+var+log+syslog+remote+ICDST+uucp.log=3 Q3
destination dKnetKdebug P file(=+var+log+syslog+remote+ICDST+debug=3 Q3
destination dKnetKerror P file(=+var+log+syslog+remote+ICDST+error=3 Q3
destination dKnetKmessages P file(=+var+log+syslog+remote+ICDST+messages=3 Q3
destination dKnetKmailinfo P file(=+var+log+syslog+remote+ICDST+mail+mail.info=3 Q3
destination dKnetKmailwarn P file(=+var+log+syslog+remote+ICDST+mail+mail.warn=3 Q3
destination dKnetKmailerr P file(=+var+log+syslog+remote+ICDST+mail+mail.err=3 Q3
destination dKnetKerror P file(=+var+log+syslog+remote+ICDST+error=3 Q3
destination dKnetKppp P file(=+var+log+syslog+remote+ICDST+ppp.log=3 Q3
destination dKnewscrit P file(=+var+log+syslog+remote+ICDST+news+news.crit=3 Q3
destination dKnewserr P file(=+var+log+syslog+remote+ICDST+news+news.err=3 Q3
destination dKnewsnotice P file(=+var+log+syslog+remote+ICDST+news+news.notice=3 Q3
destination dKconsoleKall P file(=+dev+tty-6=3 Q3
N*efine ,ilters
Nfilter fKdbg P level(debug3 Q3
filter fKinfoP level(info3 Q3
filter fKnoticeP level(notice3 Q3
filter fKwarn P level(warn3 Q3
filter fKerr P level(err3 Q3
filter fKcrit P level(crit..emerg3 Q3
filter fKdebug P level(debug and not facility(auth, authpriv, news, mail3 Q3
filter fKerror P level(err .. emerg 3 Q3
Nfilter fKmessages P level(info,notice,warn and not facility(auth,authpriv,cron,daemon,mail,news3Q3
filter fKauth P facility(auth, authpriv and not filter(fKdebug3 Q3
filter fKcron P facility(cron and not filter(fKdebug3 Q3
filter fKdaemon P facility(daemon and not filter(fKdebug3 Q3
filter fKkern P facility(kern and not filter(fKdebug3 Q3
filter fKlpr P facility(lpr and not filter(fKdebug3Q3
Nfilter fKlocal P facility(local#, local-, local7, local8, local9, local:, local$ and not filter(fKdebug3 Q3
filter fKmail P facility(mail and not filter(fKdebug3 Q3
filter fKnews P facility(news and not filter(fKdebug3 Q3
filter fKsyslog P program(syslog-ng3 Q3
filter fKuser P facility(user and not filter(fKdebug3 Q3
filter fKuucp P facility(uucp and not filter(fKdebug3 Q3
filter fKppp P facility(local6 and not filter(fKdebug3 Q3
filter fKconsole P level(warn..emerg3 Q3
filter fKdefault P level(info..emerg and not (facility(mailor facility(authpriv or facility(cron3 Q3
filter fKemergency P level(emerg3 Q3
Nfilter fKnews P facility(uucp or(facility(news and level(crit..emerg3 Q3
filter fKboot P facility(local$3 Q3
N *efine logging for local logs
Nlog P source(sKsys3 filter(fKkernel3 destination(dKcons3 Q3
log P source(sKsys3 filter(fKkern3 destination(dKkern3 Q3
log P source(sKsys3 filter(fKdefault3 destination(dKmesg3 Q3
log P source(sKsys3 filter(fKauth3 destination(dKauth3 Q3
log P source(sKsys3 filter(fKmail3 destination(dKmail3 Q3
log P source(sKsys3 filter(fKemergency3 destination(dKmlal3 Q3
log P source(sKsys3 filter(fKnews3 destination(dKspol3 Q3
log P source(sKsys3 filter(fKboot3 destination(dKboot3 Q3
log P source(sKsys3 filter(fKcron3 destination(dKcron3 Q3
N*efine @ogging for )etwork logs
log P source(sKnet3 destination(dKremoteKalllogs 3 Q3
Nlog P source(sKnet3 destination(dKnetKeverything3 Q3
log P source(sKnet3 filter(fKauth3 destination(dKnetKauth3 Q3
log P source(sKnet3 filter(fKuser3 destination(dKnetKuser3 Q3
log P source(sKnet3 filter(fKauth3 destination(dKnetKauth3 Q3
log P source(sKnet3 filter(fKuser3 destination(dKnetKuser3 Q3
log P source(sKnet3 filter(fKcron3 destination(dKnetKcron3 Q3
log P source(sKnet3 filter(fKdaemon3 destination(dKnetKdaemon3 Q3
log P source(sKnet3 filter(fKkern3 destination(dKnetKkern3 Q3
log P source(sKnet3 filter(fKlpr3 destination(dKnetKlpr3 Q3
log P source(sKnet3 filter(fKmail3 destination(dKnetKmail3 Q3
log P source(sKnet3 filter(fKuucp3 destination(dKnetKuucp3 Q3
log P source(sKnet3 filter(fKmail3 filter(fKinfo3 destination(dKnetKmailinfo3 Q3
log P source(sKnet3 filter(fKmail3 filter(fKwarn3 destination(dKnetKmailwarn3 Q3
log P source(sKnet3 filter(fKmail3 filter(fKerr3 destination(dKnetKmailerr3 Q3
log P source(sKnet3 filter(fKdebug3 destination(dKnetKdebug3 Q3
log P source(sKnet3 filter(fKnews3 filter(fKcrit3 destination(dKnewscrit3 Q3
log P source(sKnet3 filter(fKnews3 filter(fKerr3 destination(dKnewserr3 Q3
log P source(sKnet3 filter(fKnews3 filter(fKnotice3 destination(dKnewsnotice3 Q3
Nlog P source(sKnet3 filter(fKmessages3 destination(dKnetKmessages3 Q3
log P source(sKnet3 destination (dKnetKmessages3 Q3
log P source(sKnet3 filter(fKppp3 destination(dKnetKppp3 Q3
log P source(sKnet3 filter(fKerror3 destination(dKnetKerror3 Q3
log P source(sKnet3 filter(fKsyslog3 destination(dKnetKsyslog3 Q3
log P source(sKnet3 destination(dKconsoleKall3 Q3
N vim4ftHsyslog-ng4ai4si4tsH84swH84et4
MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM
2.)) t*e configuration syntax in detail
%very syslog-ng configuration file must begin with a line containing the version information of syslog-ng. ,or
syslog-ng version 7.7, this line looks like4
Oversion4 7.7
Sersioning the configuration file was introduced in syslog-ng 7.#. If the configuration file does not contain the
version information, syslog-ng assumes that the file is for syslog-ng version 6.2. In this case it interprets the
configuration and sends warnings about the parts of the configuration that should be updated. Sersion 7.# and later
will correctly operate with configuration files of version 6.2, but the default values of certain parameters have
changed since 7.#.
Some options are global options, or can be set globally, for e2ample, whether syslog-ng DS% should use *)S
resolution to resolve I; addresses.
options P useKdns(no3 Q3
2.).1) +lo,al o%tions
<lobal options are detailed at below link4
http4++www.balabit.com+sites+default+files+documents+syslog-ng-ose-7.7-guides+en+syslog-ng-ose-v7.7-guide-
admin-en+html-single+inde2.htmlNchapter-global-options.
)ote4
&ll identifiers, attributes, and any other strings used in the syslog-ng configuration file are case sensitive.
To add comments to the configuration file, start a line with $ and write your comments. These lines are
ignored by syslog-ng.
Tip4 !efore activating a new configuration, check that your configuration file is syntactically correct using the
Vsyslog-ng --synta2-onlyW command.
To activate the configuration, reload the configuration of syslog-ng using V+etc+init.d+syslog-ng reload Vcommand
or you can run N systemctl reload syslog-ng
)ote4 ,or FC%@ :.2, while reloading syslog-ng , you might get below error.
;lugin module not found in Lmodule-pathL3 module-pathHL+lib:8+syslog-ngL, moduleHLafs?lL
Xust do Vyum install syslog-ng-libdbiW and then reload syslog-ng
2.).2) Syslog-ng modules
Starting with syslog-ng Dpen Source %dition version 7.7, syslog-ng DS% became modular to increase its fle2ibility
and also to simplify the development of additional modules. "ost of the functionality of syslog-ng DS% has been
moved to separate modules. That way it becomes also possible to finetune the resource re?uirements of syslog-ng
DS% for e2ample, by loading only the modules that are actually used in the configuration, or simply omitting
modules that are not used but re?uire large amount of memory.
%ach module contains one or more plugins, which add some functionality to syslog-ng DS%, for e2ample, a
destination or a source driver.
To display the list of available modules, e2ecute the syslog-ng --version command.
To the description of the available modules, e2ecute the syslog-ng --module-registry command.
To customize which modules are loaded automatically when syslog-ng DS% is started, use the --default-
modules command-line option of syslog-ng DS%.
To re?uest loading a module from the syslog-ng DS% configuration file, see below section.
2.).2.1) Loading modules
The syslog-ng Dpen Source %dition application loads every available module during startup.
To load a module that is not loaded automatically, include the following statement in the syslog-ng DS%
configuration file4
Omodule Ymodule-nameZ
)ote the following points about the %module statement4
The %module statement is a top-level statement, that is, it cannot be nested into any other statement.
1sually it is used immediately after the %"ersion statement.
%very %module statement loads a single module4 loading multiple modules re?uires a separate
%module statement for every module.
In the configuration file, the %module statement of a module must be earlier than the module is used.
2.).3) $anaging com%lex syslog-ng configurations
The following sections describe some methods that can be useful to simplify the management of large-scale
syslog-ng installations.
2.).3.1) including configuration files!
The syslog-ng application supports including e2ternal files in its configuration file, so parts of its configuration can
be managed separately. To include the contents of a file in the syslog-ng configuration, use the following synta2
include =YfilenameZ=3
(here YfilenameZ can be file name, path or even a directory (where all files are included, e2cept files beginning
with [ (tilde or . (dot. Including a directory is not recursive.
This imports the entire file into the configuration of syslog-ng, at the location of the include statement. If you
specify a directory, syslog-ng will try to include every file in alphabetic order. (hen including configuration files,
consider the following points4
If an ob0ect is defined twice (for e2ample the original syslog-ng configuration file and the file imported
into this configuration file both define the same option, source, or other ob0ect, then the ob0ect that is
defined later in the configuration file will be effective. ,or e2ample, if you set a global option at the
beginning of the configuration file, and later include a file that defines the same option with a different
value, then the option defined in the imported file will be used.
,iles can be embedded into each other4 the included files can contain include statements as well, up to a
ma2imum depth of -9 levels.
Aou cannot include complete configuration files into each other, only configuration snippets can be
included. This means that the included file cannot have a %"ersion statement.
Include statements can only be used at top level of the configuration file. ,or e2ample, the following is
correct4
Oversion4 7.7
include =e2ample.conf=3
!ut the following is not4
source sKe2ample P
include =e2ample.conf=
Q3
3arning ! 8*e syslog-ng a%%lication #ill not start if it cannot find a file t*at is to ,e included in its
configuration. 4l#ays dou,le-c*ec5 t*e filenames: %at*s: and access rig*ts #*en including
configuration files: and use t*e --syntax-only command-line o%tion to c*ec5 your configuration.
2.).) Sources!
syslog-ng receives log messages from a source. To define a source you should follow the following synta24
source YidentifierZ P source-driver(params3 source-driver(params3 ... Q3
Aou can look at the identifiers and source-drivers in the official manuals. This guide will follow the manual to
e2plain the configuration file above. The uni2-stream( source-driver opens the given &,K1)IE socket and starts
listening on it for messages. The internal( source-driver gets messages generated by syslog-ng.
Therefore, the following means4 src gets messages from the ;dev;log socket and syslog-ng.
source src P uni2-stream(=+dev+log=3 internal(3 Q3
The kernel sends log messages to +proc+kmsg and the file( driver reads log messages from files. Therefore, the
following means4 kernsrc gets messages from file +proc+kmsg
source kernsrc P file(=+proc+kmsg=3 Q3
In the default configuration file after emerging syslog-ng, the source is defined as4
source src P uni2-stream(=+dev+log=3 internal(3 pipe(=+proc+kmsg=3 Q3
Feading messages by pipe(=+proc+kmsg= gives a better performance but because it opens its argument in read-
write mode can be a security hazard as the syslog-ng admin guide states in section 7.7.74
=;ipe is very similar to the file( driver, but there are a few differences, for e2ample pipe( opens its argument in
read-write mode, therefore it is not recommended to be used on special files like +proc+kmsg
To open a port to read data from a remote server a source must be defined with this synta24
for 1*; -Z source sKnet P udp(3 Q3
or
source sKnet P tcp(3 Q3
to receive log messages via T';. !oth listen on port 9-8.
To listen on user defined ports or specific I;s of logging server, define source as below4
source sKnet P udp(ip(-.6.-:/.6.. port(9-83Q3
N Cere syslog-ng will read on I; -.6.-:/.6.. of log server on udp port 9-8 for incoming logs.
source sKnet P tcp(ip(-.6.-:/.6./ port(-9-83 Q3
N Cere syslog-ng will read on I; -.6.-:/.6./ of log server on tcp port -9-8 for incoming logs.
2.)..1) Collecting %rocess accounting logs on Linux!
Starting with version 7.6, syslog-ng DS% can collect process accounting logs on @inu2 systems.;rocess accounting
is the method of recording and summarizing commands e2ecuted on @inu2, for e2ample, the commands being run,
the user who e2ecuted the command, ';1 time used by the process, e2it code, and so on. (hen process
accounting (also called pacct is enabled on a system, the kernel writes accounting records to the
+var+log+account+pacct file (might vary between different @inu2 distributions.
To use the pacct&' driver, the following conditions must be met4
The syslog-ng DS% application must be compiled with the --enable-pacct option. %2ecute the syslog-ng
-7 command to check if your binary supports process accounting.
The pacctformat plugin must be loaded. !y default, syslog-ng DS% automatically loads the available
modules.
The scl.conf file must be included in your syslog-ng configuration4
Oinclude =scl.conf=
;rocess accounting must be running on the host. Aou can enable it with the accton command.
The pacct&' driver parses the fields of the accounting logs and transforms them into name-value pairs. The fields
are defined in the manual page of the accounting log file (man acct, syslog-ng DS% prepends every field with the
.pacct. prefi2. ,or e2ample, the ac(uid field that contains the id of the user who started the process will be
available under the ).pacct.ac(uid name. These can be used as macros in templates, in filters to select specific
messages, and so on.
To use the pacct&' driver, use the following synta2.
Oversion4 Otechversion3
Oinclude =scl.conf=
source sKpacct P pacct(3 Q3
...
log P source(sKpacct3 destination(...3 Q3
4lso see pacct( options In manual
2.)..2) &eceiving messages from external a%%lications
The program driver starts an e2ternal application and reads messages from the standard output (stdout of the
application. It is mainly useful to receive log messages from daemons that accept incoming messages and convert
them to log messages.
The program driver has a single re?uired parameter, specifying the name of the application to start.
*eclaration4
program(filename3
(xam%le >sing t*e %rogram=) driver
source sE%rogram F %rogram=G;etc;init.d;mydaemonG)H IH
2.).') -estinations
In syslog-ng, log messages are sent to files. The synta2 is very similar to sources4
destination YidentifierZ Pdestination-driver(params3 destination-driver(params3 ... Q3
Aou will be normally logging to a file, but you could log to a different destination-driver4 pipe, 1ni2 socket, T';-
1*; ports, terminals or to specific programs. Therefore, this means sending authlog messages to +var+log+auth.log4
destination authlog P file(=+var+log+auth.log=3 Q3
If the user is logged in, usertty( sends messages to the terminal of the specified user. If you want to send console
messages to rootLs terminal if it is logged in4
destination console P usertty(=root=3 Q3
"essages can be sent to a pipe with pipe(. The following sends 2console messages to the pipe +dev+2console.
destination 2console P pipe(=+dev+2console=3 Q3
To send messages on the network, use udp(. The following will send your log data out to another server.
destination remoteKserver P udp(=-#.#.#.6= port(9-83 Q3
Similarly you can use T'; for transport.
2.).)) Creating Filters for $essages
The synta2 for the filter statement is4
filter YidentifierZ P e2pression3 Q3
,unctions can be used in the e2pression, such as the function facility( which selects messages based on the facility
codes. The @inu2 kernel has a few facilities you can use for logging. %ach facility has a log-level3 where debug is
the most verbose, and panic only shows serious errors. Aou can find the facilities, log levels and priority names in
+usr+include+sys+syslog.h. e.g. To filter those messages coming from authorization, use the following4
filter fKauth P facility(auth3 Q3
The facility e2pression can use the boolean operators and, or, and not, so the following filter selects those
messages not coming from authorization, network news or mail4
filter fKdebug P not facility(auth, authpriv, news, mail3 Q3
The function level( selects messages based on its priority level, so if you want to select informational levels4
filter fKinfo P level(info3 Q3
,unctions and boolean operators can be combined in more comple2 e2pressions. The following line filters
messages with a priority level from informational to warning not coming from auth, authpriv, mail and news
facilities4
filter fKmessages P level(info..warn and not facility(auth, authpriv, mail, news3 Q3
"essages can also be selected by matching a regular e2pression in the message with the function match(=rege2=
value(=keyword=. ,or e2ample4
filter fKfailed P match(=rege2= value(=failed=3 Q3
To filter messages received from a particular remote host, the host( function must be used4
filter fKhost P host( =-.6.-:/.-.-= 3 Q3
2.)..) Log Pat*s
syslog-ng connects sources, filters and destinations with log statements. The synta2 is4
log Psource(s-3 source(s63 ...
filter(f-3 filter(f63 ...
destination(d-3 destination(d63 ...
flags(flag-R, flag6...T3 Q3
The following for e2ample sends messages from src source to mailinfo destination filtered by fKinfo filter.
log P source(src3 filter(fKmail3 filter(fKinfo3 destination(mailinfo3 Q3
2.)./) $anaging incoming and outgoing messages #it* flo#-control
This section describes the internal message-processing model of syslog-ng, as well as the flow-control feature that
can prevent message losses. To use flow-control, the flow-control flag must be enabled for the particular log path.
The syslog-ng application monitors (polls the sources defined in its configuration file, periodically checking each
source for messages. (hen a log message is found in one of the sources, syslog-ng polls every source and reads
the available messages. These messages are processed and put into the output buffer of syslog-ng (also called fifo.
,rom the output buffer, the operating system sends the messages to the appropriate destinations.
In large-traffic environments many messages can arrive during a single poll loop, therefore syslog-ng reads only a
fi2ed number of messages from each source. The log(fetch(limit&' option specifies the number of messages read
during a poll loop from a single source.
Figure 2.)./.1. $anaging log messages in syslog-ng
@ote! 8*e logEfetc*Elimit=) %arameter can ,e set as a glo,al o%tion: or for every source individually.
%very destination has its own output buffer. The output buffer is needed because the destination might not be able
to accept all messages immediately. The log(fifo(si*e&' parameter sets the size of the output buffer. The output
buffer must be larger than the log(fetch(limit&' of the sources, to ensure that every message read during the poll
loop fits into the output buffer. If the log path sends messages to a destination from multiple sources, the output
buffer must be large enough to store the incoming messages of every source.
T'; and uni2-stream sources can receive the logs from several incoming connections (for e2ample many different
clients or applications. ,or such sources, syslog-ng reads messages from every connection, thus the
log(fetch(limit&' parameter applies individually to every connection of the source.
Figure 2.)./.2 $anaging log messages of 8CP sources in syslog-ng
The flow-control of syslog-ng introduces a control window to the source that tracks how many messages can
syslog-ng accept from the source. %very message that syslog-ng reads from the source lowers the window size by
one3 every message that syslog-ng successfully sends from the output buffer increases the window size by one. If
the window is full (that is, its size decreases to zero, syslog-ng stops reading messages from the source. The initial
size of the control window is by default +0004 the log(fifo(si*e&' must be larger than this value in order for flow-
control to have any effect. If a source accepts messages from multiple connections, all messages use the same
control window.
@ote! Starting #it* syslog-ng "S( version 3.3: if t*e source can *andle multi%le connections =for exam%le:
tc%=)): t*e siJe of t*e control #indo# is divided ,y t*e value of t*e maxEconnections=) %arameter and
t*is smaller control #indo# is a%%lied to eac* connection of t*e source.
(hen flow-control is used, every source has its own control window. &s a worst-case situation, the output buffer
of the destination must be set to accommodate all messages of every control window, that is, the log(fifo(si*e&' of
the destination must be greater than number(of(sourcesMlog(iw(si*e&'. This applies to every source that sends logs
to the particular destination. Thus if two sources having several connections and heavy traffic send logs to the
same destination, the control window of both sources must fit into the output buffer of the destination. Dtherwise,
syslog-ng does not activate the flow-control, and messages may be lost.
The syslog-ng application handles outgoing messages the following way4
,igure 6.:./.7
Candling outgoing messages in syslog-ng DS%
,utput -ueue4 "essages from the output ?ueue are sent to the target syslog-ng server. The syslog-ng
application puts the outgoing messages directly into the output ?ueue, unless the output ?ueue is full. The
output ?ueue can hold :8 messages, this is a fi2ed value and cannot be modified.
,"erflow -ueue4 If the output ?ueue is full, syslog-ng puts the outgoing messages into the overflow ?ueue of
the destination. (The overflow ?ueue is identical to the output buffer used by other destinations. The
log(fifo(si*e&' parameter specifies the number of messages stored in the overflow ?ueue. ,or details on sizing
the log(fifo(si*e&' parameter,
2.).0) Log to anot*er file
In order to move some log from +var+log+messages to another file4
Nsshd configuration
destination ssh P file(=+var+log+ssh.log=3 Q3
filter fKssh P program(=sshd=3 Q3
log P source(src3 filter(fKssh3 destination(ssh3 Q3
2..) Configuring as a log*ost
'onfiguring your system to be a loghost is ?uite simple. *rop the following into your configuration, and create the
needed directory. (ith this simple configuration, log filenames will be based on the ,B*) of the remote host, and
located in +var+log+remote+. &fter creating the remote directory, reload your syslog-ng configuration.
source net P udp(3 Q3
destination remote P file(=+var+log+remote+IP,1@@CDSTQ-log=3 Q3
log P source(net3 destination(remote3 Q3
2./) Client Side Configurations!
,or rsyslog clients add the following line in +etc+rsyslog.d+rsyslog.conf4
,or T'; based transfer4 M.M OOipaddress4-9-8
,or 1*; based transfer4 M.M Oipaddress49-8
,or syslog-ng clients, configuration will be as follows4
source sKsys P
file (=+proc+kmsg= logKprefi2(=kernel4 =3
uni2-stream (=+dev+log=3
internal(3
N udp(ip(#.#.#.# port(9-83
Q3
destination dKnet Ptcp(-.6.-:/.6.. port(-9-83 Q3
Ndestination dKnetPudp(-.6.-:/.6..3Q3
log P source(sKsys3 destination(dKnet3 Q3
&lso you will have to allow above ports on firewall,
e.g.
iptables -& I);1T -m state --state )%( -p udp --dport 9-8 -0 &''%;T
iptables -& I);1T -m state --state )%( -p tcp --dport -9-8 -0 &''%;T
2.0) Im%rove Performance
Syslog-ng\s performance can be improved in different ways4
2.0.1) 3rite every so often
It seems that the old sync(E o%tion is called flushKlines(E now, where the writing to the file is buffered for E
lines. *efault is # (no buffering.
2.0.2) 4void redundant %rocessing and dis5 s%ace
& single log message can be sent to different log files several times. ,or e2ample, in the initial configuration file,
we have the following definitions4
destination cron P file(=+var+log+cron.log=3 Q3
destination messages P file(=+var+log+messages=3 Q3
filter fKcron P facility(cron3 Q3
filter fKmessages P level(info..warn
and not facility(auth, authpriv, mail, news3 Q3
log P source(src3 filter(fKcron3 destination(cron3 Q3
log P source(src3 filter(fKmessages3 destination(messages3 Q3
The same message from the cron facility will end up in both the cron.log and messages files. To change this
behavior we can use the final flag, ending up further processing with the message. Therefore, in this e2ample, if
we want messages from the cron facility not ending up in the messages file, we should change the cronLs log
sentence by4
log P source(src3 filter(fKcron3 destination(cron3 flags(final3 Q3
another way is to e2clude the cron facility from fKmessages filter4
filter fKmessages P level(info..warn and not facility(cron, auth, authpriv, mail, news3 Q3
2.12) PostgreS6L -estination
This section will use two roles4 syslog and logwriter. syslog will be the administrator of the database syslog and
logwriter will only be able to add records to the logs table.
)o longer needed to create table for logs. syslog-ng will create automatically.
ps?l -1 postgres
postgresHN 'F%&T% FD@% syslog (ITC @D<I)3
postgresHN Upassword syslog N 1sing the Upassword function is secure because
postgresHN Upassword logwriter N the password isnLt saved in history.
postgresHN 'F%&T% *&T&!&S% syslog D()%F syslog3
postgresHN U? N AouLre done here for the moment
%dit pgKhba.conf to allow syslog and logwriter to establish a connection to ;ostgreSB@.
+var+lib+postgres?l+/.8+data+pgKhba.conf
N TA;% *&T&!&S% 1S%F 'I*F-&**F%SS "%TCD*
host syslog logwriter -.6.-:/.#.-+68 md9
host syslog syslog -.6.-:/.#.-#+76 md9
Tell ;ostgreSB@ to reload the configuration files4
+etc+rc.d+postgres?l-/.8 reload
%dit +etc+syslog-ng.conf so that it knows where and how to write to ;ostgreSB@. syslog-ng will utilize the
logwriter role.
N
N SB@ logging support
N
destination dKpgs?l P
s?l(type(pgs?l
host(=-6$.#.#.-= username(=logwriter= password(=password=
database(=syslog=
table(=logsKIPCDSTQKIPFKA%&FQIPFK"D)TCQIPFK*&AQ= Nor whatever you want, e2ample IPCDSTQ=
for hosts, IP@%S%@Q= for levels.. etc
columns(=datetime timestamp with time zone=, =host varchar(76=, =program varchar(-:=, =pid varchar(-:=,
=message varchar(6##=
values(=IFKISD*&T%=, =ICDST=, =I;FD<F&"=, =I;I*=, =I"S<=
inde2es(=datetime=, =host=, =program=, =pid=, =message=3
Q3
log P source(src3 destination(dKpgs?l3 Q3
,inally, restart syslog-ng.
+etc+rc.d+syslog-ng restart
&nd check to see if things are being logged.
ps?l -1 logwriter -d syslog
syslogHZ S%@%'T M ,FD" Yyour table nameZ DF*%F !A datetime *%S' @I"IT -#3
2.11) IS" /)21 timestam%s
!efore4
)ote4 @ogger makes entries in the system log. It provides a shell command interface to the syslog system log
module.
Fun below command
Nlogger These timestamps are not optimal.
Ntail -n - +var+log+messages.log
,eb -/ -84694#- hostname logger4 These timestamps are not optimal.
Cence &dd tsEformat=iso)H to +etc+syslog-ng+syslog-ng.conf in the options section.
Then4
N +etc+rc.d+syslog-ng reload
&fter4
Nlogger )ow TC&T is a timestamp]
Ntail -n 6 +var+log+messages.log
,eb -/ -84694#- hostname logger4 These timestamps are not optimal.
6#-#-#6--/T6#46749/-#94## electron logger4 )ow TC&T is a timestamp]
N
2.12) Log Levels
@og levels are defined separately for each logged facility in syslog-ng config. &vailable log levels are listed in
+usr+include+sys+syslog.h 4
define @D<K%"%F< # +M system is unusable M+
define @D<K&@%FT - +M action must be taken immediately M+
define @D<K'FIT 6 +M critical conditions M+
define @D<K%FF 7 +M error conditions M+
define @D<K(&F)I)< 8 +M warning conditions M+
define @D<K)DTI'% 9 +M normal but significant condition M+
define @D<KI),D : +M informational M+
define @D<K*%!1< $ +M debug-level messages M+
2.13) $acros and 7aria,les
"acros can be used in both templates, and in destination file names. ,or "ore information, Fefer "acros of
syslog-ng DS% from the manual.
The following code will write the log lines to +var+log+ syslog+remote in the format of hierarchy separating logs
year wise, then month wise for each year, then day wise for each month. @og files will have information as per
template defined later in below destination file name.
destination dKalllogs P file ( =+var+log+syslog+remote+I,1@@CDST+IA%&F+I"D)TC+I*&A+I,1@@CDST-
IA%&F-I"D)TC-I*&A.log= owner(root group(root perm(#:## dirKperm(#$## createKdirs(yes
template(=I*&T% I,1@@CDST I;FD<F&" IT&< RI,&'I@ITA.I@%S%@T I"%SS&<%Un=3 Q3
log P source(sKsys3 destination(dKalllogs3 Q3
)ote4 hostname must be set in +etc+sysconfig+network for all clients3 also you need to have entries in +etc+hosts on
both client and server then only logs will be separated hostname wise.
If above settings are not done then different folders will be created for each client and folder names will be same as
I; of clients.
2.1) 8rou,les*ooting syslog-ng!
&s a general rule, first try to get logging the messages to a local file. Dnce this is working, you know that syslog-
ng is running correctly and receiving messages, and you can proceed to forwarding the messages to the server.
If the syslog-ng server does not receive the messages, use tcpdump or a similar packet sniffer tool on the client to
verify that the messages are sent correctly, and on the server to verify that it receives the messages.
e.g. create logs on client using logger command and run Vtcpdump ^vn port Yport noZW on log server to see if it is
receiving logs or not.
If syslog-ng is closing the connections for no apparent reason, be sure to check the log messages of syslog-ng. Aou
might also want to run syslog-ng with the --verbose or --debug command-line options for more-detailed log
messages. Starting from syslog-ng DS% version 7.-, you can enable these messages without restarting syslog-ng
using the syslog-ng-ctl verbose --setHon command. ,or details, see the syslog-ng-ctl man page at syslog-ng-ctl(-.
To start syslog-ng in debug mode, use command4 +sbin+syslog-ng ^,edv
Similarly, build up encrypted connections step-by-step4 first create a working unencrypted (for e2ample T';
connection, then add T@S encryption, and finally client authentication if needed.
2.1') Possi,le causes of losing log messages!
*uring the course of a message from the sending application to the final destination of the message, there are a
number of locations where a message may be lost, even though syslog-ng does its best to avoid message loss.
1sually losing messages can be avoided with careful planning and proper configuration of syslog-ng and the hosts
running syslog-ng. The following list shows the possible locations where messages may be lost, and provides
methods to minimize the risk of losing messages.
.etween the application and the syslog-ng client4 "ake sure to use an appropriate source to receive the
logs from the application (for e2ample from +dev+log. ,or e2ample, use unix-stream instead of unix-
dgram whenever possible.
/hen syslog-ng is sending messages4 If syslog-ng cannot send messages to the destination and the output
buffer gets full, syslog-ng will drop messages. The number of dropped messages is displayed per
destination in the log message statistics of syslog-ng.
,n the network4 (hen transferring messages using the 1*; protocol, messages may be lost without any
notice or feedback _ such is the nature of the 1*; protocol. &lways use the T'; protocol to transfer
messages over the network whenever possible.
0n the socket recei"e buffer4 (hen transferring messages using the 1*; protocol, the 1*; datagram
(that is, the message that reaches the receiving host placed in a memory area called the socket recei"e
buffer. If the host receives more messages than it can process, this area overflows, and the kernel drops
messages without letting syslog-ng know about it. 1sing T'; instead of 1*; prevents this issue. If you
must use the 1*; protocol, increase the size of the receive buffer using the so(rc"buf&' option.
/hen syslog-ng is recei"ing messages4 The receiving syslog-ng (for e2ample the syslog-ng server or
relay may drop messages if the fifo of the destination file gets full. The number of dropped messages is
displayed per destination in the log message statistics of syslog-ng.
/hen the destination cannot handle large load4 (hen syslog-ng is sending messages at a high rate into
an SB@ database, a file, or another destination, it is possible that the destination cannot handle the load,
and processes the messages slowly. &s a result, the buffers of syslog-ng fill up, syslog-ng cannot process
the incoming messages, and starts to loose messages. ,or details, see the previous entry. 1se the throttle
parameter to avoid this problem.
1s a result of an unclean shutdown of the syslog-ng ser"er4 If the host running the syslog-ng server
e2periences an unclean shutdown, it takes time until the clients realize that the connection to the syslog-
ng server is down. "essages that are put into the output T'; buffer of the clients during this period are
not sent to the server.
2.1)) Procedure 9 Creating syslog-ng core files
Pur%ose! (hen syslog-ng crashes for some reason, it can create a core file that contains important troubleshooting
information. To enable core files, complete the following procedure4
Ste%s!
-. 'ore files are produced only if the maximum core file si*e ulimit is set to a high value in the init script of
syslog-ng. &dd the following line to the init script of syslog-ng4
ulimit -c unlimited
6. Serify that syslog-ng has permissions to write the directory it is started from, for e2ample +opt+syslog-
ng+sbin+.
7. If syslog-ng crashes, it will create a core file in the directory syslog-ng was started from.
8. To test that syslog-ng can create a core file, you can create a crash manually. ,or this, determine the ;I*
of syslog-ng (for e2ample using the %s -4llKgre% syslog-ng command, then issue the following
command4 5ill -4?&8 Lsyslog-ng %idM
This should create a core file in the current working directory.
2.1.) Collecting de,ugging information #it* strace: truss: or tusc
To properly troubleshoot certain situations, it can be useful to trace which system calls syslog-ng DS% performs.
Cow this is performed depends on the platform running syslog-ng DS%. In general, note the following points4
(hen syslog-ng DS% is started, a supervisor process might stay in the foreground, while the actual
syslog-ng daemon goes to the background. &lways trace the background process.
&part from the system calls, the time between two system calls can be important as well. "ake sure that
your tracing tool records the time information as well. ,or details on how to do that, refer to the manual
page of your specific tool (for e2ample, strace on @inu2, or truss on Solaris and !S*.
Fun your tracing tool in verbose mode, and if possible, set it to print long output strings, so the messages
are not truncated.
(hen using strace, also record the output of lsof to see which files are accessed.
The following are e2amples for tracing system calls of syslog-ng on some platforms. The output is saved into the
+tmp+syslog-ng-trace.t2t file, sufficed with the ;I* of the related syslog-ng process.The path of the syslog-ng
binary assumes that you have installed syslog-ng DS% from the official syslog-ng DS% binaries available at the
!ala!it website _ native distribution-specific packages may use different paths.
2inux4 strace -o ;tm%;trace.txt -s2') -ff -tt8 ;o%t;syslog-ng;s,in;syslog-ng -f ;o%t;syslog-
ng;etc;syslog-ng.conf -Fdv
34-U54 tusc -f -o ;tm%;syslog-ng-trace.txt -8 ;o%t;syslog-ng;s,in;syslog-ng
0.6 105 and !olaris4 truss -f -o ;tm%;syslog-ng-trace.txt -r all -# all -u li,c!! ;o%t;syslog-
ng;s,in;syslog-ng -d -d -d
Tip4 To e2ecute these commands on an already running syslog-ng DS% process, use the -p YpidKofKsyslog-ngZ
parameter.
loggen _ <enerate syslog messages at a specified rate
The loggen application is tool to test and stress-test your syslog server and the connection to the server. It can send
syslog messages to the server at a specified rate, using a number of connection types and protocols, including T';,
1*;, and uni2 domain sockets. The messages can be generated automatically (repeating the 4177string over and
over, or read from a file or the standard input.
(hen loggen finishes sending the messages, it displays the following statistics4
a"erage rate4 &verage rate the messages were sent in messages+second.
count4 The total number of messages sent.
time4 The time re?uired to send the messages in seconds.
a"erage message si*e4 The average size of the sent messages in bytes.
bandwidth4 The average bandwidth used for sending the messages in kilobytes+second.
,or options see manual page of loggen.
(xam%les!
- The following command generates -## messages per second for ten minutes, and sends them to port 6#-# of the
localhost via T';. %ach message is 7## bytes long.
loggen --size 7## --rate -## --interval :## -6$.#.#.- 6#-#
6 The following command is similar to the one above, but uses the 1*; protocol.
loggen --inet --dgram --size 7## --rate -## --interval :## -6$.#.#.- 6#-#
7 Send a single message on T';: to the 88+ I;v: address, port +09+4
loggen --ipv: --number - 44- -#:-
8 Send a single message on 1*;: to the 88+ I;v: address, port +09+4
loggen --ipv: --dgram --number - 44- -#:-
9 Send a single message using a uni2 domain-socket4 loggen --uni2 --stream --number - Y+path+to+socketZ
: Fead messages from the standard input (stdio and send them to the localhost4
loggen -6$.#.#.- -#:- --read-file ^
2.1/) 4%ac*e and $yS6L Logging #it* Syslog-ng
http4++blog.monitis.com+inde2.php+6#--+#.+#9+apache-and-mys?l-logging-with-syslog-ng+
http4++www.linu2geek.net+apache-with-syslog-ng+
2.10) ,uild a Secure Logging Server #it* syslog-ng using SSL;8LS
http4++www.enterprisenetworkingplanet.com+netsysm+article.php+79./-8:+!uild-a-Secure-@ogging-Server-with-
syslogng-;art-6.htm
http4++www.nineproductions.com+syslog-ng-with-tls-setup-for-centralized-logging+
http4++www.balabit.com+sites+default+files+documents+syslog-ng-pe-8.#-guides+en+syslog-ng-pe-v8.#-guide-admin-
en+html+procedure-configuring-mutual-tls-server.html
http4++www.syslog.org+forum+syslog-ng+syslog-ng-using-tls-re?uire-help+
http4++www.balabit.com+sites+default+files+documents+syslog-ng-ose-7.7-guides+en+syslog-ng-ose-v7.7-guide-
admin-en+html+procedure-configuring-tls-server.html
https4++www.icts.uiowa.edu+confluence+display+I'TSit+&dd`T@S`,unctionality`to`your`syslog-ng`setup
2.22) syslog-ng < logrotate
*tt%!;;###.m%5ing.com;2213;23;logrotate-#it*-syslog-ng.*tml
*tt%!;;me#,ies.com;*o#EtoElogEusersEusingEsyslog-ngEtutorial.*tm
*tt%!;;###.t*egee5stuff.com;2212;2.;logrotate-exam%les;
*tt%s!;;#i5i.arc*linux.org;index.%*%;Logrotate
2.21) running syslog-ng in c*rooted environment and collecting logs from c*rooted a%%lications
http4++eagain.net+articles+syslog-ng-chroot+
http4++www.campin.net+syslog-ng+chroot-0ail.html
http4++www.balabit.com+sites+default+files+documents+syslog-ng-ose-7.7-guides+en+syslog-ng-ose-v7.7-guide-
admin-en+html+e2ampleKchroot.html
http4++www.balabit.com+wiki+syslog-ng-fa?-chroot
2.22) &eferences!
8*e syslog-ng "%en Source (dition 3.3 4dministrator +uide (contains every bit of information for syslog-ng
http4++www.balabit.com+sites+default+files+documents+syslog-ng-ose-7.7-guides+en+syslog-ng-ose-v7.7-guide-
admin-en+html-single+inde2.html
See also man syslog-ng, man syslog-ng.conf, man syslog-ng-ctl
List of syslog-ng "S( %arameters
http4++www.balabit.com+sites+default+files+documents+syslog-ng-ose-7.8-guides+en+syslog-ng-ose-v7.8-guide-
admin+html+syslog-ng-parameter-inde2.html
$acros of syslog-ng "S(
http4++www.balabit.com+sites+default+files+documents+syslog-ng-ose-7.8-guides+en+syslog-ng-ose-v7.8-guide-
admin+html+reference-macros.html
?oo5s a,out syslog-ng
http4++www.balabit.com+network-security+syslog-ng+opensource-logging-system+overview+books
Performane tuning syslog-ng
http4++codeascraft.com+6#-6+#/+-7+performance-tuning-syslog-ng+
1o# to Log to PostgreS6L #it* Syslog-ng

http4++blog.monitis.com+inde2.php+6#--+#.+#/+how-to-log-to-postgres?l-with-syslog-ng+
&emote Logging #it* SS1 and Syslog-@+
http4++www.deer-run.com+[hal+sysadmin+SSC-Syslog)<.html
C*a%ter 3 - &syslog
Fsyslog is the default logging program on several @inu2 distributions including *ebian and Fed Cat based
systems. &part from implementing the syslog protocol, rsyslog adds additional features such as content-based
filtering. This also uses T'; for transporting, and provides lot of configuration options.
Fsyslog uses the ?uasi-standard !S* syslog protocol, specified in F,' 7-:8. &s the te2t of F,' 7-:8 is 0ust a
vague informational description and not a standard, various incompatible e2tensions of it emerged. Fsyslog
supports many of these e2tensions. The format of relayed messages can be customized.
3.1) Features!
The most important e2tensions of the original protocol supported by rsyslog are4
ISD /:#- timestamp with millisecond granularity and timezone information
the addition of the name of relays in the host fields to make it possible to track the path a given message
has traversed
reliable transport using T';
support <SS-&;I and T@S
logging directly into various database engines.
support for F,' 9868, F,' 9869, F,' 986:
support for F%@;
support for buffered operation modes where messages are buffered locally if the receiver is not ready
,or more ,eatures of rsyslog4 see http4++www.rsyslog.com+doc+features.html
https4++wiki.archlinu2.org+inde2.php+Fsyslog
,ollowing setup will help you to analyze the log files of all the servers in your infrastructure from a central log
server.
3.2) Installation
Fsyslog comes as the default logging program in *ebian *istribution and Fed Cat based systems. If you system
doesn\t have rsyslog, install it as shown below depending on your distro.
- ,rom F;"4 Fefer @ink - http4++www.rsyslog.com+installing-rsyslog-from-rpm+
apt-get install rsyslog rsyslog-doc
(or
yum install rsyslog rsyslog-doc
see also 4 http4++www.rsyslog.com+doc+install.html
Fsyslog configurations are stored in +etc+ryslog.conf file and the files under +etc+rsyslog.d+ directory.
3.3) Configuration Structure
!efore understanding how to setup the central logging sever, it is good to understand the configuration structure of
rsyslog.
Fsyslog configuration files are structured in the following manner
-. "odules
6. 'onfiguration *irectives
7. Fule line
3.3.1) $odules
Fsyslog has a modular architecture. It enables functionalities to be added dynamically through these modules. The
modules are categorized as4
Input "odules ^ 1sed to gather messages from various sources
Dutput "odules ^ 1sed to write the messages to various places ( file, socket etc..
;arser "odules ^ 1sed to parse the message content
;lease note that there are also other categories of modules available. This is to give an overview of what modules
can do.
,or more Info see http4++www.rsyslog.com+doc+modules.html
http4++www.rsyslog.com+doc+rsyslogKconfKmodules.html
3.3.2) Configuration -irectives
&ll configuration directives must be specified one per line and must start with dollar sign (I. It affects the rules.
,or 'omplete @ist see4 http4++www.rsyslog.com+doc+rsyslogKconfKglobal.html
3.3.3) &ule line
%very rule line consists of two fields, a aselector field\ and an aaction field\. The selector field is divided into two,
afacilities b priorities\. &ction specifies what action must be taken for the matched rule.
See also4 http4++www.rsyslog.com+doc+multiKruleset.html
3.3.) 4 Sam%le Configuration
NNNNNNNNNNNNNNNNNNNNNN
"D*1@%S
NNNNNNNNNNNNNNNNNNNNNN
I"od@oad imu2sock
I"od@oad imklog
NNNNNNNNNNNNNNNNNNNNNN
*irectives
NNNNNNNNNNNNNNNNNNNNNN
N Set the default permissions for all log files.
I,ileDwner root
I,ile<roup adm
I,ile'reate"ode #:8#
I*ir'reate"ode #$99
NNNNNNNNNNNNNNNNNNNNNN
F1@%S
NNNNNNNNNNNNNNNNNNNNNN
mail.info +var+log+mail.info
mail.warn +var+log+mail.warn
mail.err +var+log+mail.err
daemon.M +var+log+daemon.log
3.3.') 8em%lates
Templates are a very important features provided by rsyslog. It allows the user to log the messages in
their desirable format. It can also be used to create dynamic file names to log the messages. In case of
database logging, the templates are used to convert the message into a proper SB@ statement.
& sample template will look like4
Itemplate mytemplate VTe2t-!efore >msg> Te2t-&fterUnW
The above template will log the message VThis is hello from rsyslogW as4
Te2t-!efore this is hello from rsyslog Te2t-&fter
(e will see how to use the template for generate the log files dynamically.
See http4++www.rsyslog.com+doc+rsyslogKconfKtemplates.html
http4++www.rsyslog.com+tag+use-a-template+
http4++www.rsyslog.com+doc+rsyslogKconfKe2amples.html
3.) Central Logging Server
The above sections should have given an overview about rsyslog and how to configure it. )ow we will move on to
setup a central logging system.
,or our discussion we will have server I; as V-.6.-:/.-.-W for the central log server, where all the log messages
from client should be forwarded.
&dd the following lines to the rsyslog.conf of the central log server servers (In this e2ample, the following line was
added on the log server with ip-address -.6.-:/.-.-4
N provides support for local system logging
I"od@oad imu2sock
N provides kernel logging support (previously done by rklogd
I"od@oad imklog
N provides 1*; syslog reception. ,or T';, load imtcp.
I"od@oad imudp
N ,or T';, InputServerFun 9-8
I1*;ServerFun 9-8
N This one is the template to generate the log filename dynamically, depending on the clientLs I; address.
Itemplate ,I@%)&"%,=+var+log+>fromhost-ip>+syslog.log=
N @og all messages to the dynamically formed file. )ow each clients log (-.6.-:/.-.6, -.6.-:/.-.7,etc..., will be
under a separate directory which is formed by the template ,I@%)&"%.
M.M G,I@%)&"%
&fter adding the above lines to the rsyslog.conf, restart the rsyslog process. )ow the rsyslog server will be ready to
accept messages.
N service rsyslog restart
Shutting down system logger4 R D5 T
Starting system logger4 R D5 T
&dd the following lines to the rsyslog.conf on the individual client machines that should send their log messages to
the central server.
I"od@oad imu2sock
I"od@oad imklog
N ;rovides 1*; forwarding. The I; is the serverLs I; address
M.M O-.6.-:/.-.-49-8
N ;rovides T'; forwarding. !ut the current server runs on 1*;
N M.M OO-.6.-:/.-.-49-8
Festart the rsyslog process on the clients. )ow the rsyslog central server (In this e2ample, -.6.-:/.-.- will
receive all the log messages from the configured clients and each client\s log will be placed under a separate
directory.
See &lso4 http4++www.howtoforge.com+centralized-rsyslog-server-monitoring (very important link
3.') (ncry%ting Syslog 8raffic #it* 8LS =SSL)
Fefer @ink4 http4++www.rsyslog.com+doc+rsyslogKtls.html
3.)) >nrelia,ility of %lain tc% syslog http4++blog.gerhards.net+6##/+#8+on-unreliability-of-plain-tcp-syslog.html
3..) 8LS secured syslog via &(LP
Fefer @ink4 http4++www.rsyslog.com+tls-secured-syslog-via-relp+
See also: SS@ %ncrypting Syslog with Stunnel http4++www.rsyslog.com+doc+rsyslogKstunnel.html
3./) &eferences!
http4++www.rsyslog.com+storing-and-forwarding-remote-messages+
https4++blog.serverdensity.com+reliable-forwarding-with-rsyslog+
http4++www.rsyslog.com+receiving-messages-from-a-remote-system+
http4++www.rsyslog.com+sending-messages-to-a-remote-syslog-server+
T'; based log transfer4 http4++www.rsyslog.com+tag+tcp+
http4++blog.gerhards.net+6##$+#/+why-does-world-need-another-syslogd.html
http4++blog.gerhards.net+6#--+--+serious-syslog-problems.html
Com%arison ,et#een syslog-ng N rsyslog
http4++www.balabit.com+syslog-ng-rsyslog-comparison
https4++czanik.blogs.balabit.com+6#-6+#-+rsyslog-vs-syslog-ng+
&Syslog 9 -ocumentation
http4++www.rsyslog.com+doc+manual.html
Part I7 - 3(? +>IS to $4@4+( Logging in Linux!
https4++czanik.blogs.balabit.com+6#--+#:+a-comparison-of-syslog-ng-web-guis+
http4++www.linu20ournal.com+content+centralized-logging-web-interface
LogJilla < syslog-ng! http4++mewbies.com+howKtoKlogzillaKlogKanalyzerKtutorial.htm
http4++mewbies.com+howKtoKsplunkKlogKanalyzerKtutorial.htm