NXLog User Guide

66. Linux System Logs

NXLog can be used to collect and process logs from a Linux system.

Linux distributions normally use a "Syslog" system logging agent to retrieve events from the kernel (/proc/kmsg) and accept log messages from user-space applications (/dev/log). Originally, this logger was syslogd; later syslog‑ng added additional features, and finally Rsyslog is the logger in common use today. For more information about Syslog, see Syslog.

Many modern Linux distributions also use the Systemd init system, which includes a journal component for handling log messages. All messages generated by Systemd-controlled processes are sent to the journal. The journal also handles messages written to /dev/log. The journal stores logs in a binary format, either in memory or on disk; the logs can be accessed with the journalctl tool. Systemd can also be configured to forward logs via a socket to a local logger like Rsyslog or NXLog.

There are several ways that NXLog can be configured to collect Linux logs. See Replacing Rsyslog for details about replacing Rsyslog altogether, handling all logs with NXLog instead. See Forwarding Messages via Socket for a simple way to forward all logs to NXLog without disabling Rsyslog (this is the least intrusive option). Finally, it is also possible to read the log files written by Rsyslog; see Reading Rsyslog Log Files.

66.1. Replacing Rsyslog

Follow these steps to disable Rsyslog and configure NXLog to collect logs in its place.

Configure NXLog to collect events from the kernel, the Systemd journal socket, and the /dev/log socket. See the example below.
Configure Systemd to forward log messages to a socket by enabling the ForwardToSyslog option.
/etc/systemd/journald.conf
```
[Journal]
ForwardToSyslog=yes
```
Stop and disable Rsyslog by running systemctl stop rsyslog and systemctl disable rsyslog as root.
Restart NXLog.
Reload the journald configuration by running systemctl force-reload systemd-journald.

Example 291. Replacing Rsyslog With NXLog

This example configures NXLog to read kernel events with the im_kernel module, read daemon messages from the Systemd journal socket with the im_uds module, and accept other user-space messages from the /dev/log socket with im_uds. In the om_tcp module instance, all of the logs are converted to JSON format, BSD Syslog headers are added, and the logs are forwarded to another host via TCP.

nxlog.conf


  1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

  <Extension _json>
    Module      xm_json
</Extension>

<Extension _syslog>
    Module      xm_syslog
</Extension>

<Input kernel>
    Module      im_kernel
    Exec        parse_syslog_bsd();
</Input>

<Input journal>
    Module      im_uds
    UDS         /run/systemd/journal/syslog
    Exec        parse_syslog_bsd();
</Input>

<Input devlog>
    Module      im_uds
    UDS         /dev/log
    FlowControl FALSE
    Exec        $raw_event =~ s/\s+$//; parse_syslog_bsd();
</Input>

<Output out>
    Module      om_tcp
    Host        192.168.1.1
    Port        1514
    Exec        $Message = to_json(); to_syslog_bsd();
</Output>

<Route r>
    Path        kernel, journal, devlog => out
</Route>

Note	Some local Syslog sources will add a trailing newline (`\n`) to each log message. The `$raw_event =~ s/\s+$//;` statement in the `devlog` input section above will automatically remove this and any other trailing whitespace before processing the message.

66.2. Forwarding Messages via Socket

By adding a short configuration file, Rsyslog can be configured to forward messages to NXLog via a Unix domain socket. This is the least intrusive of the options documented here.

Configure NXLog to accept log messages from Rsyslog via a socket. See the example below.

Configure Rsyslog to write to the socket by adding the following configuration file. See the Rsyslog documentation for more information about configuring what is forwarded to NXLog.

/etc/rsyslog.d/nxlog.conf

# Load omuxsock module
$ModLoad omuxsock

# Set socket path
$OMUxSockSocket /opt/nxlog/var/spool/nxlog/rsyslog_sock

# Configure template to preserve PRI part (must be on a single line)
$template SyslogWithPRI,"<%PRI%>%timegenerated% %HOSTNAME% %syslogtag%%msg:::drop-last-lf%"

# Forward all log messages
*.* :omuxsock:;SyslogWithPRI

# Only forward log messages of "notice" priority and higher
#*.notice :omuxsock:;SyslogWithPRI

Restart NXLog and Rsyslog in that order to create and use the socket (NXLog must create the socket before Rsyslog will write to it). Run systemctl restart nxlog and systemctl restart rsyslog.

Example 292. Collecting Logs via Socket From Rsyslog

With this example configuration, NXLog will create the socket and accept log messages from Rsyslog through the socket. The messages will then be parsed as Syslog, converted to JSON format, prefixed with a BSD Syslog header, and forwarded to another host via TLS.

nxlog.conf


  1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

  <Extension _json>
    Module      xm_json
</Extension>

<Extension _syslog>
    Module      xm_syslog
</Extension>

<Input in>
    Module      im_uds
    UDS         /opt/nxlog/var/spool/nxlog/rsyslog_sock
    Exec        parse_syslog();
</Input>

<Output out>
    Module      om_ssl
    Host        192.168.1.1
    Port        6514
    CAFile      %CERTDIR%/ca.pem
    CertFile    %CERTDIR%/client-cert.pem
    CertKeyFile %CERTDIR%/client-key.pem
    Exec        $Message = to_json(); to_syslog_bsd();
</Output>

66.3. Reading Rsyslog Log Files

NXLog can be configured to read from log messages written by Rsyslog, /var/log/messages for example. This is a slightly more intrusive option than the steps given in Forwarding Messages via Socket.

Note	NXLog will not have access to the facility and severity codes because Rsyslog, by default, follows the BSD Syslog convention of not writing the PRI code to the `/var/log/messages` file.

By default, NXLog runs as user nxlog and does not have permission to read files in /var/log. The simplest solution for this is to run NXLog as root by omitting the User option, but it is more secure to provide the necessary permissions explicitly.

Check the user or group ownership of the files in /var/log and configure if necessary. Some distributions use a group for the log files by default. On Debian/Ubuntu, for example, Rsyslog is configured to use the adm group. Otherwise, modify the Rsyslog configuration to use different ownership for log files as shown below.
/etc/rsyslog.conf or /etc/rsyslog.d/nxlog.conf
```
$FileOwner root
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022

# Default on Debian/Ubuntu
$FileGroup adm

# Or use the "nxlog" group directly
#$FileGroup nxlog
```
Run NXLog under a user or group that has permission to read the log files. Either use a user or group directly with the User or Group option in nxlog.conf, or add the nxlog user to a group that has permission. For example, on Debian/Ubuntu add the nxlog user to the adm group by running usermod -a -G adm nxlog.
If necessary, fix permissions for any files NXLog will be reading from that already exist (use the correct group for your system).
```
# chgrp adm /var/log/messages
# chmod g+r /var/log/messages
```
Configure NXLog to read from the required file(s) (see the example below). Then restart NXLog.
If the Rsyslog configuration has been modified, restart Rsyslog (systemctl restart rsyslog).

Example 293. Reading Rsyslog Log Files

With the following configuration, NXLog will read logs from /var/log/messages, parse the events as Syslog, convert them to JSON, and forward the plain JSON to another host via TCP.

nxlog.conf


  1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

  <Extension _json>
    Module  xm_json
</Extension>

<Extension _syslog>
    Module  xm_syslog
</Extension>

<Input in>
    Module  im_file
    File    '/var/log/messages'
    Exec    parse_syslog();
</Input>

<Output out>
    Module  om_tcp
    Host    192.168.1.1
    Port    1514
    Exec    $raw_event = to_json();
</Output>