NXLog User Guide

26. Forwarding and Storing Logs

This chapter discusses the configuration of NXLog outputs, including:

converting log messages to various formats,
forwarding logs over the network,
writing logs to files and sockets,
storing logs in databases, and
sending logs to an executable.

26.1. Generating Various Formats

The data format used in an outgoing log message must be considered in addition to the transport protocol. If the message cannot be parsed by the receiver, it may be discarded or improperly processed. See also Parsing Various Formats.

Syslog

There are two Syslog formats, the older BSD Syslog (RFC 3164) and the newer IETF Syslog (RFC 5424). The transport protocol in Syslog can be UDP, TCP, or SSL. The xm_syslog module provides procedures for generating Syslog messages. For more information, see Generating Syslog.

Example 129. Generating Syslog and Sending via TCP

This configuration uses the to_syslog_ietf() procedure to convert the corresponding fields in the event record to a Syslog message in IETF format. The result is forwarded via TCP by the om_tcp module.

nxlog.conf


  1
2
3
4
5
6
7
8
9
10

  <Extension _syslog>
    Module  xm_syslog
</Extension>

<Output out>
    Module  om_tcp
    Host    192.168.1.1
    Port    1514
    Exec    to_syslog_ietf();
</Output>

Syslog Snare

The Snare agent format is a special format on top of BSD Syslog which is used and understood by several tools and log analyzer frontends. This format is most useful when forwarding Windows EventLog data in conjunction with im_mseventlog and/or im_msvistalog. The to_syslog_snare() procedure can construct Syslog Snare formatted messages. For more information, see Generating Snare.

Example 130. Generating Syslog Snare and Sending via UDP

In this example, the to_syslog_snare() procedure converts the corresponding fields in the event record to Snare format. The messages are then forwarded via UDP by the om_udp module.

nxlog.conf


  1
2
3
4
5
6
7
8
9
10

  <Extension _syslog>
    Module  xm_syslog
</Extension>

<Output out>
    Module  om_udp
    Host    192.168.1.1
    Port    514
    Exec    to_syslog_snare();
</Output>

NXLog Binary format

The Binary format is only understood by NXLog. All the fields are preserved when the data is sent in this format, so there is no need to parse it again. The output module instance must contain OutputType Binary. The receiver NXLog module instance can be set to InputType Binary.

Graylog Extended Log Format (GELF)

The xm_gelf module can be used to generate GELF output.

Example 131. Generating GELF Output

With this configuration, NXLog will send the fields in the event record via UDP in GELF format.

nxlog.conf


  1
2
3
4
5
6
7
8
9
10

  <Extension _gelf>
    Module      xm_gelf
</Extension>

<Output out>
    Module      om_udp
    Host        127.0.0.1
    Port        12201
    OutputType  GELF_UDP
</Output>

26.2. Forwarding over the Network

The following network protocols can be used. There is a trade-off between speed, reliability, compatibility, and security.

UDP

To send logs in UDP datagrams, use the om_udp module.

Warning

UDP packets can be dropped by the operating system because the protocol does not guarantee reliable message delivery. It is recommended to use TCP or TLS/SSL instead if message loss is a concern.

Example 132. Using the om_udp Module

This example sends the contents of the $raw_event field to the specified host via UDP.

nxlog.conf


  1
2
3
4
5

  <Output out>
    Module  om_udp
    Host    192.168.1.1
    Port    514
</Output>

TCP

To send logs over TCP, use the om_tcp module.

Example 133. Using the om_tcp Module

In this example, the contents of the $raw_event field are sent to the specified host via TCP.

nxlog.conf


  1
2
3
4
5

  <Output out>
    Module  om_tcp
    Host    192.168.1.1
    Port    1514
</Output>

SSL/TLS

To send logs over a trusted secure SSL connection, use the om_ssl module.

Example 134. Using the om_ssl Module

This example provides nearly identical behavior to the TCP example above, but in this case SSL is used to securely transmit the data.

nxlog.conf


  1
2
3
4
5
6
7
8

  <Output out>
    Module      om_ssl
    Host        192.168.1.1
    Port        6514
    CAFile      %CERTDIR%/ca.pem
    CertFile    %CERTDIR%/client-cert.pem
    CertKeyFile %CERTDIR%/client-key.pem
</Output>

HTTP(S)

To send logs over a HTTP or HTTPS connection, use the om_http and im_http modules (available only in NXLog Enterprise Edition).

Example 135. Using the om_http Module

With this configuration, NXLog will send log messages via HTTP, using a POST request for each log message.

nxlog.conf


  1
2
3
4

  <Output out>
    Module  om_http
    URL     http://server:8080/
</Output>

26.3. Sending to Files and Sockets

Files

To store logs in local files, use the om_file module. See also Writing Syslog to File.

Example 136. Using the om_file Module

This configuration writes log messages to the specified file. No additional processing is performed by the output module instance.

nxlog.conf


  1
2
3
4

  <Output out>
    Module  om_file
    File    "/var/log/out.log"
</Output>

Unix Domain Socket

To send logs to a Unix domain socket, use the om_uds module. See also Sending Syslog to the Local Syslog Daemon via /dev/log.

Example 137. Using the om_uds Module

With this configuration, log messages are written to the specified socket without any additional processing.

nxlog.conf


  1
2
3
4

  <Output out>
    Module  om_uds
    UDS     /dev/log
</Output>

26.4. Storing in Databases

The om_dbi and om_odbc modules can be used to store logs in databases. The om_dbi module can be used on POSIX systems where libdbi is available. The om_odbc module, available in NXLog Enterprise Edition, can be used with ODBC compatible databases on Windows, Linux, and Unix.

Example 138. Using the om_dbi Module

This configuration uses libdbi and the pgsql driver to insert events into the specified database. The SQL statement references fields in the event record to be added to the database.

nxlog.conf


  1
2
3
4
5
6
7
8
9
10
11
12

  <Output out>
    Module  om_dbi
    SQL     INSERT INTO log (facility, severity, hostname, timestamp, application, \
                             message) \
            VALUES ($SyslogFacility, $SyslogSeverity, $Hostname, '$EventTime', \
                    $SourceName, $Message)
    Driver  pgsql
    Option  host 127.0.0.1
    Option  username dbuser
    Option  password secret
    Option  dbname logdb
</Output>

Example 139. Using the om_odbc Module

This example inserts events into the database specified by the ODBC connection string. In this case, the sql_exec() and sql_fetch() functions are used to interact with the database.

nxlog.conf


  1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

  <Output out>
    Module              om_odbc
    ConnectionString    DSN=mysql_ds;username=mysql;password=mysql;database=logdb;
    <Exec>
        if ( sql_exec("INSERT INTO log (facility, severity, hostname, timestamp,
                       application, message) VALUES (?, ?, ?, ?, ?, ?)",
                      1, 2, "host", now(), "app", $raw_event) == TRUE )
        {
            if ( sql_fetch("SELECT max(id) as id from log") == TRUE )
            {
                log_info("ID: " + $id);
                if ( sql_fetch("SELECT message from log WHERE id=?", $id) == TRUE )
                    log_info($message);
            }
        }
    </Exec>
</Output>

26.5. Sending to Executables

Using the om_exec module, all messages can be piped to an external program or script which will run until the module (or NXLog) is stopped.

Example 140. Using the om_exec Module

This configuration executes the specified command and writes log messages to its standard input.

nxlog.conf


  1
2
3
4
5

  <Output out>
    Module  om_exec
    Command /usr/bin/someprog
    Arg     -
</Output>