NXLog User Guide

Log messages are buffered in various places in NXLog, and buffered messages can be lost in the case of a crash. Persistent module message queues can be enabled so that these messages are stored on disk instead of in memory. Each log message is removed from the queue only after successful delivery. See the PersistLogqueue and SyncLogqueue global configuration directives, and the PersistLogqueue and SyncLogqueue module directives.

Input positions (for im_file and other modules) are saved in the cache file, and by default this file is only saved to disk on shutdown. In case of a crash some events may be duplicated or lost depending on the value of the ReadFromLast directive. This data can be periodically flushed and synced to disk using the CacheFlushInterval and CacheSync directives.

NXLog can use the HTTP/HTTPS protocol to provide guaranteed message delivery over the network, optionally with TLS/SSL. The client (om_http) sends the event in a HTTP POST request. The server (im_http, only available in NXLog Enterprise Edition) responds with a status code indicating successful message reception.

Example 152. HTTPS Log Transfer

In the following configuration example, a client reads logs from a file and transmits the logs over an SSL-secured HTTP connection.

nxlog.conf (Client/Sending)

  
1
2
3
4
5
6
7
8
9
10
11
12

  
<Input in>
    Module              im_file
    File                'input.log'
</Input>

<Output out>
    Module              om_http
    URL                 https://10.0.0.1:8080/
    HTTPSCertFile       %CERTDIR%/client-cert.pem
    HTTPSCertKeyFile    %CERTDIR%/client-key.pem
    HTTPSCAFile         %CERTDIR%/ca.pem
</Output>

The server side accepts the HTTPS connections and stores the received messages in a file. The contents of input.log will be replicated in output.log.

nxlog.conf (Server/Receiving)

  
1
2
3
4
5
6
7
8
9
10
11
12
13

  
<Input in>
    Module              im_http
    ListenAddr          0.0.0.0
    Port                8080
    HTTPSCertFile       %CERTDIR%/server-cert.pem
    HTTPSCertKeyFile    %CERTDIR%/server-key.pem
    HTTPSCAFile         %CERTDIR%/ca.pem
</Input>

<Output out>
    Module              om_file
    File                'output.log'
</Output>

The om_batchcompress and im_batchcompress modules, available in NXLog Enterprise Edition, also provide acknowledgment as part of the batchcompress protocol.

Example 153. Batched Log Transfer

With the following configuration, a client reads logs from a file and transmits the logs in compressed batches to a remote NXLog agent.

nxlog.conf (Client/Sending)

  
1
2
3
4
5
6
7
8
9
10
11
12
13

  
<Input in>
    Module         im_file
    File           'input.log'
</Input>

<Output out>
    Module         om_batchcompress
    Host           10.0.0.1
    UseSSL         true
    CertFile       %CERTDIR%/client-cert.pem
    CertKeyFile    %CERTDIR%/client-key.pem
    CAFile         %CERTDIR%/ca.pem
</Output>

The remote NXLog agent receives and decompresses the received message batches and stores the individual messages in a file. The contents of input.log will be replicated in output.log.

nxlog.conf (Server/Receiving)

  
1
2
3
4
5
6
7
8
9
10
11
12

  
<Input in>
    Module          im_batchcompress
    ListenAddr      0.0.0.0
    CertFile        %CERTDIR%/server-cert.pem
    CertKeyFile     %CERTDIR%/server-key.pem
    CAFile          %CERTDIR%/ca.pem
</Input>

<Output out>
    Module          om_file
    File            'output.log'
</Output>

Each module which receives a message directly from an input source or from another module in the route assigns a field named $__SERIAL__$ with a monotonically increasing serial number. The serial number is taken from a global generator and is increased after each fetch so that two messages received at two modules simultaneously will not have the same serial number. The serial number is initialized to the seconds elapsed since epoch when NXLog is started. This way it can provide 1,000,000 serial numbers per second without problems in case it is stopped and restarted. Otherwise the value would need to be saved and synced to disk as well after each serial number fetch and again this would result in a huge performance hit. When a module receives a message it checks the value of the field named $__SERIAL__$ against the last saved value.

The im_http module keeps the value of the last $__SERIAL__$ for each client. It is only possible to know and identify the client (om_http sender) in HTTPS mode. The Common Name (CN) in the certificate subject is used and is assumed to uniquely identify the client.

All other non-network modules use the value of SourceModuleName which is automatically set to the name of the module instance generating the log message. This value is assumed to uniquely identify the source. The value of SourceModuleName is not overwritten if it already exists. Note that this may present problems in some complex setups.

The algorithm is implemented in one procedure call named duplicate_guard(), which can be used in modules to prevent message duplication. The dropped() function can be then used to test whether the current log message has been dropped.