NXLog User Guide

Use this directive to configure a constant or macro to be used later. Refer to a define by surrounding the name with percent signs (%). Enclose a group of statements with curly braces ({}).

Example 431. Using the define Directive

This configuration shows three example defines: BASEDIR is a constant, IMPORTANT is a statement, and WARN_DROP is a group of statements.

nxlog.conf

  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

  
define BASEDIR /var/log
define IMPORTANT if $raw_event =~ /important/ \
                    $Message = 'IMPORTANT ' + $raw_event;
define WARN_DROP { log_warning("dropping message"); drop(); }

<Input messages>
    Module  im_file
    File    '%BASEDIR%/messages'
</Input>

<Input proftpd>
    Module  im_file
    File    '%BASEDIR%/proftpd.log'
    <Exec>
        %IMPORTANT%
        if $raw_event =~ /dropme/ %WARN_DROP%
    </Exec>
</Input>

This directive works like define, except that the value is retrieved from the environment.

Example 432. Using the envvar Directive

This example is like the previous one, but BASEDIR is fetched from the environment instead.

nxlog.conf

  
1
2
3
4
5
6

  
envvar BASEDIR

<Input in>
    Module  im_file
    File    '%BASEDIR%/messages'
</Input>

This directive allows a specified file to be included in the current configuration file. Wildcarded filenames are supported.

  
1

  
include modules/module1.conf

  
1

  
include /etc/nxlog.d/*.conf

This directive accepts the name of an external command or script. Configuration content will be read from the command’s standard output. Command arguments are not supported.

  
1

  
include_stdout /opt/nxset/etc/fetch_conf.sh

This directive specifies a directory where the cache file (configcache.dat) should be written. This directive has a compiled-in value which is used by default.

This directive specifies how often the in-memory position cache should be flushed to the cache file. The value of 0 indicates that the cache should only be flushed to the file when the agent shuts down; if the server or agent crashes, the current position cache will be lost. A positive integer indicates the length of the interval between flushes of the cache, in seconds. The string always specifies that the cache should be flushed to file immediately when a module sets a value. If this directive is not specified the default value of 5 seconds is used. See also the CacheSync directive below.

When the in-memory position cache is flushed to the cache file, the cache may not be immediately written to the disk due to filesystem buffering. When this directive is set to TRUE, the cache file is synced to disk immediately when it is written. The default is FALSE. CacheSync has no effect if CacheFlushInterval is set to the default of 0. Setting this to TRUE when CacheFlushInterval is set to always greatly reduces performance, though only this guarantees crash-safe operation.

This directive can be used to change the default date format as it appears in the LogFile, in the $raw_event generated by the modules, and when a datetime type value is converted to a string. The following values are accepted (corresponding to the formats accepted by the NXLog strftime() function):

This optional boolean directive specifies the flow control default for input and processor module instances. Output module instances do not inherit from this directive. By default, the global FlowControl value is TRUE. See the description of the module level FlowControl directive for more information.

If set to TRUE, this boolean directive specifies that UTC should be used when generating dates in the format YYYY-MM-DD hh:mm:ss. If set to FALSE, local time will be used when generating dates in this format. The default is FALSE. See also ParseDateInUTC.

Similar to User, NXLog will set the group ID to run under. The group can be specified by name or numeric ID. This directive has no effect when running on the Windows platform or with nxlog-processor(8).

If set to FALSE, NXLog will stop when it encounters a problem with the configuration file (such as an invalid module directive) or if there is any other problem which would prevent all modules functioning correctly. If set to TRUE, NXLog will start after logging the problem. The default value is TRUE.

NXLog will write its internal log to this file. If this directive is not specified, self logging is disabled. Note that the im_internal module can also be used to direct internal log messages to files or different output destinations, but this does not support log level below INFO. This LogFile directive is especially useful for debugging.

This directive has five possible values: CRITICAL, ERROR, WARNING, INFO, and DEBUG. It will set both the logging level used for LogFile and the standard output if NXLog is started in the foreground. The default LogLevel is INFO.

This directive specifies the directory where the files of the persistent queues are stored, for Processor and Output module instances. Even if PersistLogqueue is set to FALSE, NXLog will persist in-memory queues to the LogqueueDir on shutdown. If not specified, the default is the value of CacheDir. This directive can also be used at the module level to specify a log queue directory for a specific module instance.

This directive controls the size of the log queue for all Processor and Output module instances. The default is 100 records. See the module-level LogQueueSize for more information.

By default the NXLog binaries have a compiled-in value for the directory to search for loadable modules. This can be overridden with this directive. The module directory contains sub-directories for each module type (extension, input, output, and processor), and the module binaries are located in those.

Some modules save data to a cache file which is persisted across a shutdown/restart. Modules such as im_file will save the file position in order to continue reading from the same position after a restart as before. This caching mechanism can be explicitly turned off with this directive. This is mostly useful with nxlog-processor(8) in offline mode. If this boolean directive is not specified, it defaults to FALSE (caching is enabled). Note that many input modules, such as im_file, provide a SavePos directive that can be used to disable the position cache for a specific module instance. SavePos has no effect if the cache is disabled globally with NoCache TRUE.

This directive is for debugging. When set to TRUE, NXLog will not free module resources on exit, allowing valgrind to show proper stack trace locations in module function calls. The default value is FALSE.

A panic condition is a critical state which usually indicates a bug. Assertions are used in NXLog code for checking conditions where the code will not work unless the asserted condition is satisfied, and for security. Failing assertions result in a panic and suggest a bug in the code. A typical case is checking for NULL pointers before pointer dereference. This directive can take three different values: HARD, SOFT, or OFF. HARD will cause an abort in case the assertion fails. This is how most C based programs work. SOFT will cause an exception to be thrown at the place of the panic/assertion. In case of NULL pointer checks this is identical to a NullPointerException in Java. It is possible that NXLog can recover from exceptions and can continue to process log messages, or at least the other modules can. In case of assertion failure the location and the condition is printed at CRITICAL log level in HARD mode and ERROR log level in SOFT mode. If Panic is set to OFF, the failing condition is printed in the logs but the execution will continue on the normal code path. Most of the time this will result in a segmentation fault or other undefined behavior, though in some cases turning off a buggy assertion or panic will solve the problems caused by it in HARD/SOFT mode. The default value for Panic is SOFT.

If set to TRUE, this boolean directive specifies that dates in the format YYYY-MM-DD hh:mm:ss should be parsed as UTC. If set to FALSE, dates in this format are assumed to be in local time. The default is FALSE. See also GenerateDateInUTC.

This boolean directive specifies that log queues of Processor and Output module instances should be disk-based. See the module level PersistLogqueue directive for more information.

Under Unix operating systems, NXLog writes a PID file as other system daemons do. The default PID file can be overridden with this directive in case multiple daemon instances need to be running. This directive has no effect when running on the Windows platform or with nxlog-processor(8).

This directive is specific to nxlog-processor(8). It controls the exit condition of nxlog-processor(8). Its value is a timeout in seconds. If nxlog-processor(8) gets no data to process during this period then it will stop waiting for more data and will exit. The default value is 0.05 s. For any non-negative value which is less than 0.05 this will be 0.05. In case nxlog-processor(8) is configured to read data from the network it is recommended to set this to higher value.

NXLog will set its root directory to the value specified with this directive. If SpoolDir is also set, this will be relative to the value of RootDir (chroot() is called first). This directive has no effect when running on the Windows platform or with the nxlog-processor(8).

NXLog will change its working directory to the value specified with this directive. This is useful with files created through relative filenames (for example, with om_file) and in case of core dumps. This directive has no effect with the nxlog-processor(8).

To protect against memory exhaustion (and possibly a denial-of-service) caused by over-sized strings, there is a limit on the length of each string (in bytes). The default value is 5242880 bytes (strings will be truncated at 5 MiB).

Under some circumstances it is possible for NXLog to generate an extreme amount of internal logs consisting of the same message due to an incorrect configuration or a software bug. In this case, the LogFile can quickly consume the available disk space. With this directive, NXLog will write at most 2 lines per second if the same message is generated successively, by logging "last message repeated n times" messages. If this boolean directive is not specified, it defaults to TRUE (suppression of repeating messages is enabled).

When this directive is set to TRUE and PersistLogqueue is enabled, the disk-based queues of Processor and Output module instances will be immediately synced after each new entry is added to the queue. This greatly reduces performance but makes the queue more reliable and crash-safe. This directive can also be used at the module level.

This directive specifies the number of worker threads to use. The number of the worker threads is calculated and set to an optimal value if this directive is not defined. Do not set this unless you know what you are doing.

NXLog will drop to the user specified with this directive. This is useful if NXLog needs privileged access to some system resources (such as kernel messages or to bind a port below 1024). On Linux systems NXLog will use capabilities to access these resources. In this case NXLog must be started as root. The user can be specified by name or numeric ID. This directive has no effect when running on the Windows platform or with nxlog-processor(8).

This mandatory directive specifies which binary should be loaded. The module binary has a .so extension on Unix and a .dll on Windows platforms and resides under the ModuleDir location. Each module binary name is prefixed with im_, pm_, om_, or xm_ (for input, processor, output, and extension, respectively). It is possible for multiple instances to use the same loadable binary. In this case the binary is only loaded once but instantiated multiple times. Different module instances may have different configurations.

This directive specifies the size of the read or write buffer (in bytes) used by the Input or Output module, respectively. The BufferSize directive is only valid for Input and Output module instances. The default buffer size is 65000 bytes.

This directive specifies the directory where the files of the persistent queue are stored. LogqueueDir is only valid for Processor and Output module instances. See the description of the global LogqueueDir for more information.

Every Processor and Output instance has an input log queue for events waiting to be processed by that module. The size of the queue is measured in event records, and can be set with this directive—the default is 100 records. When the log queue of a module instance is full and FlowControl is enabled for the preceeding module, the preceeding module will be suspended. If flow control is not enabled for the preceeding module, events will be discarded. This directive is only valid for Processor and Output module instances. This directive can be used at the global level to affect all modules.

When a module passes an event to the next module along the route, it puts it into the next module’s queue. This queue can be either a memory-based or disk-based (persistent) queue. When this directive is set to TRUE, the module will use a persistent (disk-based) queue. With the default value of FALSE, the module’s incoming log queue will not be persistent (will be memory-based); however, in-memory log queues will still be persisted to disk on shutdown. This directive is only valid for Processor and Output module instances. This directive can also be used at the global level.

When this directive is set to TRUE and PersistLogqueue is enabled, the disk-based queue will be immediately synced after each new entry is added to the queue. This greatly reduces performance but makes the queue more reliable and crash-safe. This directive is only valid for Processor and Output module instances. This directive can be used at the global level to affect all modules.