NXLog User Guide

Installer packages are provided for multiple platforms, including Linux, Windows, and Android. You can use NXLog across your infrastructure, without resorting to different tools for different platforms.

Depending on the configuration, NXLog will run as a server, a client, or a combination of both. You have the freedom to choose the deployment architecture that best meets your needs. For example, NXLog can collect local log data and forward it, relay data without storing it locally, or save incoming log data to disk.

NXLog can accept data from many different sources, convert the data internally, and output it to other destinations. You can use NXLog as a single tool to process all of the different types of logs in your organization. For example, logs can be collected from files, databases, Unix domain sockets, network connections, and other sources. BSD Syslog, IETF Syslog, the Snare Agent format, Windows EventLog, JSON, and other formats are supported. NXLog can likely be configured to read or write logs in your custom application format, using the NXLog language and provided extension modules.

With an event-based architecture for processing tasks in parallel, non-blocking input and output where possible, and a worker thread pool for incoming log messages, NXLog is designed for high performance on modern multi-core and multi-processor systems. The input/output readiness notifications provided by most operating systems are used to efficiently handle large numbers of open files and network connections.

NXLog provides features throughout the application to maintain the security of your log data and systems. The core can be configured to run as an unprivileged user, and special privileges (such as binding to ports below 1024) are accessed through Linux capabilities rather than requiring the application to run as root. TLS/SSL is supported for encrypted, authenticated communications and to prevent data interception or alteration during transmission.

NXLog has a lightweight modular architecture, providing a reduced memory footprint and increased flexibility for different uses. The core handles files, events, and sockets, and provides the configuration language; modules provide the input, output, and processing capabilities. Because modules use a common API, you can write new modules to extend the features of NXLog.

Log messages can be buffered in memory or on disk. This increases reliability by holding messages in a temporary cache when a network connectivity issue or dropout occurs. Conditional buffering can be configured by using the NXLog language to define relevant conditions. For example, UDP messages may arrive faster than they can be processed, and NXLog can buffer the messages to disk for processing when the system is under less load. Conditional buffering can be used to explicitly buffer log messages during certain hours of the day or when the system load is high.

NXLog can be configured to separate high-priority log processing from low-priority log processing, ensuring that it processes the most important data first. When the system is experiencing high load, NXLog will avoid dropping important incoming messages. For example, incoming UDP messages can be prioritized to prevent dropped logs if a high volume of TCP messages overloads the system.

Built-in flow control ensures that a blocked output does not cause dropped log messages when buffers are full. In combination with the previously mentioned parallel processing, buffering, and prioritization, the possibility of message loss is greatly reduced.

NXLog uses an Apache-style configuration syntax that is easy to read and can be parsed or generated with scripts. The NXLog language supports advanced scripting and processing capabilities that are usually only found in full-fledged scripting languages. The syntax is similar to Perl, so users familiar with that language can learn it easily. It supports polymorphic functions and procedures and regular expressions with captured sub-strings. Modules can register additional functions and procedures to further extend the capabilities of the language.

NXLog includes a scheduler service. A task can be scheduled from any module without requiring an external tool such as Cron. Log files can be rotated automatically, on a time-based schedule or according to file size. The file reader and writer modules in NXLog can detect when an input or output file moves or changes its name, and re-open the file automatically.

NXLog can perform advanced processing actions on log messages, in addition to the core features already mentioned. By using additional modules, NXLog can solve many related tasks such as message classification, event correlation, pattern matching, message filtering, message rewriting, and conditional alerting. You can use a single tool for all log processing functionality.

Sometimes log messages need to be processed in batches for conversion, filtering, or analysis. NXLog provides an offline mode in which it processes all input and then exits. Because NXLog does not assume that the event time and processing time are identical, time-based correlation features can be used even during offline log processing.

NXLog supports explicit character set conversion and automatic character set detection. Log messages received in different character sets can be automatically normalized to a common standard, allowing messages to be compared across different sources.

In addition to Linux, Windows, and Android, installer packages are provided for the BSDs and the major variants of Unix (AIX, Solaris, and macOS).

Installer packages are certificate signed to ensure that the binaries are not corrupted or compromised.

Log data can be transferred in compressed batches with the im_batchcompress and om_batchcompress input/output modules. This can help in limited bandwidth scenarios.

Some SIEM and log collection systems use the IP address of the UDP Syslog packet sent by the client. When used as a server or relay, the om_udpspoof output module can be configured to retain the original IP address of the sender.

Due to vulnerabilities discovered in the SSL protocols, some protocols may need to be disabled. The various SSL/TLS networking modules in NXLog Enterprise Edition can be configured to allow only specific protocols via the SSLProtocol directive. On Windows, NXLog Enterprise Edition can utilize TLSv1.2 while NXLog Community Edition supports TLSv1.0 only.

The ODBC input and output modules, im_odbc and om_odbc, allow log data to be read from or inserted into any ODBC compliant database. The primary purpose of the im_odbc module is native Windows MSSQL support to enable log collection from Windows applications that write logs to MSSQL. The om_odbc output module can be used to insert data into an ODBC database. These modules are available on Windows as well as Linux.

A dedicated xm_soapadmin extension module allows NXLog agents to be managed remotely over a secure SOAP/SSL connection or integrated with existing monitoring and management tools. The configuration, correlation rules, patterns, and certificates can all be updated remotely from the NXLog Manager web interface or from scripts. In addition, the NXLog agent and the individual modules can be stopped/started and log collection statistics can be queried for real-time statistics.

Additional functionality is provided to guarantee a clean recovery in the case of a system crash, ensuring that no messages are lost or duplicated.

The pm_evcorr processor module can efficiently solve complex event correlation tasks, with capabilities similar to what the open-source SEC tool provides.

RESTful services are becoming increasingly popular, even for logging. The im_http and om_http input and output modules make it possible to send or receive log message data over HTTP or HTTPS.

Several compliance standards mandate file integrity monitoring. With the im_fim input module, NXLog Enterprise Edition can be used to detect modifications to files or directories. This module is available on Windows as well as Linux. The im_regmon module provides monitoring of the Windows Registry.

The xm_xml extension module can parse nested XML and data stored in XML attributes. Parsing of nested JSON has also been implemented in xm_json, and UTF-8 validation can be enforced to avoid parser failures caused by invalid UTF-8 from other tools.

The W3C format is widely used in various Microsoft products and perhaps IIS is the most well-known producer. Parsing of W3C is possible with the xm_csv extension module, but that requires defining the fields in the configuration and adjustment when the IIS configuration is changed. The xm_w3c extension module can automatically parse the logs using the field information stored in the headers. It also supports automatic parsing of the data format produced by BRO.

The xm_cef and xm_leef modules provide parsing and generation of CEF and LEEF formatted data. CEF (Common Event Format) was introduced by HP ArcSight and LEEF (Log Event Extended Format) is used by IBM Security QRadar.

Two extension modules help simplify the configuration. The xm_rewrite module allows fields to be renamed, kept (whitelisted), or deleted (blacklisted). It also supports the Exec directive so log processing logic can be localized and to avoid duplicated statements. The xm_filelist module provides two functions, contains() and matches(), which can be invoked to check whether a string is present in a text file. This can be a username, IP address, or similar. The files are cached in memory and any changes are automatically picked up without the need to reload NXLog.

Perl has a vast number of libraries that can be used to easily implement integration with various APIs, formats, and protocols. The im_perl and om_perl input and output modules make it possible to utilize Perl to collect and output data without the need to run the code as an external script.

The xm_resolver extension module provides cached DNS lookup functions for translating between IP addresses and host names. User and group names can also be mapped to/from user and group ids.

The om_elasticsearch output module allows log data to be loaded directly into an Elasticsearch server without requiring Logstash.

The im_checkpoint input module enables the remote collection of Check Point firewall logs over the OPSEC/LEA protocol. This feature is only available in the Linux version.

The im_wmi module allows remote collection of Windows EventLog over the WMI protocol on Linux hosts without the need to install an agent on the Windows target. This module is only available in the Linux version. In addition, the im_msvistalog module can query and collect Windows EventLog remotely over MSRPC on Windows Vista and later versions.

Redis is often used as an intermediate queue for log data. Two native modules, im_redis and om_redis, are available to push data to and pull data from Redis servers.

The xm_snmp extension module can be used to parse SNMP traps. The traps can then be handled like regular log messages: converted to Syslog, stored, forwarded, etc.

The im_wseventing input module can be used to collect forwarded events from Windows hosts. The Windows clients can be configured from Group Policy to send Windows EventLog using Windows Event Forwarding. While NXLog Enterprise Edition can collect Windows EventLog remotely over WMI and MSRPC, this module provides improved security for collecting from Windows machines in agent-less mode, with support for both Kerberos and HTTPS data transfer. The im_wseventing module is platform independent and available on Linux as well as Windows.

The om_webhdfs output module is available to support the Hadoop ecosystem.

The im_winperfcount input module can collect metrics from Windows Performance Counters such as CPU, disk, and memory statistics.

The im_msvistalog module can read .evt, .evtx, and .etl EventLog files directly; this is particularly useful for forensics purposes.

The im_msvistalog module retrieves the EventData and UserData parts which can contain important data in some log sources. In addition, SID values in the EventLog Message can be resolved to account names to produce the same output that EventViewer gives.

The xm_netflow extension module can parse Netflow packets received over UDP. It supports Netflow v1, v5, v7, v9, and IPFIX.

ZeroMQ is a popular high performance messaging library. The im_zmq and om_zmq modules provide input and output support for the ZeroMQ protocol.

Unlike NXLog Community Edition which is a volunteer effort, NXLog Enterprise Edition receives regular hot fixes and enhancements.