Return to

2. About NXLog

Modern IT infrastructure produces large volumes of event logging data. In a single organization, hundreds or thousands of different devices, applications, and appliances generate event log messages. These messages require many log processing tasks, including filtration, classification, correlation, forwarding, and storage. In most organizations these requirements are met with a collection of scripts and programs, each with its custom format and configuration. NXLog provides a single, high-performance, multi-platform product for solving all of these tasks and achieving consistent results.

At NXLog’s inception, there were various logging solutions available, but none with the required features. Most were single threaded and Syslog-oriented, without native support for Windows. Work on NXLog began with the goal of building a modern logger with a multi-threaded design, a clear configuration syntax, multi-platform support, and clean source code. NXLog was born in 2009 as a closed source product heavily used in several production deployments. The source code of NXLog Community Edition was released in November 2011.

NXLog can process event logs from thousands of different sources with volumes over 100,000 events per second. It can accept event logs over TCP, TLS/SSL, and UDP; from files and databases; and in Syslog, Windows EventLog, and JSON formats. NXLog can also perform advanced processing on log messages, such as rewriting, correlating, alerting, pattern matching, scheduling, and log file rotation. It supports prioritized processing of certain log messages, and can buffer messages on disk or in memory to work around problems with input latency or network congestion. After processing, NXLog can store or forward event logs in any of many supported formats. Inputs, outputs, log formats, and complex processing are implemented with a modular architecture and a powerful configuration language.

2.1. NXLog Features

This section gives an overview of some of the key advantages of NXLog over alternative systems.

Multi-platform deployment

Installer packages are provided for multiple platforms, including Linux, Windows, and Android. You can use NXLog across your infrastructure, without resorting to different tools for different platforms.

Client or server operation

Depending on the configuration, NXLog will run as a server, a client, or a combination of both. You have the freedom to choose the deployment architecture that best meets your needs. For example, NXLog can collect local log data and forward it, relay data without storing it locally, or save incoming log data to disk.

Many input and output types and formats

NXLog can accept data from many different sources, convert the data internally, and output it to other destinations. You can use NXLog as a single tool to process all of the different types of logs in your organization. For example, logs can be collected from files, databases, Unix domain sockets, network connections, and other sources. BSD Syslog, IETF Syslog, the Snare Agent format, Windows EventLog, JSON, and other formats are supported. NXLog can likely be configured to read or write logs in your custom application format, using the NXLog language and provided extension modules.

High performance, scalable architecture

With an event-based architecture for processing tasks in parallel, non-blocking input and output where possible, and a worker thread pool for incoming log messages, NXLog is designed for high performance on modern multi-core and multi-processor systems. The input/output readiness notifications provided by most operating systems are used to efficiently handle large numbers of open files and network connections.


NXLog provides features throughout the application to maintain the security of your log data and systems. The core can be configured to run as an unprivileged user, and special privileges (such as binding to ports below 1024) are accessed through Linux capabilities rather than requiring the application to run as root. TLS/SSL is supported for encrypted, authenticated communications and to prevent data interception or alteration during transmission.

Modular architecture

NXLog has a lightweight modular architecture, providing a reduced memory footprint and increased flexibility for different uses. The core handles files, events, and sockets, and provides the configuration language; modules provide the input, output, and processing capabilities. Because modules use a common API, you can write new modules to extend the features of NXLog.

Message buffering

Log messages can be buffered in memory or on disk. This increases reliability by holding messages in a temporary cache when a network connectivity issue or dropout occurs. Conditional buffering can be configured by using the NXLog language to define relevant conditions. For example, UDP messages may arrive faster than they can be processed, and NXLog can buffer the messages to disk for processing when the system is under less load. Conditional buffering can be used to explicitly buffer log messages during certain hours of the day or when the system load is high.

Prioritized processing

NXLog can be configured to separate high-priority log processing from low-priority log processing, ensuring that it processes the most important data first. When the system is experiencing high load, NXLog will avoid dropping important incoming messages. For example, incoming UDP messages can be prioritized to prevent dropped logs if a high volume of TCP messages overloads the system.

Message durability

Built-in flow control ensures that a blocked output does not cause dropped log messages when buffers are full. In combination with the previously mentioned parallel processing, buffering, and prioritization, the possibility of message loss is greatly reduced.

Familiar and powerful configuration syntax

NXLog uses an Apache-style configuration syntax that is easy to read and can be parsed or generated with scripts. The NXLog language supports advanced scripting and processing capabilities that are usually only found in full-fledged scripting languages. The syntax is similar to Perl, so users familiar with that language can learn it easily. It supports polymorphic functions and procedures and regular expressions with captured sub-strings. Modules can register additional functions and procedures to further extend the capabilities of the language.

Scheduled tasks and log rotation

NXLog includes a scheduler service. A task can be scheduled from any module without requiring an external tool such as Cron. Log files can be rotated automatically, on a time-based schedule or according to file size. The file reader and writer modules in NXLog can detect when an input or output file moves or changes its name, and re-open the file automatically.

Advanced message processing

NXLog can perform advanced processing actions on log messages, in addition to the core features already mentioned. By using additional modules, NXLog can solve many related tasks such as message classification, event correlation, pattern matching, message filtering, message rewriting, and conditional alerting. You can use a single tool for all log processing functionality.

Offline processing

Sometimes log messages need to be processed in batches for conversion, filtering, or analysis. NXLog provides an offline mode in which it processes all input and then exits. Because NXLog does not assume that the event time and processing time are identical, time-based correlation features can be used even during offline log processing.

International character and encoding support

NXLog supports explicit character set conversion and automatic character set detection. Log messages received in different character sets can be automatically normalized to a common standard, allowing messages to be compared across different sources.

2.2. Enterprise Edition Features

While the NXLog Community Edition provides all the flexibility and performance of the NXLog engine, the NXLog Enterprise Edition provides additional enhancements, including modules and core features, as well as regular hot-fixes and updates. The Enterprise Edition provides the following enhancements.

Additional platform support

In addition to Linux, Windows, and Android, installer packages are provided for the BSDs and the major variants of Unix (AIX, Solaris, and macOS).

Signed installer packages

Installer packages are certificate signed to ensure that the binaries are not corrupted or compromised.

On-the-wire compression

Log data can be transferred in compressed batches with the im_batchcompress and om_batchcompress input/output modules. This can help in limited bandwidth scenarios.

UDP source IP address spoofing

Some SIEM and log collection systems use the IP address of the UDP Syslog packet sent by the client. When used as a server or relay, the om_udpspoof output module can be configured to retain the original IP address of the sender.

Better control over SSL and TLS

Due to vulnerabilities discovered in the SSL protocols, some protocols may need to be disabled. The various SSL/TLS networking modules in NXLog Enterprise Edition can be configured to allow only specific protocols via the SSLProtocol directive. On Windows, NXLog Enterprise Edition can utilize TLSv1.2 while NXLog Community Edition supports TLSv1.0 only.

ODBC input and output

The ODBC input and output modules, im_odbc and om_odbc, allow log data to be read from or inserted into any ODBC compliant database. The primary purpose of the im_odbc module is native Windows MSSQL support to enable log collection from Windows applications that write logs to MSSQL. The om_odbc output module can be used to insert data into an ODBC database. These modules are available on Windows as well as Linux.

Remote management

A dedicated xm_soapadmin extension module allows NXLog agents to be managed remotely over a secure SOAP/SSL connection or integrated with existing monitoring and management tools. The configuration, correlation rules, patterns, and certificates can all be updated remotely from the NXLog Manager web interface or from scripts. In addition, the NXLog agent and the individual modules can be stopped/started and log collection statistics can be queried for real-time statistics.

Crash recovery

Additional functionality is provided to guarantee a clean recovery in the case of a system crash, ensuring that no messages are lost or duplicated.

Event correlation

The pm_evcorr processor module can efficiently solve complex event correlation tasks, with capabilities similar to what the open-source SEC tool provides.

HTTP(S) protocol support

RESTful services are becoming increasingly popular, even for logging. The im_http and om_http input and output modules make it possible to send or receive log message data over HTTP or HTTPS.

File integrity and registry monitoring

Several compliance standards mandate file integrity monitoring. With the im_fim input module, NXLog Enterprise Edition can be used to detect modifications to files or directories. This module is available on Windows as well as Linux. The im_regmon module provides monitoring of the Windows Registry.

Structured data formats

The xm_xml extension module can parse nested XML and data stored in XML attributes. Parsing of nested JSON has also been implemented in xm_json, and UTF-8 validation can be enforced to avoid parser failures caused by invalid UTF-8 from other tools.

Native W3C parser

The W3C format is widely used in various Microsoft products and perhaps IIS is the most well-known producer. Parsing of W3C is possible with the xm_csv extension module, but that requires defining the fields in the configuration and adjustment when the IIS configuration is changed. The xm_w3c extension module can automatically parse the logs using the field information stored in the headers. It also supports automatic parsing of the data format produced by BRO.

More support for SIEM products

The xm_cef and xm_leef modules provide parsing and generation of CEF and LEEF formatted data. CEF (Common Event Format) was introduced by HP ArcSight and LEEF (Log Event Extended Format) is used by IBM Security QRadar.

Simplified data processing configuration

Two extension modules help simplify the configuration. The xm_rewrite module allows fields to be renamed, kept (whitelisted), or deleted (blacklisted). It also supports the Exec directive so log processing logic can be localized and to avoid duplicated statements. The xm_filelist module provides two functions, contains() and matches(), which can be invoked to check whether a string is present in a text file. This can be a username, IP address, or similar. The files are cached in memory and any changes are automatically picked up without the need to reload NXLog.

Custom input and output modules in Perl

Perl has a vast number of libraries that can be used to easily implement integration with various APIs, formats, and protocols. The im_perl and om_perl input and output modules make it possible to utilize Perl to collect and output data without the need to run the code as an external script.

Name resolution

The xm_resolver extension module provides cached DNS lookup functions for translating between IP addresses and host names. User and group names can also be mapped to/from user and group ids.

Elasticsearch integration

The om_elasticsearch output module allows log data to be loaded directly into an Elasticsearch server without requiring Logstash.

Check Point LEA input

The im_checkpoint input module enables the remote collection of Check Point firewall logs over the OPSEC/LEA protocol. This feature is only available in the Linux version.

Remote Windows EventLog collection

The im_wmi module allows remote collection of Windows EventLog over the WMI protocol on Linux hosts without the need to install an agent on the Windows target. This module is only available in the Linux version. In addition, the im_msvistalog module can query and collect Windows EventLog remotely over MSRPC on Windows Vista and later versions.

Redis Support

Redis is often used as an intermediate queue for log data. Two native modules, im_redis and om_redis, are available to push data to and pull data from Redis servers.

SNMP input

The xm_snmp extension module can be used to parse SNMP traps. The traps can then be handled like regular log messages: converted to Syslog, stored, forwarded, etc.

Multi-platform support for Windows Event Forwarding

The im_wseventing input module can be used to collect forwarded events from Windows hosts. The Windows clients can be configured from Group Policy to send Windows EventLog using Windows Event Forwarding. While NXLog Enterprise Edition can collect Windows EventLog remotely over WMI and MSRPC, this module provides improved security for collecting from Windows machines in agent-less mode, with support for both Kerberos and HTTPS data transfer. The im_wseventing module is platform independent and available on Linux as well as Windows.

HDFS output

The om_webhdfs output module is available to support the Hadoop ecosystem.

Windows Performance Counters

The im_winperfcount input module can collect metrics from Windows Performance Counters such as CPU, disk, and memory statistics.

Reading Windows EventLog files directly

The im_msvistalog module can read .evt, .evtx, and .etl EventLog files directly; this is particularly useful for forensics purposes.

Additional Windows EventLog data

The im_msvistalog module retrieves the EventData and UserData parts which can contain important data in some log sources. In addition, SID values in the EventLog Message can be resolved to account names to produce the same output that EventViewer gives.

Netflow support

The xm_netflow extension module can parse Netflow packets received over UDP. It supports Netflow v1, v5, v7, v9, and IPFIX.

ZeroMQ support

ZeroMQ is a popular high performance messaging library. The im_zmq and om_zmq modules provide input and output support for the ZeroMQ protocol.

Regular hot fixes

Unlike NXLog Community Edition which is a volunteer effort, NXLog Enterprise Edition receives regular hot fixes and enhancements.

2.3. What NXLog is Not

NXLog provides a broad range of features for collecting, processing, forwarding, and storing log data. However, NXLog is not a SIEM product and does not provide:

  • a graphical interface (or "dashboard") for searching logs and displaying reports,

  • vulnerability detection or integration with external threat data,

  • automatic analysis and correlation algorithms, or

  • pre-configured compliance and retention policies.

NXLog does provide processing features that can be used to set up analysis, correlation, retention, and alerting; NXLog can be integrated with many other products to provide a complete solution for aggregation, analysis, and storage of log data.