NXLog User Guide

3.1. Event Records and Fields

In NXLog, a log message is an event, and the data relating to that event is collectively an event record. When NXLog processes an event record, it stores the various values in fields. The following sections describe event records and fields in the context of NXLog processing.

3.1.1. Event Records

There are many kinds of event records. A few important ones are listed here.

The most common event record is a single line. Thus the default is LineBased for the InputType and OutputType directives.
It is also common for an event record to use a single UDP datagram. NXLog can send and receive UDP events with the im_udp and om_udp modules.
Some event records are generated using multiple lines. These can be joined into a single event record with the xm_multiline module.
Event records may be stored in a database. Each row in the database represents an event. In this case the im_odbc and om_odbc modules can be used.
It is common for structured event records to be formatted in CSV, JSON, or XML formats. The xm_csv, xm_json, and xm_xml modules provide functions and procedures for parsing these.
NXLog provides a Binary InputType and OutputType for use when compatibility with other logging software is not required. This format preserves parsed fields and their types.

In NXLog, each event record consists of the raw event data (in a field named $raw_event) and additional fields generated during processing and parsing.

3.1.2. Fields

All event log messages contain important data such as user names, IP addresses, and application names. Traditionally, these logs have been generated as free form text messages prepended by basic metadata like the time of the event and a severity value.

While this format is easy for humans to read, it is difficult to perform log analysis and filtering on thousands of free-form logs. In contrast, structured logging provides means for matching messages based on key-value pairs. With structured logging, an event is represented as a list of key-value pairs. The name of the field is the key and the field data is the value. NXLog’s core design embraces structured logging. Using various features provided by NXLog, a message can be parsed into a list of key-value pairs for processing or as part of the message sent to the destination.

When a message is received by NXLog, it creates an internal representation of the log message using fields. Each field is typed and represents a particular attribute of the message. These fields passes through the log route, and are available in each successive module in the chain, until the log message has been sent to its destination.

The special $raw_event field contains the raw data received by the input module. Most input and output modules only transfer $raw_event by default.
The core adds a few additional fields by default. This includes, for example, the $EventReceivedTime field which is set to the time when the event is received.
The input module may add other fields. For example, the im_udp module adds a $MessageSourceAddress field.
Some input modules, such as im_msvistalog and im_odbc, map fields from the source directly to fields in the NXLog event record.
Parsers such as the parse_syslog() procedure will add more fields.
Custom fields can be added by using the NXLog language and an Exec directive.
The NXLog language or the pm_pattern module can be used to set fields using regular expressions. See Extracting Data.

When the configured output module receives the log message, in most cases it will use the contents of the $raw_event field only. If the event’s fields have been modified, it is therefore important to update $raw_event from the other fields. This can be done with the NXLog language, perhaps using a procedure like to_syslog_bsd().

A field is denoted and referenced in the configuration by a preceding dollar sign ($). See the Fields section in the Reference Manual for more information.

Example 1. Processing a Syslog Message

This example shows a Syslog event and its corresponding fields as processed by NXLog. A few fields are omitted for brevity.

NXLog receives an event:

<38>Nov 22 10:30:12 myhost sshd[8459]: Failed password for invalid user linda from 192.168.1.60 port 38176 ssh2

The raw event data is stored in the $raw_event field when NXLog receives a log message. The NXLog core and input module add additional fields.

{
  "raw_event": "<38>Nov 22 10:30:12 myhost sshd[8459]: Failed password for invalid user linda from 192.168.1.60 port 38176 ssh2",
  "EventReceivedTime": "2019-11-22 10:30:13",
  "MessageSourceAddress": "192.168.1.1",

The xm_syslog parse_syslog() procedure parses the basic format of the Syslog message, reading from $raw_event by default. This procedure adds a few more fields:

  "SyslogFacility": "USER",
  "SyslogSeverity": "NOTICE",
  "EventTime": "2019-11-22 10:30:12",
  "Hostname": "myhost",
  "SourceName": "sshd",
  "ProcessID": 8459,
  "Message": "Failed password for invalid user linda from 192.168.1.60 port 38176 ssh2",

Further metadata can be extracted from the free-form $Message field with regular expressions or other methods; see Extracting Data.

  "Status": "failed",
  "AuthenticationMethod": "password",
  "Reason": "invalid user",
  "User": "linda",
  "SourceIPAddress": "192.168.1.60",
  "SourcePort": 38176,
  "Protocol": "ssh2"
}