23. NXLog Language
The NXLog core has a built-in interpreted language. This language can be used to make complex decisions or build expressions in the NXLog configuration file. Code written in the NXLog language is similar to Perl, which is commonly used by developers and administrators for log processing tasks. When NXLog starts and reads its configuration file, directives containing NXLog language code are parsed and compiled into a pseudo-code. If a syntax error is found, NXLog will print the error. The pseudo-code is then evaluated at run-time, as with other interpreted languages.
The features of the NXLog language are not limited to those in the NXLog core: modules can register functions and procedures to supplement the built-in functions and procedures (see the xm_syslog functions, for example).
|
Note
|
Due to the simplicity of the language there is no error handling (except for function return values) available to the user. If an error occurs during the execution of the NXLog pseudo-code, usually the error is printed in the NXLog logs. If an error occurs during log message processing it is also possible for the message to be dropped. If sophisticated error handling or more complex processing is required, the message processing can be implemented in an external script or program via the xm_exec module, in a dedicated NXLog module, or in Perl via the xm_perl module. |
The NXLog language is described in five sections.
- Types
-
All fields and other expressions in the NXLog language are typed.
- Expressions
-
An expression is evaluated to a value at run-time and the value is used in place of the expression. All expressions have types. Expressions can be used as arguments for some module directives.
- Statements
-
The evaluation of a statement will cause a change in the state of the NXLog engine, the state of a module instance, or the current event. Statements often contain expressions. Statements are used as arguments for the Exec module directive, where they are then executed for each event (unless scheduled).
- Variables
-
Variables store data persistently in a module instance, across multiple event records.
- Statistical Counters
-
NXLog provides statistical counters with various algorithms that can be used for realtime analysis.
While this Guide provides many configuration examples, in some cases only statement examples are given. Statements must be used with the Exec directive (or Exec block). The following statement example shows a way to use the parsedate() function.
1
2
if $raw_event =~ /^(\w{3} \d{2} \d{2}:\d{2}:\d{2})/
$EventTime = parsedate($1);
The following configuration example uses the above statement in an Exec block.
![[Download file]](images/icons/download_icon.png "Download file"){kind=icon}
1
2
3
4
5
6
7
8
<Input in>
Module im_file
File '/var/log/app.log'
<Exec>
if $raw_event =~ /^(\w{3} \d{2} \d{2}:\d{2}:\d{2})/
$EventTime = parsedate($1);
</Exec>
</Input>
23.1. Types
The NXLog language is a typed language. Fields, literals, and other expressions evaluate to values with specific types. This allows for stricter type-safety syntax checking when parsing the configuration. Note that fields and some functions can return values with types that can only be determined at run-time.
|
Note
|
The language provides only simple types, complex types such as arrays and hashes (associative arrays) are not supported. The language does support the undefined value similar to that in Perl. See the xm_perl module if you require more complex types. |
A log format must be parsed before its individual parts can be used for processing (see Fields). But even after the message has been parsed into its parts, additional processing may still be required to, for example, prepare a timestamp for comparison with another timestamp. This is where typing is helpful: by converting all timestamps to datetime types, they can be easily compared and later converted to strings if required using provided functions and procedures. The same applies to other types.
The following input configuration, and list below, show the processing of typed fields in a Syslog event record.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
<Input in>
# 1. New event record created
Module im_udp
Host 0.0.0.0
Port 514
<Exec>
# 2. Timestamp parsed from Syslog header
if $raw_event =~ /^(\w{3} \d{2} \d{2}:\d{2}:\d{2})/
{
# 3. parsedate() function converts from string to datetime
$EventTime = parsedate($1);
# 4. Datetime fields compared
if ( $EventReceivedTime - $EventTime ) > 60000000
log_warning('Message delayed more than 1 minute');
}
</Exec>
</Input>
-
NXLog creates a new event record for the incoming log message. The new event record contains the
$raw_eventstring type field, with the contents of the entire Syslog string. -
A regular expression is used to parse the timestamp from the event. The captured sub-string is a string type, not a datetime type.
-
The parsedate() function converts the captured string to a datetime type.
-
Two datetime fields are compared to determine if the message was delayed during delivery. The datetime type
$EventReceivedTimefield is added by NXLog to each event when it is received.
|
Note
|
Normally the parse_syslog() procedure (provided by the xm_syslog extension module) would be used to parse a Syslog event. It will create fields with the appropriate types during parsing, eliminating the need to directly call the parsedate() function. See Collecting and Parsing Syslog. |
23.2. Expressions
An expression is a language element that is dynamically evaluated to a value at run-time. The value is then used in place of the expression. Each expression evaluates to a type, but not always to the same type.
The following language elements are expressions: literals, regular expressions, fields, operations, and functions.
Expressions can be bracketed by parentheses (()) to help improve
code readability.
There are three statements below, one per line. Each statement contains multiple expressions, with parentheses added in various ways.
1
2
3
if 1 + 1 == (1 + 1) log_info("2");
if (1 + 1) == (1 + 1) log_info("2");
if ((1 + 1) == (1 + 1)) log_info("2");
Expressions are often used in statements.
This simple statement uses the log_info() procedure with an expression as its argument. The expression, in this case, is a literal.
1
log_info('This message will be logged.');
Here is a function (also an expression) that is used in the same procedure. It generates an internal event with the current time when each event is processed.
1
log_info(now());
Expressions can be used with module directives that support them.
The File directive of the om_file module supports expressions. This allows the output filename to be set dynamically for each individual event.
1
2
3
4
<Output out>
Module om_file
File "/var/log/nxlog/out_" + strftime($EventTime, "%Y%m%d")
</Output>
See Using Dynamic Filenames for more information.
23.2.1. Literals
A literal is a simple expression that represents a fixed value. Common literals include booleans, integers, and strings. The type of literal is detected by the syntax used to declare it.
|
Note
|
This section demonstrates the use of literals by using examples with assignment statements. |
Boolean literals are either TRUE or FALSE, and are case-insensitively declared using one of those keywords.
1
2
$Important = FALSE;
$Local = true;
Integer literals are declared with an unquoted integer. Negative integers, hexademical notation, and base-2 modifiers (Kilo, Mega, and Giga) are supported.
1
2
3
4
$Count = 42;
$NegativeCount = -42;
$BigCount = 42M;
$HexCount = 0x2A;
String literals are declared by quoting characters with single or double quotes. Escape sequences are available when using double quotes.
1
2
3
4
$Server = 'Alpha';
$Message = 'This is a test message.';
$NumberAsString = '12';
$StringWithNewline = "This is line 1.\nThis is line 2.";
For a list of all available literals, see the Reference Manual Literals section.
23.2.2. Regular Expressions
NXLog supports regular expressions for matching, parsing, and modifying event records. In the context of the NXLog language, a regular expression is an expression that is evaluated to a boolean value at run-time. Regular expressions can be used to define complex search and replacement patterns for text matching and substitution.
|
Note
|
Examples in this section use only simple patterns. See Extracting Data and other topic-specific sections for more extensive examples. |
Matching can be used with an if statement to conditionally execute a statement.
The event record will be discarded if the $raw_event field matches
the regular expression.
1
if $raw_event =~ /TEST: / drop();
Regular expression matching can also be used for extensive parsing, by capturing sub-strings for field assignment.
If the $raw_event field contains the regular expression, the two
fields will be set to the corresponding captured sub-strings.
1
2
3
4
5
if $raw_event =~ /TEST(\d): (.+)/
{
$TestNumber = $1;
$TestName = $2;
}
Regular expression matching also supports named capturing groups. This can be useful when writing long regular expressions. Each captured group is automatically added to the event record as a field with the same name.
This regular expression uses the named groups TestNumber and TestName to
add corresponding $TestNumber and $TestName fields to the event record.
1
2
3
4
if $raw_event =~ /TEST(?<TestNumber>\d): (?<TestName>.+)/
{
$Message = $TestNumber + ' ' + $TestName;
}
Regular expression substitution can be used to modify a string. In
this case, the regular expression follows the form
s/pattern/replace/. The result of the expression will be assigned to
the field to the left of the operator.
The first regular expression match will be removed from the
$raw_event field.
1
$raw_event =~ s/TEST: //;
Global substitution is supported with the /g modifier. Without the
/g modifier, only the first match in the string will be replaced.
Every whitespace character in the $AlertType field will be replaced
with an underscore (_).
1
$AlertType =~ s/\s/_/g;
A statement can be conditionally executed according to the success of a regular expression substitution.
If the substitution succeeds, an internal log message will also be generated.
1
if $Hostname =~ s/myhost/yourhost/ log_info('Updated hostname');
For more information, see the following sections in the Reference Manual: Regular Expressions, =~, and !~.
23.2.3. Fields
When NXLog receives a log message, it creates an event record
for it. An event record is a set of fields (see Fields for more
information). A field is an expression which evaluates to a value with
a specific type. Each field has a name, and in
the NXLog language it is represented with the dollar sign
($) prepended to the name of the field, like Perl’s scalar
variables.
Fields are only available in an evaluation context which is triggered by a log message. For example, using a value of a field in the Exec directive of a Schedule block will result in a run-time error because the scheduled execution is not triggered by a log message.
Because it is through fields that the NXLog language accesses the contents of an event record, they are frequently referenced. The following examples show some common ways that fields are used in NXLog configurations.
This statement uses assignment to set the
$Department field on log messages.
1
$Department = 'customer-service';
If the $Hostname field does not match, the message will be discarded
with the drop() procedure.
1
if $Hostname != 'webserver' drop();
This statement will generate an internal event if $SeverityValue
integer field is greater than 2 (NXLog INFO severity). The
generated event will include the contents of the $Message field.
1
if $SeverityValue > 2 log_warning("ALERT: " + $Message);
23.2.4. Operations
Like other programming languages and especially Perl, the NXLog language has unary operations, binary operations, and the conditional ternary operation. These operations are expressions and evaluate to values.
- Unary Operations
-
Unary operations work with a single operand and evaluate to a boolean value.
Example 34. Using a Unary OperationThis statement uses the defined operator to log a message only if the
$Hostnamefield is defined in the event record.1
if defined $Hostname log_info('Event received');
- Binary Operations
-
Binary operations work with two operands and evaluate to a value. The type of the evaluated value depends on the type of the operands. Execution might result in a run-time error if the type of the operands are unknown at compile time and then evaluate to types which are incompatible with the binary operation when executed.
- Ternary Operation
-
The conditional or ternary operation requires three operands. The first is an expression that evaluates to a boolean. The second is an expression that is evaluated if the first expression is TRUE. The third is an expression that is evaluated if the first expression is FALSE.
Example 36. Using the Ternary OperationThis statement sets the
$Importantfield to TRUE if$SeverityValueis greater than2, or FALSE otherwise. The parentheses are optional and have been added here for clarity.1
$Important = ( $SeverityValue > 2 ? TRUE : FALSE );
For a full list of supported operations, see the Reference Manual Operations section.
23.2.5. Functions
A function is an expression which always returns a value. A function cannot be used without using its return value. Functions can be polymorphic: the same function can take different argument types.
Many NXLog language features are provided through functions. As with other types of expressions, and unlike procedures, a function never modifies the state of the NXLog engine, the state of the module, or the current event.
See the list of core functions. Modules can provide additional functions for use with the NXLog language.
These statements use the now() function (returning the current time) and the hostname() function (returning the hostname of the system running NXLog) to set fields.
1
2
$EventTime = now();
$Relay = hostname();
Here, any event with a $Message field over 4096 bytes causes an
internal log to be generated.
1
if size($Message) > 4096 log_info('Large message received.');
23.3. Statements
The evaluation of a statement will usually result in a change in the state of the NXLog engine, the state of a module, or the log message.
Statements are used with the Exec module
directive. A statement is terminated by a semicolon (;).
With this input configuration, an internal NXLog log message will be generated for each message received.
1
2
3
4
5
6
<Input in>
Module im_udp
Host 0.0.0.0
Port 514
Exec log_info("Message received on UDP port 514");
</Input>
Multiple statements can be specified, these will be evaluated and executed in order. Statements can also be given on multiple lines by using line continuation or by enclosing the statements in an Exec block.
This configuration generates an internal log message and sets the
$File field.
1
2
3
4
5
<Input in1>
Module im_file
File '/var/log/app.log'
Exec log_info("App message read from log"); $File = file_name();
</Input>
This is the same, but the backslash (\) is used to continue the
Exec directive to the next line.
1
2
3
4
5
6
<Input in2>
Module im_file
File '/var/log/app.log'
Exec log_info("App message read from log"); \
$File = file_name();
</Input>
This also is the same, but uses an Exec block instead. The backslash is not necessary here.
1
2
3
4
5
6
7
8
<Input in3>
Module im_file
File '/var/log/app.log'
<Exec>
log_info("App message read from log");
$File = file_name();
</Exec>
</Input>
Statements can also be executed based on a schedule, by using the
Exec directive of a
Schedule block. The Exec directive is
slightly different in this case: because its execution is based on a
schedule rather than a log event, there is no associated event
record. The $File field assignment in the example above would be
impossible.
This input instance will generate an hourly internal log event.
1
2
3
4
5
6
7
8
9
<Input syslog_udp>
Module im_udp
Host 0.0.0.0
Port 514
<Schedule>
When @hourly
Exec log_info("The syslog_udp input module instance is active.");
</Schedule>
</Input>
|
Note
|
Similar functionality is implemented by the im_mark module. |
23.3.1. Assignment
Each event record is made up of fields, and assignment is the
primary way that a value is written to a field in the NXLog
language. The assignment operation is declared with an equal sign
(=). This operation loads the value from the expression evaluated on
the right into an event record field on the
left.
This input instance uses assignment operations to add two fields to each event record.
1
2
3
4
5
6
7
8
<Input in>
Module im_file
File '/var/log/messages'
<Exec>
$Department = 'processing';
$Tier = 1;
</Exec>
</Input>
23.3.2. Block
Statements can be declared inside a block by surrounding them with
curly braces ({}). A statement block in the configuration is parsed
as if it were a single statement. Blocks are typically used with
conditional statements.
This statement uses a block to execute two statements if the
$Message field matches.
1
2
3
4
5
6
7
8
9
10
11
<Input in>
Module im_file
File '/var/log/messages'
<Exec>
if $Message =~ /^br0:/
{
log_warning('br0 interface state changed');
$Tag = 'network';
}
</Exec>
</Input>
23.3.3. Procedures
While functions are expressions that evaluate to values, procedures are statements that perform actions. Both functions and procedures can take arguments. Unlike functions, procedures never return values. Instead, a procedure modifies its argument, the state of the NXLog engine, the state of a module, or the current event. Procedures can be polymorphic: the same procedure can take different argument types.
Many NXLog language features are provided through procedures. See the list of available procedures. Modules can provide additional procedures for use with the NXLog language.
This example uses the parse_syslog() procedure, provided by the xm_syslog module, to parse each Syslog-formatted event record received via UDP.
1
2
3
4
5
6
<Input in>
Module im_udp
Host 0.0.0.0
Port 514
Exec parse_syslog();
</Input>
23.3.4. If-Else
The if or conditional statement allows a statement to be executed based on the boolean value of an expression. When the boolean is TRUE, the statement is executed. An optional else keyword can be followed by another statement to be executed if the boolean is FALSE.
This example uses an if statement and the drop() procedure to discard any event that matches the regular expression.
1
2
3
4
5
<Input in1>
Module im_file
File '/var/log/messages'
Exec if $raw_event =~ /junk/ drop();
</Input>
Here, any event not matching the regular expression will be dropped.
1
2
3
4
5
<Input in2>
Module im_file
File '/var/log/messages'
Exec if not ($raw_event =~ /important/) drop();
</Input>
Finally, this statement shows more extensive use of the if
statement, with an else clause and blocks
defined by curly braces ({}).
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
<Input in3>
Module im_file
File '/var/log/messages'
<Exec>
if $raw_event =~ /alert/
{
log_warning('Detected alert message');
}
else
{
log_info('Discarding non-alert message');
drop();
}
</Exec>
</Input>
23.4. Variables
While NXLog provides fields for storing data during the processing of an event, they are only available for the duration of that event record and can not be used to store a value across multiple events. For this purpose, module variables can be used. A variable stores a value for the module instance where it is set. It can only be accessed from the same module where it was created: a variable with the same name is a different variable when referenced from another module.
Each module variable can be created with an expiry value or an infinite lifetime. If an expiry is used, the variable will be destroyed automatically when the lifetime expires. This can be used as a garbage collection method or to reset variable values automatically.
A module variable is referenced by a string value and can store a value of any type. Module variables are supported by all modules. See the create_var(), delete_var(), set_var(), and get_var() procedures.
If the number of login failures exceeds 3 within 45 seconds, then an internal log message is generated.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
<Input in>
Module im_file
File '/var/log/messages'
<Exec>
if $Message =~ /login failure/
{
if not defined get_var('login_failures')
{ # create the variable if it doesn't exist
create_var('login_failures', 45);
set_var('login_failures', 1);
}
else
{ # increase the variable and check if it is over the limit
set_var('login_failures', get_var('login_failures') + 1);
if get_var('login_failures') >= 3
log_warning(">= 3 login failures within 45 seconds");
}
}
</Exec>
</Input>
|
Note
|
The pm_evcorr module is recommended instead for this case. This algorithm does not reliably detect failures because the lifetime of the variable is not affected by set_var(). For example, consider login failures at 0, 44, 46, and 47 seconds. The lifetime of the variable will be set when the first failure occurs, causing the variable to be cleared at 45 seconds. The variable is created with a new expiry at 46 seconds, but then only two failures are noticed. Also, this method can only work in real-time because the timing is not based on values available in the log message (though the event time could be stored in another variable). |
23.5. Statistical Counters
Like variables, statistical counters provide data storage for a module instance. Counters only support integers, but a counter can use an algorithm to recalculate its value every time it is updated or read. With NXLog Enterprise Edition v4.x and earlier, a statistical counter will only return a value if the time specified in the interval argument has elapsed since it was created. Statistical counters can also be created with a lifetime. When a counter expires, it is destroyed, like module variables.
A statistical counter can be created with the create_stat() procedure call. After it is created, it can be updated with the add_stat() procedure call. The value of the counter can be read with the get_stat() function call. Note that the value of the statistical counter is only recalculated during these calls, rather than happening automatically. This can result in some slight distortion of the calculated value if the add and read operations are infrequent.
A time value can also be specified during creation, updating, and reading. This makes it possible for statistical counters to be used with offline log processing.
This input configuration uses a Schedule block and a statistical counter with the RATEMAX algorithm to calculate the maximum rate of events over a 1 hour period. An internal log message is generated if the rate exceeds 500 events/second at any point during the 1 hour period.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
<Input in>
Module im_tcp
Host 0.0.0.0
Port 1514
<Exec>
parse_syslog();
if defined get_stat('eps') add_stat('eps', 1, $EventReceivedTime);
</Exec>
<Schedule>
Every 1 hour
<Exec>
create_stat('eps', 'RATEMAX', 1, now(), 3600);
if get_stat('eps') > 500
log_info('Inbound TCP rate peaked at ' + get_stat('eps')
+ ' events/second during the last hour');
</Exec>
</Schedule>
</Input>