25.7. Extracting Data
When NXLog receives an event, it creates an event record with
a $raw_event field, other core fields like
$EventReceivedTime, and any fields
provided by the particular module (see Fields for more
information). This section explores the various ways that
NXLog can be configured to extract values from the raw event.
Some log sources (like Windows EventLog collected via im_msvistalog) already contain structured data. In this case, there is often no additional extraction required; see Message Classification.
25.7.1. Parser Procedures
NXLog includes many parser procedures that will automatically extract values from a raw log event and create corresponding fields in the event record. These procedures are provided by extension modules. For example, the xm_syslog module provides the parse_syslog() procedure, which will parse a BSD or IETF Syslog formatted raw event and create fields in the event record. Other procedures will parse JSON, key-value pairs, XML, and more; many of the available extension modules provide parser procedures.
This example shows a Syslog event as it is received via UDP and processed by the parse_syslog() procedure.
<38>Nov 22 10:30:12 myhost sshd[8459]: Failed password for invalid user linda from 192.168.1.60 port 38176 ssh2The following configuration loads the xm_syslog extension module and then uses the Exec directive to executed the parse_syslog() procedure for each event.
![[Download file]](images/icons/download_icon.png "Download file"){kind=icon}
1
2
3
4
5
6
7
8
9
10
<Extension _syslog>
Module xm_syslog
</Extension>
<Input udp>
Module im_udp
Host 0.0.0.0
Port 514
Exec parse_syslog();
</Input>
This results in the following fields being added to the event record by parse_syslog().
| Field | Value |
|---|---|
$Message |
Failed password for invalid user linda from 192.168.1.60 port 38176 ssh2 |
$SyslogSeverityValue |
6 |
$SyslogSeverity |
INFO |
$SeverityValue |
2 |
$Severity |
INFO |
$SyslogFacilityValue |
4 |
$SyslogFacility |
AUTH |
$EventTime |
2016-11-22 10:30:12 |
$Hostname |
myhost |
$SourceName |
sshd |
$ProcessID |
8459 |
25.7.2. Regular Expressions via the Exec Directive
NXLog supports the use of regular expressions for parsing fields. For detailed information about regular expressions in NXLog, see the Reference Manual Regular Expressions section.
<38>Nov 22 10:30:12 myhost sshd[8459]: Failed password for invalid user linda from 192.168.1.60 port 38176 ssh2With this configuration, the above Syslog message is first parsed with
parse_syslog(). This results in a
$Message field created in the event
record. Then, a regular expression is used to further parse the
$Message field and create additional fields if it matches.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
<Extension _syslog>
Module xm_syslog
</Extension>
<Input udp>
Module im_udp
Host 0.0.0.0
Port 514
<Exec>
parse_syslog();
if $Message =~ /(?x)^Failed\ (\S+)\ for(?:\ invalid user)?\ (\S+)\ from
\ (\S+)\ port\ \d+\ ssh2$/
{
$AuthMethod = $1;
$AccountName = $2;
$SourceIPAddress = $3;
}
</Exec>
</Input>
Named capturing is supported also.
1
2
3
4
5
6
7
8
9
10
11
<Input in>
Module im_udp
Host 0.0.0.0
Port 514
<Exec>
parse_syslog();
$Message =~ /(?x)^Failed\ (?<AuthMethod>\S+)\ for(?:\ invalid\ user)?
\ (?<AccountName>\S+)\ from\ (?<SourceIPAddress>\S+)\ port
\ \d+\ ssh2$/;
</Exec>
</Input>
| Field | Value |
|---|---|
$AuthMethod |
password |
$AccountName |
linda |
$SourceIPAddress |
192.168.1.60 |
25.7.3. Pattern Matching With pm_pattern
Regular expressions are widely used in pattern matching. Unfortunately, using a large number of regular expression based patterns does not scale well, because these need to be evaluated linearly. The pm_pattern module implements a more efficient pattern matching than regular expressions used in Exec directives.
<38>Nov 22 10:30:12 myhost sshd[8459]: Failed password for invalid user linda from 192.168.1.60 port 38176 ssh2With this configuration, the above Syslog message is first parsed with parse_syslog(). This results in a $Message field created in the event record. Then, the pm_pattern module is used with a pattern XML file to further parse the record.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
<Extension _syslog>
Module xm_syslog
</Extension>
<Input in>
Module im_udp
Host 0.0.0.0
Port 514
Exec parse_syslog();
</Input>
<Processor pattern>
Module pm_pattern
PatternFile /var/lib/nxlog/patterndb.xml
</Processor>
<Output out>
Module om_null
</Output>
<Route r>
Path in => pattern => out
</Route>
The patterns for the pm_pattern module instance above
are declared in the following patterndb.xml file.
<?xml version='1.0' encoding='UTF-8'?>
<patterndb>
<created>2010-01-01 01:02:03</created>
<version>42</version>
<!-- First and only pattern group in this file -->
<group>
<name>ssh</name>
<id>42</id>
<!-- Only try to match this group if $SourceName == "sshd" -->
<matchfield>
<name>SourceName</name>
<type>exact</type>
<value>sshd</value>
</matchfield>
<!-- First and only pattern in this pattern group -->
<pattern>
<id>1</id>
<name>ssh auth failure</name>
<!-- Do regular expression match on $Message field -->
<matchfield>
<name>Message</name>
<type>regexp</type>
<value>^Failed (\S+) for(?: invalid user)? (\S+) from (\S+) port \d+ ssh2</value>
<!-- Set 3 event record fields from captured strings -->
<capturedfield>
<name>AuthMethod</name>
<type>string</type>
</capturedfield>
<capturedfield>
<name>AccountName</name>
<type>string</type>
</capturedfield>
<capturedfield>
<name>SourceIPAddress</name>
<type>string</type>
</capturedfield>
</matchfield>
<!-- Set additional fields if pattern matches -->
<set>
<field>
<name>TaxonomyAction</name>
<value>Authenticate</value>
<type>string</type>
</field>
<field>
<name>TaxonomyStatus</name>
<value>Failure</value>
<type>string</type>
</field>
</set>
</pattern>
</group>
</patterndb>| Field | Value |
|---|---|
$AuthMethod |
password |
$AccountName |
linda |
$SourceIPAddress |
192.168.1.60 |
$TaxonomyAction |
Authenticate |
$TaxonomyStatus |
Failure |
NXLog Manager provides an interface for writing pattern files, and will also test sample events to aid in establishing the correct match patterns. The pattern functions can be accessed from the PATTERNS menu in the page header.
The following instructions explain the steps required for creating the above pattern database with NXLog Manager.
-
Open . Enter a Name for the new pattern group, and optionally a Description, in the Properties section. The name is used to refer to the pattern group later. The name of the above pattern group is
ssh. -
Add a match field by clicking Add Field in the Match section. Only messages that match will be further processed by this pattern group. In the above example, there is no reason to attempt any matches if the
$SourceNamefield does not equalsshd. The above pattern group uses Field name=SourceName, Match=EXACT, and Value=sshd.
-
Save the new pattern group.
-
Open to create a new field to be used when creating new patterns. For the above example, the
$AuthMethodfield must be added because it is not in the default set provided by NXLog Manager. Set Name=AuthMethodand Field Type=STRING, then click Save. -
Open . In the Pattern Info section, enter a Pattern Name and optionally a Pattern Description. Select the correct Pattern Group from the list. In the above example, the
sshpattern group is used. -
In the Match section, set match values for the fields to be matched. If a regular expression match with captured subgroups is detected, the interface will provide a Captured fields list where target fields can be selected. The above example uses Field name=
Message, Match=REGEXP, and Value=^Failed (\S+) for(?: invalid user)? (\S+) from (\S+) port \d+ ssh2$. The three captured fields areAuthMethod,AccountName, andSourceIPAddress.
-
The Set section allows fields to be set if the match is successful. Click Add Field for each field. The above example sets
$TaxonomyStatustoFailureand$TaxonomyActiontoAuthenticate.
-
The Action section accepts NXLog language statements like would be specified in an Exec directive. Click Add action, type in the statement, and click Verify to make sure the statement is valid. The above example does not include any NXLog language statements.
-
The final tabbed section allows test messages to be entered to verify that the match works as expected. Click the + to add a test case. To test the above example, add a Value for the
Messagefield:Failed password for invalid user linda from 192.168.1.60 port 38176 ssh2. Click Update Test Cases in the Match section to automatically fill the captured fields. Verify that the fields are set as expected. Additional test cases can be added to test other events.
-
Save the new pattern. Then click Export to download the pattern.xml file or use the pattern to configure a managed agent.
See the NXLog Manager User Guide for more information.
25.7.4. Using the Extracted Fields
The previous sections explore ways that the log message can be parsed and new fields added to the event record. Once the required data has been extracted and corresponding fields created, there are various ways to use this new data.
-
A field or set of fields can be matched by string or regular expression to trigger alerts, perform filtering, or further classify the event.
-
Fields in the event record can be renamed, modified, or deleted.
-
Event correlation can be used to execute statements or suppress messages based on matching events inside a specified window.
-
Some output formats can be used to preserve the full set of fields in the event record (such as JSON and the NXLog Binary format).