NXLog User Guide

25.11. Message Classification

Pattern matching is commonly used for message classification. When certain strings are detected in a log message, the message gets tagged with classifiers. Thus it is possible to query or take action based on the classifiers only. There are several ways to classify messages based on patterns.

See also Extracting Data, a closely related topic, for more examples of classification.

25.11.1. Simple Matching on Fields

Often, message classification can be performed during parsing. However, if the required fields have already been parsed or the input module provides structured data, then it is only necessary to match the relevant fields and set classifiers.

Example 106. Classifying a Windows Security EventLog Message

This example classifies Windows Security login failure events with Event ID 4625 (controlled by the "Audit logon events" audit policy setting). If a received event has that ID, it is classified as a failed authentication attempt and the $AccountName field is set to the value of $TargetUserName.

Table 28. Sample Event via im_msvistalog (Excerpt)
Field	Value
$EventType	AUDIT_FAILURE
$EventID	4625
$SourceName	Microsoft-Windows-Security-Auditing
$Channel	Security
$Category	Logon
$TargetUserSid	S-1-0-0
$TargetUserName	linda
$TargetDomainName	WINHOST
$Status	0xc000006d
$FailureReason	%%2313
$SubStatus	0xc000006a
$LogonType	2

nxlog.conf


  1
2
3
4
5
6
7
8
9
10
11
12

  <Input in>
    Module  im_msvistalog
    <Exec>
        if ($EventID == 4625) and
           ($SourceName == 'Microsoft-Windows-Security-Auditing')
        {
            $TaxonomyAction = 'Authenticate';
            $TaxonomyStatus = 'Failure';
            $AccountName = $TargetUserName;
        }
    </Exec>
</Input>

Table 29. Fields Added to the Event Record
Field	Value
$TaxonomyAction	Authenticate
$TaxonomyStatus	Failure
$AccountName	linda

25.11.2. Regular Expressions via the Exec Directive

The =~ operator can be used for regular expression matching in an Exec directive.

Example 107. Regular Expression Message Classification

When the contents of the $Message field match against the regular expression, the $AccountName and $AccountID fields are filled with the appropriate values from the referenced captured sub-strings. Additionally the value LoginEvent is stored in the $Action field.


  1
2
3
4
5
6
7

  if $Message =~ /(?x)^pam_unix\(sshd:session\):\ session\ opened\ for\ user\ (\S+)
                \ by\ \(uid=(\d+)\)/
{
    $AccountName = $1;
    $AccountID = integer($2);
    $Action = 'LoginEvent';
}

25.11.3. Using pm_pattern

When there are a lot of patterns, writing them all in the configuration file is inefficient. Instead, the pm_pattern module can be used.

Example 108. Classifying With pm_pattern

The above pattern matching rule can be defined in the pm_pattern modules’s XML format in the following way, which will accomplish the same result.

Pattern Database (patterndb.xml)

<pattern>
   <id>42</id>
   <name>ssh_pam_session_opened</name>
   <description>ssh pam session opened</description>
   <matchfield>
      <name>Message</name>
      <type>REGEXP</type>
      <value>
      ^pam_unix\(sshd:session\): session opened for user (\S+) by \(uid=(\d+)\)
      </value>
      <capturedfield>
         <name>AccountName</name>
         <type>STRING</type>
      </capturedfield>
      <capturedfield>
         <name>AccountID</name>
         <type>INTEGER</type>
      </capturedfield>
   </matchfield>
   <set>
      <field>
          <name>Action</name>
          <type>STRING</type>
          <value>LoginEvent</value>
      </field>
   </set>
</pattern>

nxlog.conf


  1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

  <Extension _syslog>
    Module      xm_syslog
</Extension>

<Input uds>
    Module      im_uds
    UDS         /dev/log
    Exec        parse_syslog_bsd();
</Input>

<Processor pattern>
    Module      pm_pattern
    PatternFile /var/lib/nxlog/patterndb.xml
</Processor>

<Output file>
    Module      om_file
    File        "/var/log/messages"
    Exec        to_syslog_bsd();
</Output>

<Route uds_to_file>
    Path        uds => pattern => file
</Route>