NXLog User Guide

42. ArcSight Common Event Format (CEF)

NXLog can be configured to collect or forward logs in Common Event Format (CEF). NXLog Enterprise Edition provides the xm_cef module for parsing and generating CEF.

CEF is a text-based log format developed by ArcSight™ and used by HP ArcSight™ products. It uses Syslog as transport. The full format includes a Syslog header or "prefix", a CEF "header", and a CEF "extension". The extension contains a list of key-value pairs. Standard key names are provided, and user-defined extensions can be used for additional key names. In some cases, CEF is used with the Syslog header omitted.

CEF Syntax

Jan 11 10:25:39 host CEF:Version|Device Vendor|Device Product|Device Version|Device Event Class ID|Name|Severity|[Extension]

Log Sample

Oct 12 04:16:11 localhost CEF:0|nxlog.org|nxlog|2.7.1243|Executable Code was Detected|Advanced exploit detected|100|src=192.168.255.110 spt=46117 dst=172.25.212.204 dpt=80

42.1. Collecting and Parsing CEF

NXLog Enterprise Edition can be configured to collect and parse CEF logs with the xm_cef module.

The ArcSight™ Logger can be configured to send CEF logs via TCP with the following steps.

Log in to the Logger control panel.
Browse to Configuration › Data › Forwarders.
Click Add to create a new Forwarder:
- Name: nxlog
- Type: TCP Forwarder
- Type of Filter: Unified Query
Click Next to proceed to editing the new Forwarder:
- Query: (define as required)
- IP/Host: (enter the IP address or hostname of the system running NXLog)
- Port: 1514
Click Save.

Example 199. Receiving CEF Logs

With this configuration, NXLog will collect CEF logs via TCP, convert to plain JSON format, and save to file.

nxlog.conf


  1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

  <Extension _cef>
    Module  xm_cef
</Extension>

<Extension _json>
    Module  xm_json
</Extension>

<Extension _syslog>
    Module  xm_syslog
</Extension>

<Input logger_tcp>
    Module  im_tcp
    Host    0.0.0.0
    Port    1514
    Exec    parse_syslog(); parse_cef($Message);
</Input>

<Output json_file>
    Module  om_file
    File    '/var/log/json'
    Exec    to_json();
</Output>

<Route r>
    Path    logger_tcp => json_file
</Route>

42.2. Generating and Forwarding CEF

NXLog Enterprise Edition can be configured to generate and forward CEF logs with the xm_cef module.

The ArcSight™ Logger can be configured to receive CEF logs via TCP with the following steps.

Log in to the Logger control panel.
Browse to Configuration › Data › Receivers in the navigation menu.
Click Add to create a new Receiver:
- Name: nxlog
- Type: CEF TCP Receiver
Click Next to proceed to editing the new Receiver:
- Port: 574
- Encoding: UTF-8
- Source Type: CEF
Click Save.

Example 200. Sending CEF Logs

With this configuration, NXLog will read Syslog logs from file, convert them to CEF, and forward them to the ArcSight Logger via TCP. Default values will be used for the CEF header unless corresponding fields are defined in the event record (see the to_cef() procedure in the Reference Manual for a list of fields).

nxlog.conf


  1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

  <Extension _cef>
    Module  xm_cef
</Extension>

<Extension _syslog>
    Module  xm_syslog
</Extension>

<Input messages_file>
    Module  im_file
    File    '/var/log/messages'
    Exec    parse_syslog();
</Input>

<Output logger_tcp>
    Module  om_tcp
    Host    192.168.1.1
    Port    574
    Exec    $Message = to_cef(); to_syslog_bsd();
</Output>

<Route r>
    Path    messages_file => logger_tcp
</Route>

42.3. Using xm_csv and xm_kvp

Because NXLog Community Edition does not include the xm_cef module, the xm_csv and xm_kvp modules may be used instead to handle CEF logs.

Warning

The xm_csv and xm_kvp modules may not always correctly parse or generate CEF logs.

Example 201. Using CEF with NXLog Community Edition

Here, the xm_csv module is used to parse the pipe-delimited CEF header, while the xm_kvp module is used to parse the space-delimited key-value pairs in the CEF extension. The required extension configurations are shown below.

nxlog.conf Extensions


  1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

  <Extension cef_header>
    Module          xm_csv
    Fields          $Version, $Device_Vendor, $Device_Product, $Device_Version, \
                    $Signature_ID, $Name, $Severity, $_Extension
    Delimiter       |
    QuoteMethod     None
</Extension>

<Extension cef_extension>
    Module          xm_kvp
    KVDelimiter     '='
    KVPDelimiter    ' '
    QuoteMethod     None
</Extension>

<Extension syslog>
    Module          xm_syslog
</Extension>

For CEF input, use an input instance like this one.

nxlog.conf Input


  1
2
3
4
5
6
7
8
9
10

  <Input in>
    Module  im_tcp
    Host    0.0.0.0
    Port    1514
    <Exec>
        parse_syslog();
        cef_header->parse_csv($Message);
        cef_extension->parse_kvp($_Extension);
    </Exec>
</Input>

For CEF output, use an output instance like this one.

nxlog.conf Output


  1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

  <Output out>
    Module  om_tcp
    Host    192.168.1.1
    Port    574
    <Exec>
        $_Extension = cef_extension->to_kvp();
        $Version = 'CEF:0';
        $Device_Vendor = 'NXLog';
        $Device_Product = 'NXLog';
        $Device_Version = '';
        $Signature_ID = '0';
        $Name = '-';
        $Severity = '';
        $Message = cef_header->to_csv();
        to_syslog_bsd();
    </Exec>
</Output>