108.13. JSON (xm_json)
This module provides functions and procedures for processing data formatted as JSON. JSON can be generated from log data, or JSON can be parsed into fields. Unfortunately, the JSON specification does not define a type for datetime values so these are represented as JSON strings. The JSON parser in xm_json can automatically detect datetime values, so it is not necessary to explicitly use parsedate().
108.13.1. Configuration
The xm_json module accepts the following directives in addition to the common module directives.
- DateFormat
-
This optional directive can be used to set the format of the datetime strings in the generated JSON. This directive is similar to the global DateFormat, but is independent of it: this directive is defined separately and has its own default. If this directive is not specified, the default is
YYYY-MM-DDThh:mm:ss.sTZ.
- DetectNestedJSON
-
This optional directive can be used to disable the autodetection of nested JSON strings when calling the to_json() function or the to_json() procedure. For example, consider a field
$keywhich contains the string value of{"subkey":42}. If DetectNestedJSON is set to FALSE, to_json() will produce{"key":"{\"subkey\":42}"}. If DetectNestedJSON is set to TRUE (the default), the result is{"key":{"subkey":42}}—a valid nested JSON record.
- Flatten
-
This optional boolean directive specifies that the parse_json() procedure should flatten nested JSON, creating field names with dot notation. The default is FALSE. If Flatten is set to TRUE, the following JSON will populate the fields
$event.timeand$event.severity:{"event":{"time":"2015-01-01T00:00:00.000Z","severity":"ERROR"}}
- ForceUTF8
-
This optional boolean directive specifies whether the generated JSON should be valid UTF-8. The JSON specification requires JSON records to be UTF-8 encoded, and some tools fail to parse JSON if it is not valid UTF-8. If ForceUTF8 is set to TRUE, the generated JSON will be validated and any invalid character will be replaced with a question mark (
?). The default is FALSE.
- ParseDate
-
If this boolean directive is set to TRUE, xm_json will attempt to parse as a timestamp any string that appears to begin with a 4-digit year (as a regular expression,
^[12][0-9]{3}-). If this directive is set to FALSE, xm_json will not attempt to parse these strings. The default is TRUE.
- PrettyPrint
-
If set to TRUE, this optional boolean directive specifies that the generated JSON should be pretty-printed, where each key-value is printed on a new indented line. Note that this adds line-breaks to the JSON records, which can cause parser errors in some tools that expect single-line JSON. If this directive is not specified, the default is FALSE.
- UnFlatten
-
This optional boolean directive specifies that the to_json() procedure should generate nested JSON when field names exist containing the dot (
.). For example, if UnFlatten is set to TRUE, the two fields$event.timeand$event.severitywill be converted to JSON as follows:{"event":{"time":"2015-01-01T00:00:00.000Z","severity":"ERROR"}}When UnFlatten is set to FALSE (the default if not specified), the following JSON would result:
{"event.time":"2015-01-01T00:00:00.000Z","event.severity":"ERROR"}
108.13.2. Functions
The following functions are exported by xm_json.
- string
to_json() -
Convert the fields to JSON and return this as a string value. The
$raw_eventfield and any field having a leading dot (.) or underscore (_) will be automatically excluded unless IncludeHiddenFields directive is set to TRUE.
108.13.3. Procedures
The following procedures are exported by xm_json.
parse_json();-
Parse the
$raw_eventfield as JSON input. parse_json(string source);-
Parse the given string as JSON format.
to_json();-
Convert the fields to JSON and put this into the
$raw_eventfield. The$raw_eventfield and any field having a leading dot (.) or underscore (_) will be automatically excluded unless IncludeHiddenFields directive is set to TRUE.
108.13.4. Examples
The following configuration accepts Syslog (both BSD and IETF) via TCP and converts it to JSON.
![[Download file]](images/icons/download_icon.png "Download file"){kind=icon}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
<Extension syslog>
Module xm_syslog
</Extension>
<Extension json>
Module xm_json
</Extension>
<Input tcp>
Module im_tcp
Port 1514
Host 0.0.0.0
Exec parse_syslog(); to_json();
</Input>
<Output file>
Module om_file
File "/var/log/json.txt"
</Output>
<Route tcp_to_file>
Path tcp => file
</Route>
<30>Sep 30 15:45:43 host44.localdomain.hu acpid: 1 client rule loaded{
"MessageSourceAddress":"127.0.0.1",
"EventReceivedTime":"2011-03-08 14:22:41",
"SyslogFacilityValue":1,
"SyslogFacility":"DAEMON",
"SyslogSeverityValue":5,
"SyslogSeverity":"INFO",
"SeverityValue":2,
"Severity":"INFO",
"Hostname":"host44.localdomain.hu",
"EventTime":"2011-09-30 14:45:43",
"SourceName":"acpid",
"Message":"1 client rule loaded "
}The following configuration reads the Windows EventLog and converts it to the BSD Syslog format, with the message part containing the fields in JSON.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
<Extension syslog>
Module xm_syslog
</Extension>
<Extension json>
Module xm_json
</Extension>
<Input eventlog>
Module im_msvistalog
Exec $Message = to_json(); to_syslog_bsd();
</Input>
<Output tcp>
Module om_tcp
Host 192.168.1.1
Port 1514
</Output>
<Route eventlog_json_tcp>
Path eventlog => tcp
</Route>
<14>Mar 8 14:40:11 WIN-OUNNPISDHIG Service_Control_Manager: {"EventTime":"2012-03-08 14:40:11","EventTimeWritten":"2012-03-08 14:40:11","Hostname":"WIN-OUNNPISDHIG","EventType":"INFO","SeverityValue":2,"Severity":"INFO","SourceName":"Service Control Manager","FileName":"System","EventID":7036,"CategoryNumber":0,"RecordNumber":6788,"Message":"The nxlog service entered the running state. ","EventReceivedTime":"2012-03-08 14:40:12"}