NXLog User Guide

25.15. Timestamps

The NXLog core provides functions for parsing timestamps and returning datetime values, and functions for generating formatted timestamps from datetime values.

25.15.1. Parsing Timestamps

Most timestamps can be parsed with the parsedate() function, which will automatically parse any of the supported formats.

Example 120. Parsing a Timestamp With parsedate()

Consider the following line-based input sample. Each record begins with a timestamp followed by a tab.

Input Sample

2016-10-11T22:14:15.003Z⇥machine.example.com⇥An account failed to log on.

This example configuration uses a regular expression to capture the string up to the first tab. Then the parsedate() function is used to parse the resulting string and set the $EventTime field to the corresponding datetime value. This value can be converted to a timestamp string as required in later processing, either explicitly or as defined by the global DateFormat directive (see Formatting Timestamps).

nxlog.conf


  1
2
3
4
5

  <Input in>
    Module  im_file
    File    'in.log'
    Exec    if $raw_event =~ /^([^\t])\t/ $EventTime = parsedate($1);
</Input>

Tip

The parsedate() function is especially useful if the timestamp format varies within the events being processed. A timestamp of any supported format will be parsed. In this example, the timestamp must be at the beginning of the event and followed by a tab character to be matched by the regular expression.

Sometimes a log source will contain a few events with invalid or unexpected formatting. If parsedate() fails to parse the input string, it will return an undefined datetime value. This allows the user to configure a fallback timestamp.

Example 121. Using a Fallback Timestamp With parsedate()

This example statement uses a vague regular expression that may in some cases match an invalid string. If parsedate() fails to parse the timestamp, it will return an undefined datetime value. In this case, the final line below will set $EventTime to the current time.


  1
2
3
4
5

  if $raw_event =~ /^(\S+)\s+(\S+)/
    $EventTime = parsedate($1 + " " + $2);

# Make sure $EventTime is set
if not defined($EventTime) $EventTime = now();

Tip	`$EventTime = $EventReceivedTime` could be used instead to set a timestamp according to when the event was received by NXLog.

For parsing more exotic formats, the strptime() function can be used.

Example 122. Using strptime() to Parse Timestamps

In this input sample, the date and time are two distinct fields delimited by a tab. It also uses a non-standard single digit format instead of fixed width with double digits.

Input Sample

2011-5-29⇥0:3:2 GMT⇥WINDOWSDC⇥An account failed to log on.

To parse this, a regular expression can be used to capture the timestamp string. This string is then parsed with the strptime() function.


  1
2

  if $raw_event =~ /^(\d+-\d+-\d+\t\d+:\d+:\d+ \w+)/
    $EventTime = strptime($1, '%Y-%m-%d%t%H:%M:%S %Z');

25.15.2. Adjusting Timestamps

Sometimes a log source sends events with incorrect or incomplete timestamps. For example, some network devices may not have the correct time (especially immediately after rebooting); also, the BSD Syslog header provides neither the year nor the timezone. NXLog can be configured to apply timestamp corrections in various ways.

Warning

Reliably applying timezone offsets is difficult due to complications like daylight savings time (DST) and networking and processing delays. For this reason, it is best to use clock synchronization (such as NTP) and timezone-aware timestamps at the log source when possible.

The simplest solution for incorrect timestamps is to replace them with the time when the event was received by NXLog. This is a good option for devices with untrusted clocks on the local network that send logs to NXLog in real-time. The $EventReceivedTime field is automatically added to each event record by NXLog; this field can be stored alongside the event’s own timestamp (normally $EventTime) if all fields are preserved when the event is stored/forwarded. Alternatively, this field can be used as the event timestamp as shown below. This would have the effect of influencing the timestamp used on most outputs, such as with the to_syslog_ietf() procedure.

Example 123. Using $EventReceivedTime as the Event Timestamp

This configuration accepts Syslog messages via UDP with the im_udp module. Events are parsed with the parse_syslog() procedure, which adds an EventTime field from the Syslog header timestamp. The $EventTime value, however, is replaced by the timestamp set by NXLog in the $EventReceivedTime field. Any later processing that uses the $EventTime field will operate on the updated timestamp. For example, if the to_syslog_ietf() procedure is used, the resulting IETF Syslog header will contain the $EventReceivedTime timestamp.

nxlog.conf


  1
2
3
4
5
6
7
8
9
10
11

  <Extension _syslog>
    Module  xm_syslog
</Extension>

<Input syslog>
    Module  im_udp
    <Exec>
        parse_syslog();
        $EventTime = $EventReceivedTime;
    </Exec>
</Input>

In some edge cases, a UTC timestamp that does not have the timezone specified is parsed as local time. This can happen if BSD Syslog timestamps are in UTC, or when reading a non-timezone-aware ID timestamp with im_odbc. In this case, it is necessary to either manuall re-parse (see Parsing Timestamps) or apply a corresponding reverse offset.

Example 124. Reversing an Incorrect Local-to-UTC Timezone Offset

This statement uses the parsedate() and strftime() functions to apply a reverse offset after an incorrect local-to-UTC timezone conversion. To reduce the likelihood of an incorrect offset during the daylight saving time (DST) transition, this should be done in the Input module instance which is collecting the events (see the warning above).


  1

  $EventTime = parsedate(strftime($EventTime, '%Y-%m-%d %H:%M:%SZ'));

For the general case of adjusting timestamps, the plus (+) and minus (-) operators can be used to adjust a timestamp by a specified number of seconds.

Example 125. Adjusting a Datetime Value by Seconds

This statement adds two hours to the $EventTime field.

Warning

This simple method may not be suitable for correction of a timezone that uses daylight saving time (DST). In that case the required offset may change based on whether DST is in effect.


  1

  $EventTime = $EventTime + (2 * 3600);

25.15.3. Formatting Timestamps

After a timestamp has been parsed to a datetime value, it will usually need to be converted back to a string at some point before being sent to the output. This can be done automatically by the output configuration.

Example 126. Using the Default Timestamp Formatting

Consider an event record with an $EventTime field (as a datetime value) and a $Message field. Note that the table below shows the $EventTime value as it is stored internally: as microseconds since the epoch.

Table 30. Sample Event Record
Field	Value
$EventTime	1493425133541851
$Message	EXT4-fs (dm-0): mounted filesystem with ordered data mode.

The following output module instance uses the to_json() procedure without specifying the timestamp format.

nxlog.conf


  1
2
3
4
5

  <Output out>
    Module  om_file
    File    'out.log'
    Exec    to_json();
</Output>

The output of the $EventTime field in this case will depend on the DateFormat directive. The default DateFormat is YYYY-MM-DD hh:mm:ss (local time).

Output Sample

{
  "EventTime": "2017-01-02 15:19:22",
  "Message": "EXT4-fs (dm-0): mounted filesystem with ordered data mode."
}

Note	A different timestamp may be used in some cases, depending on the procedure used to convert the field and the output module. The to_syslog_bsd() procedure, for example, will use the `$EventTime` value to generate a RFC 3164 format timestamp regardless of how the DateFormat directive is set.

Alternatively, the strftime() function can be used to explicitly convert a datetime value to a string with the required format.

Example 127. Using strftime() to Format Timestamps

Again, consider an event record with an $EventTime field (as a datetime value) and a $Message field. In this example, the strftime() function is used with a format string (see the strftime(3) manual) to convert $EventTime to a string in the local time zone. Then the to_json() procedure is used to set the $raw_event field.

nxlog.conf


  1
2
3
4
5
6
7
8

  <Output out>
    Module  om_file
    File    'out.log'
    <Exec>
        $EventTime = strftime($EventTime, '%Y-%m-%dT%H:%M:%S%z');
        to_json();
    </Exec>
</Output>

Output Sample

{
  "EventTime": "2017-04-29T02:18:53+0200",
  "Message": "EXT4-fs (dm-0): mounted filesystem with ordered data mode."
}

NXLog Enterprise Edition supports a few additional format strings for formats that the stock C strftime() does not offer, including formats with fractional seconds and in UTC time. See the Reference Manual strftime() documentation for the list.

Example 128. Using strftime() Special Formats in NXLog Enterprise Edition

The following statement will convert $EventTime to a timestamp format with fractional seconds and in UTC (regardless of the current time zone).


  1

  $EventTime = strftime($EventTime, 'YYYY-MM-DDThh:mm:ss.sUTC');

The resulting timestamp string in this case would be 2017-04-29T00:18:53.541851Z.