NXLog User Guide

110.7. De-Duplicator (pm_norepeat)

This module can be used to filter out repeating messages. Like Syslog daemons, this module checks the previous message against the current. If they match, the current message is dropped. The module waits one second for duplicated messages to arrive. If duplicates are detected, the first message is forwarded, the rest are dropped, and a message containing "last message repeated n times" is sent instead.

Note	This module has been deprecated and will be removed in a future release. The functionality of this module can be implemented with Variables.

110.7.1. Configuration

The pm_norepeat module accepts the following directives in addition to the common module directives.

CheckFields: This optional directive takes a comma-separated list of field names which are used to compare log messages. Only the fields listed here are compared, the others are ignored. For example, the $EventTime field will be different in repeating messages, so this field should not be used in the comparison. If this directive is not specified, the default field to be checked is $Message.

110.7.2. Fields

The following fields are used by pm_norepeat.

$raw_event (type: string): A string containing the last message repeated n times message.

$EventTime (type: datetime): The time of the last event or the current time if EventTime was not present in the last event.

$Message (type: string): The same value as $raw_event.

$ProcessID (type: integer): The process ID of the NXLog process.

$Severity (type: string): The severity name: INFO.

$SeverityValue (type: integer): The INFO severity level value: 2.

$SourceName (type: string): Set to nxlog.

110.7.3. Examples

Example 590. Filtering Out Duplicated Messages

This configuration reads log messages from the socket. The $Hostname, $SourceName, and $Message fields are used to detect duplicates. Then the messages are written to file.

nxlog.conf


  1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

  <Input uds>
    Module      im_uds
    UDS         /dev/log
</Input>

<Processor norepeat>
    Module      pm_norepeat
    CheckFields Hostname, SourceName, Message
</Processor>

<Output file>
    Module      om_file
    File        "/var/log/messages"
</Output>


<Route uds_to_file>
    Path        uds => norepeat => file
</Route>