NXLog User Guide

117. Correlation

Event correlation is an important concept in log analysis. Each log message contains information about an event which occurred at some point. There are cases when the occurrence (or absence) of one or more events must be treated specially.

A trivial example for this is to detect 3 failed login attempts into a system. When this happens, the user will be likely locked out. If the log analysis system is capable to detect such situation, a lot of things can be automated. You will not need to wait for the user to come asking for a new password.

Event correlation in NXLog Manager is architected similarly to the Pattern system. It is performed in real-time by the NXLog agents. This way it is possible to do local event correlation on the client side (at the agent). This will not only reduce load on the central server but the system can send alerts over another channel (e.g. SMS) even if the network is down and the log messages would not reach the central log server.

For more information about the correlation capabilities of NXLog Manager, please consult the NXLog Reference Manual and see the documentation about the pm_evcorr module.

117.1. Correlation Rulesets

Correlation rules are grouped into rulesets. Because there can be different correlation rules depending on the agent or log source, rulesets ease the use of correlation rules. To view the list of correlation rulesets, click the CORRELATION menu item. The list of existing correlation rulesets will appear such as in the following screenshot:

To create a new ruleset, click the Add button.

117.2. Correlation Rules

Correlation rules check whether the conditions specified in the rule are satisfied and execute an action. Correlation rules are evaluated linearly.

Clicking the name of a correlation ruleset will show a list of correlation rules within the ruleset:

Note	The order of the rules within the ruleset matter because they are evaluated by NXLog’s pm_evcorr module in the order they appear. To change the order of the rules, use the Up and Down buttons.

117.2.1. Creating Correlation Rules

To create a new correlation rule, click the Add button on the bottom of the list. The following dialog window will appear:

Each correlation rule has a mandatory Name, Type and Action parameter and one ore more type specific parameters where the conditions can be specified. The following correlation rule types are available:

Simple
Suppressed
Pair
Absence
Thresholded

Please consult the NXLog Reference Manual and see the documentation about these rule types provided by the pm_evcorr module. There are two modes available to specify a condition.

Matched pattern is

This will generate a simple test to check whether the specified pattern matched. The generated NXLog config will contain a similar snippet:

if $PatternID == 42 {\
   ACTION            \
}

Note	For this to work, the pm_pattern module must be configured and must be in front of the pm_evcorr module in the route. The pm_pattern module is responsible for setting the PatternID field.

Expert

This field expects a statement (a boolean condition) which evaluates to TRUE or FALSE. The above expressed in Expert form would look like the following:

$PatternID == 42

Using the language constructs provided by NXLog, it is possible to specify more complex conditions here, for example:

($EventTime > now() - 10000) and ($PatternID == 42 or $PatternID == 142)

Example 620. Correlation rule for ssh bruteforce attack detection

In this example, a correlation rule is created which will detect SSH brute force attempts. The rule defines this attempt as 5 login failures within a 20 second interval. In this example, only an internal warning message is generated, but it is possible to trigger any other action such as executing an external script to block the IP or send an email alert.

This correlation rule depends on a pattern which matches the SSH authentication failure events. See the Creating patterns section on how to do this. Once the pattern is available in the database, the correlation rule should be configured as shown on the following screenshot:

117.3. Exporting and Importing Correlation Rules

NXLog Manager can export and import correlation rules and rulesets in an XML format. To export a correlation rule or a ruleset, select its checkbox in the list and click Export. You can import a correlation rule XML file by clicking on the Import button under the ruleset list.

Note	Unlike patterns, correlation rules used by NXLog are not in XML. Correlation rules exported from NXLog Manager cannot by used by NXLog because the NXLog agent uses Apache-style configuration for the rules and this is part of (or included in) the pm_evcorr module configuration block in `nxlog.conf`.

117.4. Using Correlation Rules

Similar to patterns, correlation rules are also used and executed by the NXLog engine. Unlike other log analysis solutions which utilize a single pattern correlation engine in the central log server, the architecture of NXLog Manager allows correlation rules to be evaluated on the agents as well.

To use the correlation rules in an NXLog agent, add a pm_evcorr processor module and select the appropriate correlation ruleset:

It is recommended to use only one ruleset per agent. The correlation rules are pushed to the NXLog agent by clicking Update config and they take effect after a restart. See the Agents chapter for more information about agent configuration details.

Note	In many cases correlation rules depend on patterns (and the PatternID field). For this reason a pm_pattern module should be in the processor chain before the pm_evcorr module.