Return to
Portfolio

111.4. Elasticsearch (om_elasticsearch)

This module allows logs to be stored in an Elasticsearch server. It will connect to the URL specified in the configuration in either plain HTTP or HTTPS mode. The advantage of this module over om_http is the support for bulk data operations and dynamic indexing. Event data is sent in batches, greatly reducing the latency caused by the HTTP responses and improving Elasticsearch server performance.

Note
 This module requires the xm_json extension module to be loaded in order to convert the payload to JSON. If the $raw_event field does not start with a left curly bracket ({), the module will automatically convert the data to JSON.

111.4.1. Using Elasticsearch With NXLog Enterprise Edition 3.x

Some setup is required when using Elasticsearch with NXLog Enterprise Edition 3.x. Consider the following points. None of this is required with NXLog Enterprise Edition 4.1 and later.

  • By default, Elasticsearch will not automatically detect the date format used by NXLog Enterprise Edition 3.x. As a result, NXLog datetime values, such as $EventTime, will be mapped as strings rather than dates. To fix this, add an Elasticsearch template for indices matching the specified pattern (nxlog*). Extend the dynamic_date_formats setting to include additional date formats. For compatibility with indices created with Elasticsearch 5.x or older, use _default_ instead of _doc (but _default_ will not be supported by Elasticsearch 7.0.0).

    $ curl -X PUT localhost:9200/_template/nxlog?pretty \
       -H 'Content-Type: application/json' -d '
  {
    "index_patterns" : ["nxlog*"],
    "mappings" : {
      "_doc": {
        "dynamic_date_formats": [
          "strict_date_optional_time",
          "YYYY-MM-dd HH:mm:ss.SSSSSSZ",
          "YYYY-MM-dd HH:mm:ss"
        ]
      }
    }
  }'

  • The IndexType directive should be set to _doc (the default in NXLog Enterprise Edition 3.x is logs). However, for compatibility with indices created with Elasticsearch 5.x or older, set IndexType as required for the configured mapping types. See the IndexType directive below for more information.

111.4.2. Configuration

The om_elasticsearch module accepts the following directives in addition to the common module directives. The URL directive is required.

URL

This mandatory directive specifies the URL where the module should POST the event data. The URL also indicates whether to operate in plain HTTP or HTTPS mode. If the port number is not explicitly indicated, it defaults to port 80 for HTTP and port 443 for HTTPS. The URL should point to the _bulk endpoint, or Elasticsearch will return 400 Bad Request.

AddHeader

This optional directive specifes an additional header to be added to each HTTP request.

FlushInterval

The module will send a bulk index command to the defined endpoint after this amount of time in seconds, unless FlushLimit is reached first. This defaults to 5 seconds.

FlushLimit

When the number of events in the output buffer reaches the value specified by this directive, the module will send a bulk index command to the endpoint defined in URL. This defaults to 500 events. The FlushInterval directive may trigger sending the bulk index request before this limit is reached if the log volume is low to ensure that data is promptly sent to the indexer.

Index

This directive specifies the index to insert the event data into. It must be a string type expression. If the expression in the Index directive is not a constant string (it contains functions, field names, or operators), it will be evaluated for each event to be inserted. The default is nxlog. Typically, an expression with strftime() is used to generate an index name based on the event’s time or the current time (for example, strftime(now(), "nxlog-%Y%m%d").

IndexType

This directive specifies the index type to use in the bulk index command. It must be a string type expression. If the expression in the IndexType directive is not a constant string (it contains functions, field names, or operators), it will be evaluated for each event to be inserted. The default is _doc. Note that index mapping types have been deprecated and will be removed in Elasticsearch 7.0.0 (see Removal of mapping types in the Elasticsearch Reference). IndexType should only be used if required for indices created with Elasticsearch 5.x or older.

ID

This directive allows to specify a custom _id field for Elasticsearch documents. If the directive is not defined, Elasticsearch uses a GUID for the _id field. Setting custom _id fields can be useful for correlating Elasticsearch documents in the future and can help to prevent storing duplicate events in the Elasticsearch storage. The directive’s argument must be a string type expression. If the expression in the ID directive is not a constant string (it contains functions, field names, or operators), it will be evaluated for each event to be submitted. You can use a concatenation of event fields and the event timestamp to uniquely and informatively identify events in the Elasticsearch storage.

HTTPSAllowUntrusted

This boolean directive specifies that the remote connection should be allowed without certificate verification. If set to TRUE, the connection will be allowed even if the remote HTTPS server presents an unknown or self-signed certificate. The default value is FALSE: the remote HTTPS server must present a trusted certificate.

HTTPSCADir

This specifies the path to a directory containing certificate authority (CA) certificates, which will be used to check the certificate of the remote HTTPS server. The certificate filenames in this directory must be in the OpenSSL hashed format. A remote’s self-signed certificate (which is not signed by a CA) can also be trusted by including a copy of the certificate in this directory.

HTTPSCAFile

This specifies the path of the certificate authority (CA) certificate, which will be used to check the certificate of the remote HTTPS server. To trust a self-signed certificate presented by the remote (which is not signed by a CA), provide that certificate instead.

HTTPSCertFile

This specifies the path of the certificate file to be used for the HTTPS handshake.

HTTPSCertKeyFile

This specifies the path of the certificate key file to be used for the HTTPS handshake.

HTTPSCRLDir

This specifies the path to a directory containing certificate revocation lists (CRLs), which will be consulted when checking the certificate of the remote HTTPS server. The certificate filenames in this directory must be in the OpenSSL hashed format.

HTTPSCRLFile

This specifies the path of the certificate revocation list (CRL) which will be consulted when checking the certificate of the remote HTTPS server.

HTTPSKeyPass

With this directive, a password can be supplied for the certificate key file defined in HTTPSCertKeyFile. This directive is not needed for passwordless private keys.

SNI

This optional directive specifies the host name used for Server Name Indication (SNI) in HTTPS mode.

SSLCipher

This optional directive can be used to set the permitted SSL cipher list, overriding the default. Use the format described in the ciphers(1ssl) man page.

SSLProtocol

This directive can be used to set the allowed SSL/TLS protocol(s). It takes a comma-separated list of values which can be any of the following: SSLv2, SSLv3, TLSv1, TLSv1.1, and TLSv1.2. By default, the TLSv1, TLSv1.2, and TLSv1.2 protocols are allowed. Note that the OpenSSL library shipped by Linux distributions may not support SSLv2 and SSLv3, in which case these will not work even if enabled with SSLProtocol.

ProxyAddress

This optional directive is used to specify the IP address of the proxy server in case the module should connect to the Elasticsearch server through a proxy.

Note
 The om_elasticsearch module supports HTTP proxying only. SOCKS4/SOCKS5 proxying is not supported.
ProxyPort

This optional directive is used to specify the port number required to connect to the proxy server.

SSLCompression

This boolean directive allows you to enable data compression when sending data over the network. The compression mechanism is based on the zlib compression library. If the directive is not specified, it defaults to FALSE (the compression is disabled).

Note
 Some Linux packages (for example, Debian) use the OpenSSL library provided by the OS and may not support the zlib compression mechanism. The module will emit a warning on startup if the compression support is missing. The generic deb/rpm packages are bundled with a zlib-enabled libssl library.

111.4.3. Examples

Example 598. Sending Logs to an Elasticsearch Server

This configuration reads log messages from file and forwards them to the Elasticsearch server on localhost.

nxlog.conf [Download file]

  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21


  
<Extension json>
    Module          xm_json
</Extension>

<Input file>
    Module          im_file
    File            '/var/log/myapp*.log'
    # Parse log here if needed
    # $EventTime should be set here
</Input>

<Output elasticsearch>
    Module          om_elasticsearch
    URL             http://localhost:9200/_bulk
    FlushInterval   2
    FlushLimit      100
    # Create an index daily
    Index           strftime($EventTime, "nxlog-%Y%m%d")
    # Or use the following if $EventTime is not set
    # Index         strftime(now(), "nxlog-%Y%m%d")
</Output>

← Previous: DBI (om_dbi) | ↑ Up: Output Modules | ⌂ Home: NXLog User Guide | Next: EventDB (om_eventdb) →