NXLog User Guide

109.15. Kafka (im_kafka)

This module implements an Apache Kafka consumer for collecting event records from a Kafka topic. See also the om_kafka module.

109.15.1. Configuration

The im_kafka module accepts the following directives in addition to the common module directives. The BrokerList and Topic directives are required.

BrokerList: This mandatory directive specifies the list of Kafka brokers to connect to for collecting logs. The list should include ports and be comma-delimited (for example, localhost:9092,192.168.88.35:19092).

Topic: This mandatory directive specifies the Kafka topic to collect records from.

CAFile: This specifies the path of the certificate authority (CA) certificate, which will be used to check the certificate of the remote brokers. CAFile is required if Protocol is set to ssl. To trust a self-signed certificate presented by the remote (which is not signed by a CA), provide that certificate instead.

CertFile: This specifies the path of the certificate file to be used for the SSL handshake.

CertKeyFile: This specifies the path of the certificate key file to be used for the SSL handshake.

KeyPass: With this directive, a password can be supplied for the certificate key file defined in CertKeyFile. This directive is not needed for passwordless private keys.

Option

This directive can be used to pass a custom configuration property to the Kafka library (librdkafka). For example, the group ID string can be set with Option group.id mygroup. This directive may be used more than once to specify multiple options. For a list of configuration properties, see the librdkafka CONFIGURATION.md file.

Warning

Passing librdkafka configuration properties via the Option directive should be done with care since these properties are used for the fine-tuning of the librdkafka performance and may result in various side effects.

Partition: This optional integer directive specifies the topic partition to read from. If this directive is not given, messages are collected from partition 0.

Protocol: This optional directive specifies the protocol to use for connecting to the Kafka brokers. Accepted values include plaintext (the default) and ssl. If Protocol is set to ssl, then the CAFile directive must also be provided.

109.15.2. Examples

Example 553. Using the im_kafka Module

This configuration collects events from a Kafka cluster using the brokers specified. Events are read from the first partition of the nxlog topic.

nxlog.conf


  1
2
3
4
5
6
7
8
9
10
11

  <Input in>
    Module      im_kafka
    BrokerList  localhost:9092,192.168.88.35:19092
    Topic       nxlog
    Partition   0
    Protocol    ssl
    CAFile      /root/ssl/ca-cert
    CertFile    /root/ssl/client_debian-8.pem
    CertKeyFile /root/ssl/client_debian-8.key
    KeyPass     thisisasecret
</Input>