Return to
Portfolio

3.4. Log Processing Modes

NXLog can process logs in three modes. Each mode has different characteristics, and you can use any combination of modes for your overall logging infrastructure.

3.4.1. Agent-Based Collection

With agent-based collection, NXLog runs as an agent on the system that is generating the log data. It collects the log data and sends it to another NXLog instance over the network.

Note
 We recommend agent-based log collection for most use cases. In particular, we recommend this mode if you need strong security and reliability or need to transform log data before it leaves the system on which it was generated.

Agent-based log collection offers several important advantages over agent-less collection.

  • Log data can be collected from more sources. For example, you can collect logs directly from files, instead of relying on a logging process to send log data across the network.

  • NXLog’s processing features are available. You can filter, normalize, and rewrite log data before sending it to a destination, whether a NXLog instance or a log aggregation system. This includes the ability to send structured log data, such as JSON and key-value pairs.

  • You have full control over the transfer of the log data. Messages can be sent using a variety of protocols, including over TLS/SSL encrypted connections for security. Log data can be sent in compressed batches and can be buffered if necessary.

  • Log collection in this mode is more reliable. NXLog includes delivery guarantees and flow control systems that make sure your log data gets to its destination. You can monitor the health of the NXLog agent to check that it is operating correctly.

Although agent-based collection has many compelling advantages, it is not well suited to some use cases.

  • Many network and embedded systems, such as routers and firewalls, do not support installing third-party software. In this case it would not be possible to install the NXLog agent.

  • Installing the NXLog agent on each system in a large-scale deployment may not be practical compared to reading from the existing logging daemon on each system.

3.4.2. Agent-Less Collection

With this mode of log collection, a server or device sends log data to an NXLog instance over the network, using its native protocols. NXLog collects and processes the information that it receives.

Note
 We recommend agent-less log collection in cases where agent-based log collection is not feasible, for example from legacy or embedded systems that do not support installing the NXLog agent.

Agent-less log collection has the following advantages.

  • It is not necessary to install an NXLog agent application on the target system to collect log data from it.

  • Generally, a device or system requires only minimal configuration to send log data over the network to an NXLog instance in its native format.

Agent-less log collection has some disadvantages that you should consider before deploying it.

  • Agent-less log collection may provide lower performance than agent-based collection. On Windows systems, the Windows Management Instrumentation process can consume more system resources than the NXLog agent.

  • Reliability is also a potential issue. Since most Syslog log forwarders use UDP to transfer log data, some data could be lost if the server restarts or becomes unreachable over the network. Unlike agent-based log collection, you often cannot monitor the health of the logging source.

  • The security of data transferred may be lower when using agent-less collection, since most Syslog sources transfer data over unencrypted UDP.

Agent-less collection is commonly used with the following protocols.

3.4.3. Offline Log Processing

While the other modes process log data in real-time, NXLog can also be used to perform batch log processing. This is provided by the nxlog-processor(8) tool, which is similar to the NXLog daemon and uses the same configuration file. However, it runs in the foreground and exits after all input log data has been processed.

Common input sources are files and databases. This tool is useful for log processing tasks such as:

  • loading a group of files into a database,

  • converting between different formats,

  • testing patterns,

  • doing offline event correlation, or

  • checking HMAC message integrity.

← Previous: Buffering and Flow Control | ↑ Up: System Architecture | ⌂ Home: NXLog User Guide | Next: Available Modules →