25.10. Log Rotation and Retention
NXLog can implement many kinds of log rotation and retention policies in order to prevent over-use of disk space and to organize past logs. These policies can be applied based on file size, time intervals, or even event attributes (such as severity). Log files can be rotated out to custom filenames and then compressed and/or deleted after a specified time period. The configuration is very flexible and custom policies can be easily implemented.
NXLog supports three main approaches to file rotation. In each case, policies should usually be implemented using a Schedule block.
-
Most policies are implemented under the scope of an om_file module instance, where output files are being written.
-
The im_file module can be configured to rotate log files after they have been fully read.
-
Any log file on the system can be rotated under the scope of an xm_fileop module or any other module. This includes the internal log file (specified by the LogFile directive).
Log files written by an om_file module often need to be rotated regularly. This example uses the om_file file_name() function and xm_fileop file_cycle() procedure to rotate the output file daily, keeping a total of 7 old log files.
![[Download file]](images/icons/download_icon.png "Download file"){kind=icon}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
<Extension _fileop>
Module xm_fileop
</Extension>
<Output out>
Module om_file
File '/var/log/out.log'
<Schedule>
When @daily
<Exec>
file_cycle(file_name(), 7);
reopen();
</Exec>
</Schedule>
</Output>
NXLog will write its own logs to a file specified by the LogFile directive. It is good practice to set up rotation of this file. This configuration uses the xm_fileop file_size() function. The file_cycle() procedure rotates the file if it is larger than 5 MB. The file is also rotated weekly. No more than 8 past log files are retained.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
define LOGFILE /opt/nxlog/var/log/nxlog/nxlog.log
LogFile %LOGFILE%
<Extension _fileop>
Module xm_fileop
# Check the log file size every hour and rotate if larger than 5 MB
<Schedule>
Every 1 hour
<Exec>
if (file_exists('%LOGFILE') and file_size('%LOGFILE%') >= 5M)
file_cycle('%LOGFILE%', 8);
</Exec>
</Schedule>
# Rotate log file every week on Sunday at midnight
<Schedule>
When @weekly
Exec if file_exists('%LOGFILE') file_cycle('%LOGFILE%', 8);
</Schedule>
</Extension>
There are many other ways that rotation and retention can be implemented. See the following sections for more details and examples.
25.10.1. Rotation Policies and Intervals
-
The om_file reopen() procedure will cause NXLog to reopen the output file specified by the File directive.
-
The rotate_to() procedure can be used to choose a name to rotate the current file to. This procedure will reopen the output file automatically, so there is no need to use the the reopen() procedure.
-
The file_cycle() procedure will move the selected file to "file.1". If "file.1" already exists, it will be moved to "file.2", and so on. If an integer is used as a second argument, it specifies the maximum number of previous files to keep.
WarningIf file_cycle() is used on a file that NXLog currently has open under the scope of an om_file module instance, the reopen() procedure must be used to continue logging to the file specified by the File directive. Otherwise, events will continue to be logged to the rotated file ("file.1", for example). (This is not necessary if the rotated file is the LogFile.)
25.10.1.1. Rotating by File Size
A log file can be rotated according to a pre-defined file size. This policy can be configured with the om_file file_size() function or the xm_fileop file_size() function.
This example uses the file_size() function to detect if a file has grown beyond a specified size. If it has, the file_cycle() procedure is used to rotate it. The file size is checked hourly with the When directive.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
<Extension _fileop>
Module xm_fileop
</Extension>
<Output out>
Module om_file
File '/var/log/out.log'
<Schedule>
When @hourly
<Exec>
if file_size(file_name()) >= 1M
{
file_cycle(file_name());
reopen();
}
</Exec>
</Schedule>
</Output>
25.10.1.2. Using Time-Based Intervals
For time interval based rotation policies NXLog provides two directives for use in Schedule blocks.
This example shows the use of the Every and
When directives. The output file is rotated
daily using the rotate_to() function. The name is
generated in the YYYY-MM-DD format according to the current server time.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
<Output out>
Module om_file
File '/var/log/out.log'
<Schedule>
# This can likewise be used for `@weekly` or `@monthly` time periods.
When @daily
# The following crontab-style is the same as `@daily` above.
# When "0 0 * * *"
# The `Every` directive could also be used in this case.
# Every 24 hour
Exec rotate_to(file_name() + strftime(now(), '_%Y-%m-%d'));
</Schedule>
</Output>
In this example, logs for each year and month are stored in separated sub-directories as shown below. The log file is rotated daily.
.../logs/YEAR/MONTH/YYYY-MM-DD.log
This is accomplished with the xm_fileop dir_make() procedure, the core strftime() function, and the om_file rotate_to() procedure.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
<Extension _fileop>
Module xm_fileop
</Extension>
<Output out>
define OUT_DIR /srv/logs
Module om_file
File '%OUT_DIR%/out.log'
<Schedule>
When @daily
<Exec>
# Create year/month directories if necessary
dir_make('%OUT_DIR%/' + strftime(now(), '%Y/%m'));
# Rotate current file into the correct directory
rotate_to('%OUT_DIR%/' + strftime(now(), '%Y/%m/%Y-%m-%d.log'));
</Exec>
</Schedule>
</Output>
25.10.1.3. Using Dynamic Filenames
As an alternative to traditional file rotation, output filenames can be set dynamically, based on each log event individually. This is possible because the om_file File directive supports expressions.
|
Note
|
Because dynamic filenames result in events being written to multiple
files with semi-arbitrary names, they are not suitable for scenarios
where a server or application expects events to be written to a
particular foo.log. In this case normal rotation should be used
instead.
|
Often one of now(), $EventReceivedTime, and $EventTime
are used for dynamic filenames. Consider the following points.
-
The
now()function uses the current server time, not when the event was created or when it was received by NXLog. If logs are delayed, they will be stored according to the time at which the NXLog output module instance processes them. This will not work with nxlog-processor(8) (see Offline Log Processing). -
The
$EventReceivedTimefield timestamp is set by the input module instance when an event is received by NXLog. This will usually be practically the same as usingnow(), except in cases where there are processing delays in the NXLog route (such as when using buffering). This can be used with nxlog-processor(8) if the$EventReceivedTimefield was previously set in the logs. -
The
$EventTimefield is set from a timestamp in the event, so will result in correct value even if the event was delayed before reaching NXLog. Note that some parsing may be required before this field is available (for example, the parse_syslog() procedure sets the xm_syslog EventTime field). Note also that an incorrect timestamp in an event record can cause the field to be unset or filled incorrectly resulting in data written into the wrong file.
This example accepts Syslog formatted messages via UDP. Each message is parsed by the parse_syslog() procedure. The EventTime field is set from the timestamp in the syslog header. This field is then used by the expression in the File directive to generate an output filename for the event.
Even if messages received from clients over the network are out of order or delayed, they will still be placed in the appropriate output files according to the timestamps.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
<Extension _syslog>
Module xm_syslog
</Extension>
<Input in>
Module im_udp
Port 514
Host 0.0.0.0
Exec parse_syslog();
</Input>
<Output out>
Module om_file
File '/var/log/nxlog/out_' + strftime($EventTime, '%Y-%m-%d')
Exec to_syslog_ietf();
</Output>
Dynamic filenames can be based on other fields also.
In this example, events are grouped by their source hostname.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
<Extension _syslog>
Module xm_syslog
</Extension>
<Input in>
Module im_udp
Host 0.0.0.0
Port 514
Exec parse_syslog();
</Input>
<Output out>
Module om_file
File '/tmp/logs_by_host/' + $Hostname
</Output>
25.10.1.4. Rotating Input Files
An im_file module instance can be configured to manipulate files after they are fully processed. The im_file OnEOF block can be used for this purpose.
|
Warning
|
When using OnEOF for rotation, the rotated files must be named (or placed in a directory) such that they will not be detected as new files and re-read by the module instance. |
|
Note
|
If a logging service keeps a log file open for writing, the xm_exec exec() procedure should be used to restart the service or otherwise instruct it to re-open the log file. |
In this example, files matching /var/log/app/*.log are read with an
im_file module instance. When each file has been fully read, it is
rotated. The GraceTimeout directive will
prevent NXLog from rotating the file until after there have been no
events for 10 seconds.
The input files are rotated by adding a timestamp suffix to the filename. For
example, an input file named /var/log/app/errors.log would be rotated to
/var/log/app/errors.log_20180101T130100. The new name does not match the
wildcard specified by the File directive, so the file
is not re-read.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
<Extension _fileop>
Module xm_fileop
</Extension>
<Input app_logs_rotated>
Module im_file
File '/var/log/app/*.log'
<OnEOF>
<Exec>
file_rename(file_name(),
file_name() + strftime(now(), '_%Y%m%dT%H%M%S'));
</Exec>
GraceTimeout 10
</OnEOF>
</Input>
25.10.2. Retention Policies
NXLog can be configured to keep old log files according to a particular retention policy. Functions and procedures for retention are provided by the xm_fileop module. Additional actions, such as compressing old log files, can be implemented with the xm_exec extension module.
25.10.2.1. Using Simple File Cycling
The file_cycle() procedure provides simple numbered rotation and, optionally, retention.
This example demonstrates the use of the xm_fileop file_cycle() procedure for keeping a total of 12 log files, one for each month. Log files older than 1 year will be automatically deleted.
This policy creates following log file structure: /var/log/foo.log for the
current month,/var/log/foo.log.1 for the previous month, and so on up to the
maximum of 12 files.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
<Extension _fileop>
Module xm_fileop
</Extension>
<Output out>
Module om_file
File '/var/logs/foo.log'
<Schedule>
When @monthly
<Exec>
file_cycle(file_name(), 12);
reopen();
</Exec>
</Schedule>
</Output>
Different policies for different events can be implemented in combination with dynamic filenames.
This example uses the $Severity field (such as
$Severity set by
parse_syslog()) to filter events to separate
files. Then different retention policies are applied according to
severity. Here, one week of debug logs, 2 weeks of informational logs, and 4
weeks of higher severity logs are retained.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
<Extension _syslog>
Module xm_syslog
</Extension>
<Extension _fileop>
Module xm_fileop
</Extension>
<Input logs_in>
Module im_file
File "/var/log/messages"
Exec parse_syslog();
</Input>
<Output logs_out>
define OUT_DIR /opt/nxlog/var/log
Module om_file
File '%OUT_DIR%/' + $Severity + '.log'
<Schedule>
When @daily
<Exec>
file_cycle('%OUT_DIR%/DEBUG.log', 7);
file_cycle('%OUT_DIR%/INFO.log', 14);
file_cycle('%OUT_DIR%/WARNING.log', 28);
file_cycle('%OUT_DIR%/ERROR.log', 28);
file_cycle('%OUT_DIR%/CRITICAL.log', 28);
reopen();
</Exec>
</Schedule>
</Output>
25.10.2.2. Compressing Old Log Files
The xm_exec module can be used to compress old log files to reduce disk usage.
In this example, the file size of the output file is checked hourly with the om_file file_size() function. If the size is over the limit, then:
-
a
newfilemodule variable is set to the name the current file will be rotated to, -
the om_file rotate_to() procedure renames the current output file to the name set in
newfile, -
the module re-opens the original file specified by the File directive and continue logging, and
-
the xm_exec exec_async() procedure call bzip2 on the rotated-out file (without waiting for the command to complete).
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
<Extension _exec>
Module xm_exec
</Extension>
<Extension _fileop>
Module xm_fileop
</Extension>
<Output out>
Module om_file
File '/opt/nxlog/var/log/app.log'
<Schedule>
When @hourly
<Exec>
if out->file_size() > 15M
{
set_var('newfile', file_name() + strftime(now(), '_%Y%m%d%H%M%S'));
rotate_to(get_var('newfile'));
exec_async('/bin/bzip2', get_var('newfile'));
}
</Exec>
</Schedule>
</Output>
25.10.2.3. Deleting Old Log Files
For retention policies where file deletion is not handled automatically by the xm_fileop file_cycle() procedure, the xm_fileop file_remove() can be used to delete old files. This procedure can also delete files based on their creation time.
This example uses file_remove() to remove any files that were created more than 30 days ago.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
<Input in>
Module im_null
</Input>
<Output logs_out>
Module om_file
File '/var/log/'+ strftime(now(),'%Y%m%d') + '.log'
<Schedule>
When @daily
# Delete logs older than 30 days (24x60x60x30)
Exec file_remove('/var/log/*.log', now() - 2592000);
</Schedule>
</Output>