3.8. Transaction logs

This section explains the retention and rotation policies for the Neo4j transaction logs, and how to configure them.

The transaction logs record all operations in the database. They are the "source of truth" in scenarios where the database needs to be recovered. Transaction logs are used to provide for incremental backups, as well as for cluster operations. For any given configuration, at least the latest non-empty transaction log will be kept.

Log location
By default, transaction logs are kept in the same folder as the store they belong to. The directory where transaction logs are stored is configured by dbms.directories.tx_log. For maximum performance, it is recommended to configure transaction logs to be stored on a dedicated device.
Log rotation
Log rotation is configured using the parameter dbms.tx_log.rotation.size. By default, log switches happen when log sizes surpass 250 MB.
Log retention

There are several different means of controlling the amount of transaction logs that are kept, using the parameter dbms.tx_log.rotation.retention_policy. This parameter can be configured in two different ways:

  • dbms.tx_log.rotation.retention_policy=<true/false>

    If this parameter is set to true, transaction logs will be kept indefinitely. This option is not recommended due to the effectively unbounded storage usage. Old transaction logs cannot be safely archived or removed by external jobs, since safe log pruning requires knowledge about the most recent successful checkpoint.

    If this parameter is set to false, only the most recent non-empty log will be kept. This option is not recommended in production Enterprise Edition environments, as incremental backups rely on the presence of the transaction logs since the last backup.

  • dbms.tx_log.rotation.retention_policy=<amount> <type>
Log pruning

Transaction log pruning refers to the safe and automatic removal of old, unnecessary transaction log files. The transaction log can be pruned when one or more files fall outside of the configured retention policy. Two things are necessary for a file to be removed:

  • The file must have been rotated.
  • At least one checkpoint must have happened in a more recent log file.

Observing that you have more transaction log files than you expected is likely due to checkpoints either not happening frequently enough, or taking too long. This is a temporary condition and the gap between expected and observed number of log files will be closed on the next successful checkpoint. The interval between checkpoints can be configured using dbms.checkpoint.interval.time and dbms.checkpoint.interval.tx. If your goal is to have the least amount of transaction log data, it can also help to speed up the checkpoint process itself. The configuration parameter dbms.checkpoint.iops.limit controls the number of IOs per second the checkpoint process is allowed to use. Setting the value of this parameter to -1 allows unlimited IOPS, which can speed up checkpointing. Note that disabling the IOPS limit can cause transaction processing to slow down a bit.

Table 3.6. Types that can be used to control log retention
Type Description Example

files

Number of most recent logical log files to keep

"10 files"

size

Max disk size to allow log files to occupy

"300M size" or "1G size"

txs

Number of transactions to keep

"250k txs" or "5M txs"

hours

Keep logs which contains any transaction committed within N hours from current time

"10 hours"

days

Keep logs which contains any transaction committed within N days from current time

"50 days"

Example 3.8. Configure log retention policy

This example shows some different ways to configure the log retention policy.

  • Keep transaction logs indefinitely:

    dbms.tx_log.rotation.retention_policy=true
  • Keep only the most recent non-empty log:

    dbms.tx_log.rotation.retention_policy=false
  • Keep logical logs which contain any transaction committed within 30 days:

    dbms.tx_log.rotation.retention_policy=30 days
  • Keep logical logs which contain any of the most recent 500 000 transactions:

    dbms.tx_log.rotation.retention_policy=500k txs