If one does not routinely monitor the disk space usage on a Neo4j server one may encounter a ‘No space left on device’ (for linux implementation) or a ‘Low Disk Space’ (Windows Implementations). Once you have encountered either, the following steps should be considered so as to free up enough space to start the database and allow recovery to complete. It should be noted that as you have consumed all the disk space, you cannot simply compress files to find more space, since that would require writing to the full file system.

  1. graph.db path
    DO NOT manually delete files in your database path, where the default location is $NEO4J_HOME/data/databases/graph.db. Although this path may contain the most data, do not manually remove files from this path, as manually removing files will more than likely corrupt the database and/or prevent future starts.
  2. Local backup copies
    Consider if you have written the results of neo4j-admin backup to the local file system. If so, and if that copy is not going to be necessary to restore service, can the files be moved/removed from the file system?
  3. Prior versions of Neo4j software
    If you have completed multiple upgrades of Neo4j in the past, it may be the case that you have left the prior version software on the file system. For example if you typically install software into /usr/software/ and have upgraded from Neo4j 3.2.1 to Neo4j 3.5.0, you may have a /usr/software/neo4j-enterprise-3.2.1 and /usr/software/neo4j-enterprise-3.5.0. If such an older environment exists and provided you are successfully running on the newer version, you may consider moving/removing the prior version, in this example /usr/software/neo4j-enterprise-3.2.1
  4. Log files
    During the normal course of Neo4j operation, diagnostic logs are written to $NEO4J_HOME/logs/ and specifically debug.log, ‘neo4j.log’, query.log (provided dbms.logs.query.enabled=true) and security.log (provided dbms.security.auth_enabled=true), and where the debug.log can be the largest of these files. Given these files are diagnostic logs you may consider moving/removing/truncating these log files.
  5. Plugins Your $NEO4J_HOME/plugins may contain custom plugins (JARs) for Neo4j. Check to see that your plugins are not writing to this path, or if they are, ensure they are properly managing their log files. Additionally, you might consider removing/moving the apoc* jar as it can be easily restored.
  6. Prune transaction log files through the Neo4j product
    If you have freed up a reasonable amount of space, prior to start you may want to configure the conf/neo4j.conf' parameter of `dbms.tx_log.rotation.retention_policy to a very small value (example dbms.tx_log.rotation.retention_policy=100M). In doing so after a sucessful start and then subsequent checkpoint (which defaults to every 900s), a transaction log pruning/rotation will occur and it should remove all but the last transaction log. One issue with this approach is that more than likely your next backup, if it is an incremental backup will revert to a full backup since the transaction logs between the last backup and the next backup are not without a gap. If this approach is taken, after sufficient disk space has been freed up the dbms.tx_log.rotation.retention_policy parameter should be reverted back to the value it was previously set to.
  7. Prepare for growth
    Even after freeing up sufficient space you will still need to prepare for the future in terms of making sure you have sufficient disk space for continued database growth. This is more an OS-related responsibility but should not be overlooked.
  8. Ask for guidance from Neo4j If you are still unable to free up space, ask for guidance from Neo4j.

After Neo4j has safely restarted it is strongly encouraged that you run a database consistency check. This can be performed by either running a backup bin/neo4j-admin backup …​. …​.. --check-consistency=true on a running instance or on a stopped Neo4j database by running bin/neo4j-admin check-consistency --database=graph.db

Details


Author:
Dana Canzano
Applicable versions:
3.2, 3.3, 3.4, 3.5
Keywords:
administrationdisk fullneo4j-3.2neo4j-3.3neo4j-3.4neo4j-3.5