Kubernetes maintenance operations

The section describes some maintenance operations when running Neo4j in a Kubernetes cluster.

1. Online Maintenance

Online maintenance does not require stopping the neo4j process. It is performed using the command kubectl exec.

To directly run tasks:

kubectl exec <release-name>-0 -- neo4j-admin store-info --all /var/lib/neo4j/data/databases --expand-commands

All neo4j-admin commands need the --expand-commands flag to run in the Neo4j container. This is because the Neo4j Helm chart defines the Neo4j configuration using command expansion to dynamically resolve some configuration parameters at runtime.

To run a series of commands, use an interactive shell:

kubectl exec -it <release-name>-0 -- bash

Processes executed using kubectl exec count towards the Neo4j container’s memory allocation. Therefore, running tasks that use a significant amount of memory or running Neo4j in an extremely memory-constrained configuration could cause the Neo4j container to be terminated by the underlying Operating System.

2. Offline Maintenance

You use the Neo4j offline maintenance mode to perform maintenance tasks that require Neo4j to be offline. In this mode, the neo4j process is not running. However, the Neo4j Pod does run, but it never reaches the status READY.

2.1. Put the Neo4j instance in offline mode

  1. To put the Neo4j instance in offline maintenance mode, you set the offlineMaintenanceModeEnabled: true and upgrade the helm release.

    • You can do that by using the values.yaml file:

      1. Open your values.yaml file and add offlineMaintenanceModeEnabled: true to the neo4j object:

        neo4j:
 offlineMaintenanceModeEnabled: true

      2. Run helm upgrade to apply the changes:

        helm upgrade <release-name> neo4j/neo4j-standalone -f values.yaml

    • Alternatively, you can set neo4j.offlineMaintenanceModeEnabled to true as part of the helm upgrade command:

      helm upgrade <release-name> neo4j/neo4j-standalone --version={neo4j-version-exact} --set neo4j.offlineMaintenanceModeEnabled=true

  2. Poll kubectl get pods until the pod has restarted (STATUS=Running).

    kubectl get pod <release-name>-0

  3. Connect to the pod with an interactive shell:

    kubectl exec -it "<release-name>-0" -- bash

  4. View running java processes:

    jps
    19 Jps

    The result shows no running java process other than jps itself.

2.2. Run task in offline mode

Offline maintenance tasks are performed using the command kubectl exec.

  • To directly run tasks:

    kubectl exec <release-name>-0 -- neo4j-admin store-info --all /var/lib/neo4j/data/databases --expand-commands

  • To run a series of commands, use an interactive shell:

    kubectl exec -it <release-name>-0 -- bash

  • For long running commands, use a shell and run tasks using nohup so they continue if the kubectl exec connection is lost:

    kubectl exec -it <release-name>-0 -- bash
  $ nohup neo4j-admin check-consistency --database=neo4j --expand-commands &>job.out </dev/null &
  $ tail -f job.out

2.3. Put the Neo4j instance in online mode

When you finish with the maintenance tasks, return the Neo4j instance to a normal operation:

  • You can do that by using the values.yaml file:

    1. Open your values.yaml file and add offlineMaintenanceModeEnabled: false to the neo4j object:

      neo4j:
 offlineMaintenanceModeEnabled: false

    2. Run helm upgrade to apply the changes:

      helm upgrade <release-name> neo4j/neo4j-standalone -f values.yaml

  • Alternatively, you can run helm upgrade with the flag set to false:

    helm upgrade <release-name> neo4j/neo4j-standalone --version={neo4j-version-exact} --set neo4j.offlineMaintenanceModeEnabled=false

3. Reset the neo4j user password

You reset the neo4j user password by disabling authentication and then re-enabling it.

  1. In the values.yaml file, set dbms.security.auth_enabled: to false to disable the authentication:

    All Neo4j config values must be YAML strings, not YAML booleans. Therefore, make sure you put quotes around values, such as "true" or "false", so that they are handled correctly by Kubernetes.

    		 
    config:
 dbms.security.auth_enabled: "false"

  2. Run the following command to apply the changes:

    helm upgrade <release-name> neo4j/neo4j-standalone -f values.yaml

    Authentication is now disabled.

  3. Connect with cypher-shell and set the desired password:

    ALTER USER neo4j SET PASSWORD '<new-password>'

  4. Update the Neo4j configuration to enable authentication:

    config:
 dbms.security.auth_enabled: "true"

  5. Run the following command to apply the update and re-enable authentication:

    helm upgrade <release-name> neo4j/neo4j-standalone -f values.yaml

    Authentication is now enabled, and the Neo4j user password has been reset to the desired password.

4. Dump and load databases (offline)

You can use the neo4j-admin dump command to make a full backup (an archive) of an offline database(s) and neo4j-admin load to load it back into a Neo4j deployment. These operations are performed in offline maintenance mode.

4.1. Dump the neo4j and system databases

  1. Put the Neo4j instance in offline mode.

  2. Dump neo4j and system databases:

    neo4j-admin dump --expand-commands --database=system --to /backups/system.dump && neo4j-admin dump --expand-commands --database=neo4j --to /backups/neo4j.dump

  3. Put the Neo4j instance in online mode.

  4. Verify that Neo4j is working by refreshing Neo4j Browser.

For information about the command syntax, options, and usage, see Back up an offline database.

4.2. Load the neo4j and system databases

  1. Put the Neo4j instance in offline mode.

  2. Run neo4j-admin load commands:

    neo4j-admin load --expand-commands --database=system --from /backups/system.dump && neo4j-admin load --expand-commands --database=neo4j --from /backups/neo4j.dump

    For information about the command syntax, options, and usage, see Restore a database dump.

  3. Put the Neo4j instance in online mode.

  4. Verify that Neo4j is working by refreshing Neo4j Browser.

5. Back up and restore a Neo4j database (online)

You can use the neo4j-admin backup command to make a full or incremental backup of an online database(s) and neo4j-admin restore to restore it in a live Neo4j DBMS. These operations are performed in online maintenance mode.

5.1. Back up the neo4j database

To back up the neo4j database, run the following command:

kubectl exec <release-name>-0 -- neo4j-admin backup --from=localhost:6362 --database=neo4j --backup-dir=/backups --expand-commands

You may also want to use the option --include-metadata=all. It creates a cypher script, which you can later use to restore the database’s users, roles, and privileges.

For information about the command syntax, options, memory configuration, and usage, see Back up an online database.

Note that this operation consumes resources (CPU, RAM) in the Neo4j container (competing with Neo4j itself). If your resources are constrained, you can run the backup in a sidecar container in the same pod or from a remote K8s Pod/Job.

5.2. Restore neo4j database

To restore the neo4j database, you need to execute neo4j-admin restore command and create the database in the system db.

For information about the command syntax, options, and usage, see Restore a database backup.

  1. Connect to the Neo4j standalone instance:

    kubectl exec -it <release-name>-0 -- bash

  2. Connect to your system database using cypher-shell:

    cypher-shell -u neo4j -p <password> -d system

  3. Drop the neo4j database and exit:

    DROP DATABASE neo4j;
    :exit;

  4. Run the neo4j-admin restore command:

    neo4j-admin restore --database=neo4j --from=/backups/neo4j --expand-commands

  5. Connect to your system database using cypher-shell:

    cypher-shell -u neo4j -p <password> -d system

  6. Create neo4j database.

    CREATE DATABASE neo4j;

  7. Open the browser at http://<external-ip>:7474/browser/ and check that all data is successfully restored.

  8. Execute cypher command against neo4j database:

    MATCH (n) RETURN n

If you have backed up your database with the option --include-metadata, you can manually restore the users and roles metadata. For more information, see Restore a database backup → Example.

6. Upgrade Neo4j on Kubernetes

To upgrade from Neo4j Community to Enterprise edition, run:

helm upgrade <release-name> neo4j/neo4j-standalone --set neo4j.edition=enterprise --set neo4j.acceptNeo4jLicenseAgreement=yes

To upgrade to the next patch release of Neo4j, update your Neo4j values.yaml file and upgrade the helm release.

  1. Open the values.yaml file, using the code editor of your choice, and add the following line to the image object:

    image:
  customImage: neo4j:4.3.2

  2. Run helm upgrade to apply the changes:

    helm upgrade <release-name> neo4j/neo4j-standalone -f values.yaml

7. Scale a Neo4j deployment

To increase or decrease the resources (CPU, memory) available to Neo4j, change the neo4j.resources object in the values.yaml file to set the desired resource usage, and then perform a helm upgrade.

If you change the memory allocated to the Neo4j container, you should also change the Neo4j’s memory configuration (dbms.memory.heap.max_size and dbms.memory.pagecache.size in particular). See Configure Resource Allocation for more details.

For example:

  1. Create a values.yaml file for a Neo4j instance with 1 CPU and 3 GB of memory:

    # values.yaml
neo4j:
  resources:
    cpu: "1"
    memory: "3Gi"

config:
  dbms.memory.heap.initial_size=2G
  dbms.memory.heap.max_size=2G
  dbms.memory.pagecache.size=500m

  2. Run helm install to create the instance:

    helm install  neo4j/neo4j-standalone -f values.yaml

  3. Modify the values.yaml file to increase to 2 CPU and 4 GB of memory (allocating the additional memory to the pagecache):

    # values.yaml
neo4j:
  resources:
    cpu: "2"
    memory: "4Gi"

config:
  dbms.memory.heap.initial_size=2G
  dbms.memory.heap.max_size=2G
  dbms.memory.pagecache.size=1G

  4. Run helm upgrade to apply the change:

    helm upgrade <release-name> neo4j/neo4j-standalone -f values.yaml