New server rolling upgrade of a cluster

It is recommended to read the Introduction and Upgrade or patch 5.X before continuing.

This is an example of how to do a rolling upgrade without reducing fault tolerance, by adding servers before removing old ones. This approach is suitable for deployments that use replaceable cloud or container resources.

Before you begin

  1. Ensure that the databases cannot be stopped, created, or dropped during the rolling upgrade by using the following command:

    DENY STOP ON DATABASE * TO PUBLIC
DENY DATABASE MANAGEMENT ON DBMS TO PUBLIC

  2. Verify that all databases are online. See Cypher Manual → Starting databases for details.

Upgrade steps (to be repeated for each server)

Perform these steps in full for each server, before moving on to the next server. You must ensure that the cluster returns to a stable state before moving on to the next.

  1. Check the cluster is healthy.

    1. Before replacing the server, ensure that the cluster is in a healthy state. Use the following query to check servers are hosting all their assigned databases. The query should return no results:

      SHOW SERVERS YIELD name, hosting, requestedHosting, serverId WHERE requestedHosting <> hosting

      Up to version 5.9, if you have composite databases, they will not appear in requestedHosting, and so all servers are returned by this query. Thus, you must manually compare the hosting and requestedHosting fields, excluding any composite databases. From version 5.10 onwards, composite databases appear in both hosting and requestedHosting fields.

    2. Use the following query to check all databases are in their expected state. The query should return no results:

      SHOW DATABASES YIELD name, address, currentStatus, requestedStatus, statusMessage WHERE currentStatus <> requestedStatus RETURN name, address, currentStatus, requestedStatus, statusMessage

  2. Create the new server.

  3. Add the new server to the cluster:

    1. Start the new server and wait for it to be visible to the cluster.

    2. Use SHOW SERVERS to see the new server, it should have the status Free. Make a note of the new server’s ID.

    3. Add the server to the DBMS by running ENABLE SERVER [new-server-id]. This allows the DBMS to use the server to host databases.

  4. Prepare a server for removal:

    1. Run SHOW SERVERS to retrieve the ID of one of the old servers. The server must be Available.

    2. Tell Neo4j to move any databases off the old server by running DEALLOCATE DATABASES FROM SERVER [old-server-id].

  5. Wait for all databases to be deallocated. You can use SHOW SERVER [old-server-id] to follow the process. Once the databases complete their move, the hosting column will contain only system.

    It can take time for a database to start on its new host and provide fault tolerance. If you stop the old server before this point, you reduce the fault tolerance of the database.

  6. Remove the old server:

    1. Remove the old server by running DROP SERVER [old-server-id]. It will still appear as Dropped in SHOW SERVERS until the process stops.

    2. Server can then be shut down.

  7. (Optional) Restore the privilege of the PUBLIC role to stop databases:

    REVOKE DENY STOP ON DATABASE * FROM PUBLIC

  8. (Optional) Restore the privilege of the PUBLIC role to create and drop databases:

    REVOKE DENY DATABASE MANAGEMENT ON DBMS FROM PUBLIC

  9. (Optional) If you have started offline databases during the preparation phase for a rolling upgrade, you will need to stop each of them in order to restore them to the original state.

  10. If using the LIST discovery method, after you upgrade all the servers make sure to set dbms.cluster.discovery.endpoints to include only the new servers.

    See also dbms.cluster.discovery.resolver_type.

  11. Complete the upgrade.

    If you are using version 5.8 or before, you need to manually upgrade the system database. If you are using version 5.9 or later, you can skip this step, the system database is automatically upgraded when safe to do so. Run the following Cypher against the cluster using Cypher Shell or another Cypher client:

    CALL dbms.upgrade()

Monitor the logs

When Neo4j restarts, it is a good idea to monitor the logs for any errors or warnings caused by the upgrade. The neo4j.log file contains information on the upgrade.