Upgrade a Causal Cluster to a 4.x patch release

This section describes how to upgrade a Neo4j cluster to a newer PATCH release within the same MAJOR and MINOR version.

You can upgrade your existing Neo4j Causal Cluster by either performing a rolling upgrade, or by upgrading it offline.

The prerequisites and the upgrade steps must be completed for each cluster member.

When upgrading from an earlier patch release, you might need to upgrade the system database as well. Use call dbms.upgradeStatus to check whether this is necessary or see Upgrade the system database for more information.

Offline upgrade

This variant is suitable for cases where a rolling upgrade is not possible.

It is recommended to perform a test upgrade on a production-like environment to get information on the duration of the downtime.

Prerequisites

Ensure that you have completed all the tasks on the Upgrade checklist for each cluster member.

Prepare for the upgrade

  1. Shut down all the cluster members (Cores and Read Replicas).

  2. Perform neo4j-admin unbind on each cluster member to remove cluster state data.

  3. Install the Neo4j version that you want to upgrade to on each instance. For more information on how to install the distribution that you are using, see the Operations Manual → Installation section of the version that you want to install.

  4. Replace the neo4j.conf file with the one that you have prepared for each instance in section Prepare a new neo4j.conf file to be used by the new installation.

  5. Copy all the files used for encryption, such as private key, public certificate, and the contents of the trusted and revoked directories (located in <neo4j-home>/certificates/).

  6. Restore each of your backed-up databases and transactions on each cluster member, including the system database, by either using neo4j-admin restore (online) or neo4j-admin load (offline), depending on your backup approach. If you are running a Debian/RPM distribution, you can skip this step.

    If your old installation has modified configurations starting with dbms.directories.* or the setting dbms.default_database, verify that the new neo4j.conf file is configured properly to find these directories.

Upgrade your cluster

  1. Start Neo4j by running the following command from <neo4j-home>:

    bin/neo4j start

    The upgrade takes place during startup.

  2. Monitor the neo4j.log file for information on how many steps the upgrade will involve and how far it has progressed.

  3. Verify that the cluster forms and the upgraded Neo4j DBMS deployment comes online.

  4. Start each Read Replica and wait for it to catch up with the rest of the cluster members.

  5. Verify that the Read Replicas join the cluster.

Post-upgrade

It is recommended to perform a full backup, using an empty target directory.

Rolling upgrade

Rolling upgrade is a zero-downtime method for upgrading a Causal Cluster. You upgrade one member at a time, while the rest of the members are running. However, if during a rolling upgrade the cluster loses quorum and cannot be recovered, then downtime may be required to do a disaster recovery.

Recommendations
  • The critical point during the upgrade is knowing when it is safe to switch off the original member.
    It is highly recommended to monitor the status endpoint before each removal, in order to decide which member to switch off and when it is safe to do so.

  • To reduce the risk of failure during a rolling upgrade, make sure the cluster is not under any heavy load during the upgrade. If possible, the safest would be to disable writes entirely.

  • There should be no changes to database administration during a rolling upgrade. For more information, see Operations Manual → Manage databases.

Rolling upgrade for a fixed number of servers

This variant is suitable for deployments where there is a fixed number of servers and they have to be updated in-place.

When performing a rolling upgrade for a fixed number of servers, it is not possible to increase the cluster size. Therefore, the cluster fault tolerance level will be reduced while replacing the members.

Prerequisites

  1. Ensure that you have completed all the tasks on the Upgrade checklist for each cluster member.

  2. Verify that all databases are online by running SHOW DATABASES in Cypher® Shell or Neo4j Browser. Offline databases can be started using START DATABASE [database-name].

    All databases must be started before you start a rolling upgrade. If you have to keep а database inaccessible during the rolling upgrade, you can disable access to it by using one of following ways:

    You must never run DENY ACCESS ON DATABASE system TO PUBLIC or DENY ACCESS ON DATABASE * TO PUBLIC because you will lock yourself out of the system database. If you do lock yourself out, follow the disable authentication steps in the Operations Manual to recover and prevent outside access to the instance or cluster.

    For PATCH upgrades within 4.0 version
    1. Deny all roles access to each database except the system database. You can query all available roles with SHOW ROLES.

      DENY ACCESS ON DATABASE [database-name] TO [role1],[role2]
    2. Ensure that the databases cannot be stopped during the rolling upgrade by using the following command:

      DENY STOP ON DATABASE * TO admin

      This must be done for the admin role and all other roles that have the privilege to stop databases. For more information about listing privileges, see Cypher Manual → Managing privileges.

    For PATCH upgrades within a 4.x version::
    1. Deny the PUBLIC role access to each database except the system database.

      DENY ACCESS ON DATABASE [database-name] TO PUBLIC
    2. Ensure that the databases cannot be stopped, created, or dropped during the rolling upgrade by using the following command:

      DENY STOP ON DATABASE * TO PUBLIC
      DENY DATABASE MANAGEMENT ON DBMS TO PUBLIC

Upgrade the cluster

You upgrade one cluster member at a time, while the rest of the members are running.

If during a rolling upgrade the cluster loses quorum and cannot be recovered, then downtime may be required to do disaster recovery.

For each cluster member
  1. (Recommended) Use the process described in the status endpoint to evaluate whether it is safe to remove the old instance.

  2. Shut down the instance.

  3. Install the Neo4j version that you want to upgrade to. For more information on how to install the distribution that you are using, see the Operations Manual → Installation section of the version that you want to install.

  4. Replace the neo4j.conf file with the one that you have prepared for this instance in section Prepare a new neo4j.conf file to be used by the new installation.

  5. Start the new instance and wait for it to catch up with the rest of the cluster members.

  6. Verify that the new instance has successfully joined the cluster and caught up with the rest of the members, by using the status endpoint.

Because Read Replicas are not part of the cluster consensus group, their replacement during an upgrade does not affect the cluster availability and fault tolerance level. However, it is still recommended to incrementally add Read Replicas for a structured and maintainable upgrade process.

Post-upgrade steps

The following steps must be performed after a rolling upgrade.

For PATCH upgrades within 4.0 version
  1. Restore the privilege of the admin role to stop databases.

    REVOKE DENY STOP ON DATABASE * FROM admin

    This must be done for all roles for which the privilege to stop databases has been denied (see step 6 of Rolling upgrade for a fixed number of servers). For more information about listing privileges, see Cypher Manual 4.0 → Graph and sub-graph access control.

  2. (Optional) If you have started offline databases and denied some access rights during the preparation phase for a rolling upgrade, you should also restore them to the original state:

    1. Stop each of the databases by running the following command:

      STOP DATABASE [database-name]
    2. Re-enable access to the databases by running the following command:

      REVOKE DENY ACCESS ON DATABASE [database-name] FROM [role1],[role2]
For PATCH upgrades within a 4.x version
  1. Restore the privilege of the PUBLIC role to stop databases:

    REVOKE DENY STOP ON DATABASE * FROM PUBLIC
  2. Restore the privilege of the PUBLIC role to create and drop databases:

    REVOKE DENY DATABASE MANAGEMENT ON DBMS FROM PUBLIC
  3. (Optional) If you have started offline databases during the preparation phase for a rolling upgrade, you stop each of them to restore them to the original state:

    STOP DATABASE [database-name]

Rolling upgrade for cloud infrastructure

This variant is suitable for deployments that use replaceable cloud or container resources. It follows the same steps as for the fixed number of servers, but you can add the new members before you shut down the old ones, thus preserving the cluster fault tolerance level. Because Read Replicas are not part of the cluster consensus group, their replacement during the upgrade will not affect the cluster availability and fault tolerance level. However, it is still recommended to incrementally add Read Replicas for a structured and maintainable upgrade process.