Disaster recoveryEnterprise Edition
A database can become unavailable due to issues on different system levels. For example, a data center failover may lead to the loss of multiple servers, which may cause a set of databases to become unavailable.
This section contains a step-by-step guide on how to recover unavailable databases that are incapable of serving writes and/or reads. The guide recovers the unavailable databases and make them fully operational, with minimal impact on the other databases in the cluster. However, if a database is not performing as expected for other reasons, this section cannot help.
|
If all servers in a Neo4j cluster are lost in a disaster, it is not possible to recover the current cluster. You have to create a new cluster and restore the databases, see Deploy a basic cluster and Seed a database for more information. |
Faults in clusters
Databases in clusters may be allocated differently within the cluster and may also have different numbers of primaries and secondaries.
The consequence of this is that all servers may be different in which databases they are hosting. Losing a server in a cluster may cause some databases to lose a member while others are unaffected. Therefore, in a disaster where one or more servers go down, some databases may keep running with little to no impact, while others may lose all their allocated resources.
Figure 2 shows the disaster when three servers are lost, demonstrating that this situation impacts databases in different ways.
| Database | Disaster scenario | Recovery strategy |
|---|---|---|
Database A |
All allocations are lost. |
The database needs to be recreated from a backup since there are no available allocations left in the cluster. |
Database B |
The primary allocation is lost, and the secondary allocation is available. |
The database needs to be recreated since it has lost a majority of primary allocations and is therefore write-unvailable. However, the recreation can be based on the secondary allocation still present on a healthy server, so a backup is not required. The recreated database will be as up-to-date as the secondary allocation was at the time of the disaster. |
Database C |
Two primary allocations and a secondary one are lost. |
The database needs to be recreated since it has lost a majority of primary allocations and is therefore write-unavailable. However, the recreation can be based on the primary and secondary allocations still present on healthy servers, so a backup is not required. The recreated database will reflect the state of the most up-to-date surviving primary or secondary allocation. |
Database D |
One primary allocation and two secondary allocations are lost. |
The database remains write-available, allowing it to automatically move allocations from lost servers to available ones when the lost servers are deallocated. Therefore, the database does not need to be recreated even though some allocations have been lost. |
Database E |
Stays unaffected. |
None of the database’s allocations were affected by the disaster, so no action is required. |
Although databases C and D share the same topology, their primaries and secondaries are allocated differently, requiring distinct recovery strategies in this disaster example.
Guide overview
|
In this guide the following terms are used:
|
There are four steps to recovering a cluster from a disaster:
-
Start the Neo4j process on all servers which are not lost. See Start the Neo4j process for more information.
-
Make the
systemdatabase able to serve write operations, so that the cluster can be modified. See Make thesystemdatabase write-available for more information. -
Detach any potential lost servers from the cluster and replace them by new ones. See Make servers available for more information.
-
Finish disaster recovery by starting or continuing to manage databases and verify that they are write-available. See Make databases write-available for more information.
Each step is described in the following three sections:
-
Objective — a state that the cluster needs to be in, with optional motivation.
-
Verifying the state — an example of how the state can be verified.
-
Path to correct state — a proposed series of steps to get to the correct state.
|
Verifying each state before continuing to the next step, regardless of the disaster scenario, is recommended to ensure the cluster is fully operational. |
Disaster recovery steps for a single cluster
|
Disasters may sometimes affect the routing capabilities of the driver and may prevent the use of the |
Make the system database write-available
Objective
The system database is able to serve write operations.
The system database contains the view of the cluster.
This includes which servers and databases are present, where they live and how they are configured.
During a disaster, the view of the cluster might need to change to reflect a new reality, such as removing lost servers.
Databases might also need to be recreated to regain write availability.
Because both of these steps are executed by modifying the system database, making the system database write-available is a vital first step during disaster recovery.
Verifying the state
The system database’s write availability can be verified by using the Status check procedure.
CALL dbms.cluster.statusCheck(["system"]);
|
The status check procedure cannot verify the write availability of a database configured to have a single primary.
Instead, check that the primary is allocated on an available server and that it has |
Path to correct state
Use the following steps to regain write availability for the system database if it has been lost.
They create a new system database from the most up-to-date copy of the system database that can be found in the cluster.
It is important to get a system database that is as up-to-date as possible, so it corresponds to the view before the disaster closely.
|
This section of the disaster recovery guide uses |
-
Shut down the Neo4j process on all servers. This causes downtime for all databases in the cluster until the processes are started again at the end of this section.
-
On each server, run
bin/neo4j-admin dbms unbind-system-dbto reset thesystemdatabase state on the servers. -
On each server, run
bin/neo4j-admin database info systemand compare thelastCommittedTransactionto find out which server has the most up-to-date copy of thesystemdatabase. -
On the most up-to-date server, run
bin/neo4j-admin database dump system --to-path=[path-to-dump]to take a dump of the currentsystemdatabase and store it in an accessible location. -
For every lost server, add a new unconstrained one according to Add a server to the cluster. It is important that the new servers are unconstrained, or deallocating servers in the next step of this guide might be blocked, even though enough servers were added.
In the current example, the new unconstrained servers are added in this step.
While recommended, it is not strictly necessary to add new servers in this step. There is also an option to change the
systemdatabase mode (server.cluster.system_database_mode) on secondary allocations to make them primary allocations for the newsystemdatabase. The number of primary allocations needed is defined bydbms.cluster.minimum_initial_system_primaries_count. See the Configuration settings for more information. Be aware that not replacing servers can cause cluster overload when databases are moved from lost servers to available ones in the next step of this guide. -
On each server, run
bin/neo4j-admin database load system --from-path=[path-to-dump] --overwrite-destination=trueto load the currentsystemdatabase dump.Figure 3. The unconstrained servers are added and thesystemdatabase is restored -
On each server, ensure that the discovery settings are correct. See Cluster server discovery for more information.
-
Start the Neo4j process on all servers.
Make servers available
Objective
All servers in the cluster’s view are available and enabled.
A lost server will still be in the system database’s view of the cluster, but in an unavailable state.
Furthermore, according to the view of the cluster, these lost servers are still hosting the databases they had before they became lost.
Therefore, informing the cluster of servers which are lost is not enough.
The databases hosted on lost servers also need to be moved onto available servers in the cluster, before the lost servers can be removed.
Verifying the state
The cluster’s view of servers can be seen by showing the servers.
See Showing servers for more information.
The state has been verified if all servers show health = Available and status = Enabled.
SHOW SERVERS;
Path to correct state
Use the following steps to remove lost servers and add new ones to the cluster. To remove lost servers, any allocations they were hosting must be moved to available servers in the cluster. This is done in two different steps:
-
Any allocations that cannot move by themselves require the database to be recreated so that they are forced to move.
-
Any allocations that can move will be instructed to do so by deallocating the server.
-
For each
Unavailableserver, runCALL dbms.cluster.cordonServer("unavailable-server-id")on one of the available servers. This prevents new database allocations from being moved to this server.Figure 4. Cordon unavailable serversFigure 4 shows that new unconstrained servers have been added already. It was done in the Make the
systemdatabase write-available step of this guide, and additional servers might not be needed here. -
If you have not yet added new unconstrained servers, add one for each
Cordonedserver that needs to be replaced. See Add a server to the cluster for more information. It is important that the new servers are unconstrained, or deallocating servers might be blocked even though enough servers were added.While recommended, it is not strictly necessary to add new servers in this step. However, not adding new servers reduces the capacity of the cluster to handle work. Furthermore, it might require the topology for a database to be altered to make deallocating servers and recreating databases possible.
-
For each stopped database (
currentStatus=offline), start them by runningSTART DATABASE stopped-db. This is necessary since stopped databases cannot be deallocated from a server. It is also necessary for the status check procedure to accurately indicate if this database should be recreated or not. Verify that all allocations are incurrentStatus=onlineon servers which are not lost before moving to the next step. If a database fails to start, leave it to be recreated in the next step of this guide.A database can be set to
READ-ONLYbefore it is started to avoid updates on the database with the following command:ALTER DATABASE database-name SET ACCESS READ ONLY. -
On each server, run
CALL dbms.cluster.statusCheck([])to check the write availability for all databases running in primary mode on this server. See Monitoring replication for more information.The status check procedure cannot verify the write availability of a database configured to have a single primary. Instead, check that the primary is allocated on an available server and that it has
currentStatus=onlineby runningSHOW DATABASES. -
For each database that is not write-available, recreate it to move it from lost servers and regain write availability. Go to Recreate a database for more information about recreate options. Remember to make sure there are recent backups for the databases before recreating them. See Online backup for more information. If any database has
currentStatus=quarantinedon an available server, recreate them from backup using Backup as seed.If you recreate databases using undefined servers or undefined servers with fallback backup, the store might not be recreated as up-to-date as possible in certain edge cases where the
systemdatabase has been restored.Figure 5. All write-unavailable databases were recreated -
For each
Cordonedserver, runDEALLOCATE DATABASES FROM SERVER cordoned-server-idon one of the available servers. This will move all database allocations from this server to an available server in the cluster.Figure 6. Deallocate databases from unavailable serversNote that the database D was still write-available, which means the allocations can be moved from lost servers to available ones when the lost servers are deallocated.
This operation might fail if enough unconstrained servers were not added to the cluster to replace lost servers. Another reason is that some available servers are also
Cordoned. -
For each deallocating or deallocated server, run
DROP SERVER deallocated-server-id. This removes the server from the cluster’s view.Figure 7. The fully recovered clusterAfter dropping the deallocated servers, you still have to ensure that all moved and recreated databases are write-available. For this purpose, follow the steps below.
-
Make databases write-available
Objective
All databases that are desired to be started are write-available.
Once this state is verified, disaster recovery is complete.
However, remember that previously stopped databases might have been started during this process.
If they are still desired to be in stopped state, run STOP DATABASE started-db WAIT.
|
Remember, recreating a database takes an unbounded amount of time since it may involve copying the store to a new server, as described in Recreate a database.
Therefore, an allocation with |
Verifying the state
You can verify all clustered databases' write availability by using the status check procedure.
CALL dbms.cluster.statusCheck([]);
|
The status check procedure cannot verify the write availability of a database configured to have a single primary.
Instead, check that the primary is allocated on an available server and that it has |
A stricter verification can be done to verify that all databases are in their desired states on all servers.
For the stricter check, run SHOW DATABASES and verify that requestedStatus = currentStatus for all database allocations on all servers.
Path to correct state
Use the following steps to make all databases in the cluster write-available again. They include recreating any databases that are not write-available and identifying any recreations that will not complete. Recreations might fail for different reasons, but one example is that the checksums do not match for the same transaction on different servers.
-
Identify all write-unavailable databases by running
CALL dbms.cluster.statusCheck([])as described in the Example verification part of this disaster recovery step. Filter out all databases desired to be stopped, so that they are not recreated unnecessarily. -
Recreate every database that is not write-available and has not been recreated previously. See Recreate a database for more information. Remember to make sure there are recent backups for the databases before recreating them. See Online backup for more information. If any database has
currentStatus=quarantinedon an available server, recreate them from backup using Backup as seed.If you recreate databases using undefined servers or undefined servers with fallback backup, the store might not be recreated as up-to-date as possible in certain edge cases where the
systemdatabase has been restored. -
Run
SHOW DATABASESand check any recreated databases that are not write-available. Recreating a database will not complete if one of the following messages is displayed in the message field:-
Seeders ServerId1 and ServerId2 have different checksums for transaction TransactionId. All seeders must have the same checksum for the same append index. -
Seeders ServerId1 and ServerId2 have incompatible storeIds. All seeders must have compatible storeIds. -
No store found on any of the seeders ServerId1, ServerId2…
-
-
For each database which will not complete recreation, recreate them from backup using Backup as seed.
Disaster recovery across multiple clustersIntroduced in 2026.08
In a deployment with independent clusters in different locations, you can use cross-cluster database replication for disaster recovery.
If the source database becomes unavailable, promote its replica on the downstream cluster and redirect application traffic to the promoted database. During promotion, you can modify the topology of a replica database as needed for a new deployment.
Promotion is a one-way operation. After promotion, the replica cannot be re-attached to the original source database. To resume replication, you must create a new replica.
Remember that user roles and privileges are not copied over when replicating a database. After promotion, manually recreate the required user and role metadata on the promoted database.
You can also restore the source database and the upstream cluster. After restoring the upstream cluster to service, you can re-establish replication to the downstream cluster.
For detailed steps on how to promote a replica database and perform the failback, see Replicating databases across clusters → Disaster recovery scenario.
Glossary
- allocator
-
A component in the cluster that allocates databases to servers according to the topology constraints specified and an allocation strategy.
- asynchronous replication
-
Asynchronous replication is used by secondary copies to poll for new transactions, which means they cannot be guaranteed to have received the most recent transactions. This enables efficient scale-out of read-performance.
- Aura instance
-
A fully-managed DBMS represented by a single instance ID, that is running in the Neo4j Aura cloud.
- auto-commit transaction
-
An automatically committed transaction that contains a single query.
- Bolt protocol
-
Bolt is a protocol used for interaction between Neo4j instances and drivers.
- bookmark
-
A marker the client can request from the cluster to ensure that it is able to read its own writes so that the application’s state is consistent and only databases that have a copy of the bookmark are permitted to respond.
- category (Bloom)
-
A category is based on a node label and is defined in a Perspective as a way of visually distinguishing nodes with the same label(s).
- causal consistency
-
All servers in a cluster agree on the order in which transactions take place. The position of a server on the causal chain can be guaranteed using a bookmark.
- cluster
-
A Neo4j DBMS that spans multiple servers working together to increase fault tolerance and/or read scalability. Databases on a cluster may be configured to replicate across servers in the cluster thus achieving read scalability or high availability.
- client application
-
Software that interacts with a Neo4j server.
- commit
-
A commit is the successful completion of a transaction, which ensures durability of any changes made. For more details, visit Operations Manual → Transaction management.
- composite database
-
Composite databases are the means to access partitioned graph data with a single Cypher query.
- constraint
-
Constraints are sets of data modeling rules that ensure the data is consistent and reliable.
- Cypher®
-
Neo4j’s graph query language.
- data model
-
A data model defines how information is organized in a database. A good data model will make querying and understanding your data easier. In Neo4j, the data models have a graph structure.
- database
-
A database is a container used by the DBMS to manage and store graph data. The physical structure of data is controlled by the database.
- database vs graph
-
Databases are the physical containers of graph data. Graphs are the logical structure of data in Neo4j.
- Database Management System
-
Database Management System, or DBMS, capable of managing multiple databases. A DBMS may run on a single server, or span several servers configured as a cluster.
- database schema
-
The prescribed property existence and datatypes for nodes and relationships.
- deallocate
-
An act of removing a database from a server or a server from a cluster without loss of data or reduced fault tolerance.
- degree (of a node)
-
The number of relationships of a specific node; loops are counted twice.
- disaster recovery
-
A manual intervention to restore availability of a cluster, or databases within a cluster.
- driver
-
A software library that provides access to Neo4j from a particular programming language.
- election
-
In the event that the Raft leader becomes unresponsive, followers automatically trigger an election and vote for a new leader.
- entity
-
A node or a relationship.
- expression (Cypher)
-
A component of a Cypher query which produces values. It may be used in projections, as a predicate, or when setting properties on graph elements.
- fabric
-
Fabric is the architectural design of a unified system that provides a single access point to local or distributed graph data.
- fault tolerance
-
A guarantee that a cluster can maintain a database’s persistence and availability in the event of one or more servers failing.
- follower
-
A primary copy of a database acting as a follower, receives and acknowledges synchronous writes from the leader.
- Generative AI (GenAI)
-
A type of artificial intelligence (AI) system that generates text, images, or other media in response to prompts.
- graph
-
A logical representation of a set of nodes where some pairs are connected by relationships.
- index
-
Data structure that improves read performance of a database.
- knowledge graph
-
A specific type of graph that has an organizing principle so that a user (or a computer system) can reason about the underlying data. The organizing principle provides an additional layer of structure that adds context to support knowledge discovery.
- label
-
Marks a node as a member of a named and indexed subset. A node may be assigned zero or more labels.
- leader
-
A single primary copy of a database is designated as the leader. It receives all write transactions from clients and replicates writes synchronously to followers and asynchronously to secondary copies of the database.
- main database
-
In terms of Neo4j Enterprise Studio, the database(s) containing the user’s data. Can exist in the same Neo4j deployment as the tool asset database.
- motif
-
A description of a specific pattern within a graph.
- node
-
A node represents an entity or discrete object in your graph data model. Nodes can be connected by relationships, hold data in properties, and are classified by labels.
- operator
-
A symbol representing a mathematical or logical operation.
- parameter
-
Named value provided when running a Cypher statement.
- path
-
A sequence of nodes and the relationships connecting them, that does not contain duplicate relationships. Several paths can match a pattern.
- pattern
-
A specific arrangement of nodes and relationships that can be matched in a graph. A pattern follows a motif.
- perspective (Bloom)
-
A Perspective defines a certain business view or domain that can be found in the target Neo4j graph. A single Neo4j graph can be viewed through different Perspectives, each tailored for a different business purpose.
- primary
-
A copy of the database that is able to process write transactions and is eligible to be elected as a leader. It participates in fault tolerant writes as it is part of the majority required to acknowledge and commit write transactions.
- primary vs secondary
-
In a cluster, databases can operate in either primary or secondary mode. Primary databases are able to process write and read transactions, ensuring fault tolerance. Secondary databases are replicated asynchronously from primaries, and their main purpose is to provide read scaling within the cluster.
- project (Aura)
-
An isolated environment in the unified Aura console that contains its own database instances, configurations, and resources. Preceded by tenant in the classic Aura console.
- property
-
Properties are key-value pairs that are used for storing data on nodes and relationships.
- query (Cypher)
-
A statement that retrieves or writes information to a database.
- Raft group
-
A group of servers that are participating in hosting a particular database in primary mode.
- Raft group member
-
A server that is participating in a Raft group. A server can be a member of one or more groups.
- Raft log
-
A shared log between all Raft group members that is guaranteed to be consistently updated and viewed by those members. The log contains both database data and operational state of the Raft group.
- Raft protocol
-
The networking mechanism that enables a database to replicate its data across multiple servers to give high availability for accessing the data and high durability to the data stored.
- read scaling
-
Distributing query load by creating additional database copies hosted in secondary mode (read-only).
- relationship
-
A relationship represents a connection between nodes in your graph data model. Relationships connect a source node to a target node, hold data in properties, and are classified by type.
- secondary
-
An asynchronously replicated copy of the database that provides read scaling within the cluster.
- seed
-
A seed is a database dump or a full backup used to create a database on a cluster. This is sometimes called seeding.
- server
-
A physical machine, a virtual machine, or a container running an instance of Neo4j. Servers can be standalone or part of a cluster.
- session
-
A causally linked sequence of transactions.
- session consistency
-
An alternative name for Neo4j’s causal consistency.
- standalone
-
A single server running Neo4j and not part of a cluster.
- synchronous replication
-
Synchronous replication requires the leader primary to replicate a transaction and block the commit until a quorum of the follower primaries acknowledges that the transaction is successfully replicated. Once the transaction is replicated, the commit is allowed to proceed. This ensures data durability and consistency within the cluster.
- system database
-
A database used by Neo4j to store system information.
- tenant (Aura)
-
An isolated environment in the classic Aura console that contains its own database instances, configurations, and resources. Replaced by project in the unified Aura console.
- tool asset database
-
In terms of Neo4j Enterprise Studio, the database where tools' assets are stored. This can be in the same Neo4j deployment as the main database(s) or in a separate deployment.
- topology
-
A configuration that describes how the copies of a database should be spread across the servers in a cluster, see primary mode and secondary mode.
- transaction
-
A transaction comprises a unit of work performed against a database. It is treated in a coherent and reliable way, independent of other transactions. Transactions comply with the ACID consistency model (atomic, consistent, isolated, and durable).