Managing databases in a cluster

The number of both primary and secondary servers to host a database can be set when the database is created and altered after creation. The command CREATE DATABASE can be used to specify the initial topology and ALTER DATABASE can be used to change the topology once the database is created. If a database is no longer needed, the command DROP DATABASE deletes the database from the cluster.

`CREATE DATABASE`

The command to create a database in a cluster is not significantly different from the command to create a database in a non-clustered environment (see Create, start, and stop databases for more information on database management on single servers). The difference in a clustered environment is that the topology can be specified, i.e. how many primaries and secondaries are desired for the database. To create a database foo with 3 servers hosting the database in primary mode and 2 servers in secondary mode, the command looks like this:

CREATE DATABASE foo TOPOLOGY 3 PRIMARIES 2 SECONDARIES

Alternatively, you can use parameters to provide the number of primaries and secondaries:

Parameters

{
  "dbname": "foo",
  "primary": 3,
  "secondary": 2
}

Query

CREATE DATABASE $dbname TOPOLOGY $primary PRIMARIES $secondary SECONDARIES

The command can only be executed successfully if the cluster’s servers are able to satisfy the specified topology. If they are not, the command results in an error. For example, if the cluster’s servers are set up with mode constraints to contain two primaries and three secondaries, or if only four servers exist, the command fails with an error.

If TOPOLOGY is not specified, the database is created according to initial.dbms.default_primaries_count and initial.dbms.default_secondaries_count specified in neo4j.conf. After cluster startup, these values can be overwritten using the dbms.setDefaultAllocationNumbers procedure.

A CREATE DATABASE command allocates the database, therefore there is no requirement to execute REALLOCATE DATABASES (described in Hosting databases on added servers). However, over time, or after several CREATE DATABASE commands have been issued, the distribution of databases can become unbalanced. At this point you can run REALLOCATE DATABASES to make the cluster re-balance databases across all servers that are part of the cluster.

`ALTER DATABASE`

To alter the topology of or read/write access to a database after it has been created, use the command ALTER DATABASE.

Alter database topology

To change the topology of the database foo from the previous example, the command can look like this:

ALTER DATABASE foo SET TOPOLOGY 2 PRIMARIES 1 SECONDARY

Alternatively, you can use parameters to provide the number of primaries and secondaries:

Parameters

{
  "dbname": "foo",
  "primary": 2,
  "secondary": 1
}

Query

ALTER DATABASE $dbname SET TOPOLOGY $primary PRIMARIES $secondary SECONDARIES

Like the CREATE DATABASE command, this command results in an error if the cluster does not contain sufficient servers to satisfy the requested topology.

Additionally, ALTER DATABASE is optionally idempotent and also results in an error if the database does not exist. It is possible to append the command with IF EXISTS to make sure that no error is returned if the database does not exist.

When there is more than one possible permutation of the specified topology, Neo4j uses an allocator to decide how to spread the database across the cluster. Note, like CREATE DATABASE, the ALTER DATABASE command allocates the database and there is no requirement to execute REALLOCATE DATABASES unless there is a desire to re-balance databases across all servers that are part of the cluster.

This normally happens when the cluster is configured with more servers than the sum of the number of primaries and secondaries for any one database.

It is not possible to automatically transition to a topology with a single primary host. Attempting to do so results in an error.

However, it is possible to manually do this transition. The first step is to back up the database, see Backup and restore for more information. Once the database is backed up, the next step is to drop the database, see Delete databases for more information. The last step is to either seed a cluster from the backup with the new topology, or to restore the backup on a single server. See Seed a cluster further on for information on seeding.

Also, it is possible to automatically transition from a topology with a single primary host to multiple primary hosts. Keep in mind that during such a transition, the database will be unavailable for a short period of time.

ALTER DATABASE commands are optionally idempotent, with the default behavior to fail with an error if the database does not exist. Appending IF EXISTS to the command ensures that no error is returned and nothing happens should the database not exist.

If the ALTER DATABASE command decreases the number of allocations of a database, allocations on cordoned servers are removed first.

Query

ALTER DATABASE nonExisting IF EXISTS SET TOPOLOGY 1 PRIMARY 0 SECONDARY

Alter database access

To alter the access to the database foo, the syntax looks like this:

ALTER DATABASE foo SET ACCESS {READ ONLY | READ WRITE}

By default, a newly created database has both read and write access.

Deallocate databases

To relieve the load of a specific server(s), you can use one of the following procedures to deallocate databases causing the pressure from the server(s):

You must have the SERVER MANAGEMENT privilege to execute these procedures.

For example, server01 hosts two small databases, foo and bar, and one very large database baz, while other servers contain fewer or smaller databases, and server01 is under pressure.

You can use one of the following approaches to deallocate baz from server01 or to deallocate a number of databases from server01:

Deallocating a database from a server

// With dry run
neo4j@system> CALL dbms.cluster.deallocateDatabaseFromServer("server01", "baz", true);

// Without dry run
neo4j@system> CALL dbms.cluster.deallocateDatabaseFromServer("server01", "baz");

Deallocating a database from multiple servers

// With dry run
neo4j@system> CALL dbms.cluster.deallocateDatabaseFromServers(["server01", "server02"], "baz", true);

// Without dry run
neo4j@system> CALL dbms.cluster.deallocateDatabaseFromServers(["server01", "server02"], "baz");

Deallocating three databases from a server

// With dry run
neo4j@system> CALL dbms.cluster.deallocateNumberOfDatabases("server01", 3, true);

// Without dry run
neo4j@system> CALL dbms.cluster.deallocateNumberOfDatabases("server01", 3);

Reallocate databases

To rebalance all database allocations across the cluster, for example, because you added new servers, use either procedures or Cypher commands to reallocate databases onto the new servers.

Reallocate databases using a procedure

You can use the procedure dbms.cluster.reallocateDatabase to rebalance a specific database across the cluster, or dbms.cluster.reallocateNumberOfDatabases to rebalance a number of database allocations across the cluster and relieve overloaded servers. Note that if the cluster is already balanced, no reallocations will happen when running these procedures. These procedures do not require a server name and can be executed with or without a dry run.

You must have the SERVER MANAGEMENT privilege to execute these procedures.

For example, you add three new servers and want to move a very large database, baz, from all the servers containing it to the new servers.

Reallocate one database to new servers

// With dry run
neo4j@system> CALL dbms.cluster.reallocateDatabase("baz", true);

// Without dry run
neo4j@system> CALL dbms.cluster.reallocateDatabase("baz");

Reallocating a number of databases to new servers

// With dry run
neo4j@system> CALL dbms.cluster.reallocateNumberOfDatabases(3, true);

// Without dry run
neo4j@system> CALL dbms.cluster.reallocateNumberOfDatabases(3);

Reallocate databases using a Cypher command

You can use the Cypher command REALLOCATE DATABASES to rebalance all database allocations across the cluster and relieve overloaded servers. This command can also be used with DRYRUN to preview the new allocation of databases.

REALLOCATE DATABASES on a large cluster with many databases has the potential to move a lot of allocations at once, which might stress the cluster. Consider starting with more limited reallocations, such as dbms.cluster.reallocateNumberOfDatabases with a small number, and let the databases complete their reallocation before calling it again, until no more reallocations are necessary.

neo4j@neo4j> DRYRUN REALLOCATE DATABASES;
+----------------------------------------------------------------------------------------------------------------------------------------+
| database | fromServerName | fromServerId                           | toServerName | toServerId                             | mode      |
+----------------------------------------------------------------------------------------------------------------------------------------+
| "bar"    | "server-1"     | "00000000-27e1-402b-be79-d28047a9418a" | "server-5"   | "00000003-b76c-483f-b2ca-935a1a28f3db" | "primary" |
| "bar"    | "server-3"     | "00000001-7a21-4780-bb83-cee4726cb318" | "server-4"   | "00000002-14b5-4d4c-ae62-56845797661a" | "primary" |
+----------------------------------------------------------------------------------------------------------------------------------------+

Recreate a database

The dbms.cluster.recreateDatabase() procedure allows you to:

Change the database store to a specified backup, while keeping all the associated privileges for the database.
Make your database write-available again after it has been lost (for example, due to a disaster). See Disaster recovery for more information.

The recreate procedure works only for real user databases and not for composite databases, or the system database.

Remember that the recreate procedure results in downtime while the stores get updated. The time is unbounded and may depend on different factors — for example, the size of the store, network speed, etc.

The database in question can be in an online or offline state when it is recreated, but a successful operation starts the database regardless of its previous state.

If your database has Change Data Capture (CDC) enabled, the CDC chain will stop when the database is recreated, even though CDC remains enabled in the recreated database. To restore CDC functionality, follow the guide on how to initialize CDC applications from an existing database.

Before recreating a database, any eventual quarantined states need to be addressed. For more information, see Standard databases → Error handling.

You need the CREATE DATABASE and DROP DATABASE privileges to run the recreate procedure.

To check if the recreation is successful, use the SHOW DATABASES command and verify that all allocations have been started.

Additionally, you have the option to modify the topology during the recreation process. However, note that the store format, access, and enrichment cannot be altered during recreation.

Seeding options

The store to be used during the recreation of a database can be defined in different ways. One method uses a backup, while others use available allocations in the cluster.

You can use either seedURI or seedingServers to specify the source from which the database should be recreated.

If you define neither, an error is thrown.
If you define both of them, then seedingServers must be an empty list. See Undefined servers with fallback backup for more details.
If seedingServers is not empty and seedURI is also defined, an error will occur.

Use backup as a seed

If you provide a URI to a backup or a dump, the stores on all allocations will be replaced by the backup or the dump at the given URI. The new allocations can be put on any ENABLED server in the cluster. See Seed from URI for more details.

CALL dbms.cluster.recreateDatabase("neo4j", {seedURI: "s3://myBucket/myBackup.backup"});

Use available servers as a seed

After the recreation is complete, the database will have the latest data store from the seeding servers.

Recreation is based on remaining stores or a store defined by the user. This means that stores which were lost or not defined are not used for the recreation. If not used stores were more up to date than the used ones, this results in data loss.

Specified servers

You can specify a set of available servers. The stores on all allocations will be synchronized to the most up-to-date store from the defined servers. The number of defined servers cannot exceed the number of total allocations in the desired topology.

CALL dbms.cluster.recreateDatabase("neo4j", {seedingServers: ["serverId1", "serverId2", "serverId3"]});

Undefined servers

If you provide an empty list of seeding servers and do not specify a seedURI, Neo4j automatically selects all available allocations of the database as seeders. The store will be replaced by the most up-to-date seeder available in the cluster.

CALL dbms.cluster.recreateDatabase("neo4j", {seedingServers: []});

Undefined servers with fallback backup

If both an empty list of seeding servers and a seedURI are provided, Neo4j finds all available allocations of the database and use those as seeders. However, if no available servers can be found, the database is recreated based on the backup or the dump defined by the URI. This means the store is replaced by the most up-to-date seeder if available; otherwise, the backup is used.

CALL dbms.cluster.recreateDatabase("neo4j", {seedingServers: [], seedURI: "s3://myBucket/myBackup.backup"});

Change the topology

There is an option to define a new topology when recreating a database. This can be beneficial during a disaster, if enough servers are not available to recreate the database with the original topology. When altering the total number of allocations down during a recreation, it is important to remember that the number of seeding servers cannot exceed the number of total allocations of the database. This also holds true when using recreate with an empty list of seeders. If there are more available servers in the cluster hosting the database than the number of new allocations, the recreation will fail.

CALL dbms.cluster.recreateDatabase("neo4j", {seedingServers: [], primaries: 3, secondaries: 0});

Seed a cluster

There are two different ways to seed a cluster with data:

The first option is to use a designated seeder, where a designated server is used to create a backed-up database on other servers in the cluster.
The other option is to seed the cluster from a URI, where all servers to host the database are seeded with an identical seed from an external source specified by that URI. For more details, see Create a database from a URI.

Keep in mind that using a designated seeder can be problematic in some situations as it is not known in advance how a database is going to be allocated to the servers in a cluster. Also, this method relies on the seed already existing on one of the servers.

Designated seeder

In order to designate a server in the cluster as a seeder, a database backup is transferred to that server using the neo4j-admin database restore command. Subsequently, that server is used as the source for other cluster members to copy the backed-up database from.

This example creates a user database called foo, hosted on three servers in primary mode. The foo database should not previously exist on any of the servers in the cluster.

If a database with the same name as your backup already exists, use the command DROP DATABASE to delete it and all users and roles associated with it.

Restore the foo database on one server. In this example, the server01 member is used.
```
bin/neo4j-admin database restore --from-path=/path/to/foo-backup-dir foo
```

Find the server ID of server01 by logging in to Cypher Shell and running SHOW SERVERS. Cross-reference the address to find the server ID. Use any database to connect.

SHOW SERVERS YIELD serverId, name, address, state, health, hosting;

+-----------------------------------------------------------------------------------------------------------------------------------------------------+
| serverId                               | name                                   | address          | state     | health      | hosting              |
+-----------------------------------------------------------------------------------------------------------------------------------------------------+
| "25a7efc7-d063-44b8-bdee-f23357f89f01" | "25a7efc7-d063-44b8-bdee-f23357f89f01" | "localhost:7689" | "Enabled" | "Available" | ["system",  "neo4j"] |
| "782f0ee2-5474-4250-b905-4cd8b8f586ba" | "782f0ee2-5474-4250-b905-4cd8b8f586ba" | "localhost:7688" | "Enabled" | "Available" | ["system",  "neo4j"] |
| "8512c9b9-d9e8-48e6-b037-b15b0004ca18" | "8512c9b9-d9e8-48e6-b037-b15b0004ca18" | "localhost:7687" | "Enabled" | "Available" | ["system",  "neo4j"] |
+-----------------------------------------------------------------------------------------------------------------------------------------------------+

In this case, the address for server01 is localhost:7687 and thus, the server ID is 8512c9b9-d9e8-48e6-b037-b15b0004ca18.

On one of the servers, use the system database and create the database foo using the server ID of server01. The topology of foo is stored in the system database and when you create it, it is allocated according to the default topology (which can be shown with CALL dbms.showTopologyGraphConfig). This may be different from the topology of foo when it was backed up. If you want to ensure a certain allocation across the cluster, you can specify the desired topology with the TOPOLOGY clause in the CREATE DATABASE command. See CREATE DATABASE for more information.
```
CREATE DATABASE foo
TOPOLOGY [desired number of primaries] PRIMARIES [desired number of secondaries] SECONDARIES
OPTIONS {existingData: 'use', existingDataSeedServer: '8512c9b9-d9e8-48e6-b037-b15b0004ca18'};
```

Verify that the foo database is online on the desired number of servers, in the desired roles. If the foo database is of considerable size, the execution of the command can take some time.

SHOW DATABASE foo;

+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| name  | type       | aliases | access       | address          | role      | writer | requestedStatus | currentStatus | statusMessage | default | home  | constituents |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| "foo" | "standard" | []      | "read-write" | "localhost:7687" | "primary" | FALSE  | "online"        | "online"      | ""            | FALSE   | FALSE | []           |
| "foo" | "standard" | []      | "read-write" | "localhost:7688" | "primary" | FALSE  | "online"        | "online"      | ""            | FALSE   | FALSE | []           |
| "foo" | "standard" | []      | "read-write" | "localhost:7689" | "primary" | TRUE   | "online"        | "online"      | ""            | FALSE   | FALSE | []           |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

9 rows available after 3 ms, consumed after another 1 ms

Controlling locations with allowed/denied databases

A database can by default be allocated to run on any server in a cluster. However, it is possible to constrain the servers that specific databases are hosted on. This is done with ENABLE SERVER and ALTER SERVER, described in Managing servers in a cluster. The following options are available:

allowedDatabases - a set of databases that are allowed to be hosted on a server.
deniedDatabases - a set of databases that are denied to be hosted on a server. Allowed and denied are mutually exclusive.
modeConstraint - controls in what mode (primary, secondary, or none) databases can be hosted on a server. If not set, there are no mode constraints on the server.

Change the default database

You can use the procedure dbms.setDefaultDatabase("newDefaultDatabaseName") to change the default database for a DBMS.

Ensure that the database to be set as default exists, otherwise create it using the command CREATE DATABASE <database-name>.
Show the name and status of the current default database by using the command SHOW DEFAULT DATABASE.
Stop the current default database using the command STOP DATABASE <database-name>.
Run CALL dbms.setDefaultDatabase("newDefaultDatabaseName") against the system database to set the new default database.
Optionally, you can start the previous default database as non-default by using START DATABASE <database-name>.

Be aware that the automatically created initial default database may have a different topology to the default configuration values. See Default database in a cluster for more information.

Handling errors

Databases can get into error states. Typically you can observe this with the SHOW DATABASES command, and use the error handling guidance to help.

In more serious cases you may be dealing with a disaster situation, where the whole DBMS may not be responding correctly, or some specific databases cannot be restored without downtime. Refer to the disaster recovery guide for those situations.