Managing databases in a cluster

The number of both primary and secondary servers to host a database can be set when the database is created and altered after creation. The command CREATE DATABASE can be used to specify the initial topology and ALTER DATABASE can be used to change the topology once the database is created. If a database is no longer needed, the command DROP DATABASE deletes the database from the cluster.

CREATE DATABASE

The command to create a database in a cluster is not significantly different from the command to create a database in a non-clustered environment (see Create, start, and stop databases for more information on database management on single servers). The difference in a clustered environment is that the topology can be specified, i.e. how many primaries and secondaries are desired for the database. To create a database foo with 3 servers hosting the database in primary mode and 2 servers in secondary mode, the command looks like this:

CREATE DATABASE foo TOPOLOGY 3 PRIMARIES 2 SECONDARIES

The command can only be executed successfully if the cluster’s servers are able to satisfy the specified topology. If they are not, the command results in an error. For example, if the cluster’s servers are set up with mode constraints to contain two primaries and three secondaries, or if only four servers exist, the command fails with an error.

If TOPOLOGY is not specified, the database is created according to initial.dbms.default_primaries_count and initial.dbms.default_secondaries_count specified in neo4j.conf. After cluster startup, these values can be overwritten using the dbms.setDefaultAllocationNumbers procedure.

A CREATE DATABASE command allocates the database, therefore there is no requirement to execute REALLOCATE DATABASES (described in Hosting databases on added servers). However, over time, or after several CREATE DATABASE commands have been issued, the distribution of databases can become unbalanced. At this point you can run REALLOCATE DATABASES to make the cluster re-balance databases across all servers that are part of the cluster.

ALTER DATABASE

To alter the topology of or read/write access to a database after it has been created, use the command ALTER DATABASE.

Alter topology

To change the topology of the database foo from the previous example, the command can look like this:

ALTER DATABASE foo SET TOPOLOGY 2 PRIMARIES 1 SECONDARY

Like the CREATE DATABASE command, this command results in an error if the cluster does not contain sufficient servers to satisfy the requested topology.

Additionally, ALTER DATABASE is optionally idempotent and also results in an error if the database does not exist. It is possible to append the command with IF EXISTS to make sure that no error is returned if the database does not exist.

When there is more than one possible permutation of the specified topology, Neo4j uses an allocator to decide how to spread the database across the cluster. Note, like CREATE DATABASE, the ALTER DATABASE command allocates the database and there is no requirement to execute REALLOCATE DATABASES unless there is a desire to re-balance databases across all servers that are part of the cluster.

This normally happens when the cluster is configured with more servers than the sum of the number of primaries and secondaries for any one database.

It is not possible to automatically transition to a topology with a single primary host. Attempting to do so results in an error.

However, it is possible to manually do this transition. The first step is to back up the database, see Backup and restore for more information. Once the database is backed up, the next step is to drop the database, see Delete databases for more information. The last step is to either seed a cluster from the backup with the new topology, or to restore the backup on a single server. See Seed a cluster further on for information on seeding.

Also, it is possible to automatically transition from a topology with a single primary host to multiple primary hosts. Keep in mind that during such a transition, the database will be unavailable for a short period of time.

ALTER DATABASE commands are optionally idempotent, with the default behavior to fail with an error if the database does not exist. Appending IF EXISTS to the command ensures that no error is returned and nothing happens should the database not exist.

If the ALTER DATABASE command decreases the number of allocations of a database, allocations on cordoned servers are removed first.

Query
ALTER DATABASE nonExisting IF EXISTS SET TOPOLOGY 1 PRIMARY 0 SECONDARY

0 rows

Alter access

To alter the access to the database foo, the syntax looks like this:

ALTER DATABASE foo SET ACCESS {READ ONLY | READ WRITE}

By default, a newly created database has both read and write access.

Reallocate databases

If there is a need to rebalance database allocation across the cluster, for example if new servers have been added, use the REALLOCATE DATABASE[S]. This command can be used with DRYRUN to preview the new allocation of databases.

DRYRUN is introduced in Neo4j 5.2 and thus does not work in previous versions.

 
neo4j@neo4j> DRYRUN REALLOCATE DATABASES;
+----------------------------------------------------------------------------------------------------------------------------------------+
| database | fromServerName | fromServerId                           | toServerName | toServerId                             | mode      |
+----------------------------------------------------------------------------------------------------------------------------------------+
| "bar"    | "server-1"     | "00000000-27e1-402b-be79-d28047a9418a" | "server-5"   | "00000003-b76c-483f-b2ca-935a1a28f3db" | "primary" |
| "bar"    | "server-3"     | "00000001-7a21-4780-bb83-cee4726cb318" | "server-4"   | "00000002-14b5-4d4c-ae62-56845797661a" | "primary" |
+----------------------------------------------------------------------------------------------------------------------------------------+

Seed a cluster

There are two different ways to seed a cluster with data. The first option is to use a designated seeder, where a designated server is used to create a backed-up database on other servers in the cluster. The other options is to seed the cluster from URI, where all servers to host a database are seeded with an identical seed from an external source specified by the URI. Keep in mind that using a designated seeder can be problematic in some situations as it is not known in advance how a database is going to be allocated to the servers in a cluster. Also, this method relies on the seed already existing on one of the servers.

Designated seeder

In order to designate a server in the cluster as a seeder, a database backup is transferred to that server using the neo4j-admin database restore command. Subsequently, that server is used as the source for other cluster members to copy the backed-up database from.

This example creates a user database called foo, hosted on three servers in primary mode. The foo database should not previously exist on any of the servers in the cluster.

If a database with the same name as your backup already exists, use the command DROP DATABASE to delete it and all users and roles associated with it.

  1. Restore the foo database on one server. In this example, the server01 member is used.

    bin/neo4j-admin database restore --from-path=/path/to/foo-backup-dir foo

  2. Find the server ID of server01 by logging in to Cypher Shell and running SHOW SERVERS. Cross-reference the address to find the server ID. Use any database to connect.

    SHOW SERVERS YIELD serverId, name, address, state, health, hosting;
    +-----------------------------------------------------------------------------------------------------------------------------------------------------+
| serverId                               | name                                   | address          | state     | health      | hosting              |
+-----------------------------------------------------------------------------------------------------------------------------------------------------+
| "25a7efc7-d063-44b8-bdee-f23357f89f01" | "25a7efc7-d063-44b8-bdee-f23357f89f01" | "localhost:7689" | "Enabled" | "Available" | ["system",  "neo4j"] |
| "782f0ee2-5474-4250-b905-4cd8b8f586ba" | "782f0ee2-5474-4250-b905-4cd8b8f586ba" | "localhost:7688" | "Enabled" | "Available" | ["system",  "neo4j"] |
| "8512c9b9-d9e8-48e6-b037-b15b0004ca18" | "8512c9b9-d9e8-48e6-b037-b15b0004ca18" | "localhost:7687" | "Enabled" | "Available" | ["system",  "neo4j"] |
+-----------------------------------------------------------------------------------------------------------------------------------------------------+

    In this case, the address for server01 is localhost:7687 and thus, the server ID is 8512c9b9-d9e8-48e6-b037-b15b0004ca18.

  3. On one of the servers, use the system database and create the database foo using the server ID of server01. The topology of foo is stored in the system database and when you create it, it is allocated according to the default topology (which can be shown with CALL dbms.showTopologyGraphConfig). This may be different from the topology of foo when it was backed up. If you want to ensure a certain allocation across the cluster, you can specify the desired topology with the TOPOLOGY clause in the CREATE DATABASE command. See CREATE DATABASE for more information.

    CREATE DATABASE foo
TOPOLOGY [desired number of primaries] PRIMARIES [desired number of secondaries] SECONDARIES
OPTIONS {existingData: 'use', existingDataSeedInstance: '8512c9b9-d9e8-48e6-b037-b15b0004ca18'};

  4. Verify that the foo database is online on the desired number of servers, in the desired roles. If the foo database is of considerable size, the execution of the command can take some time.

    SHOW DATABASE foo;
    +------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| name  | type       | aliases | access       | address          | role      | writer | requestedStatus | currentStatus | statusMessage | default | home  | constituents |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| "foo" | "standard" | []      | "read-write" | "localhost:7687" | "primary" | FALSE  | "online"        | "online"      | ""            | FALSE   | FALSE | []           |
| "foo" | "standard" | []      | "read-write" | "localhost:7688" | "primary" | FALSE  | "online"        | "online"      | ""            | FALSE   | FALSE | []           |
| "foo" | "standard" | []      | "read-write" | "localhost:7689" | "primary" | TRUE   | "online"        | "online"      | ""            | FALSE   | FALSE | []           |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

9 rows available after 3 ms, consumed after another 1 ms

Seed from URI

This method seeds all servers with an identical seed from an external source, specified by the URI. The seed can be either a backup or a dump from an existing database. The sources of seeds are called seed providers.

The mechanism is pluggable, allowing new sources of seeds to be supported (see Java Reference → Implement custom seed providers for more information). The product has built-in support for seed from a mounted file system (file), FTP server, HTTP/HTTPS server and Amazon S3.

S3 is supported by default, but the other providers require configuration of dbms.databases.seed_from_uri_providers. Neo4j 5 comes bundled with necessary libraries for AWS S3 connectivity and thus, aws cli is not required.

The URLConnectionSeedProvider supports the following:

  • file:

  • ftp:

  • http:

  • https:

  • URIs

Accordingly, the S3SeedProviders supports:

  • S3:

  • URIs

The URI of the seed is specified when the CREATE DATABASE command is issued:

CREATE DATABASE foo OPTIONS {existingData: 'use', seedURI:'s3://myBucket/myBackup.backup'}

Download and validation of the seed is only performed as the new database is started. If it fails, the database is not available and it has the statusMessage: Unable to start database of the SHOW DATABASES command.

neo4j@neo4j> SHOW DATABASES;
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| name    | type       | aliases | access       | address          | role      | writer | requestedStatus | currentStatus | statusMessage                                            | default | home  | constituents |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| "seed3" | "standard" | []      | "read-write" | "localhost:7682" | "unknown" | FALSE  | "online"        | "offline"     | "Unable to start database `DatabaseId{3fe1a59b[seed3]}`" | FALSE   | FALSE | []           |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

To determine the cause of the problem, it is recommended to look at the debug.log.

Certain seed providers, such as S3, may require additional configuration. This is specified with the seedConfig option. This option expects a comma-separated list of configurations. Each configuration value is specified as a name followed by = and the value, as such:

CREATE DATABASE foo OPTIONS { existingData: 'use', seedURI: 's3:/myBucket/myBackup,backup’, seedConfig: 'region=eu-west-1' }

The available configuration options are:

  • file:

  • ftp:

  • http:

  • https:

  • s3: - use this to specify the path to your S3 bucket. For example, seedURI: 's3:/myBucket/myBackup,backup'

  • region: - use this together with s3: to set the AWS region that hosts the S3 bucket. For example, seedConfig: 'region=eu-west-1'.

Some seed providers may also want to pass credentials into the provider. These are specified with the seedCredentials option. Seed credentials are securely passed from the Cypher command to each server hosting the database. For this to work, Neo4j on each server in the cluster must be configured with identical keystores. This is identical to the configuration required by remote aliases, see Configuration of DBMS with remote database alias. If this configuration is not performed, the seedCredential option fails.

For example, in the case of S3SeedProvider(the default provider), seedCredentials: [accessKey];[secretKey] where accessKey and secretKey are provided by AWS.

Seed provider reference

URL scheme Seed provider URI example

file:

URLConnectionSeedProvider

file:/tmp/backup1.backup

ftp:

URLConnectionSeedProvider

ftp:://myftp.com/backups/backup1.backup

http:

URLConnectionSeedProvider

http://myhttp.com/backups/backup1.backup

https:

URLConnectionSeedProvider

https://myhttp.com/backups/backup1.backup

S3:

S3SeedProvider

s3://mybucket/backups/backup1.backup

Controlling locations with allowed/denied databases

A database can by default be allocated to run on any server in a cluster. However, it is possible to constrain the servers that specific databases are hosted on. This is done with ENABLE SERVER and ALTER SERVER, described in Managing servers in a cluster. The following options are available:

  • allowedDatabases - a set of databases that are allowed to be hosted on a server.

  • deniedDatabases - a set of databases that are denied to be hosted on a server. Allowed and denied are mutually exclusive.

  • modeConstraint - controls in what mode (primary, secondary, or none) databases can be hosted on a server. If not set, there are no mode constraints on the server.

Change the default database

You can use the procedure dbms.setDefaultDatabase("newDefaultDatabaseName") to change the default database for a DBMS.

  1. Ensure that the database to be set as default exists, otherwise create it using the command CREATE DATABASE <database-name>.

  2. Show the name and status of the current default database by using the command SHOW DEFAULT DATABASE.

  3. Stop the current default database using the command STOP DATABASE <database-name>.

  4. Run CALL dbms.setDefaultDatabase("newDefaultDatabaseName") against the system database to set the new default database.

  5. Optionally, you can start the previous default database as non-default by using START DATABASE <database-name>.

Be aware that the automatically created initial default database may have a different topology to the default configuration values. See Default database in a cluster for more information.

Handling errors

Databases can get into error states. Typically you can observe this with the SHOW DATABASES command, and use the error handling guidance to help.

In more serious cases you may be dealing with a disaster situation, where the whole DBMS may not be responding correctly, or some specific databases cannot be restored without downtime. Refer to the disaster recovery guide for those situations.