Deploy a cluster

This section describes how to deploy different Neo4j Cluster topologies.

1. Introduction

This section describes how to set up a new cluster. Two scenarios are covered:

  1. A four-instance cluster with one Single instance as Primary server and three Read Replica instances as Secondary servers. This scenario is ideal for reporting and analytical workloads.

  2. A three-instance cluster with three Core instances as Primary servers. This scenario is ideal for transactional workloads.

Additionally, the process to turn a Secondary server into a standalone instance by detaching it from an existing cluster is also described.

2. Configure a cluster with Single and Read Replica instances

The following configuration settings are important to consider when deploying a new cluster with a Single instance as a Primary server. See also Settings reference for more detailed descriptions and examples.

This configuration is optimized for best scalability and it is recommended to be used for reporting and analytical workloads. Clusters configured in this way do not provide automatic failover and fault tolerance. In case of fault, if a cluster is not supported by appropriate external tooling, data may be lost.

In the current version of Neo4j, the clustering-related parameters use the causal_clustering namespace. This will be replaced with a more suitable namespace in an upcoming release.

Table 1. Important settings for clusters with Single instance as Primary server
Option name Servers Description

dbms.default_advertised_address

All (Primary and Secondary)

The address that other machines are told to connect to. In the typical case, this should be set to the fully qualified domain name or the IP address of this server.

dbms.mode

Primary

The operating mode of the server instance. The Primary server is set as SINGLE.

Secondary

The operating mode of the server instance. The Secondary servers are set as READ_REPLICA.

dbms.clustering.enable=true

Primary

Allows a single instance to form a cluster and is only evaluated when dbms.mode=SINGLE.

causal_clustering.initial_discovery_members

Secondary

This setting needs to be specified on Read Replica instances and contains the network address for at least the primary instance, but can also include Secondary servers. This parameter must be set to the same value on all cluster members. The behavior of this setting can be modified by configuring the setting causal_clustering.discovery_type. This is described in detail in Discovery.

The following example shows how to set up a cluster with a Single instance as Primary server and three Read Replica instances as Secondary servers.

Example 1. Configure a cluster with a Single instance as Primary server

In this example, one Primary server named single.example.com and three Secondary servers, read_replica01.example.com, read_replica02.example.com and read_replica03.example.com are configured. All instances have Neo4j Enterprise Edition installed. To form a cluster, the neo4j.conf needs to be configured on each server. The Primary server, set as Single instance, is configured as such:

neo4j.conf on single.example.com:
dbms.mode=SINGLE
dbms.clustering.enable=true
dbms.default_advertised_address=single.example.com

The neo4j.conf on the Secondary servers, set as Read Replica instances, is identical across all instances:

neo4j.conf on read_replica01.example.com, read_replica02.example.com and read_replica03.example.com:
dbms.mode=READ_REPLICA
dbms.initial_discovery_members=single.example.com:5000
dbms.default_advertised_address=read_replica<xx>.example.com

Once all neo4j.conf files have been configured, the instances can be started and the cluster is ready. After the cluster has started, it is possible to connect to any of the instances and run CALL dbms.cluster.overview() to check the status of the cluster. This shows information about each member of the cluster:

CALL dbms.cluster.overview();
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| id                                     | addresses                                                                          | databases                                       | groups |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| "8e4133d7-4de1-469e-88ac-864571cb0a92" | ["bolt://read_replica1.example.com:7687", "http://read_replica1.example.com:7474"] | {neo4j: "READ_REPLICA", system: "READ_REPLICA"} | []     |
| "eb6a4e88-9a5f-405b-b230-5bbbd681ec9e" | ["bolt://read_replica2.example.com:7687", "http://read_replica2.example.com:7474"] | {neo4j: "READ_REPLICA", system: "READ_REPLICA"} | []     |
| "274e36db-d96f-4736-8a99-68851b1bbb0b" | ["bolt://read_replica3.example.com:7687", "http://read_replica3.example.com:7474"] | {neo4j: "READ_REPLICA", system: "READ_REPLICA"} | []     |
| "6fd05bc6-760e-4644-bf02-05117a5d777d" | ["bolt://single.example.com:7687", "http://single.example.com:7474"]               | {neo4j: "LEADER", system: "LEADER"}             | []     |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
4 rows available after 8 ms, consumed after another 3 ms

3. Configure a cluster with Core instances

The following configuration settings are important to consider when deploying a new cluster with Core instances as Primary servers. See also Settings reference for more detailed descriptions and examples.

This configuration is optimized for fault tolerance, automatic failover and best scalability, and it is recommended to be used for transactional workloads. In many cases and when they are correctly configured, these clusters safeguard data and they do not require any particular external tooling.

Table 2. Important settings for clusters with Core instances as Primary servers
Option name Servers Description

dbms.default_listen_address

All (Primary and Secondary)

The address or network interface this machine uses to listen for incoming messages. Setting this value to 0.0.0.0 makes Neo4j bind to all available network interfaces.

dbms.default_advertised_address

All (Primary and Secondary)

The address that other machines are told to connect to. In the typical case, this should be set to the fully qualified domain name or the IP address of this server.

dbms.mode

Primary

The operating mode of the server instance. The Primary servers are set as CORE.

Secondary

The operating mode of the server instance. The Secondary servers are set as READ_REPLICA.

causal_clustering.minimum_core_cluster_size_at_formation

Primary

The minimum number of Core instances in the cluster at formation. A cluster will not form without the number of Cores defined by this setting, and this should in general be configured to the full and fixed amount.

causal_clustering.minimum_core_cluster_size_at_runtime

Primary

The minimum number of Core instances which will exist in the consensus group.

causal_clustering.initial_discovery_members

All (Primary and Secondary)

The network addresses of an initial set of Core cluster members that are available to bootstrap this Core or Read Replica instance. In the default case, the initial discovery members are given as a comma-separated list of address/port pairs, and the default port for the discovery service is :5000. It is good practice to set this parameter to the same value on all Core Servers.

The behavior of this setting can be modified by configuring the setting causal_clustering.discovery_type. This is described in detail in Discovery.

Listen configuration

Listening on 0.0.0.0 makes the ports publicly available. Make sure you understand the security implications and strongly consider setting up encryption.

The following example shows how to set up a simple cluster with three Core servers:

Example 2. Configure a Core-only cluster

In this example, three Core instances named core01.example.com, core02.example.com and core03.example.com are configured. Neo4j Enterprise Edition is installed on all three servers. They are configured by preparing neo4j.conf on each server. Note that they are all identical, except for the configuration of dbms.default_advertised_address:

neo4j.conf on core01.example.com:
dbms.default_listen_address=0.0.0.0
dbms.default_advertised_address=core01.example.com
dbms.mode=CORE
causal_clustering.initial_discovery_members=core01.example.com:5000,core02.example.com:5000,core03.example.com:5000
neo4j.conf on core02.example.com:
dbms.default_listen_address=0.0.0.0
dbms.default_advertised_address=core02.example.com
dbms.mode=CORE
causal_clustering.initial_discovery_members=core01.example.com:5000,core02.example.com:5000,core03.example.com:5000
neo4j.conf on core03.example.com:
dbms.default_listen_address=0.0.0.0
dbms.default_advertised_address=core03.example.com
dbms.mode=CORE
causal_clustering.initial_discovery_members=core01.example.com:5000,core02.example.com:5000,core03.example.com:5000

The Neo4j servers are ready to be started. The startup order does not matter.

After the cluster has started, it is possible to connect to any of the instances and run CALL dbms.cluster.overview() to check the status of the cluster. This shows information about each member of the cluster:

CALL dbms.cluster.overview();
+----------------------------------------------------------------------------------------------------------------------------------------+
| id                                     | addresses                                  | databases                               | groups |
+----------------------------------------------------------------------------------------------------------------------------------------+
| "8e07406b-90b3-4311-a63f-85c45af63583" | ["bolt://core1:7687", "http://core1:7474"] | {neo4j: "LEADER", system: "FOLLOWER"}   | []     |
| "aeb6debe-d3ea-4644-bd68-304236f3813b" | ["bolt://core3:7687", "http://core3:7474"] | {neo4j: "FOLLOWER", system: "FOLLOWER"} | []     |
| "b99ff25e-dc64-4c9c-8a50-ebc1aa0053cf" | ["bolt://core2:7687", "http://core2:7474"] | {neo4j: "FOLLOWER", system: "LEADER"}   | []     |
+----------------------------------------------------------------------------------------------------------------------------------------+
Startup time

The instance may appear unavailable while it is joining the cluster. If you want to follow along with the startup, you can follow the messages in neo4j.log.

4. Add a Core Server to an existing cluster

Core Servers are added to an existing cluster by starting a new Neo4j instance with the appropriate configuration. The new server joins the existing cluster and becomes available once it has copied the data from its peers. It may take some time for the new instance to perform the copy if the existing cluster contains large amounts of data.

The setting causal_clustering.initial_discovery_members shall be updated on all the servers in the cluster to include the new server.

Example 3. Add a Core Server to an existing cluster

In this example, a Core Server, core04.example.com, is added to the cluster created in Configure a Core-only cluster.

Configure the following entries in neo4j.conf:

neo4j.conf on core04.example.com:
dbms.default_listen_address=0.0.0.0
dbms.default_advertised_address=core04.example.com
dbms.mode=CORE
causal_clustering.minimum_core_cluster_size_at_formation=3
causal_clustering.minimum_core_cluster_size_at_runtime=3
causal_clustering.initial_discovery_members=core01.example.com:5000,core02.example.com:5000,core03.example.com:5000,core04.example.com:5000

Note that the configuration is very similar to that of the previous servers. In this example, the new server is not intended to be a permanent member of the cluster, thus it is not included in causal_clustering.initial_discovery_members on the other Core members of the cluster.

Now start the new Core Server and let it add itself to the existing cluster.

5. Add a Secondary server to an existing cluster

In the 4.3 version of Neo4j, all Secondary servers are Read Replica instances. The initial configuration for Read Replica instances is provided via neo4j.conf, as mentioned above in Configure a cluster with Single and Read Replica instances. Since Read Replicas do not participate in cluster quorum decisions, their configuration is shorter; they only need to know the addresses of at least one primary instance which they can bind to in order to discover the cluster.

It is recommended to specify the addresses for all existing primary instances in a cluster when adding a Read Replica. They can then select an appropriate Primary server from which to copy data.

Example 4. Add a Secondary server to an existing cluster with a Single instance as Primary server

In this example, a Read Replica instance, replica04.example.com, is added to the cluster created in Configure a cluster with a Single instance as Primary server.

Configure the following entries in neo4j.conf:

neo4j.conf on replica04.example.com:
dbms.default_advertised_address=read_replica04.example.com
dbms.mode=READ_REPLICA
causal_clustering.initial_discovery_members=single.example.com:5000

Now start the new Read Replica and let it add itself to the existing cluster.

Example 5. Add a Secondary server to an existing cluster with Core servers as Primary servers

In this example, a Read Replica, replica05.example.com, is added to the cluster created in Configure a Core-only cluster.

Configure the following entries in neo4j.conf:

neo4j.conf on replica05.example.com:
dbms.default_advertised_address=read_replica05.example.com
dbms.mode=READ_REPLICA
causal_clustering.initial_discovery_members=core01.example.com:5000,core02.example.com:5000,core03.example.com:5000

Now start the new Read Replica and let it add itself to the existing cluster.

When adding a Secondary server to an existing cluster, only Primary servers need to be listed in causal_clustering.initial_discovery_members. It is not necessary to include existing Secondary servers, i.e. other Read Replica instances.

6. Detach a Secondary server from an existing cluster

It is possible to turn a Secondary server into a standalone instance that thus contains a snapshot of the data in the cluster. This can, in theory, be done for a Core Server as well, but this is not recommended for performance and safety reasons. As mentioned above, in the 4.3 version of Neo4j, all Secondary servers are Read Replica instances.

Example 6. Detach a Read Replica and turn it into a stand alone instance

In this example, a Read Replica, replica01.example.com, is detached from a cluster. See [causal-clustering-add-read-replica] above on how to add a Read Replica to a cluster.

First, ensure that the Read Replica is up-to-date, then shut it down. Once the Read Replica is shut down, configure the following entry in neo4j.conf:

neo4j.conf on replica01.example.com:
dbms.mode=SINGLE

Start the instance again. It is now a standalone instance containing the same data as the cluster (at the time of shutting down the Read Replica).

There is always a chance that the Read Replica is behind the Core Servers at any time. If a transaction is being processed at the time of the shutdown of the Read Replica, this transaction is eventually reflected in the remaining Cluster, but not on the detached Read Replica. A way to ensure that a Read Replica contains a snapshot of a database in the cluster at a point in time, is to pause the read Replica before shutting it down. See dbms.cluster.readReplicaToggle() for more information.

7. Connect to a Read Replica

It is important to use the correct URI scheme when connecting to a Read Replica since it only allows read sessions/transactions. The following table illustrates the supported URI schemes (the +s indicates that TLS has been configured for the cluster):

SSR disabled SSR enabled

bolt+s://

neo4j+s:// read session/tx

neo4j+s:// write session/tx

neo4j+s:// read session/tx

neo4j+s:// write session/tx

Cypher Shell

Browser

Neo4j Driver

In addition to dbms.routing.enabled=true, the configuration setting dbms.routing.default_router needs to be set to SERVER in order to connect to a Read Replica, where applicable. See Server-side routing for more information.