This section describes how to deploy a new Neo4j Causal Cluster.
In this section we will learn how to set up a brand new Causal Cluster consisting of three Core instances; the minimum number of servers needed for a fault-tolerant Core cluster. We then learn how to add more Core servers as well as Read Replicas. From this basic pattern, we can extend what we have learned here to create any sized cluster.
The following subjects are described:
For a description of the clustering architecture and cluster concepts encountered here, please refer to Introduction to Causal Clustering. We will not describe how to import data from an existing Neo4j instance; for help on using an existing Neo4j database to seed a new Causal Cluster, please see Seed a Causal Cluster.
If you want to try to set up a Causal Cluster on your local machine, refer to Section B.1, “Set up a local Causal Cluster”.
The instructions in this section assume that you have already downloaded the appropriate version of Neo4j Enterprise Edition from the Neo4j download site and have installed it on the servers that will make up the cluster.
When running Neo4j on three separate machines, the most basic configuration requires changing five different parameters. The settings below are located in neo4j.conf, under the header "Network connector configuration".
0.0.0.0which allows Neo4j to bind to any and all network interfaces. Uncomment the line
The settings below are located in neo4j.conf under the header "Causal Clustering Configuration".
READ_REPLICA. On three of our instances, we will use the
COREmode. Uncomment the line:
causal_clustering.expected_core_cluster_size=3will specify that the cluster has three Core members.
:5000. You should include the address of the local machine in this setting as well.
Apply these settings to each configuration file on each instance. The values will be the same for each.
Start the Neo4j servers as usual. Note that the startup order does not matter.
server-1$ ./bin/neo4j start
server-2$ ./bin/neo4j start
server-3$ ./bin/neo4j start
If you want to follow along with the startup of a server you can follow the messages in neo4j.log. While an instance is joining the cluster, the server may appear unavailable.
Now you can access the three servers and check their status.
Open the locations below in a web browser and issue the following query:
This will show you the status of the cluster and information about each member of the cluster.
You now have a Neo4j Causal Cluster of three instances running.
Adding instances to the Core cluster is simply a matter of starting a new database server with the appropriate configuration as described in Section 126.96.36.199, “Configure a Core-only cluster”. Following those instructions, we need to change neo4j.conf to reflect the new Core Server’s desired configuration like so:
causal_clustering.initial_discovery_membersto be identical to the corresponding parameter on the existing Core servers.
Once we’ve done that, start the server and the new server will integrate itself with the existing cluster. In the case where an instance is joining a cluster with lots of data, it may take a number of minutes for the new instance to download the data from the cluster and become available. When the server has copied over the graph data from its peers it will become available.
Initial Read replica configuration is provided similarly to Core Servers via neo4j.conf. Since Read Replicas do not participate in cluster quorum decisions, their configuration is shorter. They simply need to know the addresses of some of the Core Servers which they can bind to in order to run the discovery protocol (see: Section 188.8.131.52, “Discovery protocol” for details). Once it has completed the initial discovery the Read replica becomes aware of the available Core Servers and can choose an appropriate one from which to catch up (see: Section 184.108.40.206, “Catchup protocol” for how that happens).
In the neo4j.conf file in the section "Causal Clustering Configuration", the following settings need to be changed:
causal_clustering.initial_discovery_membersto be identical to the corresponding parameter on the Core servers.
Should we wish to downgrade a Core server to a standalone instance, it can be done with the following command on the shutdown database:
neo4j-admin unbind [--database=<name>]
Once a server has been unbound from a cluster, the store files are equivalent to a Neo4j standalone instance.
From this point those files could be used to run a standalone instance by restarting it in
SINGLE mode, or be used to seed a new cluster.
The on-disk state of Core server instances is different to that of standalone server instances. It is important to understand that once an instance unbinds from a cluster, it cannot be re-integrated with that cluster. This is because both the cluster and the single instance are now separate databases with different and irreconcilable writes having been applied to them. Technically the cluster will have written entries to its Raft log, whilst the standalone instance will have written directly to the transaction log (since there is no Raft log in a standalone instance).
If we try to reinsert the standalone instance into the cluster, then the logs and stores will mismatch. Operators should not try to merge standalone databases into the cluster in the optimistic hope that their data will become replicated. That will not happen and will likely lead to unpredictable cluster behavior.
The Core servers in a Causal Cluster use the Raft protocol to ensure consistency and safety. An implementation detail of Raft is that it uses a Leader role to impose an ordering on an underlying log with other instances acting as Followers that replicate the leader’s state. In Neo4j terms this means writes to the database are ordered by the Core instance currently playing the leader role.
In some situations, operators might want to actively prevent some instances from taking on the leader role.
For example in a multi-data center scenario, we might want to ensure that the leader remains in a favored geographic location
for performance or governance reasons.
In Neo4j Causal Clustering we can configure instances to refuse to become leader, which is equivalent to always remaining a follower.
This is done by configuring the setting
It is not generally advisable to use this option, since the first priority in the cluster is to maximize availability.
The highest availability stems from having any healthy instance take leadership of the cluster on pathological failure.
Despite superficial similarities, a non-leader capable Core instance is not the same as a Read Replica. Read Replicas do not participate in transaction processing, nor are they permitted to be involved in cluster topology management.
Conversely, a follower-only Core instance will still process transactions and cluster membership requests as per Raft to ensure consistency and safety.