Troubleshooting

The following information can help you diagnose and correct a problem.

Locate and investigate problems with the Neo4j Helm chart

The rollout of the Neo4j Helm chart in Kubernetes can be thought of in these approximate steps:

Neo4j Pod is created.
Neo4j Pod is scheduled to run on a specific Kubernetes Node.
All Containers in the Neo4j Pod are created.
InitContainers in the Neo4j Pod is run.
Containers in the Neo4j Pod are run.
Startup and Readiness probes are checked.

After all these steps are completed successfully, the Neo4j StatefulSet, Pod, and Services must be in a ready state. You should be able to connect to and use your Neo4j database.

If the Neo4j Helm chart is installed successfully, but Neo4j is not starting and reaching a ready state in Kubernetes, then troubleshooting has two steps:

Check the state of resources in Kubernetes using kubectl get commands. This will identify which step has failed.
Collect the information relevant to that step.

Depending on the failed step, you can collect information from Kubernetes (e.g., using kubectl describe) and from the Neo4j process (e.g., checking the Neo4j debug log).

The following table provides simple steps to get started investigating problems with the Neo4j Helm chart rollout. For more information on how to debug applications in Kubernetes, see the Kubernetes documentation.

Table 1. Investigating problems with the Neo4j Helm chart rollout
Step	Diagnosis	Further investigation
Neo4j Pod created	If `kubectl get pod <release-name>-0` does not return a single Pod result, there is a problem with the pod creation.	Describe the Neo4j StatefulSet — check the output of `kubectl describe statefulset <release-name>`.
Neo4j Pod scheduled	If the state, shown in `kubectl get pod <release-name>-0`, is stuck in `Pending`, there is a problem with pod scheduling.	Describe the Neo4j Pod `kubectl describe pod <release-name>-0` and check the output.
Containers in the Neo4j Pod created	If the state, shown in `kubectl get pod <release-name>-0`, is stuck in `Waiting`, there is a problem with creating or starting containers.	Describe the Neo4j Pod — check the output of `kubectl describe pod <release-name>-0`, paying particular attention to `Events`.
InitContainers in the Neo4j Pod	If the state, shown in `kubectl get pod <release-name>-0`, is stuck in `Init:` (e.g.,`Init:CrashLoopBackOff`, `Init:Error` etc.), there is a problem with `InitContainers`. Note that if the pod `Status` is `PodInitializing` or `Running`, then `InitContainers` have already finished successfully.	Describe the Neo4j Pod — check the output of `kubectl describe pod <release-name>-0`, paying particular attention to `InitContainer` (note the `InitContainer` names) and `Events`. Fetch `InitContainer` logs using `kubectl logs <pod-name> -c <init-container-name>`.
Containers in the Neo4j Pod running	If the state, shown in `kubectl get pod <release-name>-0`, does NOT match any of the states listed above, but the Pod still does not reach `Running`, then there is a problem running containers in the Neo4j Pod.	Describe the Neo4j Pod — check the output of `kubectl describe pod <release-name>-0`, paying particular attention to the `Container` state (note the `Container` names) and `Events`. Fetch `Container` logs using `kubectl logs <pod-name> -c <init-container-name>`. If the Neo4j `Container` is starting but exits unexpectedly (e.g., the state is `CrashLoopBackOff`), follow the instructions for Neo4j crashes or restarts unexpectedly.
Startup and Readiness Probes	If the state, shown in `kubectl get pod <release-name>-0`, is `Running`, but the pod does not become `ready`, there is a problem with `Startup` or `Readiness` probes.	Describe the Neo4j Pod — check the output of `kubectl describe pod <release-name>-0`, paying particular attention to `Events` and probes. Check the pod log `kubectl logs <release-name>-0`, the Neo4j log `kubectl exec <release-name>-0 -- tail -n 100 /logs/neo4j.log`, and the Neo4j debug log `kubectl exec <release-name>-0 -- tail -n 500 /logs/debug.log`.

Neo4j crashes or restarts unexpectedly

If the Neo4j Pod starts but then crashes or restarts unexpectedly, there are a range of possible causes. Known causes include:

An invalid or incorrect configuration of Neo4j, causing it to shut down shortly after the container is started.
The Neo4j Java process runs out of memory and exits with OutOfMemoryException.
There has been some disruption affecting the Kubernetes Node where the Neo4j Pod is scheduled, e.g., it is being shut drained or has shut down.
Containers in the Neo4j Pod are shut down by the operating system for using more memory than the resource limit configured for the container (OOMKilled).
Very long Garbage Collection pauses cause the Neo4j Pod LivenessProbe to fail, causing Kubernetes to restart Neo4j.

OOMKILLED and OutOfMemoryException appear very similar, but they appear in different places and have different fixes. It is important to be aware of this and be sure of what you are dealing with.

Here are some checks to help troubleshoot crashes and unexpected restarts:

Describe the Neo4j Pod

Use kubectl to describe the Neo4j Pod:

kubectl describe pod <release-name>-0

Check the Neo4j Container state

Check the State and Last State of the container. This shows how the Last State of a container that has restarted after being OOMKilled appears:

$ kubectl describe pod neo4j-0

State:          Running
  Started:      Mon, 1 Jan 2021 00:02:00 +0000
Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137
  Started:      Mon, 1 Jan 2021 00:00:00 +0000
  Finished:     Mon, 1 Jan 2021 00:01:00 +0000

Exit Code: 137 is indicative of OOMKilled if it appears here or in other logs, even if the "OOMKilled" string is not present.

Check recent `Events`

The kubectl describe output shows older events at the top and more recent events at the bottom. Generally, you can ignore older events.

A Killing event that shows that the Neo4j container was killed by the Kubernetes kubelet:

$ kubectl describe pod neo4j-0

Events:
Type    Reason       Age      From                  Message
----    ------       ----     ----                  -------
Normal  Scheduled    6m30s    default-scheduler     Successfully assigned default/neo4j-0 to k8s-node-a
...
Normal  Killing        56s    kubelet, k8s-node-a   Killing container with id docker://neo4j-0-neo4j:Need to kill Pod

It is not clear from this event log alone why Kubernetes decided that the Neo4j container should be killed.

The next steps in this example could be to check:

if the container was OOMKilled.
if the container failed Liveness or Startup probes.
investigate the node to see if there was some reason why it might kill the container, e.g.,kubectl describe node <k8s node>.

Check Neo4j logs and metrics

The Neo4j Helm chart configures Neo4j to persist logs and metrics on provided volumes. If no volume is explicitly configured for logs or metrics, they are stored persistently on the Neo4j data volume. This ensures that the logs and metrics outputs from a Neo4j instance that crashes or shuts down unexpectedly are preserved.

Collect data from a running Neo4j Pod

Download all Neo4j logs from a pod using kubectl cp commands:
```
kubectl cp <neo4j-pod-name>:/logs neo4j-logs/
```
If CSV metrics collection is enabled for Neo4j (the default), download all Neo4j metrics from a pod using:
```
kubectl cp <neo4j-pod-name>:/metrics neo4j-metrics/
```

Collect data from a not running Neo4j Pod

If the Neo4j Pod is not running or is crashing so frequently that kubectl cp is not feasible, the Neo4j deployment should be put into offline maintenance mode to collect logs and metrics.

Check container logs

The logs for the main Neo4j DBMS process are persisted to disk and can be accessed as described in Check Neo4j logs and metrics. However, the logs for Neo4j startup and logs for other Containers in the Neo4j Pod are sent to the container’s stdout and stderr streams. These container logs can be viewed using kubectl logs <pod name> -c <container name>.

Unfortunately, if the container has restarted following a crash or unexpected shutdown, typically, kubectl logs shows the logs for the new container instance (following the restart), and the logs for the previous container instance (the instance that shut down unexpectedly) are not available via kubectl logs.

To capture the logs for a crashing container, you can try:

View the container logs in a log collector/aggregator that is connected to your Kubernetes cluster, e.g., Stackdriver, Cloudwatch Logs, Logstash, etc. If you are using a managed Kubernetes platform, this is usually enabled by default.
Use kubectl logs --follow to stream the logs of a running container until it crashes again.

Glossary

allocator: A component in the cluster that allocates databases to servers according to the topology constraints specified and an allocation strategy.
asynchronous replication: Asynchronous replication is used by secondary copies to poll for new transactions, which means they cannot be guaranteed to have received the most recent transactions. This enables efficient scale-out of read-performance.
Aura instance: A fully-managed DBMS represented by a single instance ID, that is running in the Neo4j Aura cloud.
auto-commit transaction: An automatically committed transaction that contains a single query.
Bolt protocol: Bolt is a protocol used for interaction between Neo4j instances and drivers.
bookmark: A marker the client can request from the cluster to ensure that it is able to read its own writes so that the application’s state is consistent and only databases that have a copy of the bookmark are permitted to respond.
category (Bloom): A category is based on a node label and is defined in a Perspective as a way of visually distinguishing nodes with the same label(s).
causal consistency: All servers in a cluster agree on the order in which transactions take place. The position of a server on the causal chain can be guaranteed using a bookmark.
cluster: A Neo4j DBMS that spans multiple servers working together to increase fault tolerance and/or read scalability. Databases on a cluster may be configured to replicate across servers in the cluster thus achieving read scalability or high availability.
client application: Software that interacts with a Neo4j server.
commit: A commit is the successful completion of a transaction, which ensures durability of any changes made. For more details, visit Operations Manual → Transaction management.
composite database: Composite databases are the means to access partitioned graph data with a single Cypher query.
constraint: Constraints are sets of data modeling rules that ensure the data is consistent and reliable.
Cypher®: Neo4j’s graph query language.
data model: A data model defines how information is organized in a database. A good data model will make querying and understanding your data easier. In Neo4j, the data models have a graph structure.
database: A database is a container used by the DBMS to manage and store graph data. The physical structure of data is controlled by the database.
database vs graph: Databases are the physical containers of graph data. Graphs are the logical structure of data in Neo4j.
Database Management System: Database Management System, or DBMS, capable of managing multiple databases. A DBMS may run on a single server, or span several servers configured as a cluster.
database schema: The prescribed property existence and datatypes for nodes and relationships.
deallocate: An act of removing a database from a server or a server from a cluster without loss of data or reduced fault tolerance.
degree (of a node): The number of relationships of a specific node; loops are counted twice.
disaster recovery: A manual intervention to restore availability of a cluster, or databases within a cluster.
driver: A software library that provides access to Neo4j from a particular programming language.
election: In the event that the Raft leader becomes unresponsive, followers automatically trigger an election and vote for a new leader.
entity: A node or a relationship.
expression (Cypher): A component of a Cypher query which produces values. It may be used in projections, as a predicate, or when setting properties on graph elements.
fabric: Fabric is the architectural design of a unified system that provides a single access point to local or distributed graph data.
fault tolerance: A guarantee that a cluster can maintain a database’s persistence and availability in the event of one or more servers failing.
follower: A primary copy of a database acting as a follower, receives and acknowledges synchronous writes from the leader.
Generative AI (GenAI): A type of artificial intelligence (AI) system that generates text, images, or other media in response to prompts.
graph: A logical representation of a set of nodes where some pairs are connected by relationships.
index: Data structure that improves read performance of a database.
knowledge graph: A specific type of graph that has an organizing principle so that a user (or a computer system) can reason about the underlying data. The organizing principle provides an additional layer of structure that adds context to support knowledge discovery.
label: Marks a node as a member of a named and indexed subset. A node may be assigned zero or more labels.
leader: A single primary copy of a database is designated as the leader. It receives all write transactions from clients and replicates writes synchronously to followers and asynchronously to secondary copies of the database.
main database: In terms of Neo4j Enterprise Studio, the database(s) containing the user’s data. Can exist in the same Neo4j deployment as the tool asset database.
motif: A description of a specific pattern within a graph.
node: A node represents an entity or discrete object in your graph data model. Nodes can be connected by relationships, hold data in properties, and are classified by labels.
operator: A symbol representing a mathematical or logical operation.
parameter: Named value provided when running a Cypher statement.
path: A sequence of nodes and the relationships connecting them, that does not contain duplicate relationships. Several paths can match a pattern.
pattern: A specific arrangement of nodes and relationships that can be matched in a graph. A pattern follows a motif.
perspective (Bloom): A Perspective defines a certain business view or domain that can be found in the target Neo4j graph. A single Neo4j graph can be viewed through different Perspectives, each tailored for a different business purpose.
primary: A copy of the database that is able to process write transactions and is eligible to be elected as a leader. It participates in fault tolerant writes as it is part of the majority required to acknowledge and commit write transactions.
primary vs secondary: In a cluster, databases can operate in either primary or secondary mode. Primary databases are able to process write and read transactions, ensuring fault tolerance. Secondary databases are replicated asynchronously from primaries, and their main purpose is to provide read scaling within the cluster.
project (Aura): An isolated environment in the unified Aura console that contains its own database instances, configurations, and resources. Preceded by tenant in the classic Aura console.
property: Properties are key-value pairs that are used for storing data on nodes and relationships.
query (Cypher): A statement that retrieves or writes information to a database.
Raft group: A group of servers that are participating in hosting a particular database in primary mode.
Raft group member: A server that is participating in a Raft group. A server can be a member of one or more groups.
Raft log: A shared log between all Raft group members that is guaranteed to be consistently updated and viewed by those members. The log contains both database data and operational state of the Raft group.
Raft protocol: The networking mechanism that enables a database to replicate its data across multiple servers to give high availability for accessing the data and high durability to the data stored.
read scaling: Distributing query load by creating additional database copies hosted in secondary mode (read-only).
relationship: A relationship represents a connection between nodes in your graph data model. Relationships connect a source node to a target node, hold data in properties, and are classified by type.
secondary: An asynchronously replicated copy of the database that provides read scaling within the cluster.
seed: A seed is a database dump or a full backup used to create a database on a cluster. This is sometimes called seeding.
server: A physical machine, a virtual machine, or a container running an instance of Neo4j. Servers can be standalone or part of a cluster.
session: A causally linked sequence of transactions.
session consistency: An alternative name for Neo4j’s causal consistency.
standalone: A single server running Neo4j and not part of a cluster.
synchronous replication: Synchronous replication requires the leader primary to replicate a transaction and block the commit until a quorum of the follower primaries acknowledges that the transaction is successfully replicated. Once the transaction is replicated, the commit is allowed to proceed. This ensures data durability and consistency within the cluster.
system database: A database used by Neo4j to store system information.
tenant (Aura): An isolated environment in the classic Aura console that contains its own database instances, configurations, and resources. Replaced by project in the unified Aura console.
tool asset database: In terms of Neo4j Enterprise Studio, the database where tools' assets are stored. This can be in the same Neo4j deployment as the main database(s) or in a separate deployment.
topology: A configuration that describes how the copies of a database should be spread across the servers in a cluster, see primary mode and secondary mode.
transaction: A transaction comprises a unit of work performed against a database. It is treated in a coherent and reliable way, independent of other transactions. Transactions comply with the ACID consistency model (atomic, consistent, isolated, and durable).