Transaction management

This topic describes transactional management and behavior.

To fully maintain data integrity and ensure good transactional behavior, Neo4j DBMS supports the ACID properties:

  • Atomicity — If a part of a transaction fails, the database state is left unchanged.

  • Consistency — Every transaction leaves the database in a consistent state.

  • Isolation — During a transaction, modified data cannot be accessed by other operations.

  • Durability — The DBMS can always recover the results of a committed transaction.

Specifically:

  • All database operations that access the graph, indexes, or schema must be performed in a transaction.

  • The default isolation level is read-committed isolation level.

  • Data retrieved by traversals is not protected from modification by other transactions.

  • Non-repeatable reads may occur (i.e., only write locks are acquired and held until the end of the transaction).

  • One can manually acquire write locks on nodes and relationships to achieve a higher level of isolation — serialization isolation level.

  • Locks are acquired at the Node and Relationship levels.

  • Deadlock detection is built into the core transaction management.

Interaction cycle

There are database operations that must be performed in a transaction to ensure the ACID properties. Specifically, operations that access the graph, indexes, or schema are such operations. Transactions are single-threaded, confined, and independent. Multiple transactions can be started in a single thread and they are independent of each other.

The interaction cycle of working with transactions follows the steps:

  1. Begin a transaction.

  2. Perform database operations.

  3. Commit or roll back the transaction.

It is crucial to finish each transaction. The locks or memory acquired by a transaction are only released upon completion.

The idiomatic use of transactions in Neo4j is to use a try-with-resources statement and declare transaction as one of the resources. Then start the transaction and try to perform graph operations. The last operation in the try block should commit or roll back the transaction, depending on the business logic. In this scenario, try-with-resources is used as a guard against unexpected exceptions and as an additional safety mechanism to ensure that the transaction gets closed no matter what happens inside the statement block. All non-committed transactions will be rolled back as part of resource cleanup at the end of the statement. No resource cleanup is required for a transaction that is explicitly committed or rolled back, and the transaction closure is an empty operation.

All modifications performed in a transaction are kept in memory. This means that very large updates must be split into several transactions to avoid running out of memory.

Isolation levels

Transactions in Neo4j use a read-committed isolation level, which means they see data as soon as it has been committed but cannot see data in other transactions that have not yet been committed. This type of isolation is weaker than serialization but offers significant performance advantages while being sufficient for the overwhelming majority of cases.

In addition, the Neo4j Java API enables explicit locking of nodes and relationships. Using locks allows simulating the effects of higher levels of isolation by obtaining and releasing locks explicitly. For example, if a write lock is taken on a common node or relationship, then all transactions are serialized on that lock — giving the effect of a serialization isolation level.

Lost updates in Cypher

In Cypher it is possible to acquire write locks to simulate improved isolation in some cases. Consider the case where multiple concurrent Cypher queries increment the value of a property. Due to the limitations of the read-committed isolation level, the increments might not result in a deterministic final value. If there is a direct dependency, Cypher automatically acquires a write lock before reading. A direct dependency is when the right-hand side of SET has a dependent property read in the expression or the value of a key-value pair in a literal map.

For example, if you run one of the following queries by one hundred concurrent clients, it will increment the property n.prop to 100 because Cypher will automatically acquire a write lock.

Example 1. Cypher can acquire a write lock

The following example requires a write lock, and Cypher automatically acquires one:

MATCH (n:Example {id: 42})
SET n.prop = n.prop + 1
Example 2. Cypher can acquire a write lock

This example also requires a write lock, and Cypher automatically acquires one:

MATCH (n)
SET n += {prop: n.prop + 1}

Due to the complexity of determining such a dependency in the general case, Cypher does not cover all cases. If you run one of the following queries concurrently 100 times, the final value of n.prop will most probably be less than 100.

Example 3. Complex Cypher

Variable depending on results from reading the property in an earlier statement:

MATCH (n)
WITH n.prop AS p
// ... operations depending on p, producing k
SET n.prop = k + 1
Example 4. Complex Cypher

Circular dependency between properties read and written in the same query:

MATCH (n)
SET n += {propA: n.propB + 1, propB: n.propA + 1}

To ensure deterministic behavior also in the more complex cases, it is necessary to explicitly acquire a write lock on the node in question. In Cypher there is no explicit support for this, but it is possible to work around this limitation by writing to a temporary property.

Example 5. Explicitly acquire a write lock

This example acquires a write lock for the node by writing to a dummy property before reading the requested value:

MATCH (n:Example {id: 42})
SET n._LOCK_ = true
WITH n.prop AS p
// ... operations depending on p, producing k
SET n.prop = k + 1
REMOVE n._LOCK_

The existence of the SET n._LOCK_ statement before the read of the n.prop read ensures the lock is acquired before the read action, and no updates are lost due to enforced serialization of all concurrent queries on that specific node.

Default locking behavior

  • When adding, changing, or removing a property on a node or relationship, a write lock is taken on the specific node or relationship.

  • When creating or deleting a node a write lock is taken for the specific node.

  • When creating or deleting a relationship a write lock is taken on the specific relationship and both its nodes.

The locks are added to the transaction and released when the transaction finishes.

Deadlocks

Since locks are used, deadlocks can happen. Neo4j, however, detects any deadlock (caused by acquiring a lock) before they happen and throws an exception. The transaction is marked for rollback before the exception is thrown. All locks acquired by the transaction are still held but will be released when the transaction finishes. Once the locks are released, other transactions that were waiting for locks held by the transaction causing the deadlock can proceed. You can then retry the work performed by the transaction causing the deadlock if needed.

Experiencing frequent deadlocks is an indication of concurrent write requests happening in such a way that it is not possible to execute them while at the same time living up to the intended isolation and consistency. The solution is to make sure concurrent updates happen reasonably. For example, given two specific nodes (A and B), adding or deleting relationships to both these nodes in random order for each transaction results in deadlocks when two or more transactions do that concurrently. One option is to make sure that updates always happen in the same order (first A then B). Another option is to make sure that each thread/transaction does not have any conflicting writes to a node or relationship as some other concurrent transaction. This can, for example, be achieved by letting a single thread do all updates of a specific type.

Deadlocks caused by the use of other synchronization than the locks managed by Neo4j can still happen. Since all operations in the Neo4j API are thread-safe unless specified otherwise, there is no need for external synchronization. Other code that requires synchronization should be synchronized in such a way that it never performs any Neo4j operation in the synchronized block.

Deadlock handling an example

The following is an example of how deadlocks can be handled in procedures, server extensions, or when using Neo4j embedded.

The full source code used for the code snippet can be found in DeadlockDocTest.java.

When dealing with deadlocks in code, there are several issues you may want to address:

  • Only do a limited amount of retries, and fail if a threshold is reached.

  • Pause between each attempt to allow the other transaction to finish before trying again.

  • A retry loop can be useful not only for deadlocks but for other types of transient errors as well.

Below is an example that shows how this can be implemented.

Example 6. Handling deadlocks using a retry loop

This example shows how to use a retry loop for handling deadlocks:

Throwable txEx = null;
int RETRIES = 5;
int BACKOFF = 3000;
for ( int i = 0; i < RETRIES; i++ )
{
    try ( Transaction tx = databaseService.beginTx() )
    {
        Object result = doStuff(tx);
        tx.commit();
        return result;
    }
    catch ( Throwable ex )
    {
        txEx = ex;

        // Add whatever exceptions to retry on here
        if ( !(ex instanceof DeadlockDetectedException) )
        {
            break;
        }
    }

    // Wait so that we don't immediately get into the same deadlock
    if ( i < RETRIES - 1 )
    {
        try
        {
            Thread.sleep( BACKOFF );
        }
        catch ( InterruptedException e )
        {
            throw new TransactionFailureException( "Interrupted", e );
        }
    }
}

if ( txEx instanceof TransactionFailureException )
{
    throw ((TransactionFailureException) txEx);
}
else if ( txEx instanceof Error )
{
    throw ((Error) txEx);
}
else
{
    throw ((RuntimeException) txEx);
}

Delete semantics

When deleting a node or a relationship all properties for that entity will be automatically removed but the relationships of a node will not be removed. Neo4j enforces a constraint (upon commit) that all relationships must have a valid start node and end node. In effect, this means that trying to delete a node that still has relationships attached to it will throw an exception upon commit. It is, however, possible to choose in which order to delete the node and the attached relationships as long as no relationships exist when the transaction is committed.

The delete semantics can be summarized as follows:

  • All properties of a node or relationship will be removed when it is deleted.

  • A deleted node cannot have any attached relationships when the transaction commits.

  • It is possible to acquire a reference to a deleted relationship or node that has not yet been committed.

  • Any write operation on a node or relationship after it has been deleted (but not yet committed) will throw an exception.

  • Trying to acquire a new or work with an old reference to a deleted node or relationship after commit, will throw an exception.

Creating unique nodes

In many use cases, a certain level of uniqueness is desired among entities. For example, only one user with a certain email address may exist in a system. If multiple concurrent threads naively try to create the user, duplicates will be created.

The following are the main strategies for ensuring uniqueness, and they all work across cluster and single-instance deployments.

Single thread

By using a single thread, no two threads even try to create a particular entity simultaneously. In a cluster, an external single-threaded client can perform the operations.

Get or create

Defining a uniqueness constraint and using the Cypher MERGE clause is the most efficient way to get or create a unique node. See Unique nodes for more information.

Transaction events

A neo4j.org.graphdb.event.TransactionEventListener can be registered to receive Neo4j database transaction events. Once it has been registered at a org.neo4j.dbms.api.DatabaseManagementService instance, it receives transaction events for the database with which it was registered. Listeners get notified about transactions that have performed any write operation, and that will be committed. If Transaction#commit() has not been called, or the transaction was rolled back with Transaction#rollback(), it will be rolled back and no events are sent to the listener.

Before a transaction is committed, the listeners' beforeCommit method is called with the entire diff of modifications made in the transaction. At this point the transaction is still running, so changes can still be made. The method may also throw an exception, which prevents the transaction from being committed. If the transaction is rolled back, a call to the listener’s afterRollback method will follow.

The order in which listeners are executed is undefined — there is no guarantee that changes made by one listener will be seen by other listeners.

If beforeCommit is successfully executed in all registered listeners, the transaction is committed and the afterCommit method is called with the same transaction data. This call also includes the object returned from beforeCommit.

In afterCommit, the transaction is closed and access to anything outside org.neo4j.graphdb.event.TransactionData requires a new transaction to be opened. A neo4j.org.graphdb.event.TransactionEventListener gets notified about transactions that have any changes accessible via org.neo4j.graphdb.event.TransactionData. Some indexing and schema changes will not trigger these events.

The following example shows how to register a listener for a specific database and perform basic operations on top of the transaction change set.

The full source code used for the code snippet can be found in TransactionEventListenerExample.java.
Example 7. TransactionEventListener

Register a transaction event listener and inspect the change set:

public static void main( String[] args ) throws IOException
{
    FileUtils.deleteDirectory( HOME_DIRECTORY );
    var managementService = new DatabaseManagementServiceBuilder( HOME_DIRECTORY ).build();
    var database = managementService.database( DEFAULT_DATABASE_NAME );

    var countingListener = new CountingTransactionEventListener();
    managementService.registerTransactionEventListener( DEFAULT_DATABASE_NAME, countingListener );

    var connectionType = RelationshipType.withName( "CONNECTS" );
    try ( var transaction = database.beginTx() )
    {
        var startNode = transaction.createNode();
        var endNode = transaction.createNode();
        startNode.createRelationshipTo( endNode, connectionType );
        transaction.commit();
    }
}

private static class CountingTransactionEventListener implements TransactionEventListener<CreatedEntitiesCounter>
{
    @Override
    public CreatedEntitiesCounter beforeCommit( TransactionData data, Transaction transaction, GraphDatabaseService databaseService ) throws Exception
    {
        return new CreatedEntitiesCounter( size( data.createdNodes() ), size( data.createdRelationships() ) );
    }

    @Override
    public void afterCommit( TransactionData data, CreatedEntitiesCounter entitiesCounter, GraphDatabaseService databaseService )
    {
        System.out.println( "Number of created nodes: " + entitiesCounter.getCreatedNodes() );
        System.out.println( "Number of created relationships: " + entitiesCounter.getCreatedRelationships() );
    }

    @Override
    public void afterRollback( TransactionData data, CreatedEntitiesCounter state, GraphDatabaseService databaseService )
    {
    }
}

private static class CreatedEntitiesCounter
{
    private final long createdNodes;
    private final long createdRelationships;

    public CreatedEntitiesCounter( long createdNodes, long createdRelationships )
    {
        this.createdNodes = createdNodes;
        this.createdRelationships = createdRelationships;
    }

    public long getCreatedNodes()
    {
        return createdNodes;
    }

    public long getCreatedRelationships()
    {
        return createdRelationships;
    }
}