Transactional and concurrent Cypher ingestion

A Cypher® ingestion statement wrapped in a bare CALL subquery runs inside a single transaction, accumulating uncommitted writes in heap memory until the final commit. On small batches this is not a problem, but on larger ones it can require Neo4j to hold a very large transaction state in memory before writing anything to disk. This can result in:

An OutOfMemoryError that kills the import mid-run.
A transaction that runs for hours and then fails, leaving nothing committed.

Adding IN TRANSACTIONS to the CALL subquery solves this by committing after every N rows. The working set stays small, the heap stays healthy, and a partial failure leaves data that can be resumed from.

Understanding batch sizes

CALL { … } IN TRANSACTIONS by default commits every 1000 rows and clears the transaction state from heap before starting the next batch. Given a subquery that performs 5 write operations per row (e.g. one node write, three property writes, and one index update), a batch of 5000 rows accumulates 5000 state entries before commit. A file with 1,000,000 rows thus never holds more than those 5000 entries in heap at any point (without batching, the same import would accumulate 5 000 000 state entries before a single commit, which is where heap exhaustion occurs). The right batch size is determined by what the subquery does per row, not by how many rows are in the file. If your subquery is creating nodes with a single label, one suggestion is to start with the default of 1000 rows and double until you encounter problems. If the subquery is doing more than that, it can be worthwhile to start lower.

To alter the batch size of a CALL { … } IN TRANSACTIONS, append OF n ROWS. For example, CALL { … } IN TRANSACTIONS OF 500 ROWS will commit every 500 rows rather than the default 1000, reducing the transactional state held in heap at any one time.

Structure `CALL { … } IN TRANSACTIONS` correctly: reads outside the subquery, writes inside

An important performance principle when using CALL { … } IN TRANSACTIONS is to perform any read operations and filtering before the subquery, and limit the subquery itself to write operations only.

The subquery runs inside a transaction. Every operation inside it contributes to the transaction’s memory footprint and holds locks until commit. Any filtering or matching you can do outside the subquery reduces both the number of rows entering it and the lock duration inside it.

Recommended: filter outside, write inside

LOAD CSV WITH HEADERS FROM 'file:///transactions.csv' AS row
WITH row
WHERE row.status = 'COMPLETED' AND row.amount IS NOT NULL
CALL (row) {
  MERGE (t:Transaction {id: row.id})
  SET t.amount = toFloat(row.amount)
} IN TRANSACTIONS

Not recommended: filtering inside the subquery

LOAD CSV WITH HEADERS FROM 'file:///transactions.csv' AS row
CALL (row) {
  WITH row
  WHERE row.status = 'COMPLETED' AND row.amount IS NOT NULL
  MERGE (t:Transaction {id: row.id})
  SET t.amount = toFloat(row.amount)
} IN TRANSACTIONS

Use multiple sequential subqueries for complex writes

If your import involves several related write operations, break them into several sequential subqueries. This pattern also helps when the first write is a prerequisite for the second: the nodes created in batch one are committed and visible before batch two tries to match them.

Example: sequential subqueries, second step depends on the first

// Step 1: create nodes
UNWIND $rows AS row
CALL (row) {
  MERGE (c:Customer {id: row.customerId})
  SET c.name = row.name
} IN TRANSACTIONS;

// Step 2: create relationships, relying on Step 1 already being committed
UNWIND $rows AS row
MATCH (c:Customer {id: row.customerId})
CALL (row, c) {
  MERGE (o:Order {id: row.orderId})
  MERGE (c)-[:PLACED]->(o)
} IN TRANSACTIONS;

Monitor transactions

Heap usage can be monitored while the import runs with the SHOW TRANSACTIONS command.

Monitor the transactional state of a database

SHOW TRANSACTIONS
YIELD transactionId, currentQuery, status, estimatedUsedHeapMemory

For more information about configuring memory usage with Neo4j, see the Operations Manual → Memory configuration.

You can also prepend ingestion statements with PROFILE, which will show the query-wide peak memory. For more information, see the Cypher Manual → Understanding query plans.

Monitor transaction log disk usage

CALL { … } IN TRANSACTIONS commits after every batch. Each commit writes to the transaction log. On a large import with a small batch size, this generates a high volume of commits and transaction log entries that can accumulate quickly and exhaust disk space before the import completes.

There are two possible mitigation strategies:

First, increase your batch size where possible. Fewer commits means less log volume.

Second, if disk space is a concern during a large initial import, consider temporarily adjusting the retention policy in neo4j.conf:

db.tx_log.rotation.retention_policy=false

This keeps only the most recent non-empty log. Restore the default after the import completes, particularly if you rely on transaction logs for backups or cluster operations.

Monitor disk usage at the OS level during large imports:

du -sh $NEO4J_HOME/data/transactions/<database-name>

For more information, see the Operations Manual → Transaction logging.

Relevant settings

The following settings are relevant when ingesting data with CALL { … } IN TRANSACTIONS:

db.transaction.timeout: the maximum time a single transaction is allowed to run, unlimited by default. If a batch size is set too large, or write throughput is slower than expected, a single batch’s transaction can hit this timeout and fail partway through rather than committing.
db.memory.transaction.max / db.memory.transaction.total.max: caps on the memory a single transaction, or all transactions combined, can consume. Relevant if a batch grows larger than expected, or a query still buffers more than intended despite being wrapped in CALL { … } IN TRANSACTIONS.

Concurrent transactions

CALL subqueries can be run concurrently, with the following command: CALL { … } IN CONCURRENT TRANSACTIONS. This allows you to run batches in parallel across multiple threads, which can dramatically improve throughput on hardware with multiple CPU cores.

CALL { … } IN CONCURRENT TRANSACTIONS runs batches in parallel across multiple threads, which can dramatically improve throughput on hardware with multiple CPU cores. The benefit compounds with dataset size. A sequential CALL { … } IN TRANSACTIONS run is bound by how fast one thread can process one batch at a time, however many cores the machine has sitting idle. Running batches concurrently lets that same hardware actually get used, rather than paying for cores that a sequential ingestion never touches.

But it introduces a new problem: lock contention.

Identifying deadlocks

Deadlocks can be identified in the ON ERROR events in the status report.

Detect deadlocks

LOAD CSV WITH HEADERS FROM 'file:///follows.csv' AS row
WITH row.follower_id AS followerId, row.followee_id AS followeeId
CALL (followerId, followeeId) {
  MATCH (follower:User {id: followerId})
  MATCH (followee:User {id: followeeId})
  MERGE (follower)-[:FOLLOWS]->(followee)
} IN CONCURRENT TRANSACTIONS OF 500 ROWS
  ON ERROR RETRY FOR 30 SECONDS THEN CONTINUE
  REPORT STATUS AS status
RETURN status.errorMessage AS error, status.committed AS committed

Pre-creating dense nodes

Dense nodes are nodes that have a large number of connected relationships. An example could be nodes with labels like Currency, Country, or Category that potentially thousands or millions of transactions point to. If they do not exist before import, and you let a concurrent import create them, a deadlock is likely. It is, therefore, a good idea to pre-create dense nodes in a first, separate step.

Step 1: pre-create potentially dense nodes before concurrent import

UNWIND ['USD', 'EUR', 'GBP', 'JPY'] AS code
MERGE (:Currency {code: code})

Now the concurrent import only ever matches them:

Step 2: concurrent import — Currency nodes already exist, no write lock needed

LOAD CSV WITH HEADERS FROM 'file:///currencies.csv' AS row
WITH row.id AS txId,
     row.amount AS amount,
     row.currency AS currencyCode
MATCH (c:Currency {code: currencyCode})
CALL (*) {
  MERGE (o:Order {id: txId, amount: toFloat(amount)})
  MERGE (o)-[:PURCHASED_WITH]->(c)
} IN CONCURRENT TRANSACTIONS

If you skipped step 1 and used MERGE on Currency inside the subquery, hundreds of concurrent batches would be racing to create or find, for example, USD nodes simultaneously.

Ignoring/retrying deadlocks

Deadlocks can be ignored with the ON ERROR CONTINUE option.

Ignore deadlocks with ON ERROR CONTINUE

LOAD CSV WITH HEADERS FROM 'file:///transactions.csv' AS row
WITH row.id AS txId,
     row.customer_id AS custId,
     toFloat(row.amount) AS amount
MATCH (c:Customer {id: custId})
CALL (*) {
  MERGE (t:Transaction {id: txId})
  SET t.amount = amount
  MERGE (c)-[:HAS_TRANSACTION]->(t)
} IN CONCURRENT TRANSACTIONS ON ERROR CONTINUE
  REPORT STATUS AS status
RETURN status.errorMessage AS error, status.committed AS committed

Deadlocks can also be retried with the ON ERROR RETRY option. The retry window defaults to 30 seconds with exponential backoff between attempts. To specify a different duration: ON ERROR RETRY FOR n SECONDS. If the batch still fails after the window expires, the THEN clause determines what happens next: THEN CONTINUE, THEN BREAK, or THEN FAIL (the default if omitted).

If the same batch keeps failing after retries, the error message will describe a data problem such as a constraint violation or type error. Retrying will not help in that case.

Retrying deadlocks with ON ERROR RETRY

LOAD CSV WITH HEADERS FROM 'file:///transactions.csv' AS row
WITH row.id AS txId,
     row.customer_id AS custId,
     toFloat(row.amount) AS amount
MATCH (customer:Customer {id: custId})
CALL (*) {
  MERGE (order:Order {id: txId})
  SET order.amount = amount
  MERGE (customer)-[:ORDERED]->(order)
} IN CONCURRENT TRANSACTIONS OF 500 ROWS
  ON ERROR RETRY FOR 10 SECONDS THEN CONTINUE
  REPORT STATUS AS status
RETURN status.errorMessage AS error, status.committed AS committed

Preventing deadlocks with `DISJOINT BY`
Cypher 25 onlyIntroduced in 2026.06

Rather than relying on ON ERROR RETRY to handle deadlocks after they occur, DISJOINT BY prevents them by ensuring batches that share resources never run at the same time.

Explicitly declaring lock-prone resources with DISJOINT BY (<expr>, …)

LOAD CSV WITH HEADERS FROM 'file:///transactions.csv' AS row
MATCH (customer:Customer {id: row.customer_id})
CALL (row, customer) {
   MERGE (order:Order {id: row.transaction_id})
   SET order.amount = toFloat(row.amount)
   MERGE (customer)-[:PLACED]->(order)
} IN CONCURRENT TRANSACTIONS
DISJOINT BY (row.customer_id, row.transaction_id) ON ERROR RETRY THEN CONTINUE

DISJOINT BY (row.customer_id, row.transaction_id) declares the lock prone resources (these must be variables passed to the subquery from the outer scope). Batches sharing the same customer_id or transaction_id are accordingly serialized, but everything else runs concurrently.

Alternatively, DISJOINT BY AUTO lets the runtime infer the lock expressions automatically from the MERGE patterns. This is useful when the locked resources are straightforward.

Automatic lock inference with DISJOINT BY AUTO

LOAD CSV WITH HEADERS FROM 'file:///transactions.csv' AS row
WITH row.transaction_id AS txId,
     row.customer_id AS custId,
     toFloat(row.amount) AS amount
MATCH (customer:Customer {id: custId})
CALL (txId, amount, customer, custId) {
    MERGE (order:Order {id: txId})
    SET order.amount = amount
    MERGE (customer)-[:PLACED]->(order)
} IN CONCURRENT TRANSACTIONS
DISJOINT BY AUTO

ON ERROR RETRY THEN CONTINUE is used in both examples as a fallback for transient errors.

Use EXPLAIN to verify which lock expressions the runtime inferred (they appear in the TransactionForeach operator Details column).

Sorting CSV files before import

When creating or updating nodes and relationships from CSV data, the order rows are processed can affect import performance with concurrent transactions. If the same node appears across many rows scattered throughout the file, multiple batches will compete for a write lock on that node simultaneously, turning a dense node into a bottleneck that serializes what should be parallel work.

This is most likely to cause problems when creating relationships. A dense node with thousands of incoming relationships will have many batches competing for it at once, whereas a node that only receives property updates is less likely to generate the same level of contention. Sorting your CSV by the column that maps to the dense node groups all affected rows into consecutive batches. One batch acquires the write lock, does all its work, commits, and releases it before the next batch needs it.

Sort CSV file

import pandas as pd

df = pd.read_csv('transactions.csv')

# Sort by the column that maps to the dense node
df_sorted = df.sort_values('customer_id')
df_sorted.to_csv('transactions_sorted.csv', index=False)

Import using sorted CSV file

LOAD CSV WITH HEADERS FROM 'file:///transactions_sorted.csv' AS row
WITH row.id AS orderId,
     row.customer_id AS custId,
     toFloat(row.amount) AS amount
MATCH (customer:Customer {id: custId})
CALL (*) {
  MERGE (order:Order {id: orderId})
  SET order.amount = amount
  MERGE (customer)-[:PLACED]->(order)
} IN CONCURRENT TRANSACTIONS OF 500 ROWS

Batch size and outliers

Sorting helps, but it does not fully solve the problem for extreme outliers. If your batch size is 1,000 rows and your top customer has 80,000 transactions, their rows still spill across 80 batches all competing for that one Customer dense node. A few options:

Increase batch size to cover your densest node’s row count in a single batch. Not always practical, but eliminates contention for that node entirely.
Import dense nodes separately: identify your top-N customers upfront, import their transactions in a dedicated sequential pass, then run the concurrent import for the rest.
Accept residual contention for true outliers and rely on ON ERROR RETRY to handle the deadlocks that result.

Partitioning instead of sorting

If your data pipeline makes sorting awkward, partitioning achieves the same goal. Split the file into N separate files based on a hash of the column that maps to your dense node. This guarantees that all rows referencing the same node always end up in the same file:

Partition CSV file

import pandas as pd

df = pd.read_csv('transactions.csv')
for partition, group in df.groupby(df['customer_id'].apply(lambda x: hash(x) % 4)): group.to_csv(f'transactions_{partition}.csv', index=False)

Since the same node can never appear in two different partition files, there is no lock contention between files and each can be imported concurrently without competing for the same node:

Import using partitioned CSV files

LOAD CSV WITH HEADERS FROM 'file:///transactions_0.csv' AS row
WITH row.id AS orderId,
     row.customer_id AS custId,
    toFloat(row.amount) AS amount
MATCH (customer:Customer {id: custId})
CALL (*) {
  MERGE (order:Order {id: orderId})
  SET order.amount = amount
  MERGE (customer)-[:PLACED]->(order)
} IN CONCURRENT TRANSACTIONS OF 500 ROWS

Run the same query for transactions_1.csv, transactions_2.csv, and transactions_3.csv from separate sessions simultaneously.