Best practices for Source connector

This page collects recommendations and operational practices for running the Source connector. Each recommendation explains the trade-offs so you can decide whether it applies to your use case.

Prefer Avro over JSON

The serialization format you choose for messages has a direct impact on connector throughput. Avro is a compact binary format whose schema is stored separately in Schema Registry, so each message carries only its values rather than repeating field names as text. Compared to JSON, this reduces payload size and the serialization and deserialization overhead the connector incurs for every change event, resulting in better throughput and lower memory pressure.

For these reasons, Avro is the recommended format when your environment supports it.

Avro requires a Schema Registry to store and resolve schemas. Before adopting it, make sure the structure of your exported data is suitable for schema enforcement, and review the guidance in Working with Schema Registry — in particular the caveats around unstructured graph data, where a schema-less format may still be the right choice.

Choose the CDC key strategy deliberately

For the Change Data Capture strategy, neo4j.cdc.topic.{NAME}.key-strategy controls how the Kafka message key is serialized. This setting affects both connector overhead and how messages are distributed across topic partitions, so it is worth choosing the key strategy deliberately.

The default value, WHOLE_VALUE, serializes the entire change event into the message key in addition to the message value. This makes each message self-contained, but means every change event is serialized and deserialized twice, which adds overhead without providing a useful key for partitioning.

Use the following table as a guidance to determine the best strategy.

Strategy Use case

Strategy	Use case
`SKIP`	Best for throughput. No key is produced, so the connector avoids the cost of serializing the key and Kafka spreads messages across the topic’s partitions without grouping them by entity. Use this when you do not need key-based routing or per-entity ordering.
`ENTITY_KEYS`	Use when you need per-entity ordering. The entity’s key properties are used as the message key, so all changes to the same entity are routed to the same partition and their relative order is preserved.
`ELEMENT_ID`	Use when you need per-entity ordering but no key properties are available. The element ID is used as the message key, with the same partitioning and ordering behavior as `ENTITY_KEYS`.
`WHOLE_VALUE`	The default — most complete, but lower throughput. The entire change event is serialized into the message key as well as the value, so the message is self-contained without relying on key properties or element IDs. This completeness comes at the cost of double serialization without grouping changes by entity, so prefer one of the other strategies when throughput is the priority.

SKIP

Best for throughput.

No key is produced, so the connector avoids the cost of serializing the key and Kafka spreads messages across the topic’s partitions without grouping them by entity. Use this when you do not need key-based routing or per-entity ordering.

ENTITY_KEYS

Use when you need per-entity ordering.

The entity’s key properties are used as the message key, so all changes to the same entity are routed to the same partition and their relative order is preserved.

ELEMENT_ID

Use when you need per-entity ordering but no key properties are available.

The element ID is used as the message key, with the same partitioning and ordering behavior as ENTITY_KEYS.

WHOLE_VALUE

The default — most complete, but lower throughput.

The entire change event is serialized into the message key as well as the value, so the message is self-contained without relying on key properties or element IDs. This completeness comes at the cost of double serialization without grouping changes by entity, so prefer one of the other strategies when throughput is the priority.

The key strategy determines which partition a change event is routed to, and Kafka only guarantees ordering within a single partition. With SKIP, changes to the same entity may be processed out of order, which can cause issues if your downstream consumers depend on the order of operations (for example, a creation followed by an update or deletion). Only use SKIP when your data model and consumers tolerate this.

CDC cursor recovery

While the CDC source connector tracks CDC change identifier in Kafka Connect managed offset storage, it also keeps the offset in memory while it is up and running. If the database invalidates CDC change identifiers, the connector can keep retrying with a change identifier that the database can no longer resolve and will eventually fail, which needs to be recovered manually.

CDC change identifiers can become invalid after database operations that change data outside the transaction layer, such as restoring a backup, restoring from a snapshot, copying the database, importing data through CLI commands, or pausing and resuming a Neo4j Aura database. They can also become invalid when the transaction log gets truncated and no longer contains the transaction referenced by the stored change identifier. For the underlying CDC behavior and error codes, see Change Data Capture > Known Issues and Change Data Capture > Troubleshooting.

Pausing and resuming a Neo4j Aura database can disable CDC by resetting txLogEnrichment to OFF. After resuming an Aura database, verify that CDC is enabled before restarting the source connector. If CDC is disabled, re-enable it using the Enable CDC on Neo4j Aura instructions, then restart the source connector so it can establish a valid CDC change identifier.

To recover, restart the source connector so it can establish a new valid CDC change identifier from its configured initial offset.

The neo4j.start-from setting controls the initial offset used when the source connector starts:

NOW: Start from the current end of the CDC stream. This skips changes between the invalidated offset and the restart point, but resumes ingestion from a valid change identifier.
EARLIEST: Start from the earliest CDC change identifier that is still available for the database. When CDC has been enabled continuously, this resumes consumption from the earliest transaction that has not been truncated by the database.
USER_PROVIDED: Start from a specific CDC change identifier. Before restarting the source connector, refetch the starting offset and set neo4j.start-from.value to a valid change identifier obtained from the same database using the CDC procedures, for example db.cdc.current or db.cdc.earliest. You can also use a valid change identifier returned by db.cdc.query. See Change Data Capture > Use CDC to query changes for details.

For example, get the current CDC change identifier from Neo4j:

CALL db.cdc.current() YIELD id
RETURN id;

Then update the source connector configuration with the refreshed starting offset before restarting it:

"neo4j.start-from": "USER_PROVIDED",
"neo4j.start-from.value": "<change identifier from db.cdc.current>"