This chapter provides an introduction to the Neo4j Connector for Apache Spark
The Neo4j Connector for Apache Spark is intended to make integrating graphs together with spark easy. There are effectively two ways of using the connector:
As a data source: read any set of nodes or relationships as a DataFrame in Spark
As a sink: write any DataFrame to Neo4j as a collection of nodes or relationships, or alternatively; use a Cypher statement to process records in a DataFrame into the graph pattern of your choice.
Because the connector is based on the new Spark DataSource API, other spark interpreters for languages such as Python and R will work.
The API remains the same, and mostly only slight syntax changes are necessary to accomodate the differences between (for example) Python and Scala.
This connector works with Neo4j 3.5, and the entire 4+ series of Neo4j, whether run as a single instance, in causal cluster mode, or run as a managed service in Neo4j Aura. The connector does not rely on enterprise features, and as such will work with Neo4j Community as well, with the appropriate version number.
|Neo4j versions prior to 3.5 are not supported|
This connector currently supports Spark 2.4.5+ with Scala 2.11 and Scala 2.12 and Spark 3.0+ with Scala 2.12.
Depending on the combination of Spark and Scala version you’ll need a different JAR.
JARs are named in the form
Here’s a compatibility table to help you choose the correct JAR.
|Spark 2.4||Spark 3.0+|
If you want an introduction on the Neo4j Connector for Apache Spark, take a look at the training that Andrea Santurbano presented at NODES2020.
This connector is provided under the terms of the Apache 2.0 license, which can be found in the GitHub repository.
For Neo4j Enterprise and Neo4j Aura customers, official releases of this connector are supported under the terms of your existing Neo4j support agreement. This support extends only to regular releases, and excludes alpha, beta, and pre-releases. If you have any questions about the support policy, please get in touch with Neo4j.
Was this page helpful?