Goals There are various ways to beneficially use Neo4j with Apache Spark, here we will list some approaches and point to solutions that enable you to leverage your Spark infrastructure with Neo4j. Prerequisites You should have a sound understanding of… Read more →
There are various ways to beneficially use Neo4j with Apache Spark, here we will list some approaches and point to solutions that enable you to leverage your Spark infrastructure with Neo4j.
You should have a sound understanding of both Apache Spark and Neo4j, each data model, data processing paradigm and APIs to leverage them effectively together.
Apache Spark is a clustered, in-memory data processing solution that scales processing of large datasets easily across many machines. It also comes with GraphX and GraphFrames two frameworks for running graph compute operations on your data.
You can integrate with Spark in a variety of ways. Either to pre-process (aggregate, filter, convert) your raw data to be imported into Neo4j.
Spark can also serve as external Graph Compute solution, where you
- export data of selected subgraphs from Neo4j to Spark,
- compute the analytic aspects, and
- write the results back to Neo4j
- to be used in your Neo4j operations and Cypher queries.
|Neo4j itself is capable of running graph processing on medium to large graphs quickly. For instance the graph-processing project demonstrates that we can run PageRank (5 iterations) on the dbpedia dataset (10M nodes, 125M relationships) in 20 seconds as a Neo4j server extension or user defined procedure. Spark might be better suited for larger datasets or more intensive compute operations.|
The Neo4j Spark Connector uses the binary Bolt protocol to transfer data from and to a Neo4j server.
It offers Spark-2.0 APIs for RDD, DataFrame, GraphX and GraphFrames, so you’re free to chose how you want to use and process your Neo4j graph data in Apache Spark.
Configure Neo4j-URL, -user and -password via
spark.neo4j.bolt.* Spark Config options.
The general usage is:
rels(query,[params])as direct query, or
pattern( ("Label1","prop1"),("REL","prop"),("Label2","prop2") )
choose which datatype to return
Here is a basic example for loading a
Similar operations are available for DataFrames and GraphX.
The GraphX integration also allows to write data back to Neo4j with a
To use GraphFrames you have to declare it as package. Then you can load a GraphFrame with graph data from Neo4j and run graph algorithms or pattern matchin on it (the latter will be slower than in Neo4j).
More examples and details can be found in the docs of the GitHub repository.
An interest in analytical graph processing led Kenny Bastani to work on an integration solution. It allows to export dedicated datasets, e.g. node or relationship-lists to Spark.
It supports these algorithms:
- Closeness Centrality
- Betweenness Centrality
- Triangle Counting
- Connected Components
- Strongly Connected Components
After running graph processing algorithms the results are written back concurrently and transactionally to Neo4j.
One focus of this approach is on data safety, that’s why it uses a persistent queue (RabbitMQ) to communicate data between Neo4j and Spark.
The infrastructure is set up using Docker containers, there are dedicated containers for Spark, RabbitMQ, HDFS and Neo4j with the Mazerunner Extension.
More details can be found on the project’s GitHub page.
Spark for Data Preprocessing
One example of pre-processing raw data (Chicago Crime dataset) into a format that’s well suited for import into Neo4j, was demonstrated by Mark Needham. He combined a number of functions into a Spark-job that takes the existing data, cleans and aggregates it and outputs fragments which are recombined later to larger files.
The approach is detailed in his blog post: Spark: Generating CSV Files to import into Neo4j.