Databricks quickstart

This page includes instructions on the usage of a third-party platform, which may be subject to changes beyond our control. In case of doubt, refer to the third-party platform documentation.

Prerequisites

  • A Databricks workspace must be available on an URL like https://dbc-xxxxxxxx-yyyy.cloud.databricks.com.

  • A Databricks Runtime compatible with your connector version. See the connector compatibility table, which lists only LTS runtimes.

Set up a compute cluster

  1. Create a compute cluster with Dedicated access mode, Unrestricted policy, and your preferred Scala runtime.

    The Dedicated access mode is required. The Standard and other shared access modes are not supported. See Limitations.

  2. Once the cluster is available, open its page and select the Libraries tab.

  3. Select Install new and choose Maven as the library source.

  4. In the Coordinate field, enter org.neo4j.connectors:spark:6.0.0-s_2.13.

    Do not use Search Packages, as the connector has not been published to Spark Packages since version 5.4.0. Make sure the Scala suffix in the coordinate matches the Scala version of the cluster runtime.

  5. Select Install.

Installation methods

The Maven coordinate is the default and covers most workspaces. It requires the cluster, or its Maven proxy, to reach Maven Central at startup, and with Unity Catalog the coordinate may need to be allowlisted.

When a cluster policy blocks Maven resolution, or the workspace has no outbound network access, install a downloaded JAR as a cluster library instead. Store it in a Unity Catalog volume or in cloud object storage, as JARs kept in workspace files cannot be used with the Dedicated access mode. See Installation for where to download the JAR.

The downloadable JAR bundles its dependencies, including the Neo4j Java Driver, which the Maven coordinate resolves transitively instead. Install one or the other, never both, and never alongside a standalone neo4j-java-driver library.

Cluster libraries are installed after the driver and executor JVMs start. An init script is the only option when the connector must be on the classpath before Spark initializes, for example when a Spark configuration set at cluster startup references connector classes. The script must live in a Unity Catalog volume or in cloud object storage.

Example init script stored in a Unity Catalog volume
#!/bin/bash
cp /Volumes/<catalog>/<schema>/<volume>/neo4j-spark-connector-6.0.0-s_2.13.jar /databricks/jars/

Use a cluster library unless you have that specific requirement. JARs placed in /databricks/jars/ join the runtime classpath and can shadow the JARs shipped with Databricks Runtime. Copy the connector JAR only, never its Spark or Scala dependencies.

Unity Catalog

Neo4j supports the Unity Catalog in Dedicated access mode only. Refer to the Databricks documentation for further information.

Session configuration

You can set the Spark configuration on the cluster you are running your notebooks on by doing the following:

  1. Open the cluster configuration page.

  2. Select the Advanced Options toggle under Configuration.

  3. Select the Spark tab.

For example, you can add Neo4j Bearer authentication configuration in the text area as follows:

Bearer authentication example
neo4j.url neo4j://<host>:<port>
neo4j.authentication.type bearer
neo4j.authentication.bearer.token <token>

Databricks advises against storing secrets such as passwords and tokens in plain text. A secure alternative is to use secrets instead.

Authentication methods

All the authentication methods supported by the Neo4j Java Driver (version 6 and higher) are supported.

See the Neo4j driver options for more details on authentication configuration.

Set up secrets

You can add secrets to your environment using the Secrets API via the Databricks CLI. If you use a Databricks runtime version 15.0 or above, you can add secrets directly from a notebook terminal.

After setting secrets up, you can access them from a Databricks notebook using the Databricks Utilities (dbutils). For example, given a neo4j scope and the username and password secrets for basic authentication, you can do the following in a Python notebook:

from pyspark.sql import SparkSession

url = "neo4j+s://xxxxxxxx.databases.neo4j.io"
username = dbutils.secrets.get(scope="neo4j", key="username")
password = dbutils.secrets.get(scope="neo4j", key="password")
dbname = "neo4j"

spark = (
    SparkSession.builder.config("neo4j.url", url)
    .config("neo4j.authentication.basic.username", username)
    .config("neo4j.authentication.basic.password", password)
    .config("neo4j.database", dbname)
    .getOrCreate()
)

Delta tables

You can use the Spark connector to read from and write to Delta tables from a Databricks notebook. This does not require any additional setup.

Basic roundtrip

The following example shows how to read a Delta table, write it as nodes and node properties to Neo4j, read the corresponding nodes and node properties from Neo4j, and write them to a new Delta table.

Content of the Delta table

The example assumes that a Delta table users_example exists and contains the following data:

Table 1. users_example table
name surname age

John

Doe

42

Jane

Doe

40

 
# Read the Delta table
tableDF = spark.read.table("users_example")

# Write the DataFrame to Neo4j as nodes
(
    tableDF
    .write.format("org.neo4j.spark.DataSource")
    .mode("Append")
    .option("labels", ":User")
    .save()
)

# Read the nodes with `:User` label from Neo4j
neoDF = (
    spark.read.format("org.neo4j.spark.DataSource")
    .option("labels", ":User")
    .load()
)

# Write the DataFrame to another Delta table,
# which will contain the additional columns
# `<id>` and `<labels>`
neoDF.write.saveAsTable("users_new_example")

Delta tables to Neo4j nodes and relationships

To avoid deadlocks, always use a single partition (with coalesce(1) or repartition(1)) before writing relationships to Neo4j.

The following example shows how to read a Delta table and write its data as both nodes and relationships to Neo4j. See the Writing page for details on using the Overwrite mode and on writing nodes only.

Content of the Delta table

The example assumes that a Delta table customers_products_example exists and contains the following data:

Table 2. customers_products_example table
name surname customerID product quantity order

John

Doe

1

Product 1

200

ABC100

Jane

Doe

2

Product 2

100

ABC200

 
# Read the Delta table into a DataFrame
relDF = spark.read.table("customers_products_example")

# Write the table to Neo4j using the
# `relationship` write option
(
    relDF
    # Use a single partition
    .coalesce(1)
    .write
    # Create new relationships
    .mode("Append")
    .format("org.neo4j.spark.DataSource")
    # Assign a type to the relationships
    .option("relationship", "BOUGHT")
    # Create source nodes and assign them a label
    .option("relationship.source.save.mode", "Append")
    .option("relationship.source.labels", ":Customer")
    # Map DataFrame columns to source node properties
    .option("relationship.source.node.properties", "name,surname,customerID:id")
    # Create target nodes and assign them a label
    .option("relationship.target.save.mode", "Append")
    .option("relationship.target.labels", ":Product")
    # Map DataFrame columns to target node properties
    .option("relationship.target.node.properties", "product:name")
    # Map DataFrame columns to relationship properties
    .option("relationship.properties", "quantity,order")
    .save()
)

Neo4j nodes to Delta tables

The following example shows how to read nodes from Neo4j and write them to a Delta table. See the Reading page for details on reading relationships.

# Read the nodes with `:Customer` label from Neo4j
df = (
    spark.read.format("org.neo4j.spark.DataSource")
    .option("labels", ":Customer")
    .load()
)

# Write the DataFrame to another Delta table
df.write.saveAsTable("customers_status_example")

Limitations

The connector runs on Databricks classic compute with the Dedicated access mode only. This means the connector does not support serverless compute, for either notebooks or jobs. The Standard and no-isolation-shared access modes and legacy High Concurrency clusters are not supported either. Databricks sets spark.databricks.cluster.profile from the cluster type: unset or singleNode are supported, while serverless selects a legacy High Concurrency cluster and, despite its name, has nothing to do with serverless compute.

For limitations that are not specific to Databricks, see Known issues and limitations.