HDBSCAN

Glossary

Directed: Directed trait. The algorithm is well-defined on a directed graph.
Directed: Directed trait. The algorithm ignores the direction of the graph.
Directed: Directed trait. The algorithm does not run on a directed graph.
Undirected: Undirected trait. The algorithm is well-defined on an undirected graph.
Undirected: Undirected trait. The algorithm ignores the undirectedness of the graph.
Heterogeneous nodes: Heterogeneous nodes fully supported. The algorithm has the ability to distinguish between nodes of different types.
Heterogeneous nodes: Heterogeneous nodes allowed. The algorithm treats all selected nodes similarly regardless of their label.
Heterogeneous relationships: Heterogeneous relationships fully supported. The algorithm has the ability to distinguish between relationships of different types.
Heterogeneous relationships: Heterogeneous relationships allowed. The algorithm treats all selected relationships similarly regardless of their type.
Weighted relationships: Weighted trait. The algorithm supports a relationship property to be used as weight, specified via the relationshipWeightProperty configuration parameter.
Weighted relationships: Weighted trait. The algorithm treats each relationship as equally important, discarding the value of any relationship weight.

Introduction

HDBSCAN, which stands for Hierarchical Density-Based Spatial Clustering of Applications with Noise, is a clustering algorithm used to identify clusters of similar data points within a dataset. It builds upon the DBSCAN algorithm but adds a hierarchical structure, making it more robust to varying densities within the data.

Unlike DBSCAN, HDBSCAN does not require tuning a specific density parameter; instead, it runs DBSCAN over a range of parameters, creating a hierarchy of clusters. This hierarchical approach allows HDBSCAN to find clusters of varying densities and to be more adaptable to real-world data.

HDBSCAN is known for its ease of use, noise tolerance, and ability to handle data with varying densities, making it a versatile tool for clustering tasks, especially when dealing with complex, high-dimensional datasets.

For more information on this algorithm, see:

HDBSCAN: Density based Clustering over Location Based Services

Considerations

In order for HDBSCAN to work properly, the property arrays for all nodes must have the same number of elements. Also, they should contain exclusively numbers and not contain any NaN values.

Syntax

HDBSCAN syntax per mode

Run HDBSCAN in stream mode on a named graph.

CALL gds.hdbscan.stream(
  graphName: String,
  configuration: Map
)
YIELD
  nodeId: Integer,
  label: Integer

Table 1. Parameters
Name	Type	Default	Optional	Description
graphName	String	`n/a`	no	The name of a graph stored in the catalog.
configuration	Map	`{}`	yes	Configuration for algorithm-specifics and/or graph filtering.

Table 2. Configuration
Name	Type	Default	Optional	Description
nodeLabels	List of String	`['*']`	yes	Filter the named graph using the given node labels. Nodes with any of the given labels will be included.
relationshipTypes	List of String	`['*']`	yes	Filter the named graph using the given relationship types. Relationships with any of the given types will be included.
concurrency	Integer	`4`	yes	The number of concurrent threads used for running the algorithm.
jobId	String	`Generated internally`	yes	An ID that can be provided to more easily track the algorithm’s progress.
logProgress	Boolean	`true`	yes	If disabled the progress percentage will not be logged.
nodeProperty	String	`n/a`	no	A node property corresponding to an array of floats used by HDBSCAN to compute clusters.
minClusterSize	Integer	`5`	yes	The minimum number of nodes that a cluster should contain.
samples	Integer	`10`	yes	The number of neighbours to be considered when computing the core distances of a node.
leafSize	Integer	`1`	yes	The number of leaf nodes of the supporting tree data structure.

Table 3. Results
Name	Type	Description
nodeId	Integer	Node ID.
label	Integer	The label ID, `-1` if the node is considered `noise`.

Run HDBSCAN in stats mode on a named graph.

CALL gds.hdbscan.stats(
  graphName: String,
  configuration: Map
)
YIELD
  nodeCount: Integer,
  numberOfClusters: Integer,
  numberOfNoisePoints: Integer,
  preProcessingMillis: Integer,
  computeMillis: Integer,
  postProcessingMillis: Integer,
  configuration: Map

Table 4. Parameters
Name	Type	Default	Optional	Description
graphName	String	`n/a`	no	The name of a graph stored in the catalog.
configuration	Map	`{}`	yes	Configuration for algorithm-specifics and/or graph filtering.

Table 5. Configuration
Name	Type	Default	Optional	Description
nodeLabels	List of String	`['*']`	yes	Filter the named graph using the given node labels. Nodes with any of the given labels will be included.
relationshipTypes	List of String	`['*']`	yes	Filter the named graph using the given relationship types. Relationships with any of the given types will be included.
concurrency	Integer	`4`	yes	The number of concurrent threads used for running the algorithm.
jobId	String	`Generated internally`	yes	An ID that can be provided to more easily track the algorithm’s progress.
logProgress	Boolean	`true`	yes	If disabled the progress percentage will not be logged.
nodeProperty	String	`n/a`	no	A node property corresponding to an array of floats used by HDBSCAN to compute clusters.
minClusterSize	Integer	`5`	yes	The minimum number of nodes that a cluster should contain.
samples	Integer	`10`	yes	The number of neighbours to be considered when computing the core distances of a node.
leafSize	Integer	`1`	yes	The number of leaf nodes of the supporting tree data structure.

Table 6. Results
Name	Type	Description
nodeCount	Integer	The number of nodes the algorithm ran on.
numberOfClusters	Integer	The number of clusters found by the algorithm.
numberOfNoisePoints	Integer	The number of noise points found by the algorithm.
preProcessingMillis	Integer	Milliseconds for preprocessing the data.
computeMillis	Integer	Milliseconds for running the algorithm.
postProcessingMillis	Integer	Milliseconds for computing percentiles and community count.
configuration	Map	The configuration used for running the algorithm.

Run HDBSCAN in mutate mode on a named graph.

CALL gds.hdbscan.mutate(
  graphName: String,
  configuration: Map
)
YIELD
  nodeCount: Integer,
  numberOfClusters: Integer,
  numberOfNoisePoints: Integer,
  preProcessingMillis: Integer,
  computeMillis: Integer,
  postProcessingMillis: Integer,
  mutateMillis: Integer,
  nodePropertiesWritten: Integer,
  configuration: Map

Table 7. Parameters
Name	Type	Default	Optional	Description
graphName	String	`n/a`	no	The name of a graph stored in the catalog.
configuration	Map	`{}`	yes	Configuration for algorithm-specifics and/or graph filtering.

Table 8. Configuration
Name	Type	Default	Optional	Description
mutateProperty	String	`n/a`	no	The node property in the GDS graph to which the cluster is written.
nodeLabels	List of String	`['*']`	yes	Filter the named graph using the given node labels.
relationshipTypes	List of String	`['*']`	yes	Filter the named graph using the given relationship types.
concurrency	Integer	`4`	yes	The number of concurrent threads used for running the algorithm.
jobId	String	`Generated internally`	yes	An ID that can be provided to more easily track the algorithm’s progress.
nodeProperty	String	`n/a`	no	A node property corresponding to an array of floats used by HDBSCAN to compute clusters.
minClusterSize	Integer	`5`	yes	The minimum number of nodes that a cluster should contain.
samples	Integer	`10`	yes	The number of neighbours to be considered when computing the core distances of a node.
leafSize	Integer	`1`	yes	The number of leaf nodes of the supporting tree data structure.

Table 9. Results
Name	Type	Description
nodeCount	Integer	The number of nodes the algorithm ran on.
numberOfClusters	Integer	The number of clusters found by the algorithm.
numberOfNoisePoints	Integer	The number of noise points found by the algorithm.
preProcessingMillis	Integer	Milliseconds for preprocessing the data.
computeMillis	Integer	Milliseconds for running the algorithm.
mutateMillis	Integer	Milliseconds for adding properties to the projected graph.
postProcessingMillis	Integer	Milliseconds for computing percentiles and community count.
nodePropertiesWritten	Integer	Number of properties added to the projected graph.
configuration	Map	The configuration used for running the algorithm.

Run HDBSCAN in write mode on a named graph.

CALL gds.hdbscan.write(
  graphName: String,
  configuration: Map
)
YIELD
  nodeCount: Integer,
  numberOfClusters: Integer,
  numberOfNoisePoints: Integer,
  preProcessingMillis: Integer,
  computeMillis: Integer,
  postProcessingMillis: Integer,
  writeMillis: Integer,
  nodePropertiesWritten: Integer,
  configuration: Map

Table 10. Parameters
Name	Type	Default	Optional	Description
graphName	String	`n/a`	no	The name of a graph stored in the catalog.
configuration	Map	`{}`	yes	Configuration for algorithm-specifics and/or graph filtering.

Table 11. Configuration
Name	Type	Default	Optional	Description
nodeLabels	List of String	`['*']`	yes	Filter the named graph using the given node labels. Nodes with any of the given labels will be included.
relationshipTypes	List of String	`['*']`	yes	Filter the named graph using the given relationship types. Relationships with any of the given types will be included.
concurrency	Integer	`4`	yes	The number of concurrent threads used for running the algorithm.
jobId	String	`Generated internally`	yes	An ID that can be provided to more easily track the algorithm’s progress.
logProgress	Boolean	`true`	yes	If disabled the progress percentage will not be logged.
writeConcurrency	Integer	`value of 'concurrency'`	yes	The number of concurrent threads used for writing the result to Neo4j.
writeProperty	String	`n/a`	no	The node property in the Neo4j database to which the cluster is written.
nodeProperty	String	`n/a`	no	A node property corresponding to an array of floats used by HDBSCAN to compute clusters.
minClusterSize	Integer	`5`	yes	The minimum number of nodes that a cluster should contain.
samples	Integer	`10`	yes	The number of neighbours to be considered when computing the core distances of a node.
leafSize	Integer	`1`	yes	The number of leaf nodes of the supporting tree data structure.

Table 12. Results
Name	Type	Description
nodeCount	Integer	The number of nodes the algorithm ran on.
numberOfClusters	Integer	The number of clusters found by the algorithm.
numberOfNoisePoints	Integer	The number of noise points found by the algorithm.
preProcessingMillis	Integer	Milliseconds for preprocessing the data.
computeMillis	Integer	Milliseconds for running the algorithm.
writeMillis	Integer	Milliseconds for adding properties to the Neo4j database.
postProcessingMillis	Integer	Milliseconds for computing percentiles and community count.
nodePropertiesWritten	Integer	Number of properties added to the projected graph.
configuration	Map	The configuration used for running the algorithm.

Examples

All the examples below should be run in an empty database.

The examples use Cypher projections as the norm. Native projections will be deprecated in a future release.

In this section we will show examples of running the HDBSCAN algorithm on a concrete graph. The intention is to illustrate what the results look like and to provide a guide in how to make use of the algorithm in a real setting. We will do this on a small cities graph of a handful nodes connected in a particular pattern. The example graph looks like this:

The following Cypher statement will create the example graph in the Neo4j database:

CREATE
  (:City {name: 'Surbiton', coordinates: [51.39148, -0.29825]}),
  (:City {name: 'Liverpool', coordinates: [53.41058, -2.97794]}),
  (:City {name: 'Kingston upon Thames', coordinates: [51.41259, -0.2974]}),
  (:City {name: 'Sliven', coordinates: [42.68583, 26.32917]}),
  (:City {name: 'Solna', coordinates: [59.36004, 18.00086]}),
  (:City {name: 'Örkelljunga', coordinates: [56.28338, 13.27773]}),
  (:City {name: 'Malmö', coordinates: [55.60587, 13.00073]}),
  (:City {name: 'Xánthi', coordinates: [41.13488, 24.888]});

This graph is composed of various City nodes, in three global locations - United Kingdom, Sweden and the Balkan region in Europe.

We can now project the graph and store it in the graph catalog. We load the City node label with coordinates node property.

The following statement will project the graph and store it in the graph catalog.

MATCH (c:City)
RETURN gds.graph.project(
  'cities',
  c,
  null,
  {
    sourceNodeProperties: c { .coordinates },
    targetNodeProperties: {}
  }
)

In the following examples we will demonstrate using the HDBSCAN algorithm on this graph to find communities of cities that are close to each other geographically.

Memory Estimation

First off, we will estimate the cost of running the algorithm using the estimate procedure. This can be done with any execution mode. We will use the write mode in this example. Estimating the algorithm is useful to understand the memory impact that running the algorithm on your graph will have. When you later actually run the algorithm in one of the execution modes the system will perform an estimation. If the estimation shows that there is a very high probability of the execution going over its memory limitations, the execution is prohibited. To read more about this, see Automatic estimation and execution blocking.

For more details on estimate in general, see Memory Estimation.

The following will estimate the memory requirements for running the algorithm:

CALL gds.hdbscan.write.estimate('cities', {
  writeProperty: 'label',
  nodeProperty: 'coordinates'
})
YIELD nodeCount, bytesMin, bytesMax, requiredMemory

Table 13. Results
nodeCount	bytesMin	bytesMax	requiredMemory
8	9920	9920	"9920 Bytes"

Stream

In the stream execution mode, the algorithm returns the cluster for each node. This allows us to inspect the results directly or post-process them in Cypher without any side effects.

For more details on the stream mode in general, see Stream.

The following will run the algorithm and stream results:

CALL gds.hdbscan.stream('cities', {
  nodeProperty: 'coordinates',
  minClusterSize: 2,
  samples: 2
})
YIELD nodeId, label
RETURN gds.util.asNode(nodeId).name AS name, label
ORDER BY label, name ASC

Table 14. Results
name	label
"Sliven"	-1
"Xánthi"	-1
"Kingston upon Thames"	1
"Liverpool"	1
"Surbiton"	1
"Malmö"	2
"Solna"	2
"Örkelljunga"	2

In the example above we can see that the cities are geographically clustered together.

Stats

In the stats execution mode, the algorithm returns a single row containing a summary of the algorithm result. This execution mode does not have any side effects. It can be useful for evaluating algorithm performance by inspecting the computeMillis return item. In the examples below we will omit returning the timings. The full signature of the procedure can be found in the syntax section.

For more details on the stats mode in general, see Stats.

The following will run the algorithm and returns the result in form of statistical and measurement values

CALL gds.hdbscan.stats('cities', {
  nodeProperty: 'coordinates',
  minClusterSize: 2,
  samples: 2
})
YIELD nodeCount, numberOfClusters, numberOfNoisePoints

Table 15. Results
nodeCount	numberOfClusters	numberOfNoisePoints
8	2	2

Mutate

The mutate execution mode extends the stats mode with an important side effect: updating the named graph with a new node property containing the cluster for that node. The name of the new property is specified using the mandatory configuration parameter mutateProperty. The result is a single summary row, similar to stats, but with some additional metrics. The mutate mode is especially useful when multiple algorithms are used in conjunction.

For more details on the mutate mode in general, see Mutate.

The following will run the algorithm and store the results in cities graph:

CALL gds.hdbscan.mutate('cities', {
  nodeProperty: 'coordinates',
  minClusterSize: 2,
  samples: 2,
  mutateProperty: 'label'
})
YIELD nodeCount, numberOfClusters, nodePropertiesWritten, numberOfNoisePoints

Table 16. Results
nodeCount	numberOfClusters	nodePropertiesWritten	numberOfNoisePoints
8	2	8	2

In mutate mode, only a single row is returned by the procedure. The result is written to the GDS in-memory graph instead of the Neo4j database.

Write

The write execution mode extends the stats mode with an important side effect: writing the cluster for each node as a property to the Neo4j database. The name of the new property is specified using the mandatory configuration parameter writeProperty. The result is a single summary row, similar to stats, but with some additional metrics. The write mode enables directly persisting the results to the database.

For more details on the write mode in general, see Write.

The following will run the algorithm and write the results back to Neo4j:

CALL gds.hdbscan.write('cities', {
  nodeProperty: 'coordinates',
  minClusterSize: 2,
  samples: 2,
  writeProperty: 'label'
})
YIELD nodeCount, numberOfClusters, nodePropertiesWritten, numberOfNoisePoints

Table 17. Results
nodeCount	numberOfClusters	nodePropertiesWritten	numberOfNoisePoints
8	2	8	2

In write mode, only a single row is returned by the procedure. The result is written to the Neo4j database instead of the GDS in-memory graph.