Meltdown & Spectre: Current Results from Neo4j Performance Testing


Learn how the Meltdown and Spectre security vulnerabilities affect Neo4j graph database performanceAs mentioned in our earlier blog post about Meltdown and Spectre, we have been running tests to discover the impact on Neo4j of the patches and workarounds for the Meltdown and Spectre vulnerabilities.

The Neo4j engineering team has now completed a series of tests – in various environments and under a wide range of workloads – and we’d like to share the results.

Server Types


To try to understand the performance impact across different Neo4j deployment configurations, we have tested in three different environments, as follows:

  1. AWS Ubuntu instances
    Instance type: M3 Large
    Meltdown patch tested with: Linux, 4.4.0-1047-aws
  2. Dedicated hardware, low end servers
    Instance type: 1 x 4-core Xeon Skylake-DT – 64GB RAM – SSD
    Meltdown patch tested with: Linux, 4.4.0-109
  3. Dedicated hardware, medium level servers
    Instance type: 1 x 22-core Xeon Gold 6152 (22C/44T) @ 2.1Ghz – 512GB RAM – SSD
    Meltdown patch tested with: Linux, 4.4.0-109

Workloads


We tested with several performance suites, each generating a distinctly different set of loads:

  1. Internal performance testing workloads based on the LDBC SNB Interactive benchmark.
    • Store sizes varying between 1-200GB
    • Tested with up to 64 concurrent clients
  2. Realistic read and write workloads against real (customer-provided) database stores
  3. Targeted micro-benchmarks, designed to stress the major internal components of Neo4j
  4. Very large data imports, using datasets ranging from 100 million entities to 100 billion entities

Neo4j Versions


We ran the workloads against the latest patch release of all the supported versions of Neo4j:
    • 3.0.12
    • 3.1.7
    • 3.2.9
    • 3.3.2

Results


The combination of server types, workloads and Neo4j versions creates a large matrix of tested scenarios. For every scenario we captured results from both before and after applying the relevant Meltdown patch.

Our analysis of the results shows only negligible performance impacts. Any change is within the range of variance we normally observe. These charts show results for two of the scenarios in the context of normal variation:

Neo4j Meltdown patch performance test results


Neo4j LDBC performance benchmark testing for Meltdown and Spectre


The results suggest that Neo4j users using similar servers should not experience any significant performance degradation. However, we are continuing our testing and will report back if we find scenarios where there is a measurable impact.

For advice about your specific deployment, please contact Neo4j Support.

Unrelated to Meltdown, the second chart shows some substantial improvements in Neo4j performance from versions 3.1 → 3.2 and from 3.2 → 3.3. This is as expected and reflects ongoing efforts to improve the product across multiple dimensions.

Future Work


We expect further OS patches and firmware patches to become available over the coming weeks and months. Especially, we are waiting for stable patches from Intel. We’ll continue our testing to evaluate these patches as the become available.