Customer Metrics Integration (CMI)

An application performance monitoring system can be configured to fetch metrics of Neo4j Aura Enterprise instances. This gives users access to their Neo4j Aura instance metric data for monitoring purposes. Analyzing the metrics data allows users to:

  • Optimize their Neo4j load

  • Adjust Aura instance sizing

  • Set up notifications

Process overview

process

Detailed steps

  1. Log in to Aura as tenant admin.

  2. Make sure there is a dedicated Aura user to use for fetching metrics. You can either:

    • Create a new user:

      1. In "User Management" of Neo4j Aura, invite a new user, selecting "Metrics Integration Reader" as a role. invite user

      2. Follow the invitation link and log in to Neo4j Aura.

      3. Confirm the tenant membership.

    • Or you can find an existing user in "User Management" and change its role to "Metrics Integration Reader"

      Capabilities of users with the role "Metrics Integration Reader" are limited to fetching the metrics and getting a read-only view of the tenant.

  3. Ensure you are logged in to Aura as the user selected in the previous step. In "Account Details", create new Aura API credentials. Save client secret.
    api credentials

  4. Configure the APM system to fetch metrics from the URL(s) or configuration templates shown in "Metrics Integration" of Neo4j Aura. Use oauth2 type of authentication specifying the Client ID and Client Secret created in the previous step. See examples for Prometheus and Grafana and Datadog below.

  5. Use the APM system to create visualizations, dashboards, and alarms based on Neo4j metrics.

Security

Metrics for a Neo4j Aura instance are only returned if all the following are true:

  • Authorization header of the metrics request contains a valid token.

  • The token was issued for an Aura user with "Metrics Integration Reader" role.

  • Tenant is of type enterprise.

  • The specified instance belongs to the specified tenant.

Revoke access to metrics

To revoke a user’s access to metrics of a specific tenant, remove the user from that tenant in "User Management". After that, the user still exists but its connection to the tenant is removed.

The revocation described takes effect after the authorization caches expire, which takes approximately 5 minutes. It results in HTTP 401 being returned, along with the message User doesn’t have access to Metrics resources. However, if you remove only the Aura API credentials used to retrieve metrics, the revocation will take effect only after the tokens issued with these credentials expire, as no new token can be issued anymore. Currently used token expiration time is 1 hour.

Metric labels

Depending on the metric, the following labels are applied:

  • aggregation: the aggregation used to calculate the metric value, set on every metric. Since the Neo4j instance is deployed as a Neo4j cluster, aggregations are performed to combine values from all relevant cluster nodes. The following aggregations are used: MIN, MAX, AVG and SUM.

  • instance_id: the Aura instance ID the metric is reported for, set on every metric.

  • database: the name of the Neo4j database the metric is reported for. Set to neo4j by default.

Example
# HELP neo4j_database_count_node The total number of nodes in the database.
# TYPE neo4j_database_count_node gauge
neo4j_database_count_node{aggregation="MAX",database="neo4j",instance_id="78e7c3e0"} 778114.000000 1711462853000
Looking up metric name in Neo4j Aura Advanced Metrics

In Neo4j Aura Advanced Metrics, it is possible to find out the metric name that corresponds to the chart, by using the chart menu item "Metrics Integration" as shown.

advanced metrics

Metric scrape interval

Recommended scrape interval for metrics is in the range of 30 seconds up to 2 minutes, depending on requirements. The metrics endpoint caches metrics for 30 seconds.

Example using Prometheus

Install Prometheus

One way is to get a tarball from https://prometheus.io/docs/prometheus/latest/installation/

Configure Prometheus

To monitor one or more instances, add a section to the Prometheus configuration file prometheus.yml.

Copy the configuration section proposed in Metrics Integration, as shown.

Replace the placeholders <AURA_CLIENT_ID> and <AURA_CLIENT_SECRET> with corresponding values created in the previous step.

metrics integration
Start Prometheus
./prometheus --config.file=prometheus.yml
Test that metrics are fetched

Open http://localhost:9090 and enter a metric name or expression in the search field (ex. neo4j_aura_cpu_usage).

Use Grafana

Install and configure Grafana, adding the endpoint of the Prometheus instance configured in the previous step as a data source. You can create visualizations, dashboards, and alarms based on Neo4j metrics.

Example using Datadog

Configure an endpoint with token authentication

Edit /etc/datadog-agent/conf.d/openmetrics.d/conf.yaml as follows:

Replace the placeholders <ENDPOINT_URL>, <AURA_CLIENT_ID> and <AURA_CLIENT_SECRET> with corresponding values from the previous steps.

/etc/datadog-agent/conf.d/openmetrics.d/conf.yaml
init_config:
instances:
  - openmetrics_endpoint: <ENDPOINT_URL>
    metrics:
      - neo4j_.*
    auth_token:
      reader:
        type: oauth
        url: https://api.neo4j.io/oauth/token
        client_id: <AURA_CLIENT_ID>
        client_secret: <AURA_CLIENT_SECRET>
      writer:
        type: header
        name: Authorization
        value: "Bearer <TOKEN>"
Test that metrics are fetched
  • sudo systemctl restart datadog-agent

  • Watch /var/log/datadog/* to see if fetching metrics happens or if there are warnings regarding parsing the config.

  • Check in Datadog metric explorer to see if metrics appear (after a couple of minutes).

Programmatic support

Aura API for Metrics Integration
  • Aura API (v1beta5) now supports fetching metrics integration endpoints using:

    • endpoint /tenants/{tenantId}/metrics-integration (for tenant metrics)

    • JSON property metrics_integration_url as part of /instances/{instanceId} response (for instance metrics)

  • Reference: Aura API Specification

Aura CLI for Metrics Integration
  • Aura CLI has a subcommand for tenants command to fetch tenant metrics endpoint:

    aura tenants get-metrics-integration --tenant-id <YOUR_TENANT_ID>
    
    # example output
    {
      endpoint: "https://customer-metrics-api.neo4j.io/api/<YOUR_TENANT_ID>/metrics"
    }
    
    # extract endpoint
    aura tenants get-metrics-integration --tenant-id <YOUR_TENANT_ID> | jq '.endpoint'
  • For instance metrics endpoint, Aura CLI instances get command JSON output includes a new property metrics_integration_url:

    aura instances get --instance-id <YOUR_INSTANCE_ID>
    
    # example output
    {
        "id": "id",
        "name": "Production",
        "status": "running",
        "tenant_id": "YOUR_TENANT_ID",
        "cloud_provider": "gcp",
        "connection_url": "YOUR_CONNECTION_URL",
        "metrics_integration_url": "https://customer-metrics-api.neo4j.io/api/<YOUR_TENANT_ID>/<YOUR_INSTANCE_ID>/metrics",
        "region": "europe-west1",
        "type": "enterprise-db",
        "memory": "8GB",
        "storage": "16GB"
      }
    
    # extract endpoint
    aura instances get --instance-id <YOUR_INSTANCE_ID> | jq '.metrics_integration_url'
  • Reference: Aura CLI cheetsheet

Metric Definitions

Title

Description

MetricName

MetricType

DefaultAggregation

Out of Memory errors

The total number of Out of Memory errors for the instance. Consider increasing the size of the instance if any OOM errors.

neo4j_aura_out_of_memory_errors_total

Counter

SUM

CPU Available

The total CPU cores assigned to the instance nodes.

neo4j_aura_cpu_limit

Gauge

MAX

CPU Usage

CPU usage (cores). CPU is used for planning and serving queries. If this metric is constantly spiking or at its limits, consider increasing the size of your instance.

neo4j_aura_cpu_usage

Gauge

MAX

Memory Used

Memory in use by Neo4j.

neo4j_aura_memory_usage

Gauge

MAX

Memory Total

The total memory assigned to the instance.

neo4j_aura_memory_limit

Gauge

MAX

Storage Total

The total disk storage assigned to the instance.

neo4j_aura_storage_limit

Gauge

MAX

Heap Used

The percentage of configured heap memory in use. The heap space is used for query execution, transaction state, management of the graph etc.
The size needed for the heap is very dependent on the nature of the usage of Neo4j. For example, long-running queries, or very complicated queries, are likely to require a larger heap than simpler queries. To improve performance, the heap should be large enough to sustain concurrent operations.
This value should not exceed 80% for long periods, short spikes can be normal. In case of performance issues, you may have to tune your queries and monitor their memory usage, to determine whether the heap needs to be increased. If the workload of Neo4j and performance of queries indicates that more heap space is required, consider increasing the size of your instance. This helps avoid unwanted pauses for garbage collection.

neo4j_dbms_vm_heap_used_ratio

Gauge

MAX

Page Cache Usage Ratio

The percentage of the allocated page cache in use. If this is close to or at 100%, then it is likely that the hit ratio will start dropping, and you should consider increasing the size of your instance so that more memory is available for the page cache.

neo4j_dbms_page_cache_usage_ratio

Gauge

MIN

Bolt Connections Running

The total number of Bolt connections that are currently executing Cypher transactions and returning results. This is a set of snapshots over time and may appear to spike if workloads are all completed quickly.

neo4j_dbms_bolt_connections_running

Gauge

MAX

Bolt Connections Idle

The total number of Bolt connections that are connected to the Aura database but not currently executing Cypher or returning results.

neo4j_dbms_bolt_connections_idle

Gauge

MAX

Bolt Connections Closed

The total number of Bolt connections closed since startup. This includes both properly and abnormally ended connections. This value may drop if background maintenance is performed by Aura.

neo4j_dbms_bolt_connections_closed_total

Counter

MAX

Bolt Connections Opened

The total number of Bolt connections opened since startup. This includes both successful and failed connections. This value may drop if background maintenance is performed by Aura.

neo4j_dbms_bolt_connections_opened_total

Counter

MAX

Garbage Collection Young Generation

Shows the total time since startup spent clearing up heap space for short lived objects. Young garbage collections typically complete quickly, and the Aura instance waits while the garbage collector is run. High values indicate that the instance is running low on memory for the workload and you should consider increasing the size of your instance.

neo4j_dbms_vm_gc_time_g1_young_generation_total

Counter

MAX

Garbage Collection Old Generation

Shows the total time since startup spent clearing up heap space for long-lived objects. Old garbage collections can take time to complete, and the Aura instance waits while the garbage collector is run. High values indicate that there are long-running processes or queries that could be optimized, or that your instance is running low on CPU or memory for the workload and you should consider reviewing these metrics and possibly increasing the size of your instance.

neo4j_dbms_vm_gc_time_g1_old_generation_total

Counter

MAX

Replan Events

The total number of times Cypher has replanned a query since the server started. If this spikes or is increasing, check that the queries executed are using parameters correctly. This value may drop if background maintenance is performed by Aura.

neo4j_database_cypher_replan_events_total

Counter

MAX

Active Read Transactions

The number of currently active read transactions.

neo4j_database_transaction_active_read

Gauge

MAX

Active Write Transactions

The number of currently active write transactions.

neo4j_database_transaction_active_write

Gauge

MAX

Committed Transactions

The total number of committed transactions since the server was started. This value may drop if background maintenance is performed by Aura.

neo4j_database_transaction_committed_total

Counter

MAX

Peak Concurrent Transactions

The highest number of concurrent transactions detected since the server started. This value may drop if background maintenance is performed by Aura.

neo4j_database_transaction_peak_concurrent_total

Counter

MAX

Transaction Rollbacks

The total number of rolled-back transactions. This value may drop if background maintenance is performed by Aura.

neo4j_database_transaction_rollbacks_total

Counter

MAX

Checkpoint Events

The total number of checkpoint events executed since the server started. This value may drop if background maintenance is performed by Aura.

neo4j_database_check_point_events_total

Counter

MAX

Checkpoint Events Cumulative Time

The total time in milliseconds spent in checkpointing since the server started. This value may drop if background maintenance is performed by Aura.

neo4j_database_check_point_total_time_total

Counter

MAX

Last Checkpoint Duration

The duration of the last checkpoint event. Checkpoints should typically take several seconds to several minutes. Values over 30 minutes warrant investigation.

neo4j_database_check_point_duration

Gauge

MAX

Relationships

The total number of relationships in the database.

neo4j_database_count_relationship

Gauge

MAX

Nodes

The total number of nodes in the database.

neo4j_database_count_node

Gauge

MAX

Store Size Database

Amount of disk space used to store user database data, in bytes. Ideally, the database should all fit into memory (page cache) for the best performance. Keep an eye on this metric to make sure you have enough storage for today and for future growth. Check this metric with page cache usage to see if the data is too large for the memory and consider increasing the size of your instance in this case.

neo4j_database_store_size_database

Gauge

MAX

Page Cache Evictions

The number of times data in memory is being replaced in total. A spike can mean your workload is exceeding the instance’s available memory, and you may notice a degradation in performance or query execution errors. Consider increasing the size of your instance to improve performance if this metric remains high.

neo4j_dbms_page_cache_evictions_total

Counter

MAX