Large installations

NOM agents and NOM server are designed to cope with the load of a typical Neo4j deployment without any additional configuration. This section documents the configuration options that are relevant when running NOM against Neo4j deployments with a high query or metric throughput.

All of the settings described here are optional. The defaults are safe for typical deployments and only need to be adjusted when you observe signs of overload, such as buffer overflow warnings in the server or agent logs, or a NOM persistence database that grows rapidly.

Query log ingestion

NOM agents listen on a local TCP port for Neo4j query log events and forward them in batches to NOM server, which writes them to the NOM persistence database. Under high query load, either the incoming buffer of the agent or the server-side processing queue can overflow.

Limiting the query log volume with a minimum duration

The most effective lever for controlling the query log volume is the minimum duration filter, CONFIG_INSTANCE_N_QUERY_LOG_MIN_DURATION. Only queries whose elapsed time is greater than or equal to this value, in milliseconds, are ingested.

  • The default is 50 ms, which means that queries shorter than 50 ms are discarded by the agent before they are forwarded.

  • Setting the value to 0 means that all queries are forwarded, without any filtering.

  • A good starting point for high-load environments is 100 ms. Lower the threshold gradually once you have confirmed that NOM server can keep up.

  • Errors, that is, queries that produced a failure reason in the Neo4j query log, bypass the duration filter by default, so they are always ingested regardless of their elapsed time. To filter errors by duration as well, set CONFIG_INSTANCE_N_QUERY_LOG_MIN_DURATION_FILTER_ERRORS to true. Make sure to replace N with the number of the dbms on your system you want to monitor

The minimum duration can be set in either of the following ways:

Where set How to set

Agent environment variable

CONFIG_INSTANCE_1_QUERY_LOG_MIN_DURATION=50

Agent configuration file

queryLogMinDuration: "50"

For the complete list of agent query log variables, see Query log collection configuration. For the agent configuration file, see Agent configuration file.

Use parameterized queries. If a workload issues many structurally identical queries with different literal values, for example MATCH (n) WHERE n.id = 42, switching to parameterized queries, for example MATCH (n) WHERE n.id = $id, reduces the number of distinct query log entries that NOM needs to store, and makes the aggregated statistics more meaningful.

If you see the agent log with the following message, it means that the query log ingestion rate is too high for the agent to keep up:

QueryLogReceiverService buffer overflow - dropping message

Seeing this message means you should adjust the CONFIG_INSTANCE_N_QUERY_LOG_MIN_DURATION setting on the agent to reduce the ingestion rate.

Query log retention

Individual queries are retained for 24 hours, after which they are aggregated into one-hour blocks. You can still view information, such as the average time to execute the query and how many times it was executed within the time window. After 7 days, the queries are further aggregated into 24-hour blocks and retained for an additional 30 days.

For more information about how query logs are presented and aggregated, see Log manager.

Metrics aggregation and cleanup

Raw metric data points emitted by the monitored Neo4j instances are stored as individual :Metric nodes in the NOM persistence database. To prevent unbounded growth, NOM periodically aggregates them into hourly :Metric:Aggregate nodes and then deletes the raw nodes.

How aggregation works

  • Aggregation runs on a cron schedule, by default five minutes past every hour (0 05 * * * *).

  • Raw metric nodes are retained for three days (P3D) after they are written. Once a metric node is older than three days, the next aggregation run includes it in the hourly rollup and then deletes the original.

  • The resulting :Metric:Aggregate nodes are retained for 31 days (P31D). A separate cleanup job, running daily at 02:00 (0 0 2 * * *), deletes the aggregate nodes that are older than this window.

For a description of how the aggregated metrics are presented, see Metric manager.

Aggregation and cleanup configuration reference

All the schedules and retention windows above are configurable through NOM server properties. They can be set in application.properties, as environment variables, or as command line arguments to java -jar server.jar, in the same way as any other server property. See Server configuration reference.

Property Environment variable Default Description

metrics.cleanup.schedule

METRICS_CLEANUP_SCHEDULE

0 0 2 * * *

Cron expression that controls how often the deletion of old aggregate nodes runs.

metrics.cleanup.retention-time

METRICS_CLEANUP_RETENTION_TIME

P31D

ISO-8601 duration. Aggregated metric nodes older than this are deleted. P31D is the maximum retention time supported by the UI.

metrics.cleanup.batch-size

METRICS_CLEANUP_BATCH_SIZE

3000

Number of nodes deleted per transaction during cleanup.

The following example passes overrides as Java arguments:

java -jar server.jar \
  --metrics.cleanup.retention-time=P14D

Tuning guidance for large deployments

  • If individual cleanup transactions are too large and cause contention, reduce metrics.cleanup.batch-size, for example to 1000. This makes each transaction smaller, at the cost of more round trips.

  • Running aggregation more frequently, for example every 30 minutes with 0 0/30 * * * *, reduces the number of raw nodes that accumulate between runs, which in turn makes each aggregation pass faster.

Cron expressions follow the Spring Framework six-field format: seconds minutes hours day-of-month month day-of-week.