Java Garbage Collection (gc) pauses are monitored by the MonitorGc process in Neo4j, and recorded in the $NEO4J_HOME/logs/debug.log ( or $NEO4J_HOME/data/graph.db/messages.log for Neo4j v2.3.x and prior). To quickly find the 10 longest pauses in

Neo4j 3.0 through Neo4j 3.3 one can run:

grep -n -i blocked debug.log | sort -r -n -k 11 | head -10

which will result in output similar to:

7830:2016-12-05 19:04:23.865+0000 WARN  [o.n.k.i.c.MonitorGc] GC Monitor: Application threads blocked for 707ms.
3283:2016-12-02 13:25:52.264+0000 WARN  [o.n.k.i.c.MonitorGc] GC Monitor: Application threads blocked for 579ms.
6360:2016-12-02 15:38:35.502+0000 WARN  [o.n.k.i.c.MonitorGc] GC Monitor: Application threads blocked for 381ms.
6349:2016-12-02 15:02:33.156+0000 WARN  [o.n.k.i.c.MonitorGc] GC Monitor: Application threads blocked for 285ms.
6361:2016-12-02 15:50:45.560+0000 WARN  [o.n.k.i.c.MonitorGc] GC Monitor: Application threads blocked for 274ms.
6920:2016-12-05 16:57:47.170+0000 WARN  [o.n.k.i.c.MonitorGc] GC Monitor: Application threads blocked for 251ms.

For Neo4j 3.4 and above run

grep -n "VM stop-the-world pause" debug.log | sort -r -k 10 | head -10

which will result in output similar to

27675:2019-04-09 10:39:27.115+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=516, gcTime=508, gcCount=1}
2671:2019-04-08 08:03:12.921+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=483, gcTime=487, gcCount=1}
27885:2019-04-09 11:01:07.033+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=371, gcTime=422, gcCount=1}
16302:2019-04-09 09:53:31.066+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=338, gcTime=372, gcCount=1}
14283:2019-04-08 16:54:03.108+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=272, gcTime=358, gcCount=1}
27904:2019-04-09 11:26:29.463+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=346, gcTime=355, gcCount=1}
4006:2019-04-08 12:14:33.514+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=352, gcTime=349, gcCount=1}
14519:2019-04-08 20:37:12.126+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=253, gcTime=345, gcCount=1}
3983:2019-04-08 11:17:23.039+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=301, gcTime=322, gcCount=1}
16110:2019-04-09 07:19:25.626+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=318, gcTime=314, gcCount=1}

The example grep command will perform a case-insensitive search (as a result of the -i argument) of the debug.log for the string ‘blocked’ and return the respective line number (as a result of the -n argument). These results are then sorted by way of the field representing the time (as a result of -k 11 and -k 10 respectively) and sorted in numeric order rather than ASCII order (as a result of the -n argument) and then presented in reverse chronological order (as a result of the -r argument). Lastly the head -10 will only display the top 10 results after the sort.

Details


Author:
Dana Canzano
Applicable versions:
2.3, 3.0, 3.1
Keywords:
long GC pauseneo4j-2.3neo4j-3.0neo4j-3.1