Evaluating Memory Quality

Available on NAMS: Yes — with backend-specific differences (some steps run server-side on NAMS, and a few sub-features are bolt-only). See the capability matrix for the details.

How to run labeled regression tests against your memory graph using the evaluation harness.

The harness is a scaffold, not a benchmark — three dimensions, simple metrics, no opinionated dataset. Use it to detect regressions when you change extraction settings, dedup thresholds, or schema, not to make "library X scores Y on benchmark Z" claims.

Dimensions

Dimension Metric

Retrieval relevance

Recall@k of client.long_term.search_entities(query, limit=k) against a labeled set of expected entity ids per query.

Audit completeness

Recall of (:Entity)←[:TOUCHED]-(:ReasoningStep) traversal against a labeled set of expected step ids per entity.

Preference fidelity

F1 score of client.long_term.get_preferences_for(user, active_only=True) against the expected active preference ids per user.

Goal

Run a suite of labeled cases and score each dimension:

from neo4j_agent_memory.memory.eval import (
    AuditCase,
    EvalSuite,
    PreferenceCase,
    RetrievalCase,
)

suite = EvalSuite(
    retrieval=[
        RetrievalCase(
            query="healthcare consultants",
            expected_entity_ids={"entity-anthem", "entity-sara"},
            k=5,
        ),
    ],
    audit=[
        AuditCase(
            entity_id="entity-anthem",
            expected_step_ids={"step-1", "step-2"},
        ),
    ],
    preference=[
        PreferenceCase(
            user_identifier="[email protected]",
            expected_active_pref_ids={"pref-senior-healthcare"},
        ),
    ],
)

report = await client.eval.run(suite)
print(f"Overall: {report.overall_score:.2f}")
print(f"Retrieval recall: {report.retrieval.score:.2f}")
print(f"Audit recall:     {report.audit.score:.2f}")
print(f"Pref F1:          {report.preference.score:.2f}")

Steps

1. Build a labeled seedset

The labels are the hard part. Two reasonable starting points:

  • Capture from production: pick a handful of representative retrieval queries; for each, record the entity ids your team agrees are correct hits. Re-evaluate periodically.

  • Synthesize from fixtures: seed the database with a known graph (the examples/audit-trail/ pattern works well), then label expectations explicitly in test code.

The harness doesn’t know how you produced the labels — it just compares to whatever you provide.

Two properties of the library shape retrieval labels: add_entity embeds the entity name (the description is metadata, not retrieval signal), and search_entities applies a 0.7 similarity floor. A labelled query that is semantically close to the description but far from the name scores 0.

Make preference cases falsifiable — seed the superseded state too, and leave the superseded id out of the expected set:

old = await client.long_term.add_preference(
    "consultants", "Any seniority is fine for healthcare engagements",
    user_identifier=user,
)
new = await client.long_term.add_preference(
    "consultants", "Only principals and senior managers on payer accounts",
    user_identifier=user,
)
# Two positional ids — there is no `new_preference=` keyword.
await client.long_term.supersede_preference(old.id, new.id)

case = PreferenceCase(
    user_identifier=user,
    expected_active_pref_ids={str(new.id)},  # `old.id` must NOT come back
)

With MemorySettings.memory.multi_tenant=True, every scoped write must carry user_identifier= or raise — so seed two tenants and give each its own case, and a cross-tenant leak shows up as an F1 miss rather than a silent pass.

2. Run the suite

report = await client.eval.run(suite)

By default every dimension with cases is evaluated. To run a subset:

report = await client.eval.run(suite, dimensions=["audit"])

Skipped dimensions show as None on the report.

3. Inspect per-case detail

DimensionReport.details lists each case with its expected vs. actual ids, recall (or precision/recall/F1 for the preference dimension), and the case parameters. Useful for debugging regressions:

for d in report.audit.details:
    if d["recall"] < 1.0:
        print(f"Audit miss for entity {d['entity_id']}:")
        print(f"  expected = {d['expected']}")
        print(f"  actual   = {d['actual']}")

4. Wire into CI

Treat the suite as a regression test: fail the build below a threshold, and write a machine-readable report so the failure is diffable.

report = await client.eval.run(suite)
payload = {
    "overall": report.overall_score,
    "audit": report.audit.score if report.audit else None,
    "retrieval": report.retrieval.score if report.retrieval else None,
    "preference": report.preference.score if report.preference else None,
}
Path("eval-report.json").write_text(json.dumps(payload, indent=2))

if report.overall_score < MIN_SCORE:
    raise SystemExit(f"Memory-quality regression: {report.overall_score:.2f}")

Then call it from the workflow and keep the report as an artifact:

- run: uv run python examples/eval-harness/ci_gate.py --min-score 0.9 --report eval-report.json
- uses: actions/upload-artifact@v4
  if: always()
  with:
    name: eval-report
    path: eval-report.json

examples/eval-harness/ci_gate.py is that script, ready to copy: it exits 0 at or above the threshold, 1 below it, and 2 when the run itself failed. Appending one JSONL row per run (main.py --out trend.jsonl) gives you the score over time instead of a single snapshot.

What the harness is not

  • Not a public benchmark — your labels are domain-specific.

  • Not a replacement for hand inspection — high recall@k can hide systematic bias toward popular entities.

  • Not a substitute for the ConsolidationRun audit trail — the eval harness measures current state; audit nodes record change over time.

See Also