The payload: the semantic layer, memory, and the loop that fills it
Graph ML and GenAI Research, Neo4j
28 min read
Part 2 of the self-learning multi-agent series. Part 1 wired an agent to the graph; this post is about what travels the wires.
Every useful AI agent runs into the same wall: it is brilliant in the abstract and ignorant in the specifics. It can reason, but it does not know what your revenue column nets out, which account is up for renewal, that last night’s load finished late, or that the join it is about to write was wrong the last three times someone tried it. That knowledge exists, in your warehouse schemas, your glossaries, your runbooks, and mostly in people’s heads, but it is not where the agent can reach it, so the agent guesses, starts every session cold, and repeats a mistake a colleague made a week ago.
The Meta Knowledge Graph (MKG) is the layer that closes that gap. It is a shared, graph-structured context layer for AI agents, and it reaches an agent through the whole harness wiring working together: an MCP server exposing the tools the agent calls to read and write the graph, and the lifecycle hooks that capture what happens and inject what’s relevant before the model answers. It does one job: give whatever is consuming context (an agent, a co-pilot, a BI tool, an application) the operating knowledge it needs to act correctly, and capture what it learns so the next consumer starts further ahead than the last. It is harness-agnostic by design by using Agent Plugins specification; the same graph serves Claude Code, Codex, or a custom loop, so a team of agents shares one memory instead of each drifting in its own silo.
Inside the MKG: The Semantic Layer and the Memory Agents Earn
Part 1A made the architectural argument and Part 1B built the wires, the tools and hooks that connect an agent to the graph. This post is about what travels over them.
And the first thing to say about the payload is what it is not. It is not the data. That is the rule the whole design turns on, carried straight from Part 1A: the graph does not copy what the enterprise’s stores already hold.
The lakehouse, the warehouse, the operational stores, the document repositories keep their data where it already sits, governed and regulated and far too large to migrate into some new agent-memory store. What the graph carries instead is metadata, the operating knowledge an agent needs to act on data it never has to see a copy of, and that metadata lives in two homes. Most of it lives in the semantic layer, the part that describes the data itself: what it is, whether it is healthy, what it means. The rest lives in memory, the part the agents write by working: what was tried, what failed, what the human corrected. The two halves of this post are those two homes, and the closing section draws the edge where this post stops and the next one starts. The semantic layer is authored. Memory is earned.
The semantic layer: meaning before structure
Most of the metadata lives here, because most of what an agent needs to know is about the data, not in it. A schema carries almost none of that. A column called util_pct, a table called acct_pr_usage, tell a model nothing it can trust. An agent left staring at raw DDL will guess the join, guess what one row means, guess what active means, and guess wrong in ways that look entirely plausible.
The semantic layer is the translation between how the business talks and how the warehouse stores. It says this table is monthly account-product utilization, one row is one account per product per month, revenue is already net of discount, the table loaded clean this morning, and at risk means a renewal inside ninety days with utilization trending down. None of those last clauses is in any schema; they are technical, operational, and business facts layered on top of a pointer to the data, and together they are the bulk of what the graph carries.
In this system the semantic layer over the warehouse is Neocarta. We built the first agent for renewals and upsell on a B2B vehicle-rental book, so the demo warehouse is full of rental accounts and utilization tables, but nothing in the approach is about vehicles.
Neocarta is a set of connectors that introspect a source, write a catalog of it into Neo4j as a uniform shape of databases, tables, and columns, and then embed the descriptions so the catalog is searchable by meaning. Point the connectors at your warehouse and you get the same nodes and relationships you would get from any other source; the physical store underneath is an implementation detail the agent never has to know. That uniformity is the whole point. Instead of one bespoke retrieval path per system, every enterprise source is projected into a single representation, and that representation becomes the single source of truth for data search. The agent does not ask which of my N systems might hold renewal data and guess; it asks the catalog where is renewal data and gets account_renewalsback by semantic similarity.
That is the inversion the semantic layer buys you: the agent meets meaning first and structure second. It searches the catalog in business terms, the layer resolves those terms to physical tables and columns, and only then is a query built and run against the source through the source tool. Find first, then fetch. Neo4j plays the same role for the data that is natively graph-shaped, the accounts and contacts no warehouse table ever modeled as connections, where a schema tool and a read tool let the agent learn the shape before it traverses it.
Running Neocarta: connectors in, tools out
Neocarta ships as two things.
First, the connectors, the ingestion step you met above: one per source, run once or on a schedule. Point one at a warehouse and another at an operational store, and both land in the same representation; the physical store stops mattering the moment the catalog is written.
Second, Neocarta ships an MCP server on top of that catalog, which is what makes it easy to adopt. The agent does not have to be taught how the graph is laid out or which Cypher to write to search it. It gets tools out of the box. Some are always mounted: list_schemasto enumerate what is connected, list_tables_by_schemato drill into one. On top of those sit the retrieval tools that find tables and columns by meaning. Mount the server and any harness has a working data-discovery surface with no bespoke integration, and one set of tools works across every source you connect because every source was projected into the same shape going in.
Neocarta now also ships a CLI.
One shape, four kinds of metadata
Under the tools is a single, uniform shape. Every source, whatever it is underneath, becomes the same skeleton in the MKG: a database holds schemas, a schema holds tables, a table holds columns. Columns carry their type, their nullability, and their key flags, and where one column is a foreign key into another the catalog draws that as a reference edge, so the join the agent would otherwise guess is already written down. A handful of example values hang off each column too, so the model can see that a status field actually reads active or churned before it filters on it. That skeleton is the technical metadata: the structure itself, the answer to what is the shape of this thing.
Layered on top of the same nodes is the metadata a schema never carries. Business metadata is the meaning: a description on every table and column saying what it is in the language people actually use, and a business glossary of terms held as their own nodes, each tagged onto the tables and columns it applies to, so at risk or net revenue is a first-class thing in the graph rather than a phrase buried in a comment. The definition of at risk, the fact that revenue is already net of discount, one row meaning one account per product per month, all of it lives here, on nodes that sit on top of a pointer to the data and never the data itself.
Operational metadata is where the catalog is heading next: freshness, when a table last loaded, whether it loaded clean, how healthy it is, the answer to whether the agent can trust a table right now rather than only what it is. That is the layer that turns “this is monthly account-product utilization” into “and it loaded clean this morning,” so the catalog will increasingly carry not just what a table means but whether to rely on it today.
And because meaning and structure live on the same nodes, the catalog is searchable without a second system. Each description is embedded into a vector index and each name, description, and glossary term into a full-text index, so the nodes that hold what a table means are the same nodes you retrieve it by. Technical and business metadata today, operational metadata increasingly, one shape, one place, one thing to search.
We have available an introduction course to Neocarta on our GraphAcademy as well.
Memory: The part no catalog has
The other home for metadata is the part no catalog can author, because it is not about the data, it is about the work. Which tool answered this kind of question. Which join was wrong last time. Which query came back empty and why. What the user corrected. The grain that finally worked for renewal-risk after three tries. This is agentic metadata, the fourth category from the very first post, and it lives in memory rather than the semantic layer for a simple reason: the semantic layer is authored and its meaning changes only when the data platform changes, whereas memory is written by the agents themselves, every session, as a byproduct of doing the job. It is also the only part of the payload that compounds. A glossary is as good on its hundredth day as its first, and that is correct, you do not want your definitions drifting. Memory is supposed to drift, upward, every corrected answer a note the next agent reads before it acts.
Memory fills itself off the hooks from Part 1b, and the shape is one stream in, several consumers out. Every event of the session, the prompts, the tool calls, the results, the failures, is logged into one ordered stream as it happens, with no model judgment yet about what matters. Capture is passive, deterministic, and complete, and the thinking comes later. Later means off the turn: at Stop, rate-limited so it never blocks you, background passes read that raw corpus and make something of it. What differs is the question each pass asks of it, and there are three: what happened, what went wrong, and what is worth remembering.
Episodic memory: what happened
The first pass is unconditional. Every work window becomes one to three Observation nodes on a project timeline: a type (change, bugfix, feature, refactor, discovery, decision, problem), a one-line title, a handful of concrete self-contained facts, and a two-to-four sentence narrative of what was asked, what was done, and how it ended. Observations are chained NEXTto their predecessor and the project keeps a pointer at the head, so the timeline is walkable in order. They are append-only, never gated, never deduplicated, and embedded so they are searchable.
The episodic timeline is the one part we borrowed ideas from claude-mem, which captures a session’s work as append-only observations and replays them at the start of the next one. claude-mem is built around a developer’s files and coding sessions; we kept the mechanism but aimed it at an agent working over data and a domain, whatever that domain happens to be, rather than only a codebase.
Tool errors: capture, consolidation, recall
The second pass is the part most agent systems throw away, and it is a loop with three stages of its own.
Capture is deterministic and out of band. Every time the agent runs a query through the BigQuery or Neo4j tool, a hook inspects the result without taking anything from the model’s turn and classifies what went wrong from the response text and shape, with no model in the path. Not just hard errors. It separates permission failures, timeouts, result-size and resource limits, missing capabilities, schema drift where a column or table does not exist, and plain syntax and parser errors, and it catches the two quiet failures that never raise an exception at all: a query that succeeded but returned zero rows, and a Neo4j result that came back with an empty object where a temporal or spatial value should have been projected. Each becomes a :QueryExecutionnode with typed :QueryIssuechildren, hung off both the project and the session, carrying the query text and an excerpt of the response. It hangs off two hook events rather than one, because harnesses disagree about what a failed tool call even is: Claude Code fires a live PostToolUseFailure, while Codex hands back an error-shaped result on ordinary PostToolUse. And it survives harnesses that fire neither, because the same Stop-time pass re-reads the transcript and reconstructs the failed queries from it, converging on the same node ids instead of duplicating them.
Consolidation is what turns that evidence into guidance. A raw failure log tells the next agent that a query failed. It does not tell it why, or what to do instead, and expecting the model to infer the lesson by reading a pile of stack traces at recall time is asking the expensive path to redo work the cheap path could have done once. So a rate-limited service folds unconsolidated failures into :QueryErrorPatternnodes, each carrying a normalized error signature, the root cause, the actionable fix, and where possible a corrected version of the query. That is the difference between this query failed and queries of this shape fail because X, do Y instead. It runs on every Stop and SessionEnd but does real work rarely: a per-tool cooldown window (six hours by default) and a threshold, so a tool consolidates only once it has more than a trivial number of pending failures to learn from.
What it refuses to consolidate matters as much as what it does. Only failures deterministically caused by the query text itself are eligible: syntax errors, schema mismatches, missing procedures and functions, serialization projections. Timeouts, rate limits, resource exhaustion, and permission errors are excluded twice over, once by hard-coded issue class so they never enter the queue, and again in the prompt, which instructs the model to discard transient one-offs hiding in the generic error bucket. The reason is that a timeout is a fact about the afternoon, not a fact about the query. Consolidating it would teach the next agent a superstition, and a superstition in a memory store is worse than an empty one, because it is confidently wrong and it compounds.
Patterns are anchored per tool rather than per project. A :ToolErrorProfilenode hangs off the project for each tool that has ever failed, the patterns hang off the profile, and each pattern links DERIVED_FROMback to the executions it was distilled from, so every piece of guidance can be traced to the failures that earned it. The grouping is what keeps the advice honest: BigQuery’s dialect quirks are not advice for a Cypher failure, and one tool’s hard-won lesson never surfaces as noise in another’s.
Recall is the payoff, and it closes faster than you would expect. The next time that same tool fails with an invalid-query error, the recall hook searches the tool’s own pattern library, matching hybrid on both the failing query text and the error message, against the vector index and the fulltext index, fused with reciprocal rank fusion, always scoped to the failing tool. Searching on both is what makes it work in both directions: a pattern is found when the error looks familiar, and also when the query looks familiar but the error is phrased differently. The retrieved fix is handed straight back to the model, in the same turn it failed, before it starts retrying blind variations of a query that cannot work.
That is the whole point, and it is a sharper claim than “the graph remembers its failures.” The loop closes inside the turn, not merely across sessions. The agent that hits the error is the agent that gets the fix. Across sessions you get the weaker, still-valuable version: before the next agent writes the join that was wrong last time, the pattern that join produced is already sitting in the tool’s profile waiting to be matched. A failure remembered is a failure the next session does not repeat; a failure consolidated is a failure this session does not repeat twice.
There is one wrinkle worth naming for anyone building this, because it is the kind of thing you only find by trying. PostToolUsesupports returning additionalContext, which is the clean way to put text in front of the model. PostToolUseFailuredoes not; its documented channel back to the model is a top-level decision: “block” whose reason is shown. So the hook “blocks” a tool call that already failed, purely to get the fix into context. Nothing is actually blocked. It is a small abuse of a control-flow verb, and it is the only door in the wall.
The same hook, different use-case
The PostToolUsehook also turns out to be a general way to materialise structured tool output into the graph, not only failures and not only memory. The implementation in MKG is specifically for Diffbot in the B2B sales demo. Some tools do not return an error to classify, they return structured facts worth keeping. In that demo the agent can call two Diffbot tools, enhance_entityto resolve a company or a person and search_newsto pull recent coverage, and both come back as structured JSON rather than prose. A PostToolUsehook matched on those tools unwraps that JSON and merges it straight into the domain graph: DiffbotOrganization and DiffbotPerson nodes hung off the internal Account they enrich, NewsArticle nodes linked to the accounts and organizations they mention, each keyed on its real Diffbot id so the same entity seen twice lands on one node rather than a duplicate. The effect is that an external lookup is paid for once. The first session to ask about an account calls the API and the answer is written into the graph as durable, queryable structure, and every session after that reads the enrichment from the graph, next to the account record, and can traverse and join and reason over it the way it would any other graph-shaped data. The API call becomes the exception rather than the habit, and the enrichment stops evaporating with the conversation that fetched it.
Learnings: the durable tier
The third pass asks the selective question, and its output is small and typed on purpose. Reusable facts, environment quirks, durable preferences, and task patterns all become :Learning candidates. Each candidate carries an action rather than just text: create a new item, update one that already says materially the same thing, or ignore work that was routine and transient. That last verb carries the design, because the store is meant to consolidate rather than accrete, so a handful of durable notes beats a transcript nobody will read, exactly as a lean tool surface beat a bloated one in Part 1.
Two constraints on this pass matter more than they look. The extractor reads conversation text and never tool outputs. That is a security boundary, not a performance tweak: a tool result is attacker-reachable in a way the user’s own words are not, and a poisoned web page, a hostile row, or a doctored API response should not be able to write itself into durable memory simply by being returned to the model. Structured tool results do reach the graph, as the Diffbot path above shows, but they arrive as typed nodes with real ids and known provenance, where a hostile value is a bad row you can find and delete, not as free-text belief the agent treats as its own. And the extraction prompt is a fixed constant in code: changing what the extractor looks for means editing the template, deliberately, in a commit. Nothing tunes it at runtime, and nothing should, because a prompt that rewrites itself on the strength of its own output is a feedback loop with no ground truth anywhere in it, and its failure mode is silent and slow.
User scope versus project scope
One distinction runs through all of it, and it decides whether a note follows you or stays put. Not everything an agent learns belongs to the repository it learned it in. A durable fact about the person, their name and role, that they prefer terse answers or British spelling, that they always want the query cost before it runs, a recurring constraint or a standing priority, is user-scoped, and it should follow them into the next project rather than dying with this one. A fact about the warehouse, the domain, the codebase, the way this renewals agent joins usage to renewals, is project-scoped and stays where it is. The extractor makes that call as it writes: it defaults to project only when the signal is genuinely ambiguous, but scopes to the person the moment a signal is unmistakably about them, on first mention, without waiting for it to repeat. Repetition is required only for borderline traits it is inferring rather than ones the person actually stated.
The two scopes are recalled on different wires. User-scoped memory is injected at session start, because it is true before you have typed anything. Project-scoped memory is injected against the specific prompt you just sent, because its relevance depends on what you are asking. One store, two lifetimes: the person’s preferences compound across every project while each project’s hard-won specifics stay local to it.
Beyond automated injection, the stores are also exposed as MCP tools, an explicit runtime interface for the agent to query, append, or patch memory as it works. Recall, though, is the easy half. What separates the two scopes is what it takes to get written in.
Extraction never writes memory directly, because an agent that simply saved everything it did would consolidate its mistakes as faithfully as its wins, and frequency is not correctness. So it writes candidates, and between a candidate and the durable store sit two gates.
The first is automatic, and the useful thing about it is that it is not a new system bolted on the side: it runs on the machinery memory already uses. Every learning is embedded the moment it is written, which is exactly what lets recall find it by meaning later, and the gate simply turns that search inward. It embeds the fresh candidate, retrieves its nearest neighbours in the same project and the same scope, and asks an LLM judge two narrow questions: does this genuinely contradict anything already held, and is it merely a restatement of something already held? The answers fan out into the outcomes in the diagram: clean candidates are approved, restatements fold into the item they repeat and reinforce it rather than piling up beside it, a winner supersedes a loser, a well-supported incumbent vetoes a speculative newcomer, and anything genuinely ambiguous is left alone rather than guessed at. Rejected and folded items lose their embedding, which drops them out of the vector index while their text and provenance edges stay behind: forgetting, here, is dropping the embedding rather than the row, so what the graph rejected and why is still there to audit.
The second gate is a person, and it exists because the first one is narrow on purpose. The automatic gate answers can these two both be true. It has nothing whatsoever to say about is this true at all, and a confident, coherent, entirely wrong learning that contradicts nothing sails straight through it, because the graph has no ground truth to check it against. Only a person does. So exactly two populations are handed to a human, and only those two: project-scoped candidates the judge marked ambiguous, and every user-scoped candidate without exception. The queue shows both sides of a conflict rather than a decontextualised assertion, and the reviewer can approve, reject, fix the extractor’s wording and promote the corrected version, pick a winner, or keep both, which fits the common case where the conflict was apparent rather than real, two things true in different circumstances.
Why every user fact goes through this, even the obvious ones, is the load-bearing decision in the whole design. A project learning gets recalled: it shows up in context, the model reads it, the model can weigh it against everything else it can see. A user fact does something categorically stronger. It gets folded into the persona, into the agent’s own operating instructions, in every project, permanently, where it is no longer a retrieved claim the model can evaluate but a premise it reasons from. That is the highest-privilege write in the system, and the blast radius of one bad entry is every future session everywhere. So nothing reaches it without a person saying yes. An unreviewed candidate cannot rewrite the persona on its own no matter how confident the extractor was, and that is precisely what stops a single poisoned fact, or just a sloppily-inferred one, from becoming permanent.
Past that boundary the consolidation runs, and it is the user’s own memory. Every so often, off the turn and rate-limited, a reflection pass (a kind of dreaming) gathers the approved user facts that have piled up unfolded and rewrites the agent’s system prompt to account for who it now knows it is working with. It insists on an edit rather than a rewrite, so the persona does not drift into something nobody chose, and it fences the facts as an untrusted description of the user: data to summarise, never instructions to follow. That fence sits behind the human gate, which is the point of defence in depth: the reviewer approved a fact as true, not as a command, and the two are different claims. The result is versioned, the outgoing prompt archived before the new one takes over, and the active version injected at the start of every session. It is memory graduating from something the agent looks up into something the agent already is by the time it says hello.
That is the shape of it, and deliberately only the shape. How consolidation really works once it is doing more than folding duplicates and rewriting a persona, what it means to compress a project’s memory rather than a person’s, is the subject of the next post, in a good deal more depth than a diagram can carry.
Closing the loop
Put the three passes together and the write side has a shape. The timeline records what happened, unconditionally, because a project’s history should not be curated by a model deciding what was interesting. Tool failures distill into per-tool patterns that are replayed the moment the same tool fails the same way. And durable learnings are earned rather than accumulated: scoped to the person or the project, checked mechanically for contradiction, reviewed by a human wherever a machine has no standing to decide, and folded into the persona only once someone has said yes.
What is still ahead is the edge, and it is worth being precise about where it falls. Project learnings are gated, human-reviewable, deduplicated, and reinforced. Tool failures are genuinely consolidated. What project memory still lacks is higher-order consolidation. Nothing clusters related learnings into the theme they are all circling. Nothing compresses fifty specific notes into the five principles behind them. And nothing forgets: a learning approved once stays exactly as loud on its thousandth day as its first, whether or not anything has reinforced it since.
If you remember one thing: the timeline records everything, tool failures teach the same session that hit them, and nothing becomes durable memory until it earns its place, mechanically or through a person. One graph carries all of it. We are building this in the open. The repository is linked below, everything in this post is in it, and we would like you to build the next iteration with us.
The payload: the semantic layer, memory, and the loop that fills it was originally published in Neo4j Developer Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.
Share Article
Explore
Related Articles
Secure GraphRAG: Wiring Microsoft Copilot Studio to Neo4j with per-user Okta SSO
From first graph algorithm experiment to production with Aura Graph Analytics
One starter repo, three winners: Neo4j at Hack the Video Agent Context Graph
