Going Meta: A Season of Building (and Grading) Ontologies

We take a look back at the past season of Going Meta, this time with ontologies as the thread running through nearly every episode. This is our recap of Season 3, which Jesús Barrasa and I officially closed out this month.

As always, thank you for sticking with us. Our GitHub repository, where we gather the code, queries, datasets and ontologies for each episode, continues to grow alongside the series.

We’ve been building with ontologies on this show for years. This season, the rest of the industry caught up. At the AI Engineer World’s Fair this year, one full session (Why Agentic Systems Need Ontologies by Frank Coyle, UC Berkeley) argued that a proper AI agent doesn’t really work without an ontology behind it. We heard versions of that same argument in hallway conversations, unconference sessions, and Slack threads all season. Season 3 is our answer to the follow-up questions: how do you actually create a good ontology, how do you know it’s good, and once you’ve got agents in the picture, what do you use it for?

Building the ontology itself

Two episodes tackled the same hard problem from different angles: how do you get from “here’s some data and some questions I need answered” to an actual ontology, without hand-rolling one from scratch?

In episode 1, we ran Neo4j’s data modeling MCP server against three inputs: sample data, a set of competency questions describing what the model needs to answer, and any pre-existing ontologies worth reusing. The server loops through candidate models, gives you a Mermaid visualization to react to, and once you’re happy, serializes the result as OWL in Turtle. We ran it against one of our favorite datasets, the blue plaques of London.

Episode 7 packaged the same methodology as an agent skill rather than an MCP server, with one important addition: at each iteration, we ran the quality metrics from episode 4 (more on this below) against the ontology in progress. That way, as the model gets richer, you catch it if it starts getting worse.

Episodes:

• S03E01 — Ontology Creation with the Neo4j MCP Server
• S03E07 — Agent Skills for Creating and Using Ontologies

The ontology-as-guardrail myth

Episode 3 pushed back on a claim making the rounds: that passing an ontology to an LLM during entity extraction guarantees a perfectly aligned graph and eliminates hallucinations. It doesn’t, and we showed why in detail. Once an ontology reaches an LLM, it’s just text. The model isn’t a reasoning engine applying your OWL axioms deterministically; it’s using the ontology as guidance, with results that vary by model quality.

The most interesting find: we compared passing the ontology as serialized OWL/Turtle against describing the same classes and properties in plain, human-readable language. The plain-language version performed just as well, sometimes better. Structure alone doesn’t make the difference. What actually helps is validating the output against your schema afterward, with something like Pydantic, so you can force structured output to comply even when the model’s reasoning doesn’t.

Episode:

• S03E03 — Ontologies, LLMs, Property Graphs: Do We Really Have a “Jaguar Problem”?

Measuring ontology quality

If ontologies are artifacts you build, you need a way to grade them, so episode 4 went looking for one. We pulled metrics from the academic literature and grouped them into four categories: structural (orphan classes, overly connected nodes, and other graph-shape issues, since an ontology is a graph too), requirement alignment (scoring how well the ontology answers your competency questions, judged by an LLM), pragmatic (documentation quality, naming conventions), and logical (catching inconsistencies with a formal reasoner once you introduce axioms). We turned all four into a set of Cypher-based checks and a dashboard, both of which are in the GitHub repo.

Episode:

• S03E04 — Evaluating the Quality of Ontologies

Validating the shape of your graph

Episode 6 brought back SHACL, a vocabulary for defining constraints on your graph: cardinality, value types and more, which we regularly covered during the seasons. We built on that foundation, taking constraints defined in SHACL and generating Cypher-based validation reports that flag exactly where your graph breaks the shape you defined.

Episode:

• S03E06 — Validating Neo4j Graphs with SHACL

Ontologies driving agents

A run of episodes explored ontologies as something an agent actively uses, not just a diagram sitting next to your graph. In episode 2, we used Aura Agents in their early days to define tools generically instead of hardcoding them. Rather than writing separate vector search, text-to-Cypher, or parameterised query tools for every use case, we built a small set of generic tools that checked an ontology to decide what to do. Change the ontology, and you change the agent’s behavior without touching a single tool definition. We ran this against the Tate Modern art catalogue.

Episodes 5, 8, and 9 took that same idea and applied it to agent memory, and that’s when Will Lyon joined the show as a regular co-host. Episode 5 was our introduction, where we walked through the three memory types Neo4j’s agent memory package supports: short-term (the conversation itself), long-term (the entity graph extracted from it) and reasoning memory (the decision traces and tool calls an agent makes along the way).

Episode 8 put reasoning memory to work on the ontology-creation problem itself. Every modeling decision the ontology-building agent makes gets traced and persisted in Neo4j. That matters because ontologies evolve. You might design the first version in one tool this month and come back to extend it in a different tool months later; having the trace means you can pick up the reasoning where you left off instead of accidentally reversing an earlier decision. We called this one the graph behind the graph.

Episode 9 made the ontology-memory connection explicit. Will framed agent memory as one category within a broader “knowledge layer”: data about the data, capturing what an agent learns from its interactions and its own reasoning. The point was that an ontology tells your agent what to remember in the first place. Feed it your domain model, and it knows a transaction and a customer matter; feed it nothing, and it falls back to a sensible default, a generic person/organisation/location/event model the agent memory package ships with (POL, or POL+O once you add objects).

We also opened up the extraction pipeline that turns unstructured conversation into a long-term memory graph. It runs in three configurable stages, each one a fallback for the one before it: fast, free statistical NLP (spaCy) first, then small local models (GLiNER for entities, GLiREL for relationships) that still run on CPU and only then an LLM for whatever’s left. You can turn the LLM off entirely for air-gapped or cost-sensitive setups. From there, Will walked through NAMS, Neo4j’s hosted agent memory service: creating a workspace, posting a message and watching the extraction pipeline turn it into a small graph in real time. Swapping the default POL ontology for a healthcare-specific one changed the extraction output immediately: sharper entities and more relevant relationships, instead of a generic person-and-organisation sketch. NAMS also accepts existing ontologies in several formats, including RDF/OWL Turtle, so you’re not locked into hand-writing a new one.

Episodes:

• S03E02 — Supercharging Neo4j Aura Agents with Ontologies
• S03E05 — Neo4j Agent Memory
• S03E08 — The Graph Behind the Graph: Tracing Ontology Design Decisions
• S03E09 — Shape Agent Memory with Ontologies

From memory to action: skills

Knowing what an agent remembers is one thing. Getting it to act on that memory is another, and that’s where episode 10 picked up. Will opened with an idea worth sitting with: descriptive knowledge tells an agent about the world, but it doesn’t tell the agent what to do. Skills close that gap.

The episode leaned on research from Neo4j’s own research team: AIP: A Graph Representation for Learning and Governing Agent Skills, which tested what happens when you model a skill (in Anthropic’s sense: a skill.md file plus reference material and scripts) as a graph of dependent steps instead of a wall of prose. Benchmarked against SkillsBench, the graph-structured version showed a statistically significant improvement in pass rate over the plain-prose version, largely because an agent doesn’t have to re-parse and re-infer the same relationships every time it reads the skill.

NAMS builds on that idea with skill distillation: point it at a successful conversation, and it collects the steps an agent actually took, grounded in the real tool calls and entities involved, into a versioned, reviewable skill. Because each skill step is grounded in real data, NAMS can also detect when a skill has drifted: if an underlying tool starts failing or an entity it depends on changes, the skill is flagged for review rather than quietly going stale. Community detection prevents skills from sprawling across unrelated topics, splitting them into a graph of skills that call each other when needed. The benefits map directly to what teams keep asking for: auditability of why an agent did what it did, more deterministic behavior and real token savings. One customer who piloted this cut token usage by roughly a third in about a week.

Episode:

• S03E10 — Actionable Knowledge for Agents with Context Graphs

Season 04

We’re taking September off. After that, Going Meta comes back for Season 4, with Will staying on the show, and we’re widening the lens: less “here’s one ontology technique” and more a broader look across the whole AI-and-ontologies landscape as it keeps evolving. See you in October or November.

Resources

• All Going Meta episodes
• GitHub repo
• Knowledge Layer for Enterprise AI

Going Meta: A Season of Building (and Grading) Ontologies was originally published in Neo4j Developer Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.