Neo4j Live: From Language to Logic – Why Graph AI Succeeds Where LLMs Fail

When an LLM reads a law, it sees tokens. When a knowledge graph reads a law, it sees structure, hierarchy, and relationships. That distinction between fluent language and reliable logic is at the heart of this Neo4j Live session with Matthias Buchhorn-Roth, AI and Cloud Engineer at Sopra Steria.

Matthias specialises in multi-agent systems, AI frameworks, and cloud data platforms. Here he shares hard-won lessons from building a Graph AI solution for a German government agency navigating one of the most legally complex environments imaginable: the evolving social law landscape of the SGB (Sozialgesetzbuch), Germany’s social code.

The Problem: LLMs Struggle with Legal Hierarchy

The challenge that triggered this project was deceptively simple: build a tool to help government case workers understand how process changes ripple through constantly evolving legislation. The reality turned out to be anything but simple.

German social law is a multi-dimensional matrix. At the top sit supranational frameworks like the EU AI Act. Below that come constitutional basics (Grundgesetz), then federal laws like the BGB, then ordinances, and finally operational guidelines (Handlungsanweisungen) specific to agencies. Each layer is hierarchically ranked, and that hierarchy has legal consequences.

Vector search, the backbone of standard RAG systems, is blind to this structure. When a caseworker asks “who approved this regulation?”, a similarity-based retrieval will surface documents containing the words “approval” or “regulation” – regardless of whether those documents sit at the correct level of the hierarchy, cover the correct time period, or carry the correct legal authority. The result is an answer that sounds reasonable but may be legally incorrect.

Three specific failure modes stand out:

Temporal blindness: Laws have start and end dates. Benefit amounts change. A caseworker asking about a benefit entitlement needs an answer valid when the case was opened, not when the query is run. LLMs frequently cannot resolve this kind of calendar arithmetic, especially when deadlines involve business-day rules or statutory exemptions.

Missing hierarchy: A municipal by-law that conflicts with constitutional law is invalid. But a vector search cannot detect that conflict – it has no concept of legal precedence. A knowledge graph can traverse the hierarchy and flag the collision.Non-determinism: Legal reasoning must be auditable. In regulated environments, every answer needs a traceable chain of justification. Probabilistic token prediction cannot provide that.

The Solution: Symbolic Graph AI

Matthias describes his approach as symbolic Graph AI – a hybrid architecture that keeps LLMs in their lane (intent extraction, entity recognition, summarisation) while delegating logic, hierarchy, and retrieval to a knowledge graph.

The result is a layered system with three architectural pillars:

Normative skeleton: The raw XML source of German social law (available from official legislative portals in well-structured form) is parsed and loaded directly into Neo4j as a hierarchy of nodes: Book → Chapter → Paragraph → Sub-paragraph. This avoids the noise introduced by PDF extraction entirely, including the notorious two-column layout problem that scrambles left-to-right reading of interleaved text.

Temporal dimension: Each legal expression is modelled with validity dates. A paragraph granting €449 in child benefit from 2022 until October 2024, and €452 thereafter, exists as two separate nodes linked by time-bounded relationships. Cypher queries can then ask: What was the valid expression of this rule at the time this case was opened? Procedural state machine: Administrative workflows – the step-by-step processes a caseworker must follow, potentially over months – are modelled as directed graphs. Every step links to its legal justification. This makes the process both executable and auditable: you can mathematically verify that a procedure was followed correctly by analysing the graph.

From Document Management to Knowledge Management

A key insight from the session is the distinction between storing documents and encoding knowledge. A document management system tells you where information lives. A knowledge graph tells you how information relates.

By combining structured XML ingestion with LLM-assisted entity extraction (normalised to Schema.org vocabularies to avoid synonym fragmentation), Matthias’s pipeline builds a graph where “benefit amount”, “entitlement calculation”, and “payment threshold” all resolve to the same canonical concept – and where each concept links directly to the paragraph that defines it.

Hybrid search (graph traversal combined with full-text and vector search) handles cases where exact structural matches are insufficient, providing both precision and recall.

Integration Patterns: MCP, Azure, and Air-Gap Deployments

The initial deployment used Azure AI Foundry with Neo4j as a managed database, connected via the Model Context Protocol (MCP). Matthias notes a practical trade-off with MCP: it is the most flexible integration pattern, but it consumes significantly more tokens than direct SDK or HTTP query API integration – an important consideration when working with many tools or cost-sensitive deployments.

When the client required a fully air-gapped system, the stack was rebuilt with local Docker containers, open-source models (including 1-bit quantised LLMs for an 87% memory reduction), and a self-hosted Neo4j instance. Neo4j Bloom and Neodash were used for prototyping dashboards and visualising the legal graph with case workers, enabling productive workshops where non-technical stakeholders could see and interrogate the model directly.

Matthias also demonstrated how Claude was used to generate Cypher queries from natural-language descriptions of analytical questions, dramatically reducing the barrier for developers unfamiliar with Cypher to explore the graph.

Graph AI vs. LLM-Only: A Direct Comparison

Dimension	LLM Only	Graph + LLM
Sourcing	Probabilistic, hallucination-prone	Verifiable, path-traceable
Reasoning	Associative	Deterministic, rule-based
Time awareness	Limited	Explicit temporal modelling
Explainability	Black box	Expandable path
Hierarchy	Not represented	First-class structure

Key Takeaways for Developers

Hierarchy is a first-class concern: If your domain has ranked authorities – legal, organisational, ontological – model them explicitly in your graph schema. Vector search cannot infer authority from similarity.
Use structured sources where possible: Official XML feeds for legislation or well-structured JSON APIs produce far cleaner knowledge graphs than PDF extraction. Reserve PDF pipelines for supplementary material that has no structured equivalent.
Model time explicitly: Validity windows on nodes and relationships unlock temporal reasoning that LLMs alone cannot perform reliably. This matters anywhere facts change over time: regulations, pricing, policies, personnel.
MCP is flexible but token-costly: For production systems with many tools or tight token budgets, consider direct SDK or HTTP API integration and reserve MCP for prototyping or lower-volume use cases.
Combine graph traversal and vector search: Pure graph queries miss some semantically equivalent terms; pure vector search misses structural relationships. Hybrid retrieval captures both.
Normalise entities with shared vocabularies: Domain-specific ontology prevents synonym fragmentation – where the same concept appears as multiple disconnected nodes – and makes traversal reliable.

Additional Resources

Matthias’s blog post: Overcoming LLM Deficits with Multi-Layered Ontologies
Matthias on GitHub: github.com/ma3u
Neo4j AI Overview: https://neo4j.com/use-cases/ai-systems/
Essential GraphRAG: https://neo4j.com/essential-graphrag/
MCP and Neo4j: https://neo4j.com/developer/genai-ecosystem/model-context-protocol-mcp/
More on Ontologies and AI: https://neo4j.com/video/going-meta-a-series-on-graphs-semantics-and-knowledge/

Timestamps:

0:00 – Welcome & intro
3:00 – Live quiz: Where vector search falls short
10:25 – The legal challenge: Why LLMs struggle with law & hierarchy
16:30 – Vector search trap: Similarity ≠ authority
21:00 – Introducing symbolic Graph AI for legal reasoning
25:50 – Temporal dimension & state machines in legal workflows
32:10 – Architecture overview: Hybrid Graph + LLM approach
41:00 – From document management to knowledge management
49:50 – Legal AI in practice: Lessons learned & tooling
57:00 – Q&A: Hierarchies, compliance & implementation
1:06:30 – Wrap-up & upcoming Neo4j events