The AI application your team can actually stand behind

The demo looked great. Stakeholders were impressed. Then someone asked the chatbot a real question — and it gave a confident, wrong answer. Now you’re fielding emails, your team is manually reviewing outputs, and the question you dread most is the one your VP keeps asking: “How do you know this is right?”

It’s a control problem. And it’s what we set out to solve at Neo4j.

Why grounding matters

AI that generates answers without grounding them in verified knowledge doesn’t just produce errors. It erodes user trust, creates manual review overhead, and leaves you accountable for answers you had no way to audit.

The goal is to give the model a knowledge base it can draw from, and build a feedback loop so you can measure, over time, whether it’s getting better or worse. That’s the architecture.

How it works

Neo4j tech stack lets you combine the reasoning capabilities of LLM with a structured knowledge of graph databases to build systems that offer highly accurate and verifiable answers.

The technical breakdown below explores how Neo4j is solving a business problem within its organization by building a Neo4j-powered Generative AI framework to deliver accurate and grounded responses.

• The Data Layer (Powered by Neo4j Aura) : To maintain security, performance, and separation of concerns, the architecture relies on the Knowledge Graph: This acts as the single source of truth for the agent. It is a highly structured architecture holding vectors, and an entity layer comprising entity types (such as Document, Product, PricingTier, Region, and Concept). This structure allows the agent to follow conceptual relationships rather than just matching keywords.
• Neo4j GraphRAG Python Package : Python library that allows building GenAI applications. It supports knowledge graph creation through a pipeline that extracts entities from unstructured text, generates embeddings, and creates a graph in Neo4j. The package also provides a number of retrievers, for graph search, vector search, integration with vector databases and agentic tools calls.
• Neo4j Needle StarterKit : The Neo4j Needle StarterKit 2.1 gives teams a working front-end from the start. It ships with a built-in Chatbot component and domain-specific templates and integration for the GraphRAG package — so your team skips the blank-canvas phase and spends time on the actual problem, not the plumbing. And because it’s modular, you can update the UI independently from the knowledge layer, which preserves the same iteration safety that runs through the rest of this architecture.
• GenAI Eval Framework : DeepEval framework, an automated test suite continuously runs expert-graded questions against the live backend to score answers for correctness, faithfulness, and reasoning quality.

1. Structured Data Track (Top Layer)

This track handles predictable, formatted data (like CSVs, SQL tables, or JSON records).

• Data Processing: The structured data enters an Extract and Transform phase. Guided by a predefined Graph Schema (which defines how data categories connect), the system maps the structured data into nodes and relationships.
• Ingestion: A Data Loader takes this transformed data and pushes it directly into Neo4j Aura.

2. Unstructured Data Track (Bottom Layer)

This track handles raw, unformatted data (like PDFs, articles, or transcripts) and uses machine learning models to extract meaning.

Phase 1: Data Processing & Models

• Text Chunker: Breaks long blocks of unstructured text into smaller, readable pieces.
• Entity Extraction: Uses an LLM and a Graph Schema to identify specific people, places, things, or concepts (entities) within the text chunks and determine how they relate to one another.
• Text Embedder: Passes the text through an Embedding Model to convert the words into numerical vectors, capturing their semantic meaning.
• Note: There is also an Image Recognition node in the models block, implying the pipeline can handle multimodal data if needed.

Phase 2: Ingestion

• The numerical vectors are sent via a Data Loader to a Vector Store (which can be hosted natively inside Neo4j or alongside it) for similarity searches.
• The extracted entity relationships are sent to Neo4j to build out the structural Knowledge Graph.

Phase 3: Post-Processing

Once the raw data is inside the graph, it passes through refinement steps:

• Entity Resolver: Identifies duplicates (e.g., realizing “Neo4j Inc.” and “Neo4j” refer to the same entity) and merges them.
• Entity Linker: Connects newly extracted entities to existing nodes in the graph to ensure the network is unified.

Phase 4: Validation

• Extraction Validator: Performs a final quality check to ensure the LLM-extracted data is accurate, correctly formatted against the schema, and free of hallucinations before being finalized.

What this means for your application roadmap: Each phase ships independently and has a clear handoff. You can validate the ingestion layer before the reasoning agent is ready, and iterate on evaluation criteria without blocking either work-stream. For a TPM driving its cross-functional delivery, that’s a phased rollout plan, not a big-bang launch.

What we built and then rebuilt to improve the numbers :

The v1 chatbot application was a straightforward vector RAG pipeline + expansion through graph: embed the question, pull the top-10 most similar chunks from Neo4j, expand 1 hop through a graph query, and hand them to the LLM with a prompt that politely asked it not to hallucinate. The graph was barely involved — retrieval surfaced neighboring entity IDs as bare strings, with no relationship types and no entity context. And critically, there was no evaluation of any kind: no test set, no scoring, no way to tell whether a prompt tweak made things better or worse.

V2 (below) had 3 architectural changes accounting for the gains, and each one maps to a specific metric.

1. A knowledge graph built against a strict schema.

Instead of free-form extraction, v2’s ingestion pipeline extracts entities against a domain schema of node types (Product, PricingTier, Feature, SecurityMechanism, Region, SupportTier ) and relationships types (HAS_TIER, AVAILABLE_ON_TIER, CERTIFIED, SECURES). Every extracted fact has to fit a shape we defined, with canonical values enforced at extraction time — “BC”, “AuraDB BC” and “Business Critical” all resolve to the same node. That structural discipline is what drives correctness: the model can’t conflate pricing tiers or security certifications when they’re distinct, typed nodes.

2. Retrieval became agentic and graph-native.

V1 had one fixed retrieval strategy for every question. V2 runs a ReAct agent with two tools it chooses between: a vector-plus-graph retriever that follows relationships out from matched chunks (returning entity context, relationship types, and source documents not just similar text), and a Text2Cypher retriever that translates precise questions (“which regions support X?”) into direct graph queries. Broad questions get semantic search grounded in graph context; factual lookups get exact answers from the graph. That’s what moved relevancy and kept faithfulness at 100% — every answer traces back to retrieved, verifiable context.

3. Evaluation became part of the architecture, not an afterthought.

V2 ships with an automated eval suite: 80 expert-curated questions spanning real support categories (security, pricing, development, infrastructure, GenAI), each with an expected answer and the retrieval tool the agent should use. Every run fires the full set against the live backend and scores four metrics with an LLM judge correctness (does the answer match the expert answer?), answer relevancy, faithfulness (is every claim supported by the retrieved context?), and tool correctness (did the agent pick the right retrieval strategy?). Results land in a Neo4j eval database with the run’s full configuration attached, so any two runs are comparable, and a dashboard tracks trends, per-question failures, cost per run, and regressions.

The V2 architecture is defensible because an automated test suite runs expert-graded questions against the live backend, scoring every answer for correctness, faithfulness, and reasoning quality. You set the questions. You grade the standard. And you see the results.

Evaluation proves the system works, but the Neo4j Knowledge Graph is what actually does the heavy lifting. By forcing the LLM to pull from a strict, typed schema rather than a pool of raw text vectors, we reduced the model’s ability to guess.

When we implemented this architecture compared to V1, here’s what happened to our metrics and real work impact:

• Pass/fail rate: 68.75% → 100% (+31 points)
• Correctness: 64.75% → 97.88% (+33 points)
• Relevancy: 86% → 97.34% (+11 points)
• Faithfulness to source material: 100%

In the first week alone, the system handled 1,500 user messages with no manual review escalations. That’s the adoption signal that matters: users trusted the answers enough to keep asking.

In Conclusion

For a decision maker, choosing Neo4j means you aren’t building a GraphRAG system from scratch. Between AuraDB, the GraphRAG Python Package, and the Needle StarterKit, your teams start with the plumbing already connected, allowing you to focus entirely on data quality and user adoption.

So, when your VP asks, ‘How do you know this is right?’ you finally have an architectural answer: ‘Our LLM retrieves facts from a Neo4j Knowledge Graph where every entity and relationship is strictly mapped, and we continuously prove its accuracy with automated evaluations against a pass-rate standard.’

Authored by –

• Morgan Senechal, Principal Professional Services Architect, Neo4j
• Sumeet Toprani, Strategic Programs Lead, Neo4j

---

The AI application your team can actually stand behind was originally published in Neo4j Developer Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.