Cut Text2SQL token costs up to 81%, and stop paying for wrong answers: a Neo4j semantic layer across BigQuery and Databricks

Across two deliberately hostile catalogs (278 Census tables on BigQuery and 264 legacy tables on Databricks), a Neocarta semantic layer in Neo4j makes Text2SQL agents faster, cheaper, and more accurate. On Census it cut tokens up to 81% while lifting a cheap model from 2/4 to 4/4; on Databricks flagships it saved up to 54% (roughly $1,527 → $693 per month at 10K queries on Opus 4.5 as of August 2026), with every model at 11/11. And without the layer, that same cheap model burned 66,687 tokens and still gave up with no answer.

First, what a semantic layer is

A semantic layer sits between your data and whatever consumes it (a BI tool, or now an AI agent) and turns opaque physical schema into business meaning. And its inputs aren’t only databases: it harvests metadata from warehouses and lakes, data catalogs, query logs (real join paths and usage), and business glossaries or ontologies. Point an agent straight at raw tables and it re-derives your joins, definitions, and table choices on every prompt, guessing which of a dozen “revenue” tables is the real one. The semantic layer is what removes the guessing.

This has become the standard way to ground agents in business context, and every major platform ships one: dbt’s Semantic Layer, Snowflake Semantic Views, Databricks Unity Catalog Metric Views, Looker’s model, and so on. Almost all of them are metric-first and, for the warehouse-native ones, single-platform: you pre-define certified measures and dimensions over a curated handful of tables, and the layer answers “how is revenue calculated” and “which metric is trusted.”

This article is about a different, complementary shape of semantic layer, one aimed at the problem those layers assume you have already solved: discovery. Before you can define a metric over “revenue,” something has to find which of hundreds of opaquely-named tables holds it, across however many warehouses your data actually lives in. That layer is a graph of the whole estate’s metadata an agent searches and traverses, not a set of hand-authored metrics. It is metadata-only (Neo4j + Neocarta), it works across warehouses, and it sits underneath the metric layers rather than competing with them: the graph finds the right table; a Metric View can still own the certified formula on it.

Two recent Neo4j articles made the case that a graph-backed semantic layer beats dumping your schema into the prompt:

• Laurent Tandé showed that replacing static YAML schema files with a Neo4j knowledge graph cuts tokens 20–30% and lifts accuracy about 10 points on complex questions, over a ~50-table HR schema in PostgreSQL.
• Alex Gilmore showed how to build that layer with Neocarta from GCP, ingesting BigQuery and a Dataplex glossary into Neo4j and serving it to an agent over MCP, on the 33-table ACME dataset.

Both demos are convincing at 30–50 tables. The obvious question: does it still hold when the schema is far too big to fit in a prompt at all? I ran the same idea on two much larger, deliberately messy catalogs (one on BigQuery, one on Databricks) and open-sourced both. The recipe is identical (retrieve the right tables from a Neo4j graph, then let the model write SQL); only the warehouse and the connector change.

The two results are complementary:

• Census ACS on BigQuery: the accuracy story. 278 near-identical tables, a schema so wide it can’t be pasted into any model. A cheap model (gpt-4o-mini) went from 2/4 to 4/4 correct, and stopped burning tokens on dead-end runs that returned nothing. → census-neocarta-benchmark
• A legacy lakehouse on Databricks: the cost story. 264 opaque tables across 12 schemas. Across eight Anthropic models, every one hit 11/11, using 19–54% fewer tokens and running about 2× faster; the biggest models saved the most money. → databricks-neocarta-benchmark

Put together: accuracy up on cheap models, tokens and dollars down on expensive ones, on two different clouds, with one Neocarta semantic layer.

The problem: access is solved, understanding isn’t

In Agents on Databricks: the platform is ready, your business context is not, Arun Srinivasan frames it well: “Access is no longer the hard part… Your agent can reach the data but it cannot reliably understand your business.” His example: ask an agent for last quarter’s revenue and “your estate has 47 tables with revenue in the name, so which one is the trusted source? … The agent will answer. It just won’t tell you which of those choices it made.” Swap 47 for 264 and you have my Databricks catalog exactly.

That is a context problem, not an access one, and it only gets worse as the schema grows. The standard Text-to-SQL recipe hands the model the schema (a big text blob, or a search index of table docs) and hopes it picks the right tables. That has three failure modes (Laurent’s article lays them out): high token cost (you ship the schema on every call), contextual noise (irrelevant tables cause hallucinated joins), and staleness (flat files rot).

These are annoying at 50 tables. At a few hundred they are fatal. The full Census schema is about 475K tokens: it does not fit in a 128K-context model, full stop. So the agent can’t “see” the schema; it has to discover it, and a naive agent does that by listing tables and guessing, re-sending growing metadata each turn until it succeeds, fails, or times out.

A semantic layer flips this: instead of giving the agent everything, you give it a way to retrieve only the handful of tables a question needs.

What Neocarta does

Neocarta reads your warehouse’s schema catalog and builds a graph in Neo4j: databases, schemas, tables, and columns, with foreign keys as join edges, a few sample values, and embeddings over each object’s description. Only metadata crosses into Neo4j; your rows stay in the warehouse.

This benchmark implements the core path (ingest once, then retrieve and run the SQL), but the same graph is designed to sit under a whole agent stack: metadata sources (data catalogs, query logs, glossaries, and the warehouses themselves) feed the semantic layer in Neo4j; a retrieval layer exposes it to agents over MCP; and the agent and consumption layers serve users, with a feedback loop writing memory back to the graph.

The graph itself is a plain model of databases, schemas, tables, and columns, with join edges and an embedding on every description:

Here’s a slice of the real graph in the Neo4j Browser: one mart expanded to its tables, their columns, and sample values, with relations (edges) linking the shared customer key across tables. Those edges are the join paths the agent follows:

Storing it as a graph is what makes retrieval work in two moves: semantic search finds an anchor table, then the graph pulls in the related tables and the exact columns to join on. The agent doesn’t guess joins; it reads them off the edges:

At query time the agent has just two tools: one to retrieve (a semantic search in Neo4j that returns the right tables, their columns, and how they join) and one to run the SQL against the warehouse. Here’s the full path on the Databricks example (finding net revenue retention in a 264-table catalog):

Retrieval finds the table; the LLM writes the query; the warehouse holds the data. Neo4j is never queried for the business facts themselves. And because Neocarta’s connectors are warehouse-polyglot (BigQuery, Snowflake, Databricks, and more), moving from one warehouse to another is just swapping the connector; the graph, the tools, and the agent are identical. That is why one semantic layer can span both case studies below.

Case study 1: Census on BigQuery (the accuracy story)

The catalog. bigquery-public-data.census_bureau_acs is 278 near-identical tables (county_2018_5yr, cbsa_2017_1yr, and so on), about 252 columns each, roughly 68K columns in total. Tables differ only by geography, year, and survey span, with cryptic column names. Only metadata goes into Neo4j; queries run against Google’s public dataset in place.

The comparison (the only thing that changes is whether Neocarta is in the loop):

• Without the layer: the agent brute-forces the schema (list tables, read columns, guess, repeat), because 68K columns can’t be pasted into a prompt.
• With the layer: one semantic search finds the right table, then the agent runs the SQL.

Result (gpt-4o-mini, average per question, 4 verified questions):

Without the semantic layer, two of the four questions looped for 20 tool calls and gave up with no answer at all. The agent kept re-scanning metadata trying to find the right table among 278, and you still paid for those dead-end runs. With Neocarta, the same cheap model answered every question correctly.

You can watch it happen side by side. Same question (“the 10 counties with the highest median household income in the 2018 5-year survey”), same gpt-4o-mini. On the left, without the layer, the agent loops until it hits the step limit and gives up with no answer, after burning 66,687 tokens. On the right, with Neocarta, it answers correctly in two tool calls and 21,566 tokens (about 68% fewer, and the difference between an answer and none):

With plain retrieval (which returns a whole 252-column table), this is mainly an accuracy and reliability win, not a clean token cut. A strong model (gpt-4o) can often guess a table from its name, and because Census tables are so wide, retrieving even one can cost as much as brute-forcing. That’s why the repo also ships a compact-retrieval variant that returns only the columns a question needs. With it, Census becomes a cost story too: 45–81% fewer tokens across five models, every one at 4/4:

On the cheapest model the compact layer cuts tokens 81% and fixes the accuracy gap; on the priciest (Opus 4.5) it still nearly halves the bill, about $0.20 to $0.12 per question. Same recipe as the wide-retrieval run: the graph just returns less.

Case study 2: a legacy Databricks lakehouse (the cost story)

The catalog. A synthetic “legacy enterprise” Unity Catalog lakehouse (264 tables, 5,614 columns, 12 schemas), engineered to punish blind exploration.

Every physical name is an opaque legacy code (ARR is dm_fin_20.f_2001, net revenue retention is dm_agg_10.a_1007), and the business meaning lives only in the table and column comments, like a real warehouse migrated off a legacy system. The trusted tables are buried under year and region shards, a hundred-odd unlabeled extract tables, and deliberate traps: one “customer” table is really marketing audience members; one “ticket” table is really HR cases. A naive agent has to find the gold before it can query it.

The comparison:

• Without the layer: list schemas, list tables, read columns, and scan the catalog in SQL, the strongest realistic baseline.
• With the layer: one semantic search, then run the SQL.

Scoring is deterministic: each of the 11 questions in eval/questions.yaml declares the table it must use and a regex over the expected answer.

Result (8 Anthropic models × 11 questions):

Every model reaches 11/11 with the layer. Haiku 4.5 goes from 10/11 to 11/11 because of it.

• Even models that already scored 11/11 pay a brute-force tax. With the layer every question converges to two steps. Without it, dozens of calls scanning the catalog, every question, forever.
• The token cut is worth the most where it’s most expensive. The same ~54% reduction saves about $0.084 per question on Opus 4.5 but only about $0.019 on Haiku. Retrieval is cheap insurance on flagship models.

And the accuracy half echoes Census: in an earlier run using OpenAI embeddings, gpt-4o-mini went 0/11 to 8/11 with the layer (73% fewer tokens, 5× faster), because without it half its runs hit the step limit paging through junk extract tables.

What the warehouse actually sees

The aggregate numbers hide where the work goes. Here is the Databricks query history for a single question (“What was net revenue retention for EMEA in December 2024?”), run the same way in each mode:

Both runs end at the exact same SQL and the same answer (108%). The difference is everything before it:

• Without the layer, the warehouse fields 18 round-trips: a stream of catalog look-ups as the agent opens one opaque table after another, hunting for the one that means “net revenue retention” (nothing is literally named that). Only then does it run the real query. That’s about 26 seconds of warehouse time spent rediscovering the catalog, and it repeats on every question.
• With the layer, that entire block is gone. The semantic search happens inside Neo4j and never touches the warehouse, so Databricks sees a single business query. Same answer, half the tokens, one round-trip instead of eighteen.

That’s the whole token-and-latency story in one line: without a semantic layer, the agent re-discovers the catalog on every question; with one, discovery was done once, offline, and each query is just a graph lookup away.

Why the graph wins on both

Same mechanism, whichever cloud:

• Opaque or cryptic names don’t matter: embeddings index the description, not the physical name, so “recognized revenue” or “rent burden by county” still lands the right table.
• Near-duplicates and homonyms get ranked out: the matching description scores highest, so “support tickets” beats the HR-cases trap and “2018 5-year county” beats 277 lookalikes.
• Joins come from the graph: declared foreign keys are edges, so the agent follows the join instead of guessing it.

If that retrieve-then-expand loop sounds familiar, it’s GraphRAG: a hybrid search resolves the question to a few entry points, then a graph traversal gathers exactly what’s needed, here pointed at your metadata instead of documents. The same pattern that grounds agents over unstructured text, applied to your schema.

From a schema map to an enterprise knowledge layer

Nothing about this stops at tables and columns. The same graph is the natural place to hold the business meaning a flat schema can’t: a glossary that settles which of 47 “revenue” tables is the trusted one, metric definitions, ownership and sensitivity, usage-inferred joins mined from query logs, and the agent’s own memory of what worked before. That is the trajectory Neo4j calls the enterprise knowledge layer: a shared, governed substrate of ontology (a live map from raw assets to business concepts and the processes on them), data for grounding, and memory for learning and traceability, making “organizational knowledge as queryable as data and as actionable as code.” A metadata graph that already knows your tables, columns, and joins is the first vertebra of that spine.

So this isn’t a rip-and-replace story, and it isn’t only about Text2SQL. The layer works across warehouses by design, and it composes with your governed layers instead of replacing them: the graph finds and joins the right tables; a Metric View or a source catalog can still own the certified formula. Discovery and retrieval is the half we measured; meaning, governance, and memory are what the same graph grows into.

Try it yourself

Both repos run locally with uv and a Neo4j instance (Aura free tier or Desktop).

BigQuery / Census (needs a GCP project; queries public data in place):

uv run module-2/build_census_semantic_layer.py    # metadata-only graph in Neo4j
uv run module-3/census_agent.py --mode without     # watch it loop / give up
uv run module-3/census_agent.py --mode with        # 2-call correct answer

Databricks (needs a SQL warehouse):

uv run python src/seed_lakehouse.py          # build the 264-table catalog
uv run python src/build_semantic_layer.py    # Neocarta ingest + embeddings
uv run python src/benchmark.py               # with vs without, all models

Takeaway

The LLM doesn’t need a bigger context window: it needs a better map. On BigQuery a graph map turned an unreliable cheap model into a 4/4 one and stopped it burning tokens on dead ends; on Databricks it cut tokens 19–54% across eight models while every one scored 11/11. Accuracy where models are weak, savings where they’re expensive, and on huge schemas, the only thing that works at all, because you can’t paste 475K tokens of schema into anything.

Neocarta makes building that map a connector call and an embedding call, once, across whatever warehouses you have. Start there with a graph that finds the right tables and joins, and you’ve also laid the foundation for an enterprise knowledge layer.

Code: census-neocarta-benchmark · databricks-neocarta-benchmark · Neocarta

---

Cut Text2SQL token costs up to 81%, and stop paying for wrong answers: a Neo4j semantic layer… was originally published in Neo4j Developer Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.