
NVIDIA Auto Ontology vs semantic layers: will data agents get the right number?
NVIDIA's open-source Auto Ontology drafts a semantic layer for data agents. Its first public score trails a plain agent, and generated layers lose to curated ones in practice. The evidence, the rivals, the defaults to change and a four-arm pilot.
Photo: Erol Ahmed / Unsplash
On this page
On 21 October 2026 NVIDIA released Auto Ontology, an open-source tool that profiles your databases, has a language model draft what the tables and columns mean in business terms, and lets AI agents ask questions through that draft. Less than three weeks later it had close to 10,000 stars on GitHub. For data and architecture leads the question is a practical one: do our agents need an ontology or semantic layer before they answer questions from the warehouse, and is this the way to build one, or should we use dbt, Cube or the layer our data platform already sells?
The short answer: curated business context helps overall in every study I found, but the large gains come from vendors or from teams grading their own systems, on small schemas. Independent benchmarks show gains of about 15 to 20 points, and on realistic warehouses the best systems still answer only about one question in seven without hints, and under a third with them. Auto Ontology's first public score, submitted by NVIDIA, is 12.4% on BEAVER, slightly below a general-purpose agent on the same model (14.0%). Treat it as an open draft-maker and a reference design, not a source of trusted numbers: version 1.2.0 is two days old and the project has been public for under three weeks, its own hallucination checks are off by default, and outside users found three connectors returning wrong values without an error. In practice, generated semantic layers score well below curated ones, and curated ones drift within weeks without upkeep. On one data platform, start with that platform's own generator; for governed metrics, use a deterministic layer such as dbt MetricFlow or Cube; try Auto Ontology only in a sandboxed pilot against the four arms at the end of this post.
What Auto Ontology is, and what it is not
In code it is five services: a web UI and API gateway, a FastAPI backend that runs a LangGraph text-to-SQL agent, an ingestion worker, an MCP server, and Postgres with pgvector holding the catalog and the vectors (README). A language model reads profiled metadata and sample values table by table and drafts glossary terms, SQL snippets and inferred joins; data experts can mark terms as certified. Agents reach it through nine read-only MCP tools, and the catalog can be exported as Apache Ossie YAML, the open interchange format for semantic models (MCP docs, Apache Ossie).
Two things are narrower than the name suggests. It reads and writes only its own YAML and Ossie YAML: no OWL, RDF or SPARQL, and no import from dbt or LookML. And the SQL is written by language model nodes and then checked, not compiled deterministically from definitions as dbt's MetricFlow does, although the repository description promises "deterministic query generation" (repository). An independent code read, which my own search of the repository agrees with, also finds that the certified flag is shown to users but does not decide which definitions the agent uses.
How Auto Ontology sits between data and agents
The language model drafts the catalog and also writes the SQL; nothing in the query path is a deterministic compiler.
Source: NVIDIA/auto-ontology README and docs/mcp.md, read 8 November 2026
Show the data
| Source | Relation | Target |
|---|---|---|
| Databases | metadata and sample values | Profiler |
| Profiler | profiles | LLM semantic compile |
| LLM semantic compile | terms, SQL snippets, inferred joins | Catalog |
| Data experts | review and certify | Catalog |
| Catalog | export and import | Apache Ossie YAML |
| Catalog | business context | Text-to-SQL agent |
| Text-to-SQL agent | generated SQL | Databases |
| Text-to-SQL agent | answers with SQL and rows | MCP server |
| MCP server | nine read-only tools | AI agents |
Its first two weeks in the open
A Hacker News thread on 3 November with 290 points (thread) took the repository from 2,134 stars to 9,863 by 8 November. Version 1.2.0 shipped as a Helm chart on 6 November, still with no GitHub release or changelog (NGC Helm index). The outside users who actually ran it reported more than the discussion did:
- The documented default model was dead at launch. The Nemotron model in the quickstart had been retired on 1 September and answered 410 Gone, so every question on the documented setup failed until an outside contributor fixed the defaults on 4 November (pull request 402).
- The hosted NVIDIA endpoint rejects its structured-output calls. Two users report a 400 error on every such call and about two minutes per pipeline step from fallbacks; no maintainer had replied by 8 November (issue 440).
- Three connectors return wrong values silently. Postgres, MySQL and SQLite lose or swap values when two result columns share a name, such as two id columns, with no error to trigger a retry (issue 408).
- It cannot always say "the data cannot answer this". Its validator has no "the data does not hold the answer" outcome: on one such question it looped for 192 seconds, twice assumed a stars column that does not exist, and ended with a generic refusal (issue 462).
The project now says it does not accept external pull requests, though it merged the one above (CONTRIBUTING.md). Apache-2.0 lets you fork it, but the roadmap and support are NVIDIA's alone.
The first accuracy number appeared on 3 November, not in NVIDIA's repositories but on the BEAVER leaderboard, a benchmark built from real query logs of private enterprise warehouses. NVIDIA's entry with Claude Opus 5.5, with no hints, scores 12.4%; ReFoRCE, a general text-to-SQL agent, scores 14.0% with the same model (BEAVER leaderboard). The entry has no ontology-off comparison, so it does not tell us what the ontology adds. NVIDIA's own evaluation harness is well designed, with one verifier for both arms and no LLM judge, but it still publishes no result (auto-ontology-eval). The only product claim, that it "reduced ontology creation time by 90%", comes with no baseline or scope (NVIDIA developer page).
Does an ontology or semantic layer make the answers right?
The idea Auto Ontology automates is well supported. No primary study I found shows curated context lowering overall accuracy, though single components sometimes did. How much it helps depends on who ran the test, how small the data was and how weak the baseline was:
Appraising the evidence
| data.world 2023 | Vendor (sells the layer) | 13 tables, 43 questions | Execution match | 16.7% | 54.2%change: +37.5 pts | Weak: changed the query language, context and engine at once |
|---|---|---|---|---|---|---|
| dbt Labs 2026 | Vendor | Same data, 11 questions x 20 runs | Expected results, method not stated | 64.5% | 72.7%change: +8.2 pts | Medium: small set, newer models |
| Cube 2026 | Vendor | Retail data, 99 questions | LLM judge, lenient | 46.5% | 68.7%change: +22.2 pts | Medium: the layer was a 4 KB Markdown file |
| Snowflake 2025 | Vendor | BIRD, 392 questions | LLM judge | 57% | 78%change: +21 pts | Medium: vendor's own product |
| GSK 2024 | Self-graded | 500+ tables, 60 logged questions | Domain expert | 8.3% | 78.3%change: +70.0 pts | Medium: real data, small set |
| LinkedIn 2025 | Self-graded | Millions of tables, 133 questions | LLM judge and experts | 9% | 48%change: +39 pts | Medium: real data, "correct or close" |
| BIRD 2023 | Independent | 95 databases, one-sentence hints | Execution match | 34.9% | 54.9%change: +20.0 pts | Strong |
| BEAVER 2026 | Independent | Real warehouses, all oracle hints | Execution match | 9.5% | 25.9%change: +16.4 pts | Strong: average of seven models |
| EntSQL 2026 | Independent | Five business domains | Execution match | 6.8% | 21.4%change: +14.6 pts | Strong |
Without and with curated business context, same model within each study.
Source: The papers and posts listed under Sources, read 3 and 8 November 2026
Three patterns matter for a buyer.
- Curation matters more than format. A hand-written 4 KB Markdown file lifted three frontier models by 17 to 23 points in Cube's study, without Cube's engine (Cube 2026); a business context document took GSK from 8.3% to 78.3% (GSK); BIRD's one-sentence hints added 20 points (BIRD). The often-repeated claim that an ontology triples accuracy rests on one vendor study of 43 questions that changed three things at once (data.world 2023).
- What formal layers add is the refusal. A deterministic layer answers from defined metrics or says it cannot; dbt's semantic layer "tells you it can't answer" instead of returning wrong data (dbt Labs 2026), and ontology checks turned some wrong answers into an explicit "unknown" (data.world 2024). Auto Ontology's version of those checks ships switched off.
- Outside the modelled scope, the layer scores zero. On dbt's questions that needed joins the semantic layer could not express, it scored 0% while plain text-to-SQL scored 70% to 100% (dbt Labs 2026). Coverage has to be measured as closely as accuracy.
For how to read claims like these, the same questions apply as in GraphRAG vs RAG: what the evidence actually says, and the label problems are covered in how to evaluate GraphRAG honestly.
How far from reliable on real warehouses
Leaderboards on curated public data look strong; anything that resembles a real company does not.
Best results on text-to-SQL benchmarks
| BIRD, with hints | 95 databases, dirty values | 83.4 |
|---|---|---|
| Spider 2.0-Snow | Enterprise schemas on Snowflake | 96.7 (best) |
| Spider 2.0-Lite | Three SQL dialects | 76.2 |
| Spider 2.0-DBT | A whole dbt project | 65.6 |
| BEAVER, with annotations | Real warehouses, schema and domain hints | 30.5 |
| EntSQL, curated evidence | Needs private business documents | 21.4 |
| BIRD-Interact | Must ask the user for missing knowledge | 17.0 |
| BEAVER, no hints | Real warehouses, real query logs | 14.0 |
Human experts score 92.96% on BIRD with hints. Annotation errors affect 52.8% of BIRD Mini-Dev and 62.8% of the checked Spider 2.0-Snow examples, so treat the top scores with care.
Source: BIRD and BEAVER leaderboards read 8 November 2026; Spider 2.0 leaderboard, BIRD-Interact and EntSQL papers read 3 November 2026; label audit arXiv 2601.08778
BIRD's own team plans to drop its hand-written hints in favour of an interactive track where the agent must ask, and there success falls to 8.7% to 17% (BIRD-Interact). That is the clearest public evidence that missing business knowledge, not SQL syntax, is the bottleneck, which is the problem Auto Ontology and its rivals are selling a fix for.
The rivals: who else builds the context layer
Generating semantics from metadata is no longer unique. Snowflake's Semantic View Autopilot builds semantic views from metadata and query history (Snowflake docs), Databricks' Genie Ontology infers context from metric views, dashboards and queries in preview (Databricks docs), and Microsoft's Fabric IQ generates an ontology from Power BI models in preview (Microsoft Learn). What Auto Ontology adds is the open, cross-platform version.
Auto Ontology and the alternatives
| NVIDIA Auto Ontology | LLM drafts from profiled metadata; experts certify | Self-hosted; 10 SQL engines | Read-only MCP, REST | Apache-2.0; models and GPUs extra | 1.2.0, 6 Nov 2026 |
|---|---|---|---|---|---|
| dbt Semantic Layer (MetricFlow) | Hand-written YAML, AI-assisted; SQL compiled deterministically | Your warehouse; hosted API on dbt platform | dbt MCP server | MetricFlow Apache-2.0; hosted from $100 per user per month | Mature |
| Cube | Data model as code, AI-assisted | Self-hosted Core or Cube Cloud | Hosted MCP, can edit the model | Core open source; Cloud from $40 per developer per month | Mature |
| Snowflake semantic views | SQL, YAML or generated by Autopilot | Snowflake only | Managed MCP, Cortex Agents | Snowflake consumption | Available; Autopilot generally available |
| Databricks metric views and Genie Ontology | YAML or UI, plus inferred context | Databricks only | Genie MCP | Databricks consumption | Ontology in preview |
| Google Looker (LookML) | Hand-written LookML | Looker | Managed MCP | Looker licence | LookML mature; MCP in preview |
| Microsoft Fabric IQ and data agent | Ontology generated from Power BI models | Fabric only | Fabric MCP servers | Needs F2 capacity or Premium P1 | Ontology in preview |
| AtScale | SML YAML with converters | Over your warehouse | MCP | No public prices | Long-standing vendor |
Prices and status as of November 2026.
Source: Vendor documentation and pricing pages listed under Sources, read 3 and 8 November 2026
Note the design difference. In dbt's design the language model only picks metrics and dimensions, and MetricFlow writes the SQL (MetricFlow). In Auto Ontology the language model writes the SQL. The evidence above credits exactly that determinism for refusing instead of guessing. On the other side, Auto Ontology keeps its MCP server read-only, while Cube's MCP server can change the data model (Cube MCP docs), a write path into production definitions that needs its own review (more in designing MCP tools).
What happens in practice
Customer stories from the platform vendors measure adoption and time saved, almost never accuracy. Webmotors reports 72% fewer manual data tickets and 200 analyst hours saved a month with Databricks Genie (Databricks customer story); Snowflake says more than 5,600 of its own employees ran over 320,000 queries through its internal semantic layer, with no accuracy figure (Snowflake blog). Where accuracy is measured, the pattern repeats: an empty or generated layer starts low, and the high numbers arrive only after people curate it.
Accuracy in practice: generated or empty layer against a curated one
| Mastech, for a client | Snowflake Cortex Analyst | 63% | 77%change: +14 pts | 30 questions; the starting view was generated by Cortex Code and gave silent wrong answers; the better one was built with a custom domain skill |
|---|---|---|---|---|
| Microsoft engineers (ISE) | Databricks Genie | 9.5% | 88.5%change: +79.0 pts | Public benchmark; an empty space, then 69.2% with descriptions and 88.5% with a feedback loop that needed manual upkeep |
| Syren | Databricks Genie | 53% | 80%change: +27 pts | 15 questions; a space with a basic model, then remodelled; the later 100% came partly from changing the expected answers |
| Databricks worked example | Databricks Genie | 0% | 100%change: +100 pts | Illustrative, unnamed team; 13 questions; 0 of 13 at first, 100% after five iterations |
| Anthropic, internal analytics | Claude with a semantic layer and skills | 21% | 95%change: +74 pts | 21% at most without the curated skills; about 95% with them, falling to about 65% in a month without upkeep; set size not disclosed |
Most of these are vendors, partners or companies describing their own systems; none is an independent head-to-head test of products.
Source: Mastech, Microsoft ISE, Syren, Databricks and Anthropic posts, read 8 November 2026
Two findings bear directly on Auto Ontology's promise. Anthropic tried having a language model generate metric definitions from raw tables and query logs, and it "was net-negative on our evals versus a smaller, human-curated layer" (Anthropic). And Snowflake's own documentation shows a first-draft semantic view that returns confident answers exactly twice the truth, because one table mixed daily and monthly rows (Snowflake cookbook). Automatic drafting saves the first pass; it does not replace the people who know that a table holds two grains. The worry is widespread: in dbt Labs' 2026 survey of 363 respondents, mostly data practitioners, 71% were concerned about wrong data reaching stakeholders (dbt Labs survey).
Defaults to change before a pilot
Out of the box, Auto Ontology assumes a trusted lab network. These were still the defaults on 8 November 2026:
Auto Ontology defaults and pilot settings
| Hallucination checks (DB_PROBE_*) off | Made-up values and joins can return wrong results instead of errors | Turn all three on and accept the extra queries |
|---|---|---|
| An sk- or unknown API key falls back to Claude Opus 5.5 through an NVIDIA gateway | Schema metadata, sample values and questions leave your network | Point every model endpoint at an approved or self-hosted model |
| Vault off unless all four variables are set | Database passwords stored as plain JSON in Postgres | Configure Vault fully |
| Helm chart enables pgAdmin with admin@example.com / admin | A database console with sample credentials on every node | Disable pgAdmin; put an ingress with TLS in front |
| The backend trusts a user-id header; ports published on the host | Anyone who reaches the port can act as any user | Keep the services on an internal network |
| API tokens never expire by default and have no scopes | A leaked token is a full user | Set expiry; issue tokens per pilot user |
One stored connection per database: database row security does not follow the end user unless each connection is scoped.
Source: model_config.py, .env.example, Helm values.yaml and docs, NVIDIA/auto-ontology, read 8 November 2026
The identity point deserves its own decision: if the agent queries with one service account, everyone who can ask it sees what that account sees. The patterns for passing the user's own rights are in delegated access for AI agents.
When not to use it
- Regulated or personal data, or anything reachable from the internet, until the defaults above are changed and you have a threat model, which the repository does not provide.
- Numbers that go into financial or regulatory reports. Use a deterministic layer, where a metric has one definition and the system refuses what it cannot compute.
- An estate that lives on one platform. Snowflake, Databricks and Fabric generate semantics inside their own governance; an extra open layer adds a second source of truth.
- No one to certify definitions. Automatic drafting saves the first pass; the scarce resource is expert time to check it, and no vendor, NVIDIA included, has measured that.
The four-arm pilot
The decision is cheap to settle on your own data. I call it the four-arm pilot:
- Build the question set. Take 100 to 500 real questions from the warehouse's query logs, each with gold SQL written or checked by a data owner. Include questions the data cannot answer.
- Run four arms with the same model: schema only; schema plus a short curated Markdown document; Auto Ontology with its checks on; and your platform's own semantic layer.
- Score more than accuracy: execution accuracy, refusals, silent wrong answers, coverage of the modelled scope, latency, cost per question, and the expert hours spent certifying.
- Decide on silent wrong answers first. A layer that answers fewer questions but never returns a wrong number beats one that answers more and sometimes lies. If the curated Markdown arm comes within a few points of the formal layers, start there.
- Re-run it every month. Data models change and context goes stale: Anthropic saw accuracy fall from about 95% to about 65% in a month when its documentation was not kept up (Anthropic).
NVIDIA's evaluation harness, with its shared verifier and its rule against headlines when runs break, is a reasonable starting point for the measurement (auto-ontology-eval). How to keep measuring after launch is in evaluating AI agents before and after launch, and how to turn the result into a budget decision in a GraphRAG business case, done the evidence-based way. Where this sits in an agent architecture as a whole is in agentic AI from demo to production, and when a knowledge graph is the right tool at all, in five questions before you build a knowledge graph for RAG.
Sources
- NVIDIA. Auto Ontology repository (README, docs, configuration), version 1.2.0.
- NVIDIA. Auto Ontology README.
- NVIDIA. Auto Ontology MCP server documentation.
- NVIDIA. Auto Ontology model defaults (model_config.py).
- NVIDIA. Auto Ontology .env.example (DB_PROBE and Vault settings).
- NVIDIA. Auto Ontology Helm chart values.
- NVIDIA. Auto Ontology CONTRIBUTING.md.
- NVIDIA NGC. Helm chart index (Auto Ontology 1.1.0 and 1.2.0).
- NVIDIA. auto-ontology-eval repository.
- NVIDIA. auto-ontology-eval: resources servers README.
- NVIDIA (21 October 2026). Launch blog, enterprise platforms and the Agent Knowledge Foundation.
- NVIDIA. Agent Knowledge Foundation developer page.
- Auto Ontology pull request 402: retired default model (merged 4 November 2026).
- Auto Ontology issue 440: structured output fails on the hosted endpoint.
- Auto Ontology issue 408: connectors return wrong values for duplicate column names.
- Auto Ontology issue 462: unanswerable questions loop.
- BEAVER leaderboard.
- BEAVER: an enterprise text-to-SQL benchmark (2026 version). arXiv 2409.02038.
- Li et al. (2023). BIRD text-to-SQL benchmark. arXiv 2305.03111.
- BIRD benchmark leaderboard.
- BIRD-Interact benchmark. arXiv 2510.05318.
- Spider 2.0 leaderboard.
- EntSQL benchmark (2026). arXiv 2606.03363.
- Audit of annotation errors in BIRD and Spider 2.0 (2026). arXiv 2601.08778.
- Sequeda, Allemang and Jacob (2023). Knowledge graphs and LLM accuracy on enterprise SQL databases. data.world. arXiv 2311.07509.
- Allemang and Sequeda (2024). Ontology checks and repair for LLM question answering. arXiv 2405.11706.
- dbt Labs (2026). Semantic layer vs text-to-SQL, 2026 benchmark.
- Rumiantsau and Fokeev (2026). Semantic layers for reliable LLM-powered data analytics. Cube. arXiv 2604.25149.
- Snowflake (2025). Agentic semantic model text-to-SQL, engineering blog.
- Painter et al. (2024). Text-to-SQL over a safety database at GSK. arXiv 2406.10690.
- LinkedIn (2025). An internal text-to-SQL assistant with a knowledge graph. arXiv 2507.14372.
- Anthropic (3 June 2026). How Anthropic enables self-service data analytics with Claude.
- Mastech (2 June 2026). Shifting Snowflake Cortex Analyst production accuracy.
- Snowflake. Cortex AI cookbook: mixed grain and verified queries.
- Microsoft ISE Developer Blog (7 May 2026). SQL query generation from natural language.
- Syren (15 April 2026). Building high-accuracy Databricks Genie spaces.
- Databricks (6 February 2026). How to build production-ready Genie spaces.
- Databricks. Webmotors customer story (AI/BI Genie).
- Snowflake (17 August 2026). Building an internal context layer for AI agents at Snowflake.
- dbt Labs (2026). State of Analytics Engineering 2026.
- Snowflake. Semantic View Autopilot documentation.
- Snowflake. Semantic views overview.
- Databricks. Managed MCP servers.
- Microsoft Learn. Fabric MCP servers.
- dbt Labs. dbt MCP server.
- AtScale. Semantic layer platform.
- Databricks. Genie Ontology documentation.
- Databricks (8 May 2026). Pushing the frontier of data agents with Genie.
- Microsoft Learn. Generate an ontology from semantic models (Fabric IQ).
- Microsoft Learn. Fabric data agent.
- Google Cloud (11 April 2026). Introducing QueryData.
- Google Cloud. Looker managed MCP server.
- dbt Labs. MetricFlow repository.
- dbt Labs. Pricing.
- Cube. MCP server documentation.
- Cube. Pricing.
- Apache Ossie. Updates.
- Hacker News discussion of Auto Ontology, 3 November 2026.
How to cite
Ivan Bardak. "NVIDIA Auto Ontology vs semantic layers: will data agents get the right number?". bardak.ai, November 8, 2026. https://bardak.ai/posts/nvidia-auto-ontology



Discussion