Drawers of a wooden library card catalog
Field note13 min read
All posts

NVIDIA Auto Ontology vs semantic layers: will data agents get the right number?

NVIDIA's open-source Auto Ontology drafts a semantic layer for data agents. Its first public score trails a plain agent, and generated layers lose to curated ones in practice. The evidence, the rivals, the defaults to change and a four-arm pilot.

Photo: Erol Ahmed / Unsplash

On this page

    On 21 October 2026 NVIDIA released Auto Ontology, an open-source tool that profiles your databases, has a language model draft what the tables and columns mean in business terms, and lets AI agents ask questions through that draft. Less than three weeks later it had close to 10,000 stars on GitHub. For data and architecture leads the question is a practical one: do our agents need an ontology or semantic layer before they answer questions from the warehouse, and is this the way to build one, or should we use dbt, Cube or the layer our data platform already sells?

    The short answer: curated business context helps overall in every study I found, but the large gains come from vendors or from teams grading their own systems, on small schemas. Independent benchmarks show gains of about 15 to 20 points, and on realistic warehouses the best systems still answer only about one question in seven without hints, and under a third with them. Auto Ontology's first public score, submitted by NVIDIA, is 12.4% on BEAVER, slightly below a general-purpose agent on the same model (14.0%). Treat it as an open draft-maker and a reference design, not a source of trusted numbers: version 1.2.0 is two days old and the project has been public for under three weeks, its own hallucination checks are off by default, and outside users found three connectors returning wrong values without an error. In practice, generated semantic layers score well below curated ones, and curated ones drift within weeks without upkeep. On one data platform, start with that platform's own generator; for governed metrics, use a deterministic layer such as dbt MetricFlow or Cube; try Auto Ontology only in a sandboxed pilot against the four arms at the end of this post.

    What Auto Ontology is, and what it is not

    In code it is five services: a web UI and API gateway, a FastAPI backend that runs a LangGraph text-to-SQL agent, an ingestion worker, an MCP server, and Postgres with pgvector holding the catalog and the vectors (README). A language model reads profiled metadata and sample values table by table and drafts glossary terms, SQL snippets and inferred joins; data experts can mark terms as certified. Agents reach it through nine read-only MCP tools, and the catalog can be exported as Apache Ossie YAML, the open interchange format for semantic models (MCP docs, Apache Ossie).

    Two things are narrower than the name suggests. It reads and writes only its own YAML and Ossie YAML: no OWL, RDF or SPARQL, and no import from dbt or LookML. And the SQL is written by language model nodes and then checked, not compiled deterministically from definitions as dbt's MetricFlow does, although the repository description promises "deterministic query generation" (repository). An independent code read, which my own search of the repository agrees with, also finds that the certified flag is shown to users but does not decide which definitions the agent uses.

    How Auto Ontology sits between data and agents

    metadata and sample valuesprofilesterms, SQL snippets, inferred joinsreview and certifyexport and importbusiness contextgenerated SQLanswers with SQL and rowsnine read-only toolsDatabasesProfilerLLM semantic compileCatalogData expertsApache Ossie YAMLText-to-SQL agentMCP serverAI agents

    The language model drafts the catalog and also writes the SQL; nothing in the query path is a deterministic compiler.

    Source: NVIDIA/auto-ontology README and docs/mcp.md, read 8 November 2026

    Download the data (CSV)

    Show the data
    SourceRelationTarget
    Databasesmetadata and sample valuesProfiler
    ProfilerprofilesLLM semantic compile
    LLM semantic compileterms, SQL snippets, inferred joinsCatalog
    Data expertsreview and certifyCatalog
    Catalogexport and importApache Ossie YAML
    Catalogbusiness contextText-to-SQL agent
    Text-to-SQL agentgenerated SQLDatabases
    Text-to-SQL agentanswers with SQL and rowsMCP server
    MCP servernine read-only toolsAI agents

    Its first two weeks in the open

    A Hacker News thread on 3 November with 290 points (thread) took the repository from 2,134 stars to 9,863 by 8 November. Version 1.2.0 shipped as a Helm chart on 6 November, still with no GitHub release or changelog (NGC Helm index). The outside users who actually ran it reported more than the discussion did:

    • The documented default model was dead at launch. The Nemotron model in the quickstart had been retired on 1 September and answered 410 Gone, so every question on the documented setup failed until an outside contributor fixed the defaults on 4 November (pull request 402).
    • The hosted NVIDIA endpoint rejects its structured-output calls. Two users report a 400 error on every such call and about two minutes per pipeline step from fallbacks; no maintainer had replied by 8 November (issue 440).
    • Three connectors return wrong values silently. Postgres, MySQL and SQLite lose or swap values when two result columns share a name, such as two id columns, with no error to trigger a retry (issue 408).
    • It cannot always say "the data cannot answer this". Its validator has no "the data does not hold the answer" outcome: on one such question it looped for 192 seconds, twice assumed a stars column that does not exist, and ended with a generic refusal (issue 462).

    The project now says it does not accept external pull requests, though it merged the one above (CONTRIBUTING.md). Apache-2.0 lets you fork it, but the roadmap and support are NVIDIA's alone.

    The first accuracy number appeared on 3 November, not in NVIDIA's repositories but on the BEAVER leaderboard, a benchmark built from real query logs of private enterprise warehouses. NVIDIA's entry with Claude Opus 5.5, with no hints, scores 12.4%; ReFoRCE, a general text-to-SQL agent, scores 14.0% with the same model (BEAVER leaderboard). The entry has no ontology-off comparison, so it does not tell us what the ontology adds. NVIDIA's own evaluation harness is well designed, with one verifier for both arms and no LLM judge, but it still publishes no result (auto-ontology-eval). The only product claim, that it "reduced ontology creation time by 90%", comes with no baseline or scope (NVIDIA developer page).

    BEAVER, no hints: Auto Ontology against a general agent

    ReFoRCE, Claude Opus 5.5
    14.0%
    NVIDIA Auto Ontology, Claude Opus 5.5
    12.4%
    ReFoRCE, Nemotron 3.5 Lightning
    9.8%

    Questions answered correctly

    Setting 0 (end to end, no annotations). Auto Ontology's entry was submitted by NVIDIA and lists no method details.

    Source: BEAVER leaderboard, read 8 November 2026

    Download the data (CSV)

    Show the data
    systemExecution accuracy (%)
    ReFoRCE, Claude Opus 5.514.0
    NVIDIA Auto Ontology, Claude Opus 5.512.4
    ReFoRCE, Nemotron 3.5 Lightning9.8

    Does an ontology or semantic layer make the answers right?

    The idea Auto Ontology automates is well supported. No primary study I found shows curated context lowering overall accuracy, though single components sometimes did. How much it helps depends on who ran the test, how small the data was and how weak the baseline was:

    Appraising the evidence

    data.world 2023Vendor (sells the layer)13 tables, 43 questionsExecution match16.7%54.2%change: +37.5 ptsWeak: changed the query language, context and engine at once
    dbt Labs 2026VendorSame data, 11 questions x 20 runsExpected results, method not stated64.5%72.7%change: +8.2 ptsMedium: small set, newer models
    Cube 2026VendorRetail data, 99 questionsLLM judge, lenient46.5%68.7%change: +22.2 ptsMedium: the layer was a 4 KB Markdown file
    Snowflake 2025VendorBIRD, 392 questionsLLM judge57%78%change: +21 ptsMedium: vendor's own product
    GSK 2024Self-graded500+ tables, 60 logged questionsDomain expert8.3%78.3%change: +70.0 ptsMedium: real data, small set
    LinkedIn 2025Self-gradedMillions of tables, 133 questionsLLM judge and experts9%48%change: +39 ptsMedium: real data, "correct or close"
    BIRD 2023Independent95 databases, one-sentence hintsExecution match34.9%54.9%change: +20.0 ptsStrong
    BEAVER 2026IndependentReal warehouses, all oracle hintsExecution match9.5%25.9%change: +16.4 ptsStrong: average of seven models
    EntSQL 2026IndependentFive business domainsExecution match6.8%21.4%change: +14.6 ptsStrong

    Without and with curated business context, same model within each study.

    Source: The papers and posts listed under Sources, read 3 and 8 November 2026

    Download the data (CSV)

    Accuracy without and with curated context, by who ran the study

    data.world 2023 (vendor)
    dbt Labs 2026 (vendor)
    Cube 2026, Sonnet 4.6 (vendor)
    Snowflake 2025 (vendor)
    GSK 2024 (self-graded)
    LinkedIn 2025 (self-graded)
    BIRD 2023, GPT-4 (independent)
    BEAVER 2026 (independent)
    EntSQL 2026 (independent)

    Questions answered correctly

    The biggest jumps come from vendors and self-graded studies; independent benchmarks show about 15 to 20 points.

    Source: Studies as in the table above

    Download the data (CSV)

    Show the data
    studyWithout contextWith context
    data.world 2023 (vendor)16.754.2
    dbt Labs 2026 (vendor)64.572.7
    Cube 2026, Sonnet 4.6 (vendor)46.568.7
    Snowflake 2025 (vendor)5778
    GSK 2024 (self-graded)8.378.3
    LinkedIn 2025 (self-graded)948
    BIRD 2023, GPT-4 (independent)34.954.9
    BEAVER 2026 (independent)9.525.9
    EntSQL 2026 (independent)6.821.4

    Three patterns matter for a buyer.

    • Curation matters more than format. A hand-written 4 KB Markdown file lifted three frontier models by 17 to 23 points in Cube's study, without Cube's engine (Cube 2026); a business context document took GSK from 8.3% to 78.3% (GSK); BIRD's one-sentence hints added 20 points (BIRD). The often-repeated claim that an ontology triples accuracy rests on one vendor study of 43 questions that changed three things at once (data.world 2023).
    • What formal layers add is the refusal. A deterministic layer answers from defined metrics or says it cannot; dbt's semantic layer "tells you it can't answer" instead of returning wrong data (dbt Labs 2026), and ontology checks turned some wrong answers into an explicit "unknown" (data.world 2024). Auto Ontology's version of those checks ships switched off.
    • Outside the modelled scope, the layer scores zero. On dbt's questions that needed joins the semantic layer could not express, it scored 0% while plain text-to-SQL scored 70% to 100% (dbt Labs 2026). Coverage has to be measured as closely as accuracy.

    For how to read claims like these, the same questions apply as in GraphRAG vs RAG: what the evidence actually says, and the label problems are covered in how to evaluate GraphRAG honestly.

    How far from reliable on real warehouses

    Leaderboards on curated public data look strong; anything that resembles a real company does not.

    Best results on text-to-SQL benchmarks

    BIRD, with hints95 databases, dirty values83.4
    Spider 2.0-SnowEnterprise schemas on Snowflake96.7 (best)
    Spider 2.0-LiteThree SQL dialects76.2
    Spider 2.0-DBTA whole dbt project65.6
    BEAVER, with annotationsReal warehouses, schema and domain hints30.5
    EntSQL, curated evidenceNeeds private business documents21.4
    BIRD-InteractMust ask the user for missing knowledge17.0
    BEAVER, no hintsReal warehouses, real query logs14.0

    Human experts score 92.96% on BIRD with hints. Annotation errors affect 52.8% of BIRD Mini-Dev and 62.8% of the checked Spider 2.0-Snow examples, so treat the top scores with care.

    Source: BIRD and BEAVER leaderboards read 8 November 2026; Spider 2.0 leaderboard, BIRD-Interact and EntSQL papers read 3 November 2026; label audit arXiv 2601.08778

    Download the data (CSV)

    BIRD's own team plans to drop its hand-written hints in favour of an interactive track where the agent must ask, and there success falls to 8.7% to 17% (BIRD-Interact). That is the clearest public evidence that missing business knowledge, not SQL syntax, is the bottleneck, which is the problem Auto Ontology and its rivals are selling a fix for.

    The rivals: who else builds the context layer

    Generating semantics from metadata is no longer unique. Snowflake's Semantic View Autopilot builds semantic views from metadata and query history (Snowflake docs), Databricks' Genie Ontology infers context from metric views, dashboards and queries in preview (Databricks docs), and Microsoft's Fabric IQ generates an ontology from Power BI models in preview (Microsoft Learn). What Auto Ontology adds is the open, cross-platform version.

    Auto Ontology and the alternatives

    NVIDIA Auto OntologyLLM drafts from profiled metadata; experts certifySelf-hosted; 10 SQL enginesRead-only MCP, RESTApache-2.0; models and GPUs extra1.2.0, 6 Nov 2026
    dbt Semantic Layer (MetricFlow)Hand-written YAML, AI-assisted; SQL compiled deterministicallyYour warehouse; hosted API on dbt platformdbt MCP serverMetricFlow Apache-2.0; hosted from $100 per user per monthMature
    CubeData model as code, AI-assistedSelf-hosted Core or Cube CloudHosted MCP, can edit the modelCore open source; Cloud from $40 per developer per monthMature
    Snowflake semantic viewsSQL, YAML or generated by AutopilotSnowflake onlyManaged MCP, Cortex AgentsSnowflake consumptionAvailable; Autopilot generally available
    Databricks metric views and Genie OntologyYAML or UI, plus inferred contextDatabricks onlyGenie MCPDatabricks consumptionOntology in preview
    Google Looker (LookML)Hand-written LookMLLookerManaged MCPLooker licenceLookML mature; MCP in preview
    Microsoft Fabric IQ and data agentOntology generated from Power BI modelsFabric onlyFabric MCP serversNeeds F2 capacity or Premium P1Ontology in preview
    AtScaleSML YAML with convertersOver your warehouseMCPNo public pricesLong-standing vendor

    Prices and status as of November 2026.

    Source: Vendor documentation and pricing pages listed under Sources, read 3 and 8 November 2026

    Download the data (CSV)

    Note the design difference. In dbt's design the language model only picks metrics and dimensions, and MetricFlow writes the SQL (MetricFlow). In Auto Ontology the language model writes the SQL. The evidence above credits exactly that determinism for refusing instead of guessing. On the other side, Auto Ontology keeps its MCP server read-only, while Cube's MCP server can change the data model (Cube MCP docs), a write path into production definitions that needs its own review (more in designing MCP tools).

    What happens in practice

    Customer stories from the platform vendors measure adoption and time saved, almost never accuracy. Webmotors reports 72% fewer manual data tickets and 200 analyst hours saved a month with Databricks Genie (Databricks customer story); Snowflake says more than 5,600 of its own employees ran over 320,000 queries through its internal semantic layer, with no accuracy figure (Snowflake blog). Where accuracy is measured, the pattern repeats: an empty or generated layer starts low, and the high numbers arrive only after people curate it.

    Accuracy in practice: generated or empty layer against a curated one

    Mastech, for a clientSnowflake Cortex Analyst63%77%change: +14 pts30 questions; the starting view was generated by Cortex Code and gave silent wrong answers; the better one was built with a custom domain skill
    Microsoft engineers (ISE)Databricks Genie9.5%88.5%change: +79.0 ptsPublic benchmark; an empty space, then 69.2% with descriptions and 88.5% with a feedback loop that needed manual upkeep
    SyrenDatabricks Genie53%80%change: +27 pts15 questions; a space with a basic model, then remodelled; the later 100% came partly from changing the expected answers
    Databricks worked exampleDatabricks Genie0%100%change: +100 ptsIllustrative, unnamed team; 13 questions; 0 of 13 at first, 100% after five iterations
    Anthropic, internal analyticsClaude with a semantic layer and skills21%95%change: +74 pts21% at most without the curated skills; about 95% with them, falling to about 65% in a month without upkeep; set size not disclosed

    Most of these are vendors, partners or companies describing their own systems; none is an independent head-to-head test of products.

    Source: Mastech, Microsoft ISE, Syren, Databricks and Anthropic posts, read 8 November 2026

    Download the data (CSV)

    Two findings bear directly on Auto Ontology's promise. Anthropic tried having a language model generate metric definitions from raw tables and query logs, and it "was net-negative on our evals versus a smaller, human-curated layer" (Anthropic). And Snowflake's own documentation shows a first-draft semantic view that returns confident answers exactly twice the truth, because one table mixed daily and monthly rows (Snowflake cookbook). Automatic drafting saves the first pass; it does not replace the people who know that a table holds two grains. The worry is widespread: in dbt Labs' 2026 survey of 363 respondents, mostly data practitioners, 71% were concerned about wrong data reaching stakeholders (dbt Labs survey).

    Defaults to change before a pilot

    Out of the box, Auto Ontology assumes a trusted lab network. These were still the defaults on 8 November 2026:

    Auto Ontology defaults and pilot settings

    Hallucination checks (DB_PROBE_*) offMade-up values and joins can return wrong results instead of errorsTurn all three on and accept the extra queries
    An sk- or unknown API key falls back to Claude Opus 5.5 through an NVIDIA gatewaySchema metadata, sample values and questions leave your networkPoint every model endpoint at an approved or self-hosted model
    Vault off unless all four variables are setDatabase passwords stored as plain JSON in PostgresConfigure Vault fully
    Helm chart enables pgAdmin with admin@example.com / adminA database console with sample credentials on every nodeDisable pgAdmin; put an ingress with TLS in front
    The backend trusts a user-id header; ports published on the hostAnyone who reaches the port can act as any userKeep the services on an internal network
    API tokens never expire by default and have no scopesA leaked token is a full userSet expiry; issue tokens per pilot user

    One stored connection per database: database row security does not follow the end user unless each connection is scoped.

    Source: model_config.py, .env.example, Helm values.yaml and docs, NVIDIA/auto-ontology, read 8 November 2026

    Download the data (CSV)

    The identity point deserves its own decision: if the agent queries with one service account, everyone who can ask it sees what that account sees. The patterns for passing the user's own rights are in delegated access for AI agents.

    When not to use it

    • Regulated or personal data, or anything reachable from the internet, until the defaults above are changed and you have a threat model, which the repository does not provide.
    • Numbers that go into financial or regulatory reports. Use a deterministic layer, where a metric has one definition and the system refuses what it cannot compute.
    • An estate that lives on one platform. Snowflake, Databricks and Fabric generate semantics inside their own governance; an extra open layer adds a second source of truth.
    • No one to certify definitions. Automatic drafting saves the first pass; the scarce resource is expert time to check it, and no vendor, NVIDIA included, has measured that.

    The four-arm pilot

    The decision is cheap to settle on your own data. I call it the four-arm pilot:

    1. Build the question set. Take 100 to 500 real questions from the warehouse's query logs, each with gold SQL written or checked by a data owner. Include questions the data cannot answer.
    2. Run four arms with the same model: schema only; schema plus a short curated Markdown document; Auto Ontology with its checks on; and your platform's own semantic layer.
    3. Score more than accuracy: execution accuracy, refusals, silent wrong answers, coverage of the modelled scope, latency, cost per question, and the expert hours spent certifying.
    4. Decide on silent wrong answers first. A layer that answers fewer questions but never returns a wrong number beats one that answers more and sometimes lies. If the curated Markdown arm comes within a few points of the formal layers, start there.
    5. Re-run it every month. Data models change and context goes stale: Anthropic saw accuracy fall from about 95% to about 65% in a month when its documentation was not kept up (Anthropic).

    NVIDIA's evaluation harness, with its shared verifier and its rule against headlines when runs break, is a reasonable starting point for the measurement (auto-ontology-eval). How to keep measuring after launch is in evaluating AI agents before and after launch, and how to turn the result into a budget decision in a GraphRAG business case, done the evidence-based way. Where this sits in an agent architecture as a whole is in agentic AI from demo to production, and when a knowledge graph is the right tool at all, in five questions before you build a knowledge graph for RAG.

    Sources

    How to cite

    Ivan Bardak. "NVIDIA Auto Ontology vs semantic layers: will data agents get the right number?". bardak.ai, November 8, 2026. https://bardak.ai/posts/nvidia-auto-ontology

    Data behind the charts (JSON)

    Discussion