Conversation
Two deliverables for teams running dbt below a Malloy semantic layer. skills/malloy-dbt-review: a prior-art adapter for dbt, shaped like malloy-lookml-review. Covers reading dbt's artifacts, translating each metric type, the two naming collisions that are compile errors on any real project, and reconciling against dbt's own engine rather than against hand-written SQL. examples/dbt-jaffle-shop: two packages over dbt-labs/jaffle-shop. - semantic-layer/ is dbt's semantic layer converted to Malloy, over the marts dbt built. bindings.malloy and jaffle_shop.malloy are generated by tools/dbt_to_malloy.ts from dbt's artifacts, with a --check drift gate. - transforms/ rebuilds dbt's staging and marts in Malloy over the raw data, materialized with #@ persist. Both run with no warehouse, no credentials, and no dbt installed. Verified, not asserted: - 20 of 23 dbt metrics became measures, and all 20 match MetricFlow's output on the same data, including two ratio metrics and a derived metric whose inputs span two semantic models across a one-to-many join. - The other three are accounted for: two became views (a measure cannot hold a window), and median_revenue is deferred because this Malloy build has no scalar median. It is not silently shipped as avg. - All six marts in transforms/ agree with dbt's built marts across 25 compared aggregates, and all six materialize. The converter reads semantic_manifest.json, not the OSI export. OSI flattens metrics to SQL and loses two things that produce wrong numbers: filtered metrics reference MetricFlow's internal qualified dimension names rather than columns, and a derived metric's offset_window is dropped, turning revenue_growth_mom into (SUM(x) - SUM(x))*100/SUM(x). dbt warns about its cumulative metric losing meaning in OSI but not about that one. dbt's 27 data tests and 3 unit tests are not carried over; Malloy has no test framework today. COVERAGE.md states that, and every other difference, with the reason. Signed-off-by: James Swirhun <james@credibledata.com>
jswir
force-pushed
the
feat/dbt-jaffle-shop
branch
from
August 18, 2026 23:10
950548d to
37847b7
Compare
Reshapes the example and skill around the two motions in the dbt roadmap
instead of around my tooling.
Agentic, not generated. Deletes tools/dbt_to_malloy.ts and the vendored
semantic_manifest.json. The Malloy is written by reading dbt's artifacts;
the knowledge for doing that lives in the skills, not in a program. Also
folds bindings.malloy and analysis.malloy into one model file: the split
existed only to quarantine generated text from hand-written text, and that
distinction died with the generator.
Both packages are semantic layers. Renames semantic-layer/ -> adopt/ and
transforms/ -> convert/, because the difference is not whether there is a
semantic layer but what sits underneath it: the marts dbt built, or marts
Malloy rebuilt and persisted from raw. convert/ now carries the same
semantic layer text as adopt/, so the interface is identical and the claim
is checkable:
run: orders -> { aggregate: order_total, orders, large_orders, food_orders }
adopt 105826.18 9568 1141 2336
convert 105826.18000000007 9568 1141 2336
Integers match exactly; money differs in the last few decimal places
because convert computes dollars from cents in floating point while dbt
casts to numeric(16,2). The exact tail is not stable across runs, so the
docs no longer assert one.
convert/ splits the built table (orders_mart) from the semantic source
(orders). That is load-bearing: customers_mart is built from orders_mart
while the semantic orders joins customers, so collapsing the layers is a
cycle Malloy rejects.
One skill became two, one per motion: malloy-dbt-adopt (semantic layer on
existing marts) and malloy-dbt-convert (move the pipeline). Nine reference
files became four. What went was restatement of general Malloy knowledge
and of the house modeling workflow; what stayed is what was verified the
hard way -- the OSI export's two silent corruptions, the "Cannot redefine"
collision on six metrics, #(doc) only attaching to declared fields, the
joined-count outer-join trap, and which metric types cannot be measures.
Verified against a booted Publisher: 5 packages, load_errors=0, both
notebooks compile clean (23 and 11 cells), all six convert marts
materialize. A notebook cell referencing orders_mart through a transitive
import was caught only by that server load, not by compiling the model.
Signed-off-by: James Swirhun <james@credibledata.com>
jswir
changed the title
…rface
Converting dbt's metrics faithfully produced something that reconciled and
that nobody could use. Six co-equal sources with no entry point, money
rendering as bare floats, three views total, and dimension tables that were
two-column dead ends when opened. Picking `supply_costs` out of the Explorer
and getting two dimensions was the symptom.
Adds the authoring pass that dbt's semantic layer cannot supply:
- `order_items -> overview`, a `# dashboard` entry point: revenue KPIs, the
monthly trend, product mix, top sellers, and revenue by location and by
customer type. Its #(doc) says to start there.
- Render tags on every measure that needs one -- `# currency`, `# percent` --
plus a chart type per view. dbt records no display formatting, so none of
this could be converted.
- 14 analysis views across all six sources, so every source someone can pick
has something to say. dbt declares no metrics at all on products or
supplies, hence `product_count`, `supply_count`, `total_supply_cost`, and
`order_item_count`.
- `explores` + `queryableSources` in both manifests. convert/ was exposing 13
sources (stg_*, *_mart, supply_costs) against adopt/'s 6, which broke the
"identical interface" claim at the one surface a user actually touches.
Both now list exactly the same six.
COVERAGE.md gains an "Added beyond dbt" section, because none of the above is
converted and it should not read as though it were. The 20 measures that
reconcile against MetricFlow are unchanged and still match: adopt exactly,
convert exactly on integers and to ~1e-9 on money.
The adopt skill gains the general lesson, which is the part worth reusing: a
faithful metric conversion is not yet a usable model, and the authoring pass
is part of the job rather than a polish step.
Also found and recorded a Malloy issue: a limit refinement on a nested
dashboard (`-> overview + { limit: 2 }`) emits
`ROW_NUMBER() OVER (PARTITION BY group_set ORDER BY )` and fails to parse.
The dashboard runs fine without it.
Verified: 5 packages, load_errors=0, notebooks 25 and 11 cells with 0 errors,
all six convert marts materialize, every new view runs.
Signed-off-by: James Swirhun <james@credibledata.com>
Splits the demo along the two axes it was conflating. Each comparison now varies one thing: adopt-mechanical mechanical model, dbt's marts the ceiling of conversion adopt-rich rich model, dbt's marts diff = modelling only convert rich model, Malloy's pipeline diff = plumbing only adopt-mechanical restores the artifact the previous commit destroyed by enriching in place: the converter's raw output, and nothing else. 20 of 23 metrics, all matching dbt, 3 views from dbt's saved queries, 20 label tags, and zero chart or format tags. It reconciles and it is close to unusable, which is the point of having it next to adopt-rich. This is the ceiling of any automated translation, not a straw man: dbt's YAML does not carry the missing information, so no converter can do better. adopt-rich adds what only authoring can supply: cohort and band dimensions with their thresholds flagged as conventions, four window views as model entities rather than per-chart calculations, an entry-point dashboard, render tags, and two audience extensions over one base model. One of those is gated with #(authorize) "$role = 'finance'" and returns HTTP 403 to a caller asserting the default role, verified against a running server. convert drops the marts layer. dbt splits staging views from mart tables; pipeline.malloy is one transformation and #@ persist is a latency knob, so only order_items, orders, and customers build a table. products, supplies, and locations build nothing, because a rename does not earn one. Three tables instead of six. Found while building: an #(authorize) gate cannot sit in a materialized lineage. Publisher refuses the build and says why -- an authorize expression is evaluated per request, while a table served frozen carries no gate, so the rows would reach everyone. It fails closed rather than serving ungated data. convert therefore omits the gated source, and that is the one place the semantic layer is not portable between foundations. Documented in COVERAGE.md and the adopt skill. The skill also gains the general test, which is the reusable part: ask whether a construct can be a reusable, composable model entity or only exists inside one chart. That framing survives any single vendor shipping any single feature. Verified: 6 packages, load_errors=0; all three jaffle packages return identical integers on the same query and agree with dbt; notebooks 25 and 11 cells with 0 errors; convert materializes 3 sources; the authorize gate returns 403 and the ungated control 200. Signed-off-by: James Swirhun <james@credibledata.com>
Asked the model a real question over MCP -- are newer customer cohorts worth less -- and the first answer was wrong in a way that looked authoritative. `lifetime_spend` by first-order month reads $37,422 for the 2024-09 cohort against $904 for 2025-08, which looks like a 97% collapse. It is mostly tenure: September has had twelve months to accumulate and August has had one. Adds `tenure_months` and `spend_per_tenure_month`, with docs telling readers to prefer the normalized measure for cohort comparisons, and puts it in the cohort view alongside the raw total. The conclusion inverts: 2024-10 is the strongest cohort at $202/month, and the decline across the year is about 31% rather than 97%. The point is not the number, it is where the correction lives. A per-chart calculation would have fixed one chart; a documented measure is inherited by every future question, including an agent's. Also documents two dimensions that are degenerate on this sample rather than leaving them to mislead: 149 of 150 customers are repeat buyers, so `repeat_buyer_rate` is ~100% for nearly every cohort, and `customer_type` reflects current state so revenue splits returning $100,435 / new $6. `months()` needs dates, not timestamps: `months(a::date to b::date)`. The type error surfaced as a bogus annotation-syntax error 100 lines earlier and vanished when the real one was fixed, which is the cascade the gotchas skill warns about. Verified over MCP against a running server: reloaded in place, cohort view returns the normalized measure with its #(doc) and # currency tags attached. Signed-off-by: James Swirhun <james@credibledata.com>
jswir
changed the title
Same six tiles in all three packages, so the pages render the comparison the packages are making rather than describing it. In adopt-rich every tile names a view the model defines: revenue_by_month, revenue_growth_mom, revenue_by_product_type, top_products, cohorts_by_first_order_month. The page contains no analysis of its own. In adopt-mechanical the same six tiles need the page to supply what the mechanical conversion cannot. Four write their own Malloy, because dbt's only revenue view is grouped by day and there is no leaderboard view, so the page decides the monthly grain, the join path to product_type, and the top-N cutoff. Month-over-month is computed in JavaScript, because dbt declares the metric but its offset window cannot be a measure. And the cohort tile shows the tenure-confounded lifetime total with a note naming what is missing and why -- spend_per_tenure_month does not exist there, and asking for it is refused with "Query target is not queryable". Per the skill: do not silently drop a metric you could not model. convert/public/lib/ is byte-identical to adopt-rich/public/lib/ (verified with cmp). The app cannot tell whether its marts were built by dbt or by Malloy, which is the portability claim as a diff. Built to the html-data-apps recipe: modular (DOM-free formatters, an SVG chart layer, tiles as data, a thin entry point), per-tile loading/empty/error so one failing query cannot blank the page, missing never rendered as zero, division guarded, and assumptions stated in the app rather than buried. No chart library is loaded from a CDN or vendored at all -- the charts are inline SVG, because embedded page JavaScript runs with the viewing user's data authority. Verified three ways, not just eyeballed: - 8 unit tests over the pure logic (withMoM returns null for a first row, a missing point and a zero prior month rather than a fake 0%; latestNonNull skips trailing nulls; formatters render missing as a dash). - every tile rendered against real rows from a running server, asserting the output contains no undefined/NaN. - all three pages loaded in headless Chromium, waiting on content rather than network idle (publisher.js holds an SSE stream open): 18 tiles, none stuck loading, none in error, no JS console errors. The page-computed month-over-month in the mechanical app matches the model-defined measure in the rich one (12.9% for Oct 24), which is the check that the two paths agree on the numbers and differ only in where the analysis lives. Signed-off-by: James Swirhun <james@credibledata.com>
Two corrections and one improvement, all from exercising the model with
harder questions than "what is revenue".
`include { public: ... }` carries #(doc), which changes the guidance. The
skill said a passthrough table column cannot be annotated and a `select:`
projection is the way in. That is only half right: an `include {}` block
annotates the table's own columns in place, needs no projection, and
`public: *` covers everything you did not name, so only the documented
columns get written out. Verified, including that `internal:` does NOT free
a name for a measure ("Cannot redefine" persists), so the six renames stand.
The projection keeps one advantage worth stating: it makes the source a query
source, which is the only shape `#@ persist` can materialize.
`include {}` and `rename:` do compose, in either order, as long as each block
names a field by the name it has at that point. The review rubric's C-06 says
they do not and flags any source containing both, which would false-positive
on the shape used here.
Uses the feature to curate, which is the improvement. The `<name>_raw`
columns existed only because dbt metrics took their columns' names, and dbt's
marts carry `*_cents` duplicates of the dollar columns. Both are now
`internal:` in the rich packages: absent from the public API, still readable
by the measures. A caller naming one gets `'order_total_raw' is internal`.
The naming compromise is invisible to consumers instead of merely documented,
and every measure still reconciles against dbt exactly.
Records a coverage cliff found by asking about locations. Six locations exist
and only two have any orders, so `location_count` says 6 while
revenue-by-location returns 2 rows. It is sampling, not two stores, and it is
structural: `orders` joins `locations`, so the fact side cannot show a
location with no rows. Both rich packages carry the caveat in the source's
#(doc), and `locations` gains the `location_count` measure that `products` and
`supplies` already had.
What the harder questions did not show is any capability gap. Per-product
windowed growth, cohort revenue attribution across two joins, and percent of
total all work in the mechanical package too, because Malloy's query language
is equally available to both. The difference is only ever whether the judgment
is captured for the next asker.
Verified: 6 packages, load_errors=0; all three packages return identical
integers and still match dbt; 18 app tiles render in a browser with no JS
errors; convert materializes three tables (order_items, orders, customers).
Signed-off-by: James Swirhun <james@credibledata.com>
Answering "are there harder queries that might not be correct?" by looking for wrong numbers rather than missing ones, against the known dbt totals. Symmetric aggregates hold up. Summing one-side money from the many-side returns 105826.18 at the item grain, one hop up and two hops up, so fan-out is not a hazard here. count()-based measures are a hazard, and the model now says so. `count()` means "rows in scope", and a join narrows the scope: 77 of 9,568 orders have no items, so from `order_items` dbt's own `orders` metric reads 9,491 and its `new_customer_orders` reads 148 rather than 150. Two measures added in this example inherit the shape and shift with it -- `avg_order_value` reads 11.150 against 11.060, and `items_per_order` 1.501 against 1.489. Both figures are correct; they describe different populations, nothing in the result says which, and 0.8% passes review. Neither can be made grain-invariant, because there is no way to count all orders from the item side, so their #(doc) now states the grain they mean and the value a join returns. A ratio of two sums has no such problem, which is the pattern to prefer: `average_order_value` on customers returns 10.497596153846153 from both grains in all three packages. Worth noting the direction of the finding: dbt's metrics carry this hazard too, identically in the mechanical package, and dbt records it nowhere. But the two extra instances are mine -- authoring adds power and adds surface for this class of mistake, so the honest claim for a rich model is not that it is safer, only that it can say what it means. Two smaller results from the same audit, both documented: `internal:` propagates across joins, so curating a column hides it from neighbours too; and `all()` respects the query's filter, so `revenue / all(revenue)` under `where: is_food_item` is 1.0 rather than food's share of the total. Signed-off-by: James Swirhun <james@credibledata.com>
The earlier audit found no capability gap between the mechanical and rich models: per-product windowed growth, cohort attribution across two joins and percent-of-total all work in both, because Malloy's query language belongs to Malloy rather than to the model. So the question became whether the mechanical model lets you reach a wrong answer. It does, three times, and all three come from dbt's own metric set. "What was our revenue?" has two defensible answers 5.4% apart. Six metric names resolve to 100,441 (pretax) or 105,826.18 (with tax), and nothing in dbt's artifacts says which is the revenue figure. order_total's doc now says it is gross receipts rather than revenue and points at the right field, and a revenue_reconciliation view shows subtotal + tax_paid = order_total so the relationship is checkable. "What share of orders are food versus drink?" sums to 122%. dbt's food_orders and drink_orders overlap on 2,164 orders and omit the 77 with no items. A new order_contents dimension gives four mutually exclusive, exhaustive buckets that sum to 9,568 exactly, and it changes the story: food-only is 172 orders, 1.8%, not the 24.4% the overlapping flag implies. Both dbt metrics now carry the caveat. "How much revenue comes from new customers?" is wrong by 400x. dbt ships two definitions and reconciles neither: customer_type = 'new' is a current-state label and answers $6.36, while the customer's first order answers $2,586.13. customer_type's doc now says which it is and names the gap, orders gains is_first_order for the event definition, and an acquisition_split view answers the question with it (150 / $2,586.13 first, 9,418 / $103,240.05 repeat, summing to the full population). What the three have in common is the useful part: every number above is arithmetically correct and matches dbt exactly, so reconciliation cannot catch any of them. They are ambiguities of definition, and a faithful conversion preserves them perfectly because faithfulness is the objective. Nothing in the YAML says tax is not revenue, that two boolean flags overlap, or which "new customer" the business means. Verified: every bucket and split ties out to the known totals (9,568 orders and 105,826.18); 6 packages, load_errors=0; 18 app tiles render in a browser with no JS errors; all three packages still return identical integers and match dbt. Signed-off-by: James Swirhun <james@credibledata.com>
Audit of my own work, prompted by "are the skills good?". They were not: a
grep for the strongest findings of the last few rounds turned up nothing in
either skill. The definition ambiguities, the tenure confound and the
grain-bound measures existed only in the example's COVERAGE.md, which is one
instance. The skill is the part that transfers.
Adds reference/ambiguity.md: the class of failure reconciliation cannot catch,
as five patterns with the test for each and how to settle it.
1. Two metrics that both mean "revenue" and differ by tax or discounts.
Test: total every money metric over one population and group by value;
any group with more than one member is an ambiguity.
2. Boolean flags presented as a mix, which overlap or omit. Test: sum the
counts against the population; over 100% means overlap, under means a
missing bucket.
3. One business noun with a state definition and an event definition.
Test: for each noun, list the fields that could express it and classify
each as computed from a lifetime total (state) or a row's position in a
sequence (event).
4. Cohort comparisons confounded by tenure. Test: divide by a tenure
measure and see whether the ranking changes.
5. Measures that change value across a join, because count() means "rows in
scope". Test: total each count()-based metric at its own grain and from a
child source. Prefer a ratio of two sums, which is grain-invariant.
Each pattern carries the jaffle-shop numbers as evidence rather than as
illustration, because the reason to believe the pattern is that it was found
this way.
reconcile.md gains the limit it was missing, which was that file's real
weakness: a green reconciliation and a wrong answer are fully compatible.
Every ambiguity above matches dbt to the digit, because they *are* dbt's
definitions. It now says to report the two claims separately and never let
"20 of 20 matched" stand in for "the definitions were checked".
Signed-off-by: James Swirhun <james@credibledata.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three Malloy packages over dbt's jaffle-shop, and one skill per motion from the dbt roadmap.
Three packages, one variable at a time
adopt-mechanical/adopt-rich/convert/Everything runs with no warehouse, no credentials and no dbt installed: DuckDB reads ~1.6MB of committed Parquet in place.
mechanical → rich: what modelling buys
adopt-mechanicalis a complete, faithful conversion of dbt's semantic layer: every model a source, every column doc a#(doc), every entity a key or join, every metric a measure, every saved query a view. 20 of dbt's 23 metrics, all 20 matching dbt's own output exactly. Nothing skipped out of laziness.It is also close to unusable: no entry point, no chart or format tags, money rendering as bare floats, three of six sources with no measures at all, dimension tables that are two-column dead ends. That is the ceiling of what any automated translation reaches, because dbt's YAML does not carry the missing information.
The consumer-side difference, same question to both packages via
malloy_getContext:product_id/product_namewith empty doc strings. Nothing about month-over-month; the agent must invent the time grouping and the growth arithmetic.revenue_growth_mom("Revenue by month with month-over-month growth as a percentage"),revenue_by_month,avg_order_value_trend,size_band_by_month.adopt-richadds what only authoring can supply: cohort and band dimensions with thresholds flagged in their#(doc)as conventions rather than facts; four window views as model entities rather than per-chart calculations; anorder_items -> overviewdashboard as the entry point; render tags; and two audience extensions over one base source, one gated with#(authorize) "$role = 'finance'"(403 for the default role, rows whenrole=financeis supplied).The durable test is not "is Malloy nicer" but: can this construct be a reusable, composable model entity, or does it only exist inside one chart?
rich → convert: where the transformation lives
Same semantic layer, reading what Malloy derives from raw files instead of what dbt built:
Integers match exactly; money differs in the last few decimal places (floating point vs dbt's
numeric(16,2)), and the tail is not stable across runs, so the docs assert a range rather than a value.There is no marts layer. dbt splits staging views from mart tables;
pipeline.malloyis one transformation and#@ persistis a latency knob. Onlyorder_items,orders, andcustomersbuild a table — 3 instead of 6 — because a rename does not earn one. Evenrow_number() over (partition by customer_id order by ordered_at)is just acalculate:.Five things that bite on any real dbt project
Each was found by running something and being wrong, and each is now in a skill:
osi_document.json. dbt's OSI export flattens metrics to SQL and corrupts two: a filtered metric emitsSUM(CASE WHEN order_id__order_total_dim >= 20 ...), referencing MetricFlow's internal dimension name rather than a column; and a derived metric'soffset_windowis dropped, so month-over-month growth becomes(SUM(x) - SUM(x))*100/SUM(x), identically zero. dbt warns about its cumulative metric losing meaning in OSI but not about this one.<name>_raw; the metric keeps the name analysts and agents search for.#(doc)only attaches to a declared field. A self-referencing dimension fails and a bare field cannot be re-annotated inextend. Aselect:projection accepts annotations on passthrough columns, which is how dbt's column docs get carried across.median_revenueis deferred, because this build has no scalar median and every raw-SQL aggregate escape resolves as a scalar. Not silently shipped asavg.joined.count()counts the outer-join row. 77 of 9,568 orders have no items, so the obvious form summedcount_order_itemsto 14,327 against dbt's 14,250. Nothing errored.count(order_items.order_item_id)is correct. Caught by reconciling against dbt's marts, which is the argument for reconciling in one anecdote.Verified, not asserted
The oracle is dbt's own engine (
mf query), never SQL rewritten by hand.ratiometrics andorder_gross_profit, aderivedmetric spanning two semantic models across a one-to-many join where symmetric aggregates matter.load_errors=0; notebooks 25 and 11 cells with 0 errors;convertmaterializes 3 sources; the authorize gate returns 403 and the ungated control 200.lifetime_spendread $37,422 (2024-09) against $904 (2025-08), which is mostly tenure.spend_per_tenure_monthinverts the conclusion — 2024-10 is the strongest cohort — and the correction now lives in the model rather than in one chart.Two constraints worth knowing
An
#(authorize)gate cannot live in a materialized lineage. Publisher refuses the build, because an authorize expression is evaluated per request while a table served frozen carries no gate. It fails closed with that explanation.converttherefore omits the gated source, and that is the one place the semantic layer is not portable between foundations.Build and semantics stay separate in
convert.customers_builtis derived fromorders_builtwhile the semanticordersjoinscustomers; fusing them is a cycle Malloy rejects.What this does not do
dbt's 27 data tests and 3 unit tests are not carried over — Malloy has no test framework today, which is why
adopt-richis the recommended shape andconvertdemonstrates what moving the pipeline costs and buys.COVERAGE.mdrecords that plus every other difference: floating-point money, a differently-constructed surrogate key, 77 itemless orders that dbt records as NULL and Malloy as 0, and two dimensions that are degenerate on this sample.No code generator ships. The Malloy is written by reading dbt's artifacts, and the knowledge for doing that lives in
skills/malloy-dbt-adoptandskills/malloy-dbt-convertrather than in a program.Related
executeQuerydiscards the server's "No queryable source" message (found while curating these packages' surfaces).