diff --git a/.claude/skills/malloy-model-as-you-go b/.claude/skills/malloy-model-as-you-go new file mode 120000 index 000000000..59408fc97 --- /dev/null +++ b/.claude/skills/malloy-model-as-you-go @@ -0,0 +1 @@ +../../skills/malloy-model-as-you-go \ No newline at end of file diff --git a/packages/create-malloy-package/package.json b/packages/create-malloy-package/package.json index ff1a84c9e..849b7e154 100644 --- a/packages/create-malloy-package/package.json +++ b/packages/create-malloy-package/package.json @@ -1,7 +1,7 @@ { "name": "@malloy-publisher/create-malloy-package", "description": "Scaffold a Malloy Publisher package and a local agent workspace, so one command takes you from nothing to an agent that can query your data.", - "version": "0.0.8", + "version": "0.0.9", "license": "MIT", "type": "module", "engines": { diff --git a/packages/server/src/mcp/skills/skills_bundle.json b/packages/server/src/mcp/skills/skills_bundle.json index 993b4efea..109d45765 100644 --- a/packages/server/src/mcp/skills/skills_bundle.json +++ b/packages/server/src/mcp/skills/skills_bundle.json @@ -3,7 +3,7 @@ { "name": "malloy", "description": "Index of all Malloy skills. Use when user asks \"malloy help\", \"what malloy skills are available\", \"how do I use malloy\", or needs guidance on which Malloy skill to use.", - "body": "# Malloy Skills Index\n\n## First-Time Setup\n\n**No .malloy files in workspace?**\nSay \"model my data\" and the agent will orchestrate the full modeling workflow automatically. Make sure the Malloy Publisher MCP tools are configured first.\n\n## Skill Reference\n\nEvery skill in this deployment, by what it is for. Start at a driver; it routes to the rest.\n\n**Start here**\n\n| Skill | Use when... |\n|-------|-------------|\n| `skill:malloy-getting-started` | First contact with a Publisher: confirming the tools, finding what data exists, running a first grounded query |\n| `skill:malloy-modeling` | Building a semantic model from scratch (the modeling workflow driver) |\n| `skill:malloy-analysis` | Answering a data question or exploring data (the analysis workflow driver) |\n\n**Modeling phases** (driven by `skill:malloy-modeling`)\n\n| Skill | Use when... |\n|-------|-------------|\n| `skill:malloy-discover` | Silent data discovery: tables, schemas, distributions, prior art |\n| `skill:malloy-scope` | Presenting findings and proposing an analytical focus |\n| `skill:malloy-define` | Proposing the source plan and field definitions |\n| `skill:malloy-model` | Writing base and joined source .malloy files, review, curate (includes normalized schema support) |\n| `skill:malloy-document` | Adding `#(doc)` tags for discoverability |\n| `skill:malloy-lookml-review` | Prior-art adapter for LookML (field extraction, derived tables, visibility, docs) |\n\n**Analysis and presentation**\n\n| Skill | Use when... |\n|-------|-------------|\n| `skill:malloy-analyze` | Exploratory data analysis: profiling, building views and dashboards |\n| `skill:malloy-charts` | Chart selection and renderer reference for Malloy visualizations |\n| `skill:malloy-notebooks` | Building Malloy notebooks (.malloynb) |\n| `skill:malloy-analysis-report` | Combining validated queries into a notebook report or dashboard |\n| `skill:malloy-analysis-pitfalls` | Checking a query and its results before presenting an answer |\n| `skill:malloy-notebook-chat` | The chat is bound to a notebook or saved report; answer from its cells |\n| `skill:malloy-phrase-detection` | Turning a plain-English question into search targets for the context tool |\n\n**Writing correct Malloy** (read before writing, not after failing)\n\n| Skill | Use when... |\n|-------|-------------|\n| `skill:malloy-queries` | Query and view syntax: dates, aggregates, join paths, filters |\n| `skill:malloy-gotchas-modeling` | Before writing sources, dimensions, measures, joins |\n| `skill:malloy-gotchas-queries` | Before writing views, queries, notebooks |\n| `skill:malloy-gotchas-rendering` | Before adding chart annotations or formatting tags |\n| `skill:malloy-debug` | Fixing compile errors and interpreting diagnostics |\n| `skill:malloy-patterns` | Finding syntax/pattern docs: YoY, cohorts, percent-of-total, window functions |\n| `skill:malloy-review` | Reviewing, auditing, or critiquing existing Malloy |\n\n**Serving and operating a package**\n\n| Skill | Use when... |\n|-------|-------------|\n| `skill:malloy-publish` | Moving a finished model into a served package (local-to-served handoff) |\n| `skill:malloy-dashboards` | Building a dashboard: a tagged `.malloy` file in a package's `dashboards/` directory, with filter controls and drill-through |\n| `skill:malloy-html-data-apps` | Building an in-package HTML data app (a `public/` directory the package serves) |\n| `skill:malloy-html-data-app-runtime` | Writing the JavaScript that drives that app |\n| `skill:malloy-html-data-app-embedding` | Embedding a served page into a host application |\n| `skill:malloy-materialization` | Persisting an expensive source so queries read a pre-built table |\n| `skill:malloy-materialization-tuning` | Tuning what to persist, and on what schedule, for cost and speed |\n\n> **Adapter pattern:** Each prior art adapter (LookML, future dbt) follows the same structure: a coordinator SKILL.md plus reference files under `reference/` dispatched by phase skills.\n\n## Workflows\n\nTwo top-level workflows orchestrate the phase and support skills above:\n\n- **Model data from scratch:** load `skill:malloy-modeling`. It drives the full pipeline (discover, scope, define, build, review, curate) and routes to the phase skills.\n- **Answer a data question or explore:** load `skill:malloy-analysis`. It drives exploratory analysis, views, and notebooks, using `skill:malloy-analyze` and `skill:malloy-charts`.\n\nPublishing is out of scope for open-source Publisher v1. Self-hosters move a finished model into a served package via git and the host's publish path; see `skill:malloy-publish`.\n\n## Syntax Help\n\nCall `malloy_searchDocs` with your question. Use `skill:malloy-patterns` to discover available topics." + "body": "# Malloy Skills Index\n\n## First-Time Setup\n\n**No .malloy files in workspace?**\nSay \"model my data\" and the agent will orchestrate the full modeling workflow automatically. Make sure the Malloy Publisher MCP tools are configured first.\n\n## Skill Reference\n\nEvery skill in this deployment, by what it is for. Start at a driver; it routes to the rest.\n\n**Start here**\n\n| Skill | Use when... |\n|-------|-------------|\n| `skill:malloy-getting-started` | First contact with a Publisher: confirming the tools, finding what data exists, running a first grounded query |\n| `skill:malloy-modeling` | Building a semantic model from scratch (the modeling workflow driver) |\n| `skill:malloy-analysis` | Answering a data question or exploring data (the analysis workflow driver) |\n\n**Modeling phases** (driven by `skill:malloy-modeling`)\n\n| Skill | Use when... |\n|-------|-------------|\n| `skill:malloy-discover` | Silent data discovery: tables, schemas, distributions, prior art |\n| `skill:malloy-scope` | Presenting findings and proposing an analytical focus |\n| `skill:malloy-define` | Proposing the source plan and field definitions |\n| `skill:malloy-model` | Writing base and joined source .malloy files, review, curate (includes normalized schema support) |\n| `skill:malloy-document` | Adding `#(doc)` tags for discoverability |\n| `skill:malloy-lookml-review` | Prior-art adapter for LookML (field extraction, derived tables, visibility, docs) |\n\n**Analysis and presentation**\n\n| Skill | Use when... |\n|-------|-------------|\n| `skill:malloy-model-as-you-go` | After answering a question, writing down what it assumed: a `#(doc)`'d field in the model, an `extend` in the notebook, or a stated assumption plus snippet, depending on what the session can write |\n| `skill:malloy-analyze` | Open-ended exploration with no intent to keep anything: profiling, hypotheses, views |\n| `skill:malloy-charts` | Chart selection and renderer reference for Malloy visualizations |\n| `skill:malloy-notebooks` | Building Malloy notebooks (.malloynb) |\n| `skill:malloy-analysis-report` | Combining validated queries into a notebook report or dashboard |\n| `skill:malloy-analysis-pitfalls` | Checking a query and its results before presenting an answer |\n| `skill:malloy-notebook-chat` | The chat is bound to a notebook or saved report; answer from its cells |\n| `skill:malloy-phrase-detection` | Turning a plain-English question into search targets for the context tool |\n\n**Writing correct Malloy** (read before writing, not after failing)\n\n| Skill | Use when... |\n|-------|-------------|\n| `skill:malloy-queries` | Query and view syntax: dates, aggregates, join paths, filters |\n| `skill:malloy-gotchas-modeling` | Before writing sources, dimensions, measures, joins |\n| `skill:malloy-gotchas-queries` | Before writing views, queries, notebooks |\n| `skill:malloy-gotchas-rendering` | Before adding chart annotations or formatting tags |\n| `skill:malloy-debug` | Fixing compile errors and interpreting diagnostics |\n| `skill:malloy-patterns` | Finding syntax/pattern docs: YoY, cohorts, percent-of-total, window functions |\n| `skill:malloy-review` | Reviewing, auditing, or critiquing existing Malloy |\n\n**Serving and operating a package**\n\n| Skill | Use when... |\n|-------|-------------|\n| `skill:malloy-publish` | Moving a finished model into a served package (local-to-served handoff) |\n| `skill:malloy-dashboards` | Building a dashboard: a tagged `.malloy` file in a package's `dashboards/` directory, with filter controls and drill-through |\n| `skill:malloy-html-data-apps` | Building an in-package HTML data app (a `public/` directory the package serves) |\n| `skill:malloy-html-data-app-runtime` | Writing the JavaScript that drives that app |\n| `skill:malloy-html-data-app-embedding` | Embedding a served page into a host application |\n| `skill:malloy-materialization` | Persisting an expensive source so queries read a pre-built table |\n| `skill:malloy-materialization-tuning` | Tuning what to persist, and on what schedule, for cost and speed |\n\n> **Adapter pattern:** Each prior art adapter (LookML, future dbt) follows the same structure: a coordinator SKILL.md plus reference files under `reference/` dispatched by phase skills.\n\n## Workflows\n\nTwo top-level workflows orchestrate the phase and support skills above:\n\n- **Model data from scratch:** load `skill:malloy-modeling`. It drives the full pipeline (discover, scope, define, build, review, curate) and routes to the phase skills.\n- **Answer a data question or explore:** load `skill:malloy-analysis`. It drives exploratory analysis, views, and notebooks, using `skill:malloy-analyze` and `skill:malloy-charts`.\n\nPublishing is out of scope for open-source Publisher v1. Self-hosters move a finished model into a served package via git and the host's publish path; see `skill:malloy-publish`.\n\n## Syntax Help\n\nCall `malloy_searchDocs` with your question. Use `skill:malloy-patterns` to discover available topics." }, { "name": "malloy-analysis", @@ -18,12 +18,12 @@ { "name": "malloy-analysis-report", "description": "Combine validated Malloy queries into a notebook report or dashboard. Use when the user asks to \"create a report\", \"build a dashboard\", \"combine these into a report\", or wants a persistent multi-query artifact.", - "body": "# Creating Reports\n\nAn ad-hoc report is a `.malloynb` notebook that combines markdown narrative with live Malloy query cells. There is no dedicated report tool: you author the notebook directly. Load `skill:malloy-notebooks` for the full `.malloynb` cell format and authoring rules; this skill covers when to build one and how to design good report content (cells, chart annotations, narrative structure).\n\n> **Tool names** are written bare here - `get_context`, `execute_query`, `search_malloy_docs`. The exact prefixed name depends on the host surface; match each against the tools you actually have.\n\n## Before building a report\n\n1. **Run each query first** via `execute_query` to verify it works and returns expected results.\n2. **Explain the results** to the user as you go: walk through the analysis step by step.\n3. **Then assemble the notebook** once the analysis is validated.\n\nDo NOT build the notebook in the same turn as `execute_query`. Explain first, then build.\n\n## Filters are inherited from the model, don't declare them in the report\n\nReports do not (and cannot) define their own filters. If the source has `#(filter)` annotations, Publisher renders the filter widgets, parses caller parameters, and injects `where:` clauses server-side automatically: the report inherits and displays those filters with no extra work. If the analysis needs a knob the source doesn't expose, the right move is to add a `#(filter)` to the source itself (see your modeling workflow's parameterizable-filter guidance for `#(filter)`), not to wedge a filter widget into the report. For curated notebooks with their own per-notebook filter UI on top of the model, see `skill:malloy-notebooks` instead.\n\n## What goes in the report\n\nDo NOT add an H1 heading in any cell (use H2 and below for sections); the notebook name serves as the title. To redo the structure rather than tweak one cell, rewrite the notebook file end-to-end.\n\nMarkdown cells own narrative; query cells own a single Malloy query whose chart annotation tells the renderer how to display the result. Markdown supports H2 headings, lists, bold, and inline code. Keep narrative cells short, one idea per cell, so the rendered output reads as a story instead of a wall of text.\n\nIn a `.malloynb` file each cell is delimited by a `>>>markdown` or `>>>malloy` marker. A markdown cell looks like:\n\n```\n>>>markdown\n## Section heading\nNarrative text here.\n```\n\nA query cell looks like:\n\n```\n>>>malloy\n# bar_chart\nrun: source -> { group_by: dim; aggregate: measure }\n```\n\nEach Malloy cell must be a standalone query (for example `run: source -> { ... }`). The notebook's leading `>>>malloy` cell holds the `import` statement for the model file; individual query cells do not repeat it. If a query fails validation when executed, fix it and rerun.\n\nA well-structured report typically follows this pattern:\n\n```\n[Markdown] ## Overview: what question are we answering, what data is in scope (date range, entity count)\n[Malloy] KPI cell: headline numbers (e.g., # big_value, or # dashboard with nested # big_value cells)\n[Markdown] ## Trend: describe what we should look for over time\n[Malloy] Time-series cell (e.g., # line_chart on a date dimension)\n[Markdown] ## Breakdown: where the signal is\n[Malloy] Categorical cell (e.g., # bar_chart on a categorical dimension)\n[Markdown] ## Key takeaways: what the user should walk away with\n```\n\nUse this as a default; deviate when the analysis warrants. A grounded report names the time range and entity count up front so every number that follows has context.\n\n## Choosing chart types and annotations\n\nRead `skill:malloy-charts` before picking visualizations: it owns chart-type selection, properties, and the placement rules for chart annotations. `skill:malloy-queries` covers Malloy query patterns and the critical placement rules for chart-annotation tags.\n\nWhen in doubt:\n- KPIs / single numbers -> `# big_value`, often nested inside `# dashboard`.\n- Trend over time -> `# line_chart`, usually on the primary date dimension.\n- Category comparisons -> `# bar_chart`, ordered by the metric.\n- Tabular data with many columns -> a plain table cell with `# table.size=fill`.\n- Multiple coordinated charts -> `# dashboard` with `nest:` blocks.\n\nAnnotations go **before** `run:`, never inside curly braces:\n\n```malloy\n# bar_chart\nrun: source -> {\n group_by: category\n aggregate: revenue\n order_by: revenue desc\n limit: 10\n}\n```\n\nA `# dashboard` cell composes nested views, useful for KPIs alongside a trend in a single cell. Each `nest:` is a tile; any top-level `aggregate:` measures render as KPI cards. For a fixed grid, use `# dashboard { columns=N }` with `# colspan` on each tile (see `skill:malloy-charts`):\n\n```malloy\n# dashboard { columns=2 }\nrun: source -> {\n nest:\n # colspan=2\n # big_value\n kpis is {\n aggregate:\n # label=\"Revenue\"\n # currency\n total_revenue\n\n # label=\"Orders\"\n # number=auto\n order_count\n }\n nest:\n # line_chart\n trend is {\n group_by: order_date.month\n aggregate: total_revenue\n order_by: 1\n }\n}\n```\n\nKey rendering rules to keep in mind when shaping a cell:\n- FIRST `group_by` = x-axis, FIRST `aggregate` = y-axis.\n- Override field roles with `# x`, `# y`, `# series` on individual fields.\n- For multiple measure series, place `# y` above the `aggregate:` keyword.\n- One aggregate per chart view: use `# dashboard` with nested views for multiple charts.\n- Use `# table.size=fill` for standalone table queries.\n\n## Editing an existing report\n\nFor small targeted changes (fix one cell, insert one new cell), edit that cell in the `.malloynb` file rather than recreating the whole notebook. For structural rewrites (reordering many cells, changing the narrative arc), rewrite the notebook file.\n\n## IMPORTANT\n\nYou CANNOT see the rendered output of notebook cells. Do not claim to see charts, values, or patterns from report cells you haven't explicitly executed via `execute_query`. If you need to analyze results, run the query via `execute_query` first." + "body": "# Creating Reports\n\nAn ad-hoc report is a `.malloynb` notebook that combines markdown narrative with live Malloy query cells. There is no dedicated report tool: you author the notebook directly. Load `skill:malloy-notebooks` for the full `.malloynb` cell format and authoring rules; this skill covers when to build one and how to design good report content (cells, chart annotations, narrative structure).\n\n> **Tool names** are written bare here - `get_context`, `execute_query`, `search_malloy_docs`. The exact prefixed name depends on the host surface; match each against the tools you actually have.\n\n## Before building a report\n\n1. **Run each query first** via `execute_query` to verify it works and returns expected results.\n2. **Explain the results** to the user as you go: walk through the analysis step by step.\n3. **Then assemble the notebook** once the analysis is validated.\n\nDo NOT build the notebook in the same turn as `execute_query`. Explain first, then build.\n\n## Filters are inherited from the model, don't declare them in the report\n\nReports do not (and cannot) define their own filters. If the source declares `given:` parameters (or legacy `#(filter)` annotations), Publisher renders the controls, parses caller parameters, and applies them server-side automatically: the report inherits and displays them with no extra work. If the analysis needs a knob the source doesn't expose, the right move is to add a `given:` to the source itself, not to wedge a filter widget into the report. `#(filter)` is deprecated in favour of native Malloy `given:` parameters. Do not add new `#(filter)` annotations; the two exceptions are `required` and `implicit`, which `given:` cannot cover yet. See `skill:malloy-model` § Legacy: Parameterizable Filters. For curated notebooks with their own per-notebook filter UI on top of the model, see `skill:malloy-notebooks` instead.\n\n## What goes in the report\n\nDo NOT add an H1 heading in any cell (use H2 and below for sections); the notebook name serves as the title. To redo the structure rather than tweak one cell, rewrite the notebook file end-to-end.\n\nMarkdown cells own narrative; query cells own a single Malloy query whose chart annotation tells the renderer how to display the result. Markdown supports H2 headings, lists, bold, and inline code. Keep narrative cells short, one idea per cell, so the rendered output reads as a story instead of a wall of text.\n\nIn a `.malloynb` file each cell is delimited by a `>>>markdown` or `>>>malloy` marker. A markdown cell looks like:\n\n```\n>>>markdown\n## Section heading\nNarrative text here.\n```\n\nA query cell looks like:\n\n```\n>>>malloy\n# bar_chart\nrun: source -> { group_by: dim; aggregate: measure }\n```\n\nEach Malloy cell must be a standalone query (for example `run: source -> { ... }`). The notebook's leading `>>>malloy` cell holds the `import` statement for the model file; individual query cells do not repeat it. If a query fails validation when executed, fix it and rerun.\n\nA well-structured report typically follows this pattern:\n\n```\n[Markdown] ## Overview: what question are we answering, what data is in scope (date range, entity count)\n[Malloy] KPI cell: headline numbers (e.g., # big_value, or # dashboard with nested # big_value cells)\n[Markdown] ## Trend: describe what we should look for over time\n[Malloy] Time-series cell (e.g., # line_chart on a date dimension)\n[Markdown] ## Breakdown: where the signal is\n[Malloy] Categorical cell (e.g., # bar_chart on a categorical dimension)\n[Markdown] ## Key takeaways: what the user should walk away with\n```\n\nUse this as a default; deviate when the analysis warrants. A grounded report names the time range and entity count up front so every number that follows has context.\n\n## Choosing chart types and annotations\n\nRead `skill:malloy-charts` before picking visualizations: it owns chart-type selection, properties, and the placement rules for chart annotations. `skill:malloy-queries` covers Malloy query patterns and the critical placement rules for chart-annotation tags.\n\nWhen in doubt:\n- KPIs / single numbers -> `# big_value`, often nested inside `# dashboard`.\n- Trend over time -> `# line_chart`, usually on the primary date dimension.\n- Category comparisons -> `# bar_chart`, ordered by the metric.\n- Tabular data with many columns -> a plain table cell with `# table.size=fill`.\n- Multiple coordinated charts -> `# dashboard` with `nest:` blocks.\n\nAnnotations go **before** `run:`, never inside curly braces:\n\n```malloy\n# bar_chart\nrun: source -> {\n group_by: category\n aggregate: revenue\n order_by: revenue desc\n limit: 10\n}\n```\n\nA `# dashboard` cell composes nested views, useful for KPIs alongside a trend in a single cell. Each `nest:` is a tile; any top-level `aggregate:` measures render as KPI cards. For a fixed grid, use `# dashboard { columns=N }` with `# colspan` on each tile (see `skill:malloy-charts`):\n\n```malloy\n# dashboard { columns=2 }\nrun: source -> {\n nest:\n # colspan=2\n # big_value\n kpis is {\n aggregate:\n # label=\"Revenue\"\n # currency\n total_revenue\n\n # label=\"Orders\"\n # number=auto\n order_count\n }\n nest:\n # line_chart\n trend is {\n group_by: order_date.month\n aggregate: total_revenue\n order_by: 1\n }\n}\n```\n\nKey rendering rules to keep in mind when shaping a cell:\n- FIRST `group_by` = x-axis, FIRST `aggregate` = y-axis.\n- Override field roles with `# x`, `# y`, `# series` on individual fields.\n- For multiple measure series, place `# y` above the `aggregate:` keyword.\n- One aggregate per chart view: use `# dashboard` with nested views for multiple charts.\n- Use `# table.size=fill` for standalone table queries.\n\n## Editing an existing report\n\nFor small targeted changes (fix one cell, insert one new cell), edit that cell in the `.malloynb` file rather than recreating the whole notebook. For structural rewrites (reordering many cells, changing the narrative arc), rewrite the notebook file.\n\n## IMPORTANT\n\nYou CANNOT see the rendered output of notebook cells. Do not claim to see charts, values, or patterns from report cells you haven't explicitly executed via `execute_query`. If you need to analyze results, run the query via `execute_query` first." }, { "name": "malloy-analyze", - "description": "Explore data for insights and build views/dashboards/notebooks. Use when user asks to \"analyze this data\", \"find insights\", \"explore for patterns\", \"what's interesting\", \"what's driving X\", \"build a dashboard\", \"create views\", or any analysis task. For EDA exploration, start at Step 1. For building views on an existing model, jump to View Patterns.", - "body": "# Analysis with Malloy\n\nThis skill covers two workflows:\n- **EDA exploration** (Steps 1-6): iteratively query data, build hypotheses, validate findings\n- **View/dashboard building**: create views, dashboards, notebooks from an existing model\n\n> **Tool names** are written bare here - `get_context`, `execute_query`, `search_malloy_docs`. The exact prefixed name depends on the host surface; match each against the tools you actually have.\n\nTo formalize analysis into a polished semantic model, hand off to the modeling skill's \"Starting from Analysis\" workflow (`skill:malloy-model`).\n\n## Prerequisites\n\n- The Malloy MCP tools must be configured (`get_context`, `execute_query`, `search_malloy_docs`). If they are not available, **STOP** and ensure your host's MCP server is connected.\n- Call `search_malloy_docs` liberally: it has powerful analysis patterns (window functions, cohorts, percent-of-total, nested drill-downs).\n\n# EDA WORKFLOW\n\n```\nORIENT → PROFILE → HYPOTHESIZE → INVESTIGATE → VALIDATE → SYNTHESIZE\n (user) (user) (user)\n```\n\n## Adaptive Checkpoints\n\nThe 6-step structure is a framework, not a rigid script.\n\n| Situation | Adaptation |\n|-----------|------------|\n| **User has a clear hypothesis** (\"what's driving churn?\") | Skip HYPOTHESIZE, jump to INVESTIGATE on their question |\n| **Open-ended** (\"what's interesting?\") | Follow all steps. PROFILE and HYPOTHESIZE are essential |\n| **User wants you to just go** (\"explore and show me\") | Compress checkpoints, present findings at SYNTHESIZE |\n\n## Step 1: ORIENT: Understand the Data\n\n1. Ground yourself with `get_context`. It returns the package's sources, views, and fields (with their docs), so this is where you learn what data exists.\n2. Note the source names, the connection they sit on, and the key tables/fields they expose.\n3. Inspect the existing dimensions, measures, and views the model already defines, then query the data to confirm shape and values.\n4. Create a working analysis file (this grows throughout the session):\n ```malloy\n source: main_table is conn.table('schema.table') extend { primary_key: pk }\n ```\n\n**Output to user:** Brief summary of available data. Ask: *\"What questions are you most interested in? Or should I look for what's interesting?\"*\n\n## Step 2: PROFILE: Statistical Profiling\n\n**Directed analysis** (user has a question): Profile only columns relevant to their question.\n**Open-ended** (no question yet): Profile broadly, looking for surprises.\n\n### Key Profiling Queries\n\n**Column overview:** `run: source -> { index: * limit: 100 }`\n\n**Numeric distributions:**\n```malloy\nrun: source -> {\n aggregate: min_val is min(col), max_val is max(col), avg_val is avg(col), null_count is count() { where: col is null }\n}\n```\n\n**Categorical breakdown:** `run: source -> { group_by: col, aggregate: n is count(), order_by: n desc, limit: 20 }`\n\n**Time range:** Check earliest/latest dates, gaps, seasonality.\n\n**Duplicates:** `run: source -> { group_by: pk, aggregate: n is count(), having: n > 1, limit: 10 }`\n\n**Add useful profiling dimensions/measures to your analysis file as you go.** Build incrementally.\n\n## Step 3: HYPOTHESIZE: Form Questions\n\n**Skip presentation if user already has a clear question.** Use profiling to refine it and jump to INVESTIGATE.\n\n| Signal from profiling | Hypothesis type |\n|----------------------|-----------------|\n| Skewed distribution | Outlier analysis |\n| Time patterns | Trend/seasonality |\n| Category imbalance | Segment comparison |\n| Correlated columns | Driver analysis |\n| Unexpected NULLs | Data quality |\n\n**CHECKPOINT (open-ended only):** Present 3-5 hypotheses ranked by potential impact. Ask which to pursue.\n\n## Step 4: INVESTIGATE: Deep-Dive\n\n### Outlier Detection\nSearch `search_malloy_docs(\"window functions\")` for ranking and percentile patterns.\n\n### Trend Analysis\n```malloy\n# line_chart\nview: trend is { group_by: period is date_col.month, aggregate: key_metric, order_by: period }\n```\n\n### Segment Comparison\n```malloy\nview: segment_comparison is {\n group_by: segment_dim\n aggregate: row_count, key_metric\n nest:\n # line_chart\n trend is { group_by: period is date_col.month, aggregate: key_metric, order_by: period }\n}\n```\n\n### Driver Analysis\n```malloy\nrun: source -> {\n group_by: candidate_driver\n aggregate: row_count, avg_metric is avg(metric_col),\n high_rate is count() { where: metric_col > threshold } / nullif(count(), 0)\n order_by: high_rate desc\n}\n```\n\n### Multi-Source Comparison (Source vs Group)\n\nCompare each source to its group average using query-as-source. Name the query directly. `from(query_name)` was removed from the language and no longer parses (`unexpected 'from'`):\n\n```malloy\nquery: team_stats is source -> { group_by: team, season, aggregate: team_avg is avg(points) }\nquery: driver_stats is source -> { group_by: driver, team, season, aggregate: driver_points is sum(points) }\n\nsource: driver_vs_team is driver_stats extend {\n join_one: ts is team_stats on team = ts.team and season = ts.season\n dimension: advantage is driver_points - ts.team_avg\n}\n```\n\n### Nested Analysis (Malloy's Superpower)\n\nUse `nest:` for multi-level drill-downs in a single query:\n```malloy\n# dashboard { columns=2 }\nview: deep_dive is {\n nest: # big_value\n kpis is { aggregate: # label=\"Total\" total_metric, # label=\"Count\" row_count }\n nest: # bar_chart\n by_dim is { group_by: dim, aggregate: metric, order_by: metric desc, limit: 10 }\n nest: # line_chart\n over_time is { group_by: period is date.month, aggregate: metric, order_by: period }\n}\n```\n\n`columns=N` is what gives a dashboard a layout. Bare `# dashboard` is flex mode, where tiles just flow and wrap and `# colspan` is silently ignored, so if your tiles look wrong, that is usually why. See `skill:malloy-charts` for the full tag set.\n\n### Build As You Go\n\nEvery useful query should leave an artifact in your `.malloy` file. New dimension? Add it. New measure? Add it. Interesting view? Save it. This file becomes the input for formalizing into a model if the user wants one.\n\n## Step 5: VALIDATE: Triangulate\n\nFor each finding, validate with at least ONE of:\n- Cross-check with another metric (revenue spiking? do order counts also?)\n- Check the denominator (high rate from tiny sample?)\n- Examine time consistency (pattern or one-time event?)\n- Look at raw data (`select: * where: condition limit: 20`)\n- Check for data artifacts (NULLs, duplicates, encoding)\n\n**CHECKPOINT:** Present each finding with: the insight, the evidence, confidence level, and assumptions made.\n\n## Step 6: SYNTHESIZE: Compelling Summary\n\nBuild a dashboard view that tells the story:\n```malloy\n# dashboard { columns=2 }\nview: analysis_summary is {\n nest: # big_value\n headlines is { aggregate: ... }\n nest: # line_chart\n trend is { ... }\n nest: # bar_chart\n breakdown is { ... }\n}\n```\n\nDocument insights as view descriptions: `#(doc) Top 10% of customers drive 62% of revenue.`\n\nPresent to user: top 3-5 insights, supporting views, open questions, and recommended next steps.\n\n**Ready to formalize?** Hand off to the modeling skill's \"Starting from Analysis\" workflow (`skill:malloy-model`).\n\n# VIEW PATTERNS\n\nFor building views on an existing model (base + joined source files already exist).\n\n## Starter Views (2-3 max initially)\n\n1. **`summary`**: KPI cards (`# big_value`)\n2. **`by_time`**: Time trend (`# line_chart`)\n3. **`by_category`**: Category breakdown (`# bar_chart`)\n4. **`dashboard`**: Nested view combining the above (`# dashboard`)\n\n**DRY rule:** Do NOT define measures/dimensions inline in views. Reference existing ones from base source files.\n\n## View Annotations\n\n| Annotation | Use For | Notes |\n|-----------|---------|-------|\n| `# big_value` | KPI summary | 2-5 metrics with `# label` on each |\n| `# transpose` | Summary with group_by | Swaps rows/columns |\n| `# dashboard` | Multi-visualization | Tiles nested views |\n| `# line_chart` | Time trend | ONE aggregate only |\n| `# bar_chart` | Category breakdown | ONE aggregate only |\n| (none) | Detailed table | Supports multiple aggregates |\n\n**Rules:**\n- One tag per line, never combine annotations on one line\n- One aggregate per chart view, charts render only the first\n- No fixed scale on measures: use `# currency` (no scale); fixed scale only in views after confirming ranges\n- Place chart annotation on the nested view definition, not on `nest:` itself\n\nFor complete chart reference including scatter_chart, shape_map, sparklines, and all configuration options, see `skill:malloy-charts` or call `search_malloy_docs(\"rendering\")`.\n\n## Field-Level Formatting\n\n| Tag | Use For |\n|-----|---------|\n| `# currency` | Monetary values |\n| `# percent` | Rates/percentages |\n| `# number=auto` | Large counts (K/M/B) |\n| `# number=id` | Non-quantity numbers (years, IDs) |\n| `# label=\"Name\"` | Custom display name |\n| `# hidden` | Internal/helper fields |\n| `# duration=seconds` | Time durations |\n\n# NOTEBOOKS (.malloynb)\n\nCells delimited by `>>>markdown` or `>>>malloy`. **Never use `>>>malloysql`.**\n\n```\n>>>markdown\n# Sales Analysis\n\n>>>malloy\nimport \"order_analysis.malloy\"\n\n>>>malloy\nrun: order_analysis -> summary\n```\n\n**Compile errors in `.malloynb` are NOT shown in the linter**: only visible on cell execution.\n\nA notebook is also the home for a polished, narrated report: alternate `>>>markdown` cells (the story) with `>>>malloy` cells (the views), and let the malloy cells carry the chart tags. For the full cell-shape and report-authoring conventions, see `skill:malloy-notebooks`.\n\n### Interactive Filters\n\n**Notebooks do NOT define filters themselves.** When you import a model, the model's `#(filter)` annotations on the source are **inherited and displayed automatically**: the publisher renders the filter widgets, parses caller parameters, and injects `where:` clauses server-side. You don't redeclare them in the consumer. If the analysis needs a knob the model doesn't expose, the right move is to add a `#(filter)` to the source itself (see `skill:malloy-model` § Parameterizable Filters with `#(filter)`), not to wedge filtering into the consumer.\n\nThe notebook-level `##(filters)` annotation and the dimension-level `#(filter) {\"type\": \"...\"}` JSON-blob form are **unsupported legacy syntax**, don't use them. The only supported form is `#(filter) name=... dimension=... type=...` declared above the source.\n\n### View Refinement\n\nUse `+` to modify existing views: `run: source -> my_view + { limit: 15, where: status = 'active' }`\n\n## Done\n\nStep complete. Output: analysis `.malloy` file with views, insights, and reusable building blocks. For chart/renderer details, see `skill:malloy-gotchas-rendering` or call `search_malloy_docs`. To formalize into a model, hand off to the modeling skill (`skill:malloy-model`).\n\nPublishing is out of scope for now: open-source Publisher serves the model from disk, and self-hosters publish via git plus their host's publish path." + "description": "Open-ended exploration with no specific question to answer, and building views/dashboards/notebooks on an existing model. Use when the user asks \"what's interesting?\", \"explore this data\", \"find insights\", \"look for patterns\", \"build a dashboard\", or \"create views\". Not for a specific data question - answering one is `malloy-analysis`, and writing down what the answer assumed is `malloy-model-as-you-go`. For exploration start at Step 1; for views on an existing model jump to View Patterns.", + "body": "# Analysis with Malloy\n\nThis skill covers two workflows:\n- **EDA exploration** (Steps 1-6): iteratively query data, build hypotheses, validate findings\n- **View/dashboard building**: create views, dashboards, notebooks from an existing model\n\n> **Tool names** are written bare here - `get_context`, `execute_query`, `search_malloy_docs`. The exact prefixed name depends on the host surface; match each against the tools you actually have.\n\nTo formalize analysis into a polished semantic model, hand off to the modeling skill's \"Starting from Analysis\" workflow (`skill:malloy-model`).\n\n## Prerequisites\n\n- The Malloy MCP tools must be configured (`get_context`, `execute_query`, `search_malloy_docs`). If they are not available, **STOP** and ensure your host's MCP server is connected.\n- Call `search_malloy_docs` liberally: it has powerful analysis patterns (window functions, cohorts, percent-of-total, nested drill-downs).\n\n# EDA WORKFLOW\n\n```\nORIENT → PROFILE → HYPOTHESIZE → INVESTIGATE → VALIDATE → SYNTHESIZE\n (user) (user) (user)\n```\n\n## Adaptive Checkpoints\n\nThe 6-step structure is a framework, not a rigid script.\n\n| Situation | Adaptation |\n|-----------|------------|\n| **User has a clear hypothesis** (\"what's driving churn?\") | Skip HYPOTHESIZE, jump to INVESTIGATE on their question |\n| **Open-ended** (\"what's interesting?\") | Follow all steps. PROFILE and HYPOTHESIZE are essential |\n| **User wants you to just go** (\"explore and show me\") | Compress checkpoints, present findings at SYNTHESIZE |\n\n## Step 1: ORIENT: Understand the Data\n\n1. Ground yourself with `get_context`. It returns the package's sources, views, and fields (with their docs), so this is where you learn what data exists.\n2. Note the source names, the connection they sit on, and the key tables/fields they expose.\n3. Inspect the existing dimensions, measures, and views the model already defines, then query the data to confirm shape and values.\n4. Create a working analysis file (this grows throughout the session):\n ```malloy\n source: main_table is conn.table('schema.table') extend { primary_key: pk }\n ```\n\n**Output to user:** Brief summary of available data. Ask: *\"What questions are you most interested in? Or should I look for what's interesting?\"*\n\n## Step 2: PROFILE: Statistical Profiling\n\n**Directed analysis** (user has a question): Profile only columns relevant to their question.\n**Open-ended** (no question yet): Profile broadly, looking for surprises.\n\n### Key Profiling Queries\n\n**Column overview:** `run: source -> { index: * limit: 100 }`\n\n**Numeric distributions:**\n```malloy\nrun: source -> {\n aggregate: min_val is min(col), max_val is max(col), avg_val is avg(col), null_count is count() { where: col is null }\n}\n```\n\n**Categorical breakdown:** `run: source -> { group_by: col, aggregate: n is count(), order_by: n desc, limit: 20 }`\n\n**Time range:** Check earliest/latest dates, gaps, seasonality.\n\n**Duplicates:** `run: source -> { group_by: pk, aggregate: n is count(), having: n > 1, limit: 10 }`\n\n**Add useful profiling dimensions/measures to your analysis file as you go.** Build incrementally.\n\n## Step 3: HYPOTHESIZE: Form Questions\n\n**Skip presentation if user already has a clear question.** Use profiling to refine it and jump to INVESTIGATE.\n\n| Signal from profiling | Hypothesis type |\n|----------------------|-----------------|\n| Skewed distribution | Outlier analysis |\n| Time patterns | Trend/seasonality |\n| Category imbalance | Segment comparison |\n| Correlated columns | Driver analysis |\n| Unexpected NULLs | Data quality |\n\n**CHECKPOINT (open-ended only):** Present 3-5 hypotheses ranked by potential impact. Ask which to pursue.\n\n## Step 4: INVESTIGATE: Deep-Dive\n\n### Outlier Detection\nSearch `search_malloy_docs(\"window functions\")` for ranking and percentile patterns.\n\n### Trend Analysis\n```malloy\n# line_chart\nview: trend is { group_by: period is date_col.month, aggregate: key_metric, order_by: period }\n```\n\n### Segment Comparison\n```malloy\nview: segment_comparison is {\n group_by: segment_dim\n aggregate: row_count, key_metric\n nest:\n # line_chart\n trend is { group_by: period is date_col.month, aggregate: key_metric, order_by: period }\n}\n```\n\n### Driver Analysis\n```malloy\nrun: source -> {\n group_by: candidate_driver\n aggregate: row_count, avg_metric is avg(metric_col),\n high_rate is count() { where: metric_col > threshold } / nullif(count(), 0)\n order_by: high_rate desc\n}\n```\n\n### Multi-Source Comparison (Source vs Group)\n\nCompare each source to its group average using query-as-source. Name the query directly. `from(query_name)` was removed from the language and no longer parses (`unexpected 'from'`):\n\n```malloy\nquery: team_stats is source -> { group_by: team, season, aggregate: team_avg is avg(points) }\nquery: driver_stats is source -> { group_by: driver, team, season, aggregate: driver_points is sum(points) }\n\nsource: driver_vs_team is driver_stats extend {\n join_one: ts is team_stats on team = ts.team and season = ts.season\n dimension: advantage is driver_points - ts.team_avg\n}\n```\n\n### Nested Analysis (Malloy's Superpower)\n\nUse `nest:` for multi-level drill-downs in a single query:\n```malloy\n# dashboard { columns=2 }\nview: deep_dive is {\n nest: # big_value\n kpis is { aggregate: # label=\"Total\" total_metric, # label=\"Count\" row_count }\n nest: # bar_chart\n by_dim is { group_by: dim, aggregate: metric, order_by: metric desc, limit: 10 }\n nest: # line_chart\n over_time is { group_by: period is date.month, aggregate: metric, order_by: period }\n}\n```\n\n`columns=N` is what gives a dashboard a layout. Bare `# dashboard` is flex mode, where tiles just flow and wrap and `# colspan` is silently ignored, so if your tiles look wrong, that is usually why. See `skill:malloy-charts` for the full tag set.\n\n### Build As You Go\n\nEvery useful query should leave an artifact in your `.malloy` file. New dimension? Add it. New measure? Add it. Interesting view? Save it. This file becomes the input for formalizing into a model if the user wants one.\n\n## Step 5: VALIDATE: Triangulate\n\nFor each finding, validate with at least ONE of:\n- Cross-check with another metric (revenue spiking? do order counts also?)\n- Check the denominator (high rate from tiny sample?)\n- Examine time consistency (pattern or one-time event?)\n- Look at raw data (`select: * where: condition limit: 20`)\n- Check for data artifacts (NULLs, duplicates, encoding)\n\n**CHECKPOINT:** Present each finding with: the insight, the evidence, confidence level, and assumptions made.\n\n## Step 6: SYNTHESIZE: Compelling Summary\n\nBuild a dashboard view that tells the story:\n```malloy\n# dashboard { columns=2 }\nview: analysis_summary is {\n nest: # big_value\n headlines is { aggregate: ... }\n nest: # line_chart\n trend is { ... }\n nest: # bar_chart\n breakdown is { ... }\n}\n```\n\nDocument insights as view descriptions: `#(doc) Top 10% of customers drive 62% of revenue.`\n\nPresent to user: top 3-5 insights, supporting views, open questions, and recommended next steps.\n\n**Ready to formalize?** Hand off to the modeling skill's \"Starting from Analysis\" workflow (`skill:malloy-model`).\n\n# VIEW PATTERNS\n\nFor building views on an existing model (base + joined source files already exist).\n\n## Starter Views (2-3 max initially)\n\n1. **`summary`**: KPI cards (`# big_value`)\n2. **`by_time`**: Time trend (`# line_chart`)\n3. **`by_category`**: Category breakdown (`# bar_chart`)\n4. **`dashboard`**: Nested view combining the above (`# dashboard`)\n\n**DRY rule:** Do NOT define measures/dimensions inline in views. Reference existing ones from base source files.\n\n## View Annotations\n\n| Annotation | Use For | Notes |\n|-----------|---------|-------|\n| `# big_value` | KPI summary | 2-5 metrics with `# label` on each |\n| `# transpose` | Summary with group_by | Swaps rows/columns |\n| `# dashboard` | Multi-visualization | Tiles nested views |\n| `# line_chart` | Time trend | ONE aggregate only |\n| `# bar_chart` | Category breakdown | ONE aggregate only |\n| (none) | Detailed table | Supports multiple aggregates |\n\n**Rules:**\n- One tag per line, never combine annotations on one line\n- One aggregate per chart view, charts render only the first\n- No fixed scale on measures: use `# currency` (no scale); fixed scale only in views after confirming ranges\n- Place chart annotation on the nested view definition, not on `nest:` itself\n\nFor complete chart reference including scatter_chart, shape_map, sparklines, and all configuration options, see `skill:malloy-charts` or call `search_malloy_docs(\"rendering\")`.\n\n## Field-Level Formatting\n\n| Tag | Use For |\n|-----|---------|\n| `# currency` | Monetary values |\n| `# percent` | Rates/percentages |\n| `# number=auto` | Large counts (K/M/B) |\n| `# number=id` | Non-quantity numbers (years, IDs) |\n| `# label=\"Name\"` | Custom display name |\n| `# hidden` | Internal/helper fields |\n| `# duration=seconds` | Time durations |\n\n# NOTEBOOKS (.malloynb)\n\nCells delimited by `>>>markdown` or `>>>malloy`. **Never use `>>>malloysql`.**\n\n```\n>>>markdown\n# Sales Analysis\n\n>>>malloy\nimport \"order_analysis.malloy\"\n\n>>>malloy\nrun: order_analysis -> summary\n```\n\n**Compile errors in `.malloynb` are NOT shown in the linter**: only visible on cell execution.\n\nA notebook is also the home for a polished, narrated report: alternate `>>>markdown` cells (the story) with `>>>malloy` cells (the views), and let the malloy cells carry the chart tags. For the full cell-shape and report-authoring conventions, see `skill:malloy-notebooks`.\n\n### Interactive Filters\n\n**Notebooks do NOT define filters themselves.** When you import a model, the model's runtime parameters are **inherited and displayed automatically**: the publisher renders the controls, parses caller parameters, and applies them server-side. You don't redeclare them in the consumer. If the analysis needs a knob the model doesn't expose, the right move is to add it to the source itself, not to wedge filtering into the consumer.\n\n**Declare that knob as a `given:`.** `#(filter)` is deprecated in favour of native Malloy `given:` parameters. Do not add new `#(filter)` annotations; the two exceptions are `required` and `implicit`, which `given:` cannot cover yet. See `skill:malloy-model` § Legacy: Parameterizable Filters.\n\nThe notebook-level `##(filters)` annotation and the dimension-level `#(filter) {\"type\": \"...\"}` JSON-blob form are **unsupported legacy syntax**, don't use them. The only supported form is `#(filter) name=... dimension=... type=...` declared above the source.\n\n### View Refinement\n\nUse `+` to modify existing views: `run: source -> my_view + { limit: 15, where: status = 'active' }`\n\n## Done\n\nStep complete. Output: analysis `.malloy` file with views, insights, and reusable building blocks. For chart/renderer details, see `skill:malloy-gotchas-rendering` or call `search_malloy_docs`. To formalize into a model, hand off to the modeling skill (`skill:malloy-model`).\n\nPublishing is out of scope for now: open-source Publisher serves the model from disk, and self-hosters publish via git plus their host's publish path." }, { "name": "malloy-charts", @@ -43,7 +43,7 @@ { "name": "malloy-define", "description": "Propose a source plan and field definitions for a Malloy semantic model. Covers picking which sources to model and at what grain, then proposing the specific renames, dimensions, and measures per source, every proposal backed by querying the data.", - "body": "# Propose sources and definitions\n\nThis skill covers two consecutive activities when building or extending a Malloy semantic model:\n\n> **Tool names** are written bare here - `get_context`, `execute_query`, `search_malloy_docs`. The exact prefixed name depends on the host surface; match each against the tools you actually have.\n\n- **Propose sources**: the architectural blueprint (which sources, what grain).\n- **Propose definitions**: the specific fields per base source (renames, dimensions, measures).\n\nBoth happen in conversation. Propose, let the user confirm or adjust, then carry the confirmed plan forward into the actual `.malloy` model. There is no separate plan-file store: keep the source plan and field proposals in the conversation, and write the model itself when the user has confirmed. See your modeling workflow for the broader picture.\n\nRead the existing model first so you propose against what is really there. Use `get_context` with a plain-English description to inspect the current sources and fields and find the most relevant existing sources. Confirm the scope (which tables are in play) before proposing the source plan.\n\n## Propose a source plan\n\n**Goal:** Propose the full source architecture for the tables in scope.\n\n### Base sources\n\nOne base source per table in scope. For each, specify:\n\n| Source | Table | Grain | Primary Key | Role |\n|--------|-------|-------|-------------|------|\n| orders | sales.orders | one row per order | order_id | Fact, transactions |\n| customers | sales.customers | one row per customer | customer_id | Dimension, who |\n| products | sales.products | one row per product | product_id | Dimension, what |\n\n### Computed sources\n\nComputed sources are created from queries, not physical tables. Propose them when:\n\n1. **Grain mismatch**: the analytical scope requires a grain that no physical table provides (e.g., customer-level metrics from an order-grain table).\n2. **Repeated aggregation patterns**: the same group-by plus aggregate pattern would be used in multiple places.\n3. **Cross-entity aggregations**: inspecting the model and querying the data shows that an aggregate rolled up to a different entity would be reused.\n\nFor each computed source, explain:\n\n| Source | Source Query | Grain | Rationale |\n|--------|-------------|-------|-----------|\n| user_order_facts | orders grouped by customer_id | one row per customer | Need customer-level order metrics (LTV, order count, recency) for customer health analysis |\n\n### Dependencies\n\nShow which sources depend on which:\n\n```\ncustomers (physical) ← user_order_facts (derived, sources from orders)\norders (physical) → user_order_facts (derived)\nproducts (physical): independent\n```\n\n### Deferred sources\n\nList sources considered but not included, with reasoning:\n\n- **order_items**: bridge table, defer until line-item analysis is needed.\n- **monthly_product_facts**: derived, defer until product trend analysis is requested.\n\n### User interaction\n\nThe user will:\n- **Confirm** the source plan as-is.\n- **Add** missing sources (physical or derived).\n- **Remove** unnecessary sources.\n- **Validate** grain assignments.\n- **Defer** sources to later iterations.\n\nOnce the source plan is confirmed, carry it forward into the definitions step below. Keep the confirmed map in the conversation rather than persisting it to a separate file.\n\n## Propose definitions\n\n**Goal:** Propose specific fields per base source with data evidence, working from the confirmed source plan.\n\n### For each base source\n\nPresent a table of proposed fields.\n\n**Renames (schema cleanup):**\n\n| Raw Column | Proposed Name | Reason |\n|-----------|---------------|--------|\n| `Order Date` | order_date | Whitespace in column name |\n| `Type` | order_type | Reserved word |\n| `number` | item_number | Reserved word |\n\n**Dimensions:**\n\n| Field | Logic | Data Evidence | Priority |\n|-------|-------|---------------|----------|\n| order_status | status column | 5 distinct values: pending, processing, shipped, delivered, cancelled | must-have |\n| order_month | submitted_at.month | Time trending | must-have |\n| order_size | total buckets (data-driven) | Distribution: min $5, p25 $35, median $85, p75 $150, p95 $450, max $2,400. Proposed breaks at p25/p75: <$35, $35-$150, >$150 | nice-to-have |\n| is_returned | returned_at is not null | 8% of orders have non-null returned_at | nice-to-have |\n\n**Data-driven tiers:** For bucketed dimensions like `order_size`, always derive boundaries from the actual data distribution (percentiles, natural breaks, clustering). Query `min`, `max`, `p25`, `p50`, `p75`, `p95` and propose boundaries based on the distribution. Show the evidence so the user can confirm or adjust. Never use arbitrary hardcoded thresholds unless the user explicitly provides them.\n\n**Measures:**\n\n| Field | Logic | Data Evidence | Priority |\n|-------|-------|---------------|----------|\n| order_count | count() | Basic metric | must-have |\n| revenue | sum(total) | Total column includes tax. Range: $5 - $2,400 | must-have |\n| avg_order_value | revenue / nullif(order_count, 0) | Derived from above | must-have |\n| return_rate | returned_count / nullif(order_count, 0) | 8% overall return rate | nice-to-have |\n\n### For each computed source\n\nShow the source query and additional fields.\n\n**`user_order_facts`**, derived from `orders` grouped by `customer_id`:\n\n| Aggregated Field | Logic |\n|-----------------|-------|\n| total_orders | count() |\n| total_revenue | sum(total_price) |\n| first_order_date | min(submitted_at) |\n| last_order_date | max(submitted_at) |\n\n**Additional dimensions on top:**\n\n| Field | Logic | Evidence |\n|-------|-------|----------|\n| days_since_last_order | days(last_order_date::timestamp to now) | Recency metric (cast: a date column will not measure against a timestamp) |\n| is_repeat_buyer | total_orders > 1 | 62% of customers are repeat |\n| buyer_frequency | total_orders buckets | Distribution: 1 (38%), 2-4 (35%), 5-19 (22%), 20+ (5%) |\n\n### Business logic questions\n\nFlag decisions the agent can't make from data alone. Be specific and data-grounded:\n\n> **Q1:** Your `orders` table has both `created_at` and `submitted_at`. 87% of rows have them within 1 minute, but 13% differ by 1-3 days. Which should be the canonical order date?\n>\n> **Q2:** I'm proposing `order_size` tiers based on the data distribution: small (<$35, below p25), medium ($35-$150, p25-p75), large (>$150, above p75). Do these data-driven breaks work for you, or do you have specific business thresholds?\n>\n> **Q3:** The `status` column has 5 values. Should \"cancelled\" orders be excluded from revenue calculations, or included with a separate measure?\n\n### Priority ranking\n\nGroup proposals into:\n- **Must-have**: core metrics that every analyst needs (counts, sums, primary dimensions).\n- **Nice-to-have**: useful but not critical (bucketed dimensions, rates).\n- **Value-add**: new insights the data supports but may not be asked for yet (computed sources, complex measures).\n\n### User interaction\n\nThe user will:\n- **Confirm** business logic decisions.\n- **Adjust** thresholds and bucket boundaries.\n- **Add** missing fields.\n- **Remove** fields they don't need.\n- **Change** priorities.\n\nOnce the definitions are confirmed, write them into the `.malloy` model (see your modeling workflow). Use `#(doc)` annotations to document sources and fields, and `#(filter)` annotations to declare server-side filterable dimensions where appropriate. Keep the confirmed definitions in the conversation; there is no separate plan-file store.\n\n## Data-driven proposals\n\n**Every recommendation must be backed by a query result.** Do not propose based on column names or schema structure alone. Always run `execute_query` to check the actual data before presenting. To learn what sources and fields exist, ground yourself with `get_context`: it returns the model's sources, views, and fields, so there is no separate schema-search step.\n\n| Proposal Type | What to query first |\n|--------------|---------------------|\n| Dimension (bucketed) | Distribution: min, p25, median, p75, p95, max. Propose boundaries from natural breaks, not arbitrary values. |\n| Dimension (categorical) | Distinct values and frequencies. Show the actual categories and their counts. |\n| Measure (sum/avg) | Sample values: min, max, avg. Verify the column contains what you think (e.g., is `total` gross or net?). |\n| Measure (rate/ratio) | Query both numerator and denominator. Verify they make sense together. |\n| Denormalized field vs join | Compare the pre-computed column against the joined aggregate. Report match rate. Recommend whichever is more reliable. |\n| Computed source | Run the proposed group-by plus aggregation. Verify the grain collapses as expected and the result is useful. |\n| Date field selection | Query all candidate date columns. Show % of rows where they differ and by how much. |\n| Column rename | Verify the column has data worth exposing (not 100% NULL). |\n\n**Example, denormalized vs joined:**\n\n> \"Your `customers` table has an `order_count` column. I compared it against `count()` from the `orders` table:\n> - 94% of customers match exactly\n> - 6% have stale counts (the denormalized value is lower than the actual count)\n> - The max discrepancy is 12 orders\n>\n> I'd recommend using the joined count from `orders` rather than the denormalized `order_count`. Want to keep the denormalized column as internal, or drop it?\"\n\n## Tips\n\n- **Show data, not assumptions:** every proposed dimension or measure should have evidence (distinct values, distributions, ranges).\n- **Use `execute_query`** to verify any data questions before presenting to the user.\n- **Don't over-propose:** 5-8 dimensions and 4-6 measures per base source is usually enough to start.\n- **Rank everything:** users appreciate knowing what's essential vs. optional.\n- **Business logic questions must be specific.** \"What date should I use?\" is bad. \"Your table has `created_at` and `submitted_at` that differ by 1-3 days in 13% of rows, which is canonical?\" is good.\n\n## Output\n\nA confirmed source architecture and a confirmed set of field definitions (renames, dimensions, measures, business decisions), held in the conversation and ready to write into the `.malloy` model via your modeling workflow." + "body": "# Propose sources and definitions\n\nThis skill covers two consecutive activities when building or extending a Malloy semantic model:\n\n> **Tool names** are written bare here - `get_context`, `execute_query`, `search_malloy_docs`. The exact prefixed name depends on the host surface; match each against the tools you actually have.\n\n- **Propose sources**: the architectural blueprint (which sources, what grain).\n- **Propose definitions**: the specific fields per base source (renames, dimensions, measures).\n\nBoth happen in conversation. Propose, let the user confirm or adjust, then carry the confirmed plan forward into the actual `.malloy` model. There is no separate plan-file store: keep the source plan and field proposals in the conversation, and write the model itself when the user has confirmed. See your modeling workflow for the broader picture.\n\nRead the existing model first so you propose against what is really there. Use `get_context` with a plain-English description to inspect the current sources and fields and find the most relevant existing sources. Confirm the scope (which tables are in play) before proposing the source plan.\n\n## Propose a source plan\n\n**Goal:** Propose the full source architecture for the tables in scope.\n\n### Base sources\n\nOne base source per table in scope. For each, specify:\n\n| Source | Table | Grain | Primary Key | Role |\n|--------|-------|-------|-------------|------|\n| orders | sales.orders | one row per order | order_id | Fact, transactions |\n| customers | sales.customers | one row per customer | customer_id | Dimension, who |\n| products | sales.products | one row per product | product_id | Dimension, what |\n\n### Computed sources\n\nComputed sources are created from queries, not physical tables. Propose them when:\n\n1. **Grain mismatch**: the analytical scope requires a grain that no physical table provides (e.g., customer-level metrics from an order-grain table).\n2. **Repeated aggregation patterns**: the same group-by plus aggregate pattern would be used in multiple places.\n3. **Cross-entity aggregations**: inspecting the model and querying the data shows that an aggregate rolled up to a different entity would be reused.\n\nFor each computed source, explain:\n\n| Source | Source Query | Grain | Rationale |\n|--------|-------------|-------|-----------|\n| user_order_facts | orders grouped by customer_id | one row per customer | Need customer-level order metrics (LTV, order count, recency) for customer health analysis |\n\n### Dependencies\n\nShow which sources depend on which:\n\n```\ncustomers (physical) ← user_order_facts (derived, sources from orders)\norders (physical) → user_order_facts (derived)\nproducts (physical): independent\n```\n\n### Deferred sources\n\nList sources considered but not included, with reasoning:\n\n- **order_items**: bridge table, defer until line-item analysis is needed.\n- **monthly_product_facts**: derived, defer until product trend analysis is requested.\n\n### User interaction\n\nThe user will:\n- **Confirm** the source plan as-is.\n- **Add** missing sources (physical or derived).\n- **Remove** unnecessary sources.\n- **Validate** grain assignments.\n- **Defer** sources to later iterations.\n\nOnce the source plan is confirmed, carry it forward into the definitions step below. Keep the confirmed map in the conversation rather than persisting it to a separate file.\n\n## Propose definitions\n\n**Goal:** Propose specific fields per base source with data evidence, working from the confirmed source plan.\n\n### For each base source\n\nPresent a table of proposed fields.\n\n**Renames (schema cleanup):**\n\n| Raw Column | Proposed Name | Reason |\n|-----------|---------------|--------|\n| `Order Date` | order_date | Whitespace in column name |\n| `Type` | order_type | Reserved word |\n| `number` | item_number | Reserved word |\n\n**Dimensions:**\n\n| Field | Logic | Data Evidence | Priority |\n|-------|-------|---------------|----------|\n| order_status | status column | 5 distinct values: pending, processing, shipped, delivered, cancelled | must-have |\n| order_month | submitted_at.month | Time trending | must-have |\n| order_size | total buckets (data-driven) | Distribution: min $5, p25 $35, median $85, p75 $150, p95 $450, max $2,400. Proposed breaks at p25/p75: <$35, $35-$150, >$150 | nice-to-have |\n| is_returned | returned_at is not null | 8% of orders have non-null returned_at | nice-to-have |\n\n**Data-driven tiers:** For bucketed dimensions like `order_size`, always derive boundaries from the actual data distribution (percentiles, natural breaks, clustering). Query `min`, `max`, `p25`, `p50`, `p75`, `p95` and propose boundaries based on the distribution. Show the evidence so the user can confirm or adjust. Never use arbitrary hardcoded thresholds unless the user explicitly provides them.\n\n**Measures:**\n\n| Field | Logic | Data Evidence | Priority |\n|-------|-------|---------------|----------|\n| order_count | count() | Basic metric | must-have |\n| revenue | sum(total) | Total column includes tax. Range: $5 - $2,400 | must-have |\n| avg_order_value | revenue / nullif(order_count, 0) | Derived from above | must-have |\n| return_rate | returned_count / nullif(order_count, 0) | 8% overall return rate | nice-to-have |\n\n### For each computed source\n\nShow the source query and additional fields.\n\n**`user_order_facts`**, derived from `orders` grouped by `customer_id`:\n\n| Aggregated Field | Logic |\n|-----------------|-------|\n| total_orders | count() |\n| total_revenue | sum(total_price) |\n| first_order_date | min(submitted_at) |\n| last_order_date | max(submitted_at) |\n\n**Additional dimensions on top:**\n\n| Field | Logic | Evidence |\n|-------|-------|----------|\n| days_since_last_order | days(last_order_date::timestamp to now) | Recency metric (cast: a date column will not measure against a timestamp) |\n| is_repeat_buyer | total_orders > 1 | 62% of customers are repeat |\n| buyer_frequency | total_orders buckets | Distribution: 1 (38%), 2-4 (35%), 5-19 (22%), 20+ (5%) |\n\n### Business logic questions\n\nFlag decisions the agent can't make from data alone. Be specific and data-grounded:\n\n> **Q1:** Your `orders` table has both `created_at` and `submitted_at`. 87% of rows have them within 1 minute, but 13% differ by 1-3 days. Which should be the canonical order date?\n>\n> **Q2:** I'm proposing `order_size` tiers based on the data distribution: small (<$35, below p25), medium ($35-$150, p25-p75), large (>$150, above p75). Do these data-driven breaks work for you, or do you have specific business thresholds?\n>\n> **Q3:** The `status` column has 5 values. Should \"cancelled\" orders be excluded from revenue calculations, or included with a separate measure?\n\n### Priority ranking\n\nGroup proposals into:\n- **Must-have**: core metrics that every analyst needs (counts, sums, primary dimensions).\n- **Nice-to-have**: useful but not critical (bucketed dimensions, rates).\n- **Value-add**: new insights the data supports but may not be asked for yet (computed sources, complex measures).\n\n### User interaction\n\nThe user will:\n- **Confirm** business logic decisions.\n- **Adjust** thresholds and bucket boundaries.\n- **Add** missing fields.\n- **Remove** fields they don't need.\n- **Change** priorities.\n\nOnce the definitions are confirmed, write them into the `.malloy` model (see your modeling workflow). Use `#(doc)` annotations to document sources and fields, and `given:` parameters to declare runtime-filterable dimensions where appropriate. `#(filter)` is deprecated in favour of native Malloy `given:` parameters. Do not add new `#(filter)` annotations; the two exceptions are `required` and `implicit`, which `given:` cannot cover yet. See `skill:malloy-model` § Legacy: Parameterizable Filters. Keep the confirmed definitions in the conversation; there is no separate plan-file store.\n\n## Data-driven proposals\n\n**Every recommendation must be backed by a query result.** Do not propose based on column names or schema structure alone. Always run `execute_query` to check the actual data before presenting. To learn what sources and fields exist, ground yourself with `get_context`: it returns the model's sources, views, and fields, so there is no separate schema-search step.\n\n| Proposal Type | What to query first |\n|--------------|---------------------|\n| Dimension (bucketed) | Distribution: min, p25, median, p75, p95, max. Propose boundaries from natural breaks, not arbitrary values. |\n| Dimension (categorical) | Distinct values and frequencies. Show the actual categories and their counts. |\n| Measure (sum/avg) | Sample values: min, max, avg. Verify the column contains what you think (e.g., is `total` gross or net?). |\n| Measure (rate/ratio) | Query both numerator and denominator. Verify they make sense together. |\n| Denormalized field vs join | Compare the pre-computed column against the joined aggregate. Report match rate. Recommend whichever is more reliable. |\n| Computed source | Run the proposed group-by plus aggregation. Verify the grain collapses as expected and the result is useful. |\n| Date field selection | Query all candidate date columns. Show % of rows where they differ and by how much. |\n| Column rename | Verify the column has data worth exposing (not 100% NULL). |\n\n**Example, denormalized vs joined:**\n\n> \"Your `customers` table has an `order_count` column. I compared it against `count()` from the `orders` table:\n> - 94% of customers match exactly\n> - 6% have stale counts (the denormalized value is lower than the actual count)\n> - The max discrepancy is 12 orders\n>\n> I'd recommend using the joined count from `orders` rather than the denormalized `order_count`. Want to keep the denormalized column as internal, or drop it?\"\n\n## Tips\n\n- **Show data, not assumptions:** every proposed dimension or measure should have evidence (distinct values, distributions, ranges).\n- **Use `execute_query`** to verify any data questions before presenting to the user.\n- **Don't over-propose:** 5-8 dimensions and 4-6 measures per base source is usually enough to start.\n- **Rank everything:** users appreciate knowing what's essential vs. optional.\n- **Business logic questions must be specific.** \"What date should I use?\" is bad. \"Your table has `created_at` and `submitted_at` that differ by 1-3 days in 13% of rows, which is canonical?\" is good.\n\n## Output\n\nA confirmed source architecture and a confirmed set of field definitions (renames, dimensions, measures, business decisions), held in the conversation and ready to write into the `.malloy` model via your modeling workflow." }, { "name": "malloy-discover", @@ -53,7 +53,7 @@ { "name": "malloy-document", "description": "Add documentation with #(doc) tags to Malloy models so fields and sources are described in plain language. Use when user asks to \"add documentation\", \"add doc tags\", \"document the model\", or wants fields and sources described for natural-language search and discovery. For declaring parameterizable filters with #(filter), see the malloy-model skill. Filters are a runtime/modeling construct (governance, latency, correctness), not a documentation tag.", - "body": "# Documenting a Malloy Model\n\nAdd `#(doc)` tags to describe sources and fields in plain language so they are easy to find and understand:\n\n| Tag | Purpose | Goes on |\n|-----|---------|---------|\n| `#(doc)` | Plain-language description for natural-language search | source, dimension, measure, view, join |\n| `#(filter)` | Declare a parameterizable filter (runtime/modeling concern, see `malloy-model`) | source |\n\n`#(doc)` is a standard Malloy annotation. It documents a field or source with a human-readable description that downstream tools can surface and search against.\n\n## #(doc) Tag\n\nAdd before any source, dimension, measure, view, or join. When multiple fields share a keyword, use it once as a block header. Tags and field names are indented under the keyword; tags go on the line(s) directly above the field they annotate.\n\n**Tag ordering** (when a field has multiple tags): `#(doc)` → render tags (`# currency`, `# label`, etc.) → field name. Separate each field group with a blank line:\n\n```malloy\n#(doc) Customer who placed the order\njoin_one: users with user_id\n\ndimension:\n #(doc) Date the order was placed (UTC)\n order_date is created_at::date\n\nmeasure:\n #(doc) Total revenue from all orders in USD\n # currency\n revenue is sum(total)\n```\n\n### Writing Doc Strings for Retrieval\n\nDoc strings power natural-language search: users type plain-English questions and the system matches against your `#(doc)` strings. Write descriptions that match how analysts would search:\n\n- **Include business meaning**, not code mechanics: what it represents, not how it's implemented\n- **Include units** (USD, count, percentage): a unit is part of what a number means. For counts, name the unit being counted and say whether it counts distinct entities or events: \"total students enrolled\" on a subject×term-grain measure counts enrolments, not students, and a student taking four subjects counts four times. If the model cannot answer the distinct-entity version, say so in the doc.\n- **List a categorical field's values only while the list stays short** (roughly ten or fewer). A handful of values makes a description concrete; past that, say what the field captures instead, because the dump crowds out the meaning and goes stale the moment someone adds a value. Treat ten as a rule of thumb, not a hard cap.\n- **Avoid Malloy jargon**: never use \"filterable\", \"groupable\", \"dimension\", \"measure\", \"aggregation\"\n\n**Good examples:**\n- `#(doc) Total revenue from completed orders in USD` matches \"what was our revenue?\"\n- `#(doc) Customer signup date (UTC)` matches \"when did the customer join?\"\n- `#(doc) Order status: pending, processing, shipped, delivered, cancelled` matches \"what are the order statuses?\"\n\n**Bad examples:**\n- `#(doc) Filterable dimension for order status`: no analyst searches for \"filterable\"\n- `#(doc) Groupable by region`: \"groupable\" is a system concept\n- `#(doc) Aggregation of total sales`: \"aggregation\" doesn't match natural queries\n\n### Mark conventions as conventions\n\nA `#(doc)` must let a reader tell a **measured fact** from a **choice someone made**. Any dimension or measure encoding a threshold, bucket boundary, or business definition that the user did not explicitly confirm must say so in its own doc string:\n\n```malloy\n// WRONG - a chosen cutoff stated as fact\n#(doc) Popularity band: Hit (70+), Popular (40-69), Moderate (15-39), Obscure (<15)\n\n// RIGHT - the choice is visible and auditable\n#(doc) Popularity band: Hit (70+), Popular (40-69), Moderate (15-39), Obscure (<15).\n#(doc) 70 follows the source dataset's own high-popularity cutoff; 40 and 15 are\n#(doc) working boundaries for this model, not settled by the data.\n```\n\nThese are governed models: a threshold nobody confirmed is an assumption, and an unlabeled assumption reads as a fact to everyone downstream, including the agents that answer questions from these docs. The hedge in the `#(doc)` is the artifact-time record; the \"Flag Ambiguous Descriptions\" table below is the conversation-time surface for getting them confirmed, and `modeling-notes.md`'s \"Open decisions\" section (see `skill:malloy-modeling`) is where they wait for a subject-matter expert.\n\nDo not hedge measured facts: `avg_energy is avg(energy)` needs no caveat. Hedge only where a domain expert could reasonably choose differently.\n\n## #(filter): see `malloy-model`\n\n`#(filter)` is also a `#(...)`-shaped annotation, but unlike `#(doc)` it's a **runtime/modeling construct**: it shapes governance, query latency, and correctness, not discoverability. The full reference (syntax, filter types, `required` / `implicit` flags, and when each applies) lives in `malloy-model` § Parameterizable Filters with `#(filter)` alongside the other source-authoring constructs.\n\nOne rule worth knowing here: filters live on the source, never on the consumer. Ad-hoc reports and notebooks that import a source inherit its filters automatically; they do not (and cannot) declare new ones.\n\n## `internal:` and `private:`: column-level access in a source\n\n`#(doc)` describes what's exposed. Two access modifiers control what's exposed in the first place, and both live **inside** a source's `include {}` block. They are about the source's public API and data sensitivity, not about documentation, so reach for them when curating which columns callers can pick.\n\n| Mechanism | Layer | Why you reach for it |\n|---|---|---|\n| `internal:` | Inside a source (one column in `include {}`) | The column **isn't part of your model's public API**. Common reasons: data is messy (empty/garbage, raw JSON, duplicates), or a documented derived dimension already supersedes it, or the raw column exists only to be joined on / referenced internally and shouldn't appear as a dimension callers can pick. The data may be perfectly fine, it's just not what you want exposed. |\n| `private:` | Inside a source (one column in `include {}`) | The **data is sensitive**: SSN, raw credit card, password. Governance / security concern; a harder block than `internal:`. |\n\nIn one sentence: **`internal:` and `private:` shape what's inside a source's public API; `#(doc)` describes the fields you do expose.**\n\n### Example\n\nA base source pulled from a messy raw table often uses `internal:` to drop raw fields from the public API, while documenting the curated columns with `#(doc)`.\n\n```malloy\n// orders_base.malloy\n#(doc) Raw orders. Use orders.malloy as the entry point for analysis.\nsource: orders_base is conn.table('orders_raw')\n include {\n public: id, customer_id, order_date, total\n internal: raw_json_payload, deprecated_status_code, _temp_dedup_marker\n }\n extend {\n primary_key: id\n }\n```\n\n```malloy\n// orders.malloy\nimport \"orders_base.malloy\"\n\n#(doc) Order analysis. Use for revenue, fulfillment, and customer-order joins.\nsource: orders is orders_base extend {\n // joins, measures, curated dimensions\n}\n```\n\nThe base source stays fully queryable (`run: orders_base -> { ... }` still works); `internal:` only governs which columns appear as public dimensions callers can pick.\n\n## Annotating Columns in Include (Experimental)\n\nWith `##! experimental.access_modifiers`, you can add `#(doc)` tags to raw table columns inside `include` blocks. This documents columns without redefining them as dimensions.\n\n```malloy\n##! experimental.access_modifiers\n\nsource: orders is conn.table('orders') include {\n public:\n #(doc) Order line item identifier\n id\n\n #(doc) Customer email address\n email\n\n #(doc) Order status: pending, shipped, delivered\n status\n\n // internal: only for verified noise (empty cols, raw JSON blobs, duplicates)\n}\nextend {\n // ... dimensions and measures\n}\n```\n\n**When to use:**\n- Documenting raw columns without creating explicit dimensions\n- Curating which columns are public vs internal\n\n## Source-Level Documentation\n\nDocument **when to use** a source, not what it contains. Dimensions and measures can already be searched directly, so the source-level `#(doc)` should describe what questions/analyses this source answers.\n\n**Base source files:** Document what the table represents.\n```malloy\n#(doc) Customer records with demographics and segmentation. One row per customer.\nsource: customers is conn.table('sales.customers') extend { ... }\n```\n\n**Source files:** Document what analytical questions the source answers.\n```malloy\n#(doc) Customer health analysis. Use for retention, segmentation, churn risk, and lifetime value. For order-level analysis, use order_analysis instead.\nsource: customer_health is customers extend { ... }\n```\n\n**Best practices:**\n- Add `#(doc)` to all base source and joined source definitions\n- Base source docs: describe what the table is (one row per what)\n- Source docs: describe what questions/analyses the source answers\n- Documentation happens per-source-file, not in one monolithic file\n\n## Flag Ambiguous Descriptions\n\nAfter writing `#(doc)` tags, present any that required judgment to the user for confirmation:\n\n| Field | Proposed doc | Confidence | Uncertainty |\n|-------|-------------|------------|-------------|\n| `total` | \"Total order amount in USD\" | Medium | Could be gross or net, verified with sample query |\n| `status` | \"Order status: pending, shipped, delivered\" | High | Values confirmed via a query of distinct values |\n\nOnly flag fields where the description required assumptions about business meaning, units, or valid values. When in doubt about valid values, run a quick query against the data to confirm them before writing the description. Use `malloy_getContext` to ground yourself in the package's sources and fields and `malloy_executeQuery` to check distinct values, for example `run: source -> { group_by: status }`.\n\n## Done\n\nStep complete. Output: `#(doc)` tags added to all public fields and sources." + "body": "# Documenting a Malloy Model\n\nAdd `#(doc)` tags to describe sources and fields in plain language so they are easy to find and understand:\n\n| Tag | Purpose | Goes on |\n|-----|---------|---------|\n| `#(doc)` | Plain-language description for natural-language search | source, dimension, measure, view, join |\n| `#(filter)` | Deprecated, prefer `given:`. Parameterizable filter (runtime/modeling concern, see `malloy-model`) | source |\n\n`#(doc)` is a standard Malloy annotation. It documents a field or source with a human-readable description that downstream tools can surface and search against.\n\n## #(doc) Tag\n\nAdd before any source, dimension, measure, view, or join. When multiple fields share a keyword, use it once as a block header. Tags and field names are indented under the keyword; tags go on the line(s) directly above the field they annotate.\n\n**Tag ordering** (when a field has multiple tags): `#(doc)` → render tags (`# currency`, `# label`, etc.) → field name. Separate each field group with a blank line:\n\n```malloy\n#(doc) Customer who placed the order\njoin_one: users with user_id\n\ndimension:\n #(doc) Date the order was placed (UTC)\n order_date is created_at::date\n\nmeasure:\n #(doc) Total revenue from all orders in USD\n # currency\n revenue is sum(total)\n```\n\n### Writing Doc Strings for Retrieval\n\nDoc strings power natural-language search: users type plain-English questions and the system matches against your `#(doc)` strings. Write descriptions that match how analysts would search:\n\n- **Include business meaning**, not code mechanics: what it represents, not how it's implemented\n- **Include units** (USD, count, percentage): a unit is part of what a number means. For counts, name the unit being counted and say whether it counts distinct entities or events: \"total students enrolled\" on a subject×term-grain measure counts enrolments, not students, and a student taking four subjects counts four times. If the model cannot answer the distinct-entity version, say so in the doc.\n- **List a categorical field's values only while the list stays short** (roughly ten or fewer). A handful of values makes a description concrete; past that, say what the field captures instead, because the dump crowds out the meaning and goes stale the moment someone adds a value. Treat ten as a rule of thumb, not a hard cap.\n- **Avoid Malloy jargon**: never use \"filterable\", \"groupable\", \"dimension\", \"measure\", \"aggregation\"\n\n**Good examples:**\n- `#(doc) Total revenue from completed orders in USD` matches \"what was our revenue?\"\n- `#(doc) Customer signup date (UTC)` matches \"when did the customer join?\"\n- `#(doc) Order status: pending, processing, shipped, delivered, cancelled` matches \"what are the order statuses?\"\n\n**Bad examples:**\n- `#(doc) Filterable dimension for order status`: no analyst searches for \"filterable\"\n- `#(doc) Groupable by region`: \"groupable\" is a system concept\n- `#(doc) Aggregation of total sales`: \"aggregation\" doesn't match natural queries\n\n### Mark conventions as conventions\n\nA `#(doc)` must let a reader tell a **measured fact** from a **choice someone made**. Any dimension or measure encoding a threshold, bucket boundary, or business definition that the user did not explicitly confirm must say so in its own doc string:\n\n```malloy\n// WRONG - a chosen cutoff stated as fact\n#(doc) Popularity band: Hit (70+), Popular (40-69), Moderate (15-39), Obscure (<15)\n\n// RIGHT - the choice is visible and auditable\n#(doc) Popularity band: Hit (70+), Popular (40-69), Moderate (15-39), Obscure (<15).\n#(doc) 70 follows the source dataset's own high-popularity cutoff; 40 and 15 are\n#(doc) working boundaries for this model, not settled by the data.\n```\n\nThese are governed models: a threshold nobody confirmed is an assumption, and an unlabeled assumption reads as a fact to everyone downstream, including the agents that answer questions from these docs. The hedge in the `#(doc)` is the artifact-time record; the \"Flag Ambiguous Descriptions\" table below is the conversation-time surface for getting them confirmed, and `modeling-notes.md`'s \"Open decisions\" section (see `skill:malloy-modeling`) is where they wait for a subject-matter expert.\n\nDo not hedge measured facts: `avg_energy is avg(energy)` needs no caveat. Hedge only where a domain expert could reasonably choose differently.\n\n## #(filter): deprecated, see `malloy-model`\n\n`#(filter)` is deprecated in favour of native Malloy `given:` parameters. Do not add new `#(filter)` annotations; the two exceptions are `required` and `implicit`, which `given:` cannot cover yet. See `skill:malloy-model` § Legacy: Parameterizable Filters.\n\n`#(filter)` is also a `#(...)`-shaped annotation, but unlike `#(doc)` it's a **runtime/modeling construct**: it shapes governance, query latency, and correctness, not discoverability. The full reference (syntax, filter types, `required` / `implicit` flags, and when each applies) lives in `malloy-model` § Legacy: Parameterizable Filters alongside the other source-authoring constructs, next to the `given:` guidance that replaces it.\n\nOne rule worth knowing here: filters live on the source, never on the consumer. Ad-hoc reports and notebooks that import a source inherit its filters automatically; they do not (and cannot) declare new ones.\n\n## `internal:` and `private:`: column-level access in a source\n\n`#(doc)` describes what's exposed. Two access modifiers control what's exposed in the first place, and both live **inside** a source's `include {}` block. They are about the source's public API and data sensitivity, not about documentation, so reach for them when curating which columns callers can pick.\n\n| Mechanism | Layer | Why you reach for it |\n|---|---|---|\n| `internal:` | Inside a source (one column in `include {}`) | The column **isn't part of your model's public API**. Common reasons: data is messy (empty/garbage, raw JSON, duplicates), or a documented derived dimension already supersedes it, or the raw column exists only to be joined on / referenced internally and shouldn't appear as a dimension callers can pick. The data may be perfectly fine, it's just not what you want exposed. |\n| `private:` | Inside a source (one column in `include {}`) | The **data is sensitive**: SSN, raw credit card, password. Governance / security concern; a harder block than `internal:`. |\n\nIn one sentence: **`internal:` and `private:` shape what's inside a source's public API; `#(doc)` describes the fields you do expose.**\n\n### Example\n\nA base source pulled from a messy raw table often uses `internal:` to drop raw fields from the public API, while documenting the curated columns with `#(doc)`.\n\n```malloy\n// orders_base.malloy\n#(doc) Raw orders. Use orders.malloy as the entry point for analysis.\nsource: orders_base is conn.table('orders_raw')\n include {\n public: id, customer_id, order_date, total\n internal: raw_json_payload, deprecated_status_code, _temp_dedup_marker\n }\n extend {\n primary_key: id\n }\n```\n\n```malloy\n// orders.malloy\nimport \"orders_base.malloy\"\n\n#(doc) Order analysis. Use for revenue, fulfillment, and customer-order joins.\nsource: orders is orders_base extend {\n // joins, measures, curated dimensions\n}\n```\n\nThe base source stays fully queryable (`run: orders_base -> { ... }` still works); `internal:` only governs which columns appear as public dimensions callers can pick.\n\n## Annotating Columns in Include (Experimental)\n\nWith `##! experimental.access_modifiers`, you can add `#(doc)` tags to raw table columns inside `include` blocks. This documents columns without redefining them as dimensions.\n\n```malloy\n##! experimental.access_modifiers\n\nsource: orders is conn.table('orders') include {\n public:\n #(doc) Order line item identifier\n id\n\n #(doc) Customer email address\n email\n\n #(doc) Order status: pending, shipped, delivered\n status\n\n // internal: only for verified noise (empty cols, raw JSON blobs, duplicates)\n}\nextend {\n // ... dimensions and measures\n}\n```\n\n**When to use:**\n- Documenting raw columns without creating explicit dimensions\n- Curating which columns are public vs internal\n\n## Source-Level Documentation\n\nDocument **when to use** a source, not what it contains. Dimensions and measures can already be searched directly, so the source-level `#(doc)` should describe what questions/analyses this source answers.\n\n**Base source files:** Document what the table represents.\n```malloy\n#(doc) Customer records with demographics and segmentation. One row per customer.\nsource: customers is conn.table('sales.customers') extend { ... }\n```\n\n**Source files:** Document what analytical questions the source answers.\n```malloy\n#(doc) Customer health analysis. Use for retention, segmentation, churn risk, and lifetime value. For order-level analysis, use order_analysis instead.\nsource: customer_health is customers extend { ... }\n```\n\n**Best practices:**\n- Add `#(doc)` to all base source and joined source definitions\n- Base source docs: describe what the table is (one row per what)\n- Source docs: describe what questions/analyses the source answers\n- Documentation happens per-source-file, not in one monolithic file\n\n## Flag Ambiguous Descriptions\n\nAfter writing `#(doc)` tags, present any that required judgment to the user for confirmation:\n\n| Field | Proposed doc | Confidence | Uncertainty |\n|-------|-------------|------------|-------------|\n| `total` | \"Total order amount in USD\" | Medium | Could be gross or net, verified with sample query |\n| `status` | \"Order status: pending, shipped, delivered\" | High | Values confirmed via a query of distinct values |\n\nOnly flag fields where the description required assumptions about business meaning, units, or valid values. When in doubt about valid values, run a quick query against the data to confirm them before writing the description. Use `malloy_getContext` to ground yourself in the package's sources and fields and `malloy_executeQuery` to check distinct values, for example `run: source -> { group_by: status }`.\n\n## Done\n\nStep complete. Output: `#(doc)` tags added to all public fields and sources." }, { "name": "malloy-getting-started", @@ -158,7 +158,12 @@ { "name": "malloy-model", "description": "Build Malloy semantic models with base source and joined source files. Use when creating or modifying .malloy files, user asks to \"create a malloy model\", \"add dimensions\", \"add measures\", \"create a source\", or any Malloy model authoring task.", - "body": "# Building Malloy Models\n\n> **Tool names** are written bare here - `get_context`, `execute_query`, `search_malloy_docs`. The exact prefixed name depends on the host surface; match each against the tools you actually have.\n\n## Getting Started (New Projects)\n\nIf no `.malloy` files exist yet, do discovery and propose a structure first, then return here to build base source and joined source files. Keep proposals and the analysis behind them in the conversation.\n\n**File structure convention** (a flat layout at the package root is the simplest default):\n```\n/\n publisher.json # Required for publishing (name, version, description)\n customers.malloy # Base source: one per table\n products.malloy\n orders.malloy\n user_order_facts.malloy # Computed source\n order_analysis.malloy # Source: one per analytical domain\n customer_health.malloy\n```\n\nVersions for new packages should start at \"0.0.1\".\n\n**Before creating any files**, check for an existing `publisher.json` in the target directory. If one exists for a different package, create a new subdirectory for your package, don't overwrite another package's config.\n\nIn Publisher an environment is a project, and Publisher is single-tenant, so there is no org/tenant layer to model around: one environment holds one set of packages.\n\n## Prior Art Dispatch\n\nIf your discovery turned up existing modeling patterns to mirror (a derived table, UNNEST joins, a review or curation pass), read the relevant reference before building.\n\n| Pattern found in prior art | Reference to read |\n|---------------------|-------------------|\n| Derived table (PDT/NDT) | `skill:malloy-lookml-review` build-derived-tables guidance |\n| UNNEST joins or struct access | `skill:malloy-lookml-review` build-unnest guidance |\n| Review pass for coverage | `skill:malloy-lookml-review` review-coverage guidance |\n| Curate pass with visibility seeds | `skill:malloy-lookml-review` curate-visibility guidance |\n\n## Base Source Templates\n\n### Base Source (Simple Mode)\n\n```malloy\nsource: customers is my_conn.table('sales.customers')\nextend {\n primary_key: customer_id\n\n dimension:\n // A dimension is the lighter way to give a column a cleaner name\n order_type is `Type`\n full_name is concat(first_name, ' ', last_name)\n segment is lifetime_value ?\n pick 'enterprise' when >= 100000\n pick 'mid-market' when >= 10000\n else 'SMB'\n\n measure:\n customer_count is count()\n}\n```\n\n> **Every `dimension:` needs `name is expr`.** A bare column name like `dimension: species` is a parse error (e.g. `missing IS at ...`). Raw columns are already usable in `group_by` / `select` without any declaration, so only add a `dimension:` when deriving or renaming a field (e.g. `revenue is price * quantity`).\n\n### Base Source (Curated Mode with Access Modifiers)\n\n```malloy\n##! experimental.access_modifiers\n\nsource: orders is my_conn.table('sales.orders')\ninclude {\n public:\n #(doc) Order identifier\n order_id\n\n #(doc) Customer who placed the order\n user_id\n\n #(doc) Total sale price in USD\n sale_price\n\n #(doc) Order creation timestamp\n created_at\n\n internal:\n raw_payload_json // Verified empty via index query + user confirmation\n}\nextend {\n primary_key: order_id\n\n dimension:\n #(doc) Date the order was placed\n order_date is created_at::date\n\n measure:\n #(doc) Total number of orders\n order_count is count()\n\n #(doc) Total revenue in USD\n # currency\n revenue is sum(sale_price)\n}\n```\n\n### Computed Source (from Query)\n\n```malloy\nimport \"orders.malloy\"\n\nsource: user_order_facts is from(\n orders -> {\n group_by: customer_id\n aggregate:\n total_orders is count()\n total_revenue is sum(sale_price)\n first_order_date is min(created_at)\n last_order_date is max(created_at)\n }\n) extend {\n primary_key: customer_id\n\n dimension:\n days_since_last_order is days(last_order_date to now)\n is_repeat_buyer is total_orders > 1\n\n measure:\n buyer_count is count()\n avg_customer_ltv is avg(total_revenue)\n}\n```\n\nFor advanced query-based source patterns (window functions, pipelines), see `reference/query-sources.md`.\n\n## Joined Source File Template\n\n```malloy\nimport \"customers.malloy\"\nimport \"orders.malloy\"\nimport \"user_order_facts.malloy\"\n\n#(doc) Customer health analysis. Use for retention, segmentation, and churn risk.\nsource: customer_health is customers extend {\n join_one: user_order_facts with customer_id\n join_many: orders on customer_id = orders.customer_id\n\n dimension:\n is_at_risk is user_order_facts.days_since_last_order > 90\n and user_order_facts.total_orders > 1\n\n measure:\n revenue_per_customer is orders.sale_price.sum() / nullif(customer_count, 0)\n at_risk_count is count() { where: is_at_risk = true }\n}\n```\n\n## Base vs Joined Sources\n\n| | Base Joined Source File | Joined Source File |\n|---|---|---|\n| **Contains** | One table's fields | Joins between base sources |\n| **Dimensions** | Intrinsic to this table only | Cross-source (require joins) |\n| **Measures** | Single-table aggregations | Cross-source aggregations |\n| **Joins** | None (or only lookup joins intrinsic to the source) | Defines relationships between base sources |\n| **Views** | None (views belong in analysis) | None |\n| **One per** | Physical table or computed source | Analytical domain |\n\n## Key Rules\n\n- **Every `dimension:` needs `name is expr`**: a bare `dimension: species` is a parse error. Raw columns are queryable directly in `group_by` / `select`; only declare a `dimension:` to derive or rename a field.\n- **Define joined tables before referencing them**, use `import` statements in multi-file architecture\n- **Use `nullif(denominator, 0)` for all division**\n- **Alias joined fields before using in `order_by`**: `group_by: yr is table.year`\n- **Verify join paths** exist before referencing `a.b.field` (each hop needs explicit join)\n- **Pick syntax**: value BEFORE condition, `pick 'Small' when size < 10`\n- **`where:` vs `having:`**: Use `where:` for row filters, `having:` for aggregate filters\n- **`rename:` composes with `include {}`, but only in one order**: the `extend { rename: }` must come before the `include {}`, which then names the field by its new name. Reversed, it fails with `Can't find field 'X' to set access modifier`. For a cleaner column name without a rename, `internal:` + `dimension:` is still the lighter move (mark `` `Type` `` as `internal`, add `` dimension: order_type is `Type` ``). See `skill:malloy-gotchas-modeling` § Field Management\n- **Mark raw columns `internal` when a derived dimension replaces them**\n- **Check for duplicate rows** before building measures\n- When both a combined table (all types) and filtered/split tables exist, prefer the split tables\n- **DRY: define measures/dimensions in base source files, not inline in views**\n- **Never write a threshold, tier boundary, or bucket cutoff you chose yourself.** Every boundary in a `pick` expression or filtered measure is user-supplied, distribution-derived (query `min`/`p25`/`p50`/`p75`/`p95` first and show the evidence; see `skill:malloy-define` § Data-driven proposals), or explicitly flagged as an assumption in its `#(doc)`. A hardcoded cutoff nobody confirmed is a business decision shipped as fact.\n\n## Parameterizing sources with `given:` (preferred)\n\nNative Malloy **`given:` parameters** are the going-forward way to expose tunable knobs (date range, region, manufacturer) on a source - prefer them over `#(filter)` when you author a new model. A `given:` is a first-class runtime parameter you reference in the model's own logic; callers supply values at query time and the model uses them however it declares. Enable them with `##! experimental.givens` at the top of the model.\n\n```malloy\n##! experimental.givens\n\ngiven:\n manufacturer_filter :: filter is f''\n subject_filter :: filter is f''\n\nsource: recalls is duckdb.table('data/auto_recalls.csv') extend {\n where: Manufacturer ~ $manufacturer_filter, Subject ~ $subject_filter\n measure: recall_count is count()\n}\n```\n\nA given is **declared bare** but **referenced with a `$` sigil** in expressions (`$manufacturer_filter`), as above.\n\n- **Give every optional filter a neutral, match-all default** - a `filter<>` given defaulting to `f''` - so an unsupplied value returns unfiltered rows, matching how `#(filter)` behaves when a value is omitted. Because the given bakes an always-on `where:` into the source, a non-neutral default (e.g. a date floor) applies to *every* read of the source, not just the ones that opt in - so keep defaults neutral. Defaults must be Malloy literals.\n- **Givens don't auto-inject a `where:`.** Unlike `#(filter)`, you write the filter expression that references the given yourself (e.g. `where: dimension ~ $given_name`).\n- **Not every filter maps cleanly.** A filter with no neutral match-all literal default - e.g. a scalar date/number range like `> @2020-01-01` - is not a good `given:`; keep those on `#(filter)`. Two more cases keep using `#(filter)`: mandatory scoping filters (`required`) and system-injected row-level filters (`implicit`), both below.\n\nGivens are also the substrate for access control - see \"Access Control: Source Gating with `#(authorize)`\" below.\n\n## Legacy: Parameterizable Filters with `#(filter)`\n\n`#(filter)` is the older, Publisher-specific mechanism for the same idea. Publisher parses the annotation, exposes filter metadata via the API, renders filter widgets in the notebook UI, and **injects `where:` clauses into queries server-side** when callers supply parameters. Prefer `given:` (above) for new models; keep reading and maintaining `#(filter)` on existing models, and keep using it for the two cases `given:` can't cover yet - `required` (mandatory scoping) and `implicit` (system-injected filters), below.\n\nFilters are a **runtime/modeling construct**, not just documentation. They shape governance, query latency (forcing filters keeps result sets bounded), and correctness (see `required` below). They live on the source, never on the consumer: an ad-hoc report or notebook that imports a source inherits and displays that source's filters automatically; it does not (and cannot) declare new ones. If an existing `#(filter)`-based source needs another knob, add it to the source itself, not to the consumer.\n\n### Syntax\n\n```malloy\n#(filter) [name=NAME] dimension=DIMENSION type=TYPE [implicit] [required]\n```\n\n| Parameter | Required | Description |\n|-----------|----------|-------------|\n| `name` | No | Unique identifier for the filter; defaults to the dimension name. Used as the API parameter key. |\n| `dimension` | Yes | The source dimension this filter targets. Quote with `\"...\"` if the name contains spaces. |\n| `type` | Yes | Comparator (see below). |\n| `implicit` | No | Hides the filter from the UI and API summaries. Used for infrastructure concerns the system injects rather than the user. |\n| `required` | No | Server returns 400 if a required filter has no value at query time. Use this for governance, latency, and correctness, see below. |\n\n### Filter types\n\n| Type | Malloy clause | Use case |\n|------|---------------|----------|\n| `equal` | `dimension = 'value'` | Exact match on a single value |\n| `in` | `dimension ? 'a' \\| 'b' \\| 'c'` | Match any of multiple values |\n| `like` | `dimension ~ '%value%'` | Substring / pattern matching |\n| `greater_than` | `dimension > value` | Range floor (after, minimum) |\n| `less_than` | `dimension < value` | Range ceiling (before, maximum) |\n\n### Example\n\n```malloy\n#(filter) name=Manufacturer dimension=Manufacturer type=in\n#(filter) name=Subject dimension=Subject type=like\n#(filter) name=Major_Recall dimension=\"Major Recall\" type=equal\n#(filter) name=Recall_After dimension=\"Report Received Date\" type=greater_than\n#(filter) name=Recall_Before dimension=\"Report Received Date\" type=less_than\nsource: recalls is duckdb.table('data/auto_recalls.csv') extend {\n measure:\n recall_count is count()\n}\n```\n\nFor date-range filters, declare two filters with distinct `name` values targeting the same dimension (one `greater_than`, one `less_than`).\n\n### When to use `required`\n\n`required` filters are a correctness, latency, and governance mechanism, not just UX. Mark a filter `required` when:\n\n1. **Modeling correctness, the source's `primary_key:` is only unique under a filter.** If a high-cardinality key is not unique across the whole table but is unique within a scoping dimension, then that scoping dimension MUST be supplied for symmetric aggregation to produce correct numbers. For example, if `events.id` repeats across days but is unique within a single `event_date`, queries that don't pin the date can fan out and return hash-collision-sized garbage (~10²¹). Declare `#(filter) name=Event_Date dimension=event_date type=equal required` so the server refuses queries that don't provide it.\n2. **Query latency, the source spans more data than any single query should scan.** A multi-year, multi-region table where every reasonable analysis is scoped to a date range or region: making the date filter required prevents accidental full-table scans.\n3. **Partial views** that are only meaningful inside a date range, region, or business segment.\n4. **Governance**, an analyst should never query the raw source without a scoping filter applied.\n\nFor (1), pair the required filter with a comment explaining the cardinality dependency, and consider also declaring `#(doc)` on the source noting the constraint.\n\n### When to use `implicit`\n\nUse `implicit` for filters the *system* must inject but users should not see. The filter applies; it just doesn't appear in the UI or API filter list.\n\n### Type-aware literals\n\nPublisher formats values based on the dimension's data type, `string` → `'value'`, `boolean` → bare `true`/`false`, `date` → `@YYYY-MM-DD`. You don't quote values yourself in the API call; Publisher handles formatting.\n\n### Bypass\n\nPass `bypass_filters=true` (REST) or `bypassFilters: true` (POST body) to skip filter injection entirely. Use sparingly, required-filter governance only works if bypass is restricted to trusted callers.\n\n## Access Control: Source Gating with `#(authorize)`\n\nGate query access to a source with `#(authorize)` over declared `given:` values (`given:` is Malloy's native runtime-parameter mechanism, the going-forward replacement for `#(filter)`). A gate is an `#(authorize)` annotation on its own line directly above the `source:` line, carrying an **unquoted, ordinary Malloy boolean expression**; Publisher grafts that expression onto the source as a row filter before running the query, so a caller it admits nowhere gets **200 with zero rows**, not a 403. A **403** means only that the gate could not be attached at all. A source with no `#(authorize)` annotation of its own or inherited is unrestricted.\n\n```malloy\n##! experimental.givens\n\ngiven:\n ROLE :: string\n\n#(authorize) $ROLE = 'analyst'\nsource: orders is duckdb.table('orders.parquet') extend {\n measure: order_count is count()\n}\n```\n\n- **Any legal Malloy boolean expression is a legal gate**, over givens, row fields (including through a join), literals, functions and operators: `org_id in $GROUPS`, `upper(region) = $REGION`, `` `cost center` in $GROUPS ``, `(org_id in $GROUPS or region = $REGION) and amount > $AMOUNTMIN`. There is no allowlist of accepted comparison shapes.\n- **A source may declare at most one `#(authorize)` annotation.** Declaring a second on the same source fails the load naming both. Spell OR inside the expression rather than stacking annotations. For a condition too long to read on one line, point the gate at an ordinary boolean dimension instead: `#(authorize) authorized` above the source, over `dimension: authorized is org_id in $GROUPS` inside it; validation follows the reference through.\n- **`#(authorize)` only gates from the `source:` line.** The same annotation on a `dimension:`/`measure:`/`join_*:`/`view:` line, or on a top-level `query:`, is refused at load naming the position rather than silently protecting nothing.\n- **Every given the gate references must be declared on the entry model's own surface, and must carry no default.** A given the model cannot resolve is refused at load. So is a referenced given declared *with* a default: a caller who supplies nothing would get that default and be admitted or excluded by a value the gate's own line never shows, so it is refused rather than reasoned about case by case. This follows a bare reference through, so a given reached one hop away via `#(authorize) authorized` is checked too.\n- **Two shapes load with a warning rather than a refusal.** A gate that references **no given** at all is a fixed predicate, not an access rule keyed on the caller. A gate that **negates a membership test** (`not (org_id in $GROUPS)`) matches every row for an *empty* given instead of none. Both warn and still load, so read the load warnings.\n- **Entry point only: not joined, but inherited through `extend`.** The gate applies to the source a query enters through. A gate on a source reached only via `join_*` **never fires**, at any depth, so anything ungated that joins a locked base hands the base's rows to every caller. A source that `extend`s a locked base and declares no gate of its own **does** carry the base's gate; declaring its own annotation replaces it. A source derived from a locked base via a query (`source: z is locked -> { … }`) instead **always carries the base's gate in addition to its own**: the derivation recurses into the base unconditionally, so an own gate does not replace it, and the two combine as separate AND'd entries. Pair a locked base (`#(authorize) false`) with curated extension sources, using access modifiers (`include { public: …, private: * }`), so an extension re-exposes only a curated column surface, and keep sensitive sources out of ungated joins.\n- **A derivation that drops a column the gate reads fails CLOSED.** `extend { except: org_id }`, or an `accept:` that omits it, leaves the grafted filter unable to compile, so the request is denied rather than served ungated. The one hole to know: dropping the gated column and then `rename:`-ing a *different* column onto that exact name grafts successfully and binds the gate to the wrong column. Narrow, but real, so don't recycle a gated column's name.\n- Comparing a row field to an array-typed given with `=`/`!=` (`org_id = $GROUPS`) compiles and loads cleanly, then fails at query execution with a warehouse conversion error. Use `in` for an array-typed given, not `=`.\n- **The quoted-string and file-level forms are refused at load and no longer exist.** `#(authorize) \"\"` on the `source:` line, in either quote (`'...'` is refused the same way), a file-level `##(authorize) \"\"` applying to every source in the file, and the earlier `internal dimension: authorized is ` form are all retired; the load names the rewrite. Every `.malloy` file in a package compiles at load and any failure aborts the package, so a retired-form gate anywhere in the package is refused. Only a declaring file *outside* the package escapes that: it loads and denies every request instead, with no compile-time hint. See your deployment's reference documentation.\n- **A gated source can be persisted, but the gating column freezes.** `storage=` and `#@ preaggregate` refuse a gated source outright; a colocated `#@ persist` is admitted when the gate is provably the entry point's own row filter. The gate still runs live on every query, so rows come back filtered - but the column it filters ON is frozen at build time, so a row whose access decision changes keeps being served under the old decision until the next rebuild. Pair `#@ persist` on a gated source with a freshness window (`fallback=\"live\"`), which is the only control that bounds that - and read `skill:malloy-materialization` for where that window binds, because on a standalone Publisher it does not.\n\n> **Trust caveat.** Givens are **caller-asserted**, anyone who can reach the query API can claim a favorable given, e.g. `{\"ROLE\":\"admin\"}`. `#(authorize)` is only a real boundary when it sits behind a trusted tier that sets givens from its own verified context, never directly from an untrusted caller. It is not, on its own, end-user authentication.\n>\n> **Forward direction.** Givens are how access control is built here, and the planned next step is **identity-bound (\"secure\") givens** - reserved values a trusted tier populates from a verified token or proxy header, which the caller cannot override - turning `#(authorize)` into a standalone boundary. Model access on `given:` + `#(authorize)` now; it is the surface that carries forward.\n\nFull syntax, inheritance rules, validation, and the error contract are covered in your deployment's `#(authorize)` reference documentation.\n\n## Join Syntax\n\n- Simple join: `join_one: users with user_id`\n- Expression join: `join_one: origin is airports on origin_code = origin.code`\n- Composite key: `join_one: items on order_id = items.order_id and product_id = items.product_id`\n- Multiple joins to same table: `join_one: origin_airport is airports with origin`\n\n**Join Types:** `join_one:` (many-to-one, efficient) | `join_many:` (one-to-many, always safe) | `join_cross:` (many-to-many)\n\n**Verify cardinality** before writing joins: `run: target -> { group_by: fk_col, aggregate: n is count(), having: n > 1, limit: 5 }`. 0 results → `join_one`. Any results → `join_many`.\n\n## After Writing: Check & Review\n\nCheck diagnostics after writing. Errors cascade, fix the FIRST error only, then re-check. If errors persist, use the debugging strategy: look at first error, search docs if unsure, fix, repeat.\n\n**Validate with `execute_query`:** Run queries, check distributions, verify measures, confirm joins (no fan-out).\n\nTo inspect the sources and fields a model already defines, ground yourself with `get_context`. It returns the package's sources, views, and fields, so there is no separate schema-search step. When you're unsure of Malloy syntax, call `search_malloy_docs` rather than guessing.\n\n## Advanced Patterns\n\nLoad the relevant reference file when you encounter these scenarios:\n\n| Scenario | Read |\n|----------|------|\n| Need pre-aggregated or windowed source | `reference/query-sources.md` |\n| Curating access modifiers | `reference/access-modifiers.md` |\n| Normalized/ER-style schema (4+ tables, no clear fact table) | `reference/normalized-schemas.md` |\n| Formalizing analysis into a model | `reference/analysis-to-model.md` |\n| Many-to-many / bridge tables / composite keys | `reference/bridge-tables.md` |\n\n## Done\n\nStep complete. Output: base source files (`.malloy`, one per table) and joined source files (`.malloy`, one per analytical domain).\n\n**Suggest next steps to the user:**\n\n- Open the model in the browser to see it live: `http://localhost:4000//` for the package, or `http://localhost:4000///` for a single model file. First confirm the running server actually serves this package (it is in the loaded `publisher.config.json`, or mounted live with `--server_root . --watch-env `); a package the server has not loaded returns a 404, so do not hand over a link to a package that was just authored but never loaded.\n- Build a notebook with interactive filters over the model (see `skill:malloy-notebooks`).\n- Run analysis questions against the model (see `skill:malloy-analysis`).\n- When you're ready to serve the model, publishing is out of scope for open-source Publisher v1: self-hosters commit the package to git and use their host's publish path.\n\n## Reference files over MCP\n\nThis skill's `reference/` files are served as separate prompts, one per file, fetched only when you ask for them. Where the text above says to read `reference/.md`, get the prompt named `malloy-model/` instead.\n\nAvailable: access-modifiers, analysis-to-model, bridge-tables, normalized-schemas, query-sources." + "body": "# Building Malloy Models\n\n> **Tool names** are written bare here - `get_context`, `execute_query`, `search_malloy_docs`. The exact prefixed name depends on the host surface; match each against the tools you actually have.\n\n## Getting Started (New Projects)\n\nIf no `.malloy` files exist yet, do discovery and propose a structure first, then return here to build base source and joined source files. Keep proposals and the analysis behind them in the conversation.\n\n**File structure convention** (a flat layout at the package root is the simplest default):\n```\n/\n publisher.json # Required for publishing (name, version, description)\n customers.malloy # Base source: one per table\n products.malloy\n orders.malloy\n user_order_facts.malloy # Computed source\n order_analysis.malloy # Source: one per analytical domain\n customer_health.malloy\n```\n\nVersions for new packages should start at \"0.0.1\".\n\n**Before creating any files**, check for an existing `publisher.json` in the target directory. If one exists for a different package, create a new subdirectory for your package, don't overwrite another package's config.\n\nIn Publisher an environment is a project, and Publisher is single-tenant, so there is no org/tenant layer to model around: one environment holds one set of packages.\n\n## Prior Art Dispatch\n\nIf your discovery turned up existing modeling patterns to mirror (a derived table, UNNEST joins, a review or curation pass), read the relevant reference before building.\n\n| Pattern found in prior art | Reference to read |\n|---------------------|-------------------|\n| Derived table (PDT/NDT) | `skill:malloy-lookml-review` build-derived-tables guidance |\n| UNNEST joins or struct access | `skill:malloy-lookml-review` build-unnest guidance |\n| Review pass for coverage | `skill:malloy-lookml-review` review-coverage guidance |\n| Curate pass with visibility seeds | `skill:malloy-lookml-review` curate-visibility guidance |\n\n## Base Source Templates\n\n### Base Source (Simple Mode)\n\n```malloy\nsource: customers is my_conn.table('sales.customers')\nextend {\n primary_key: customer_id\n\n dimension:\n // A dimension is the lighter way to give a column a cleaner name\n order_type is `Type`\n full_name is concat(first_name, ' ', last_name)\n segment is lifetime_value ?\n pick 'enterprise' when >= 100000\n pick 'mid-market' when >= 10000\n else 'SMB'\n\n measure:\n customer_count is count()\n}\n```\n\n> **Every `dimension:` needs `name is expr`.** A bare column name like `dimension: species` is a parse error (e.g. `missing IS at ...`). Raw columns are already usable in `group_by` / `select` without any declaration, so only add a `dimension:` when deriving or renaming a field (e.g. `revenue is price * quantity`).\n\n### Base Source (Curated Mode with Access Modifiers)\n\n```malloy\n##! experimental.access_modifiers\n\nsource: orders is my_conn.table('sales.orders')\ninclude {\n public:\n #(doc) Order identifier\n order_id\n\n #(doc) Customer who placed the order\n user_id\n\n #(doc) Total sale price in USD\n sale_price\n\n #(doc) Order creation timestamp\n created_at\n\n internal:\n raw_payload_json // Verified empty via index query + user confirmation\n}\nextend {\n primary_key: order_id\n\n dimension:\n #(doc) Date the order was placed\n order_date is created_at::date\n\n measure:\n #(doc) Total number of orders\n order_count is count()\n\n #(doc) Total revenue in USD\n # currency\n revenue is sum(sale_price)\n}\n```\n\n### Computed Source (from Query)\n\n```malloy\nimport \"orders.malloy\"\n\nsource: user_order_facts is from(\n orders -> {\n group_by: customer_id\n aggregate:\n total_orders is count()\n total_revenue is sum(sale_price)\n first_order_date is min(created_at)\n last_order_date is max(created_at)\n }\n) extend {\n primary_key: customer_id\n\n dimension:\n days_since_last_order is days(last_order_date to now)\n is_repeat_buyer is total_orders > 1\n\n measure:\n buyer_count is count()\n avg_customer_ltv is avg(total_revenue)\n}\n```\n\nFor advanced query-based source patterns (window functions, pipelines), see `reference/query-sources.md`.\n\n## Joined Source File Template\n\n```malloy\nimport \"customers.malloy\"\nimport \"orders.malloy\"\nimport \"user_order_facts.malloy\"\n\n#(doc) Customer health analysis. Use for retention, segmentation, and churn risk.\nsource: customer_health is customers extend {\n join_one: user_order_facts with customer_id\n join_many: orders on customer_id = orders.customer_id\n\n dimension:\n is_at_risk is user_order_facts.days_since_last_order > 90\n and user_order_facts.total_orders > 1\n\n measure:\n revenue_per_customer is orders.sale_price.sum() / nullif(customer_count, 0)\n at_risk_count is count() { where: is_at_risk = true }\n}\n```\n\n## Base vs Joined Sources\n\n| | Base Joined Source File | Joined Source File |\n|---|---|---|\n| **Contains** | One table's fields | Joins between base sources |\n| **Dimensions** | Intrinsic to this table only | Cross-source (require joins) |\n| **Measures** | Single-table aggregations | Cross-source aggregations |\n| **Joins** | None (or only lookup joins intrinsic to the source) | Defines relationships between base sources |\n| **Views** | None, in schema-first (see below) | None, in schema-first (see below) |\n| **One per** | Physical table or computed source | Analytical domain |\n\n**The \"no views\" rule is schema-first only.** A schema-first model is built before anyone\nhas asked a question, so any view in it is a guess. It does **not** apply to the\nanalysis-first workflow (`skill:malloy-model-as-you-go`), where every view is a\nquestion that was asked and verified - there, **saving a view or a dashboard is the right\ncall**, in the model file next to the measures it uses. Analysis-first still models\neverything else properly: documented dimensions, measures, and joins.\n\n**Two more rules here are schema-first only.** Analysis-first should skip access modifiers\nand curation - there is no discovery surface to curate when every field was paid for by a\nquestion - and skip one-file-per-table, keeping a single domain file until it genuinely\ngets unwieldy.\n\n## Key Rules\n\n- **Every `dimension:` needs `name is expr`**: a bare `dimension: species` is a parse error. Raw columns are queryable directly in `group_by` / `select`; only declare a `dimension:` to derive or rename a field.\n- **Define joined tables before referencing them**, use `import` statements in multi-file architecture\n- **Use `nullif(denominator, 0)` for all division**\n- **Alias joined fields before using in `order_by`**: `group_by: yr is table.year`\n- **Verify join paths** exist before referencing `a.b.field` (each hop needs explicit join)\n- **Pick syntax**: value BEFORE condition, `pick 'Small' when size < 10`\n- **`where:` vs `having:`**: Use `where:` for row filters, `having:` for aggregate filters\n- **`rename:` composes with `include {}`, but only in one order**: the `extend { rename: }` must come before the `include {}`, which then names the field by its new name. Reversed, it fails with `Can't find field 'X' to set access modifier`. For a cleaner column name without a rename, `internal:` + `dimension:` is still the lighter move (mark `` `Type` `` as `internal`, add `` dimension: order_type is `Type` ``). See `skill:malloy-gotchas-modeling` § Field Management\n- **Mark raw columns `internal` when a derived dimension replaces them**\n- **Check for duplicate rows** before building measures\n- When both a combined table (all types) and filtered/split tables exist, prefer the split tables\n- **DRY: define measures/dimensions in base source files, not inline in views**\n- **Never write a threshold, tier boundary, or bucket cutoff you chose yourself.** Every boundary in a `pick` expression or filtered measure is user-supplied, distribution-derived (query `min`/`p25`/`p50`/`p75`/`p95` first and show the evidence; see `skill:malloy-define` § Data-driven proposals), or explicitly flagged as an assumption in its `#(doc)`. A hardcoded cutoff nobody confirmed is a business decision shipped as fact.\n\n## Parameterizing sources with `given:` (preferred)\n\nNative Malloy **`given:` parameters** are the going-forward way to expose tunable knobs (date range, region, manufacturer) on a source - prefer them over `#(filter)` when you author a new model. A `given:` is a first-class runtime parameter you reference in the model's own logic; callers supply values at query time and the model uses them however it declares. Enable them with `##! experimental.givens` at the top of the model.\n\n```malloy\n##! experimental.givens\n\ngiven:\n manufacturer_filter :: filter is f''\n subject_filter :: filter is f''\n\nsource: recalls is duckdb.table('data/auto_recalls.csv') extend {\n where: Manufacturer ~ $manufacturer_filter, Subject ~ $subject_filter\n measure: recall_count is count()\n}\n```\n\nA given is **declared bare** but **referenced with a `$` sigil** in expressions (`$manufacturer_filter`), as above.\n\n- **Give every optional filter a neutral, match-all default** - a `filter<>` given defaulting to `f''` - so an unsupplied value returns unfiltered rows, matching how `#(filter)` behaves when a value is omitted. Because the given bakes an always-on `where:` into the source, a non-neutral default (e.g. a date floor) applies to *every* read of the source, not just the ones that opt in - so keep defaults neutral. Defaults must be Malloy literals.\n- **Givens don't auto-inject a `where:`.** Unlike `#(filter)`, you write the filter expression that references the given yourself (e.g. `where: dimension ~ $given_name`).\n- **Not every filter maps cleanly.** A filter with no neutral match-all literal default - e.g. a scalar date/number range like `> @2020-01-01` - is not a good `given:`; keep those on `#(filter)`. Two more cases keep using `#(filter)`: mandatory scoping filters (`required`) and system-injected row-level filters (`implicit`), both below.\n\nGivens are also the substrate for access control - see \"Access Control: Source Gating with `#(authorize)`\" below.\n\n## Legacy: Parameterizable Filters with `#(filter)`\n\n`#(filter)` is the older, Publisher-specific mechanism for the same idea. Publisher parses the annotation, exposes filter metadata via the API, renders filter widgets in the notebook UI, and **injects `where:` clauses into queries server-side** when callers supply parameters. Prefer `given:` (above) for new models; keep reading and maintaining `#(filter)` on existing models, and keep using it for the two cases `given:` can't cover yet - `required` (mandatory scoping) and `implicit` (system-injected filters), below.\n\nFilters are a **runtime/modeling construct**, not just documentation. They shape governance, query latency (forcing filters keeps result sets bounded), and correctness (see `required` below). They live on the source, never on the consumer: an ad-hoc report or notebook that imports a source inherits and displays that source's filters automatically; it does not (and cannot) declare new ones. If an existing `#(filter)`-based source needs another knob, add it to the source itself, not to the consumer.\n\n### Syntax\n\n```malloy\n#(filter) [name=NAME] dimension=DIMENSION type=TYPE [implicit] [required]\n```\n\n| Parameter | Required | Description |\n|-----------|----------|-------------|\n| `name` | No | Unique identifier for the filter; defaults to the dimension name. Used as the API parameter key. |\n| `dimension` | Yes | The source dimension this filter targets. Quote with `\"...\"` if the name contains spaces. |\n| `type` | Yes | Comparator (see below). |\n| `implicit` | No | Hides the filter from the UI and API summaries. Used for infrastructure concerns the system injects rather than the user. |\n| `required` | No | Server returns 400 if a required filter has no value at query time. Use this for governance, latency, and correctness, see below. |\n\n### Filter types\n\n| Type | Malloy clause | Use case |\n|------|---------------|----------|\n| `equal` | `dimension = 'value'` | Exact match on a single value |\n| `in` | `dimension ? 'a' \\| 'b' \\| 'c'` | Match any of multiple values |\n| `like` | `dimension ~ '%value%'` | Substring / pattern matching |\n| `greater_than` | `dimension > value` | Range floor (after, minimum) |\n| `less_than` | `dimension < value` | Range ceiling (before, maximum) |\n\n### Example\n\n```malloy\n#(filter) name=Manufacturer dimension=Manufacturer type=in\n#(filter) name=Subject dimension=Subject type=like\n#(filter) name=Major_Recall dimension=\"Major Recall\" type=equal\n#(filter) name=Recall_After dimension=\"Report Received Date\" type=greater_than\n#(filter) name=Recall_Before dimension=\"Report Received Date\" type=less_than\nsource: recalls is duckdb.table('data/auto_recalls.csv') extend {\n measure:\n recall_count is count()\n}\n```\n\nFor date-range filters, declare two filters with distinct `name` values targeting the same dimension (one `greater_than`, one `less_than`).\n\n### When to use `required`\n\n`required` filters are a correctness, latency, and governance mechanism, not just UX. Mark a filter `required` when:\n\n1. **Modeling correctness, the source's `primary_key:` is only unique under a filter.** If a high-cardinality key is not unique across the whole table but is unique within a scoping dimension, then that scoping dimension MUST be supplied for symmetric aggregation to produce correct numbers. For example, if `events.id` repeats across days but is unique within a single `event_date`, queries that don't pin the date can fan out and return hash-collision-sized garbage (~10²¹). Declare `#(filter) name=Event_Date dimension=event_date type=equal required` so the server refuses queries that don't provide it.\n2. **Query latency, the source spans more data than any single query should scan.** A multi-year, multi-region table where every reasonable analysis is scoped to a date range or region: making the date filter required prevents accidental full-table scans.\n3. **Partial views** that are only meaningful inside a date range, region, or business segment.\n4. **Governance**, an analyst should never query the raw source without a scoping filter applied.\n\nFor (1), pair the required filter with a comment explaining the cardinality dependency, and consider also declaring `#(doc)` on the source noting the constraint.\n\n### When to use `implicit`\n\nUse `implicit` for filters the *system* must inject but users should not see. The filter applies; it just doesn't appear in the UI or API filter list.\n\n### Type-aware literals\n\nPublisher formats values based on the dimension's data type, `string` → `'value'`, `boolean` → bare `true`/`false`, `date` → `@YYYY-MM-DD`. You don't quote values yourself in the API call; Publisher handles formatting.\n\n### Bypass\n\nPass `bypass_filters=true` (REST) or `bypassFilters: true` (POST body) to skip filter injection entirely. Use sparingly, required-filter governance only works if bypass is restricted to trusted callers.\n\n## Access Control: Source Gating with `#(authorize)`\n\nGate query access to a source with `#(authorize)` over declared `given:` values (`given:` is Malloy's native runtime-parameter mechanism, the going-forward replacement for `#(filter)`). A gate is an `#(authorize)` annotation on its own line directly above the `source:` line, carrying an **unquoted, ordinary Malloy boolean expression**; Publisher grafts that expression onto the source as a row filter before running the query, so a caller it admits nowhere gets **200 with zero rows**, not a 403. A **403** means only that the gate could not be attached at all. A source with no `#(authorize)` annotation of its own or inherited is unrestricted.\n\n```malloy\n##! experimental.givens\n\ngiven:\n ROLE :: string\n\n#(authorize) $ROLE = 'analyst'\nsource: orders is duckdb.table('orders.parquet') extend {\n measure: order_count is count()\n}\n```\n\n- **Any legal Malloy boolean expression is a legal gate**, over givens, row fields (including through a join), literals, functions and operators: `org_id in $GROUPS`, `upper(region) = $REGION`, `` `cost center` in $GROUPS ``, `(org_id in $GROUPS or region = $REGION) and amount > $AMOUNTMIN`. There is no allowlist of accepted comparison shapes.\n- **A source may declare at most one `#(authorize)` annotation.** Declaring a second on the same source fails the load naming both. Spell OR inside the expression rather than stacking annotations. For a condition too long to read on one line, point the gate at an ordinary boolean dimension instead: `#(authorize) authorized` above the source, over `dimension: authorized is org_id in $GROUPS` inside it; validation follows the reference through.\n- **`#(authorize)` only gates from the `source:` line.** The same annotation on a `dimension:`/`measure:`/`join_*:`/`view:` line, or on a top-level `query:`, is refused at load naming the position rather than silently protecting nothing.\n- **Every given the gate references must be declared on the entry model's own surface, and must carry no default.** A given the model cannot resolve is refused at load. So is a referenced given declared *with* a default: a caller who supplies nothing would get that default and be admitted or excluded by a value the gate's own line never shows, so it is refused rather than reasoned about case by case. This follows a bare reference through, so a given reached one hop away via `#(authorize) authorized` is checked too.\n- **Two shapes load with a warning rather than a refusal.** A gate that references **no given** at all is a fixed predicate, not an access rule keyed on the caller. A gate that **negates a membership test** (`not (org_id in $GROUPS)`) matches every row for an *empty* given instead of none. Both warn and still load, so read the load warnings.\n- **Entry point only: not joined, but inherited through `extend`.** The gate applies to the source a query enters through. A gate on a source reached only via `join_*` **never fires**, at any depth, so anything ungated that joins a locked base hands the base's rows to every caller. A source that `extend`s a locked base and declares no gate of its own **does** carry the base's gate; declaring its own annotation replaces it. A source derived from a locked base via a query (`source: z is locked -> { … }`) instead **always carries the base's gate in addition to its own**: the derivation recurses into the base unconditionally, so an own gate does not replace it, and the two combine as separate AND'd entries. Pair a locked base (`#(authorize) false`) with curated extension sources, using access modifiers (`include { public: …, private: * }`), so an extension re-exposes only a curated column surface, and keep sensitive sources out of ungated joins.\n- **A derivation that drops a column the gate reads fails CLOSED.** `extend { except: org_id }`, or an `accept:` that omits it, leaves the grafted filter unable to compile, so the request is denied rather than served ungated. The one hole to know: dropping the gated column and then `rename:`-ing a *different* column onto that exact name grafts successfully and binds the gate to the wrong column. Narrow, but real, so don't recycle a gated column's name.\n- Comparing a row field to an array-typed given with `=`/`!=` (`org_id = $GROUPS`) compiles and loads cleanly, then fails at query execution with a warehouse conversion error. Use `in` for an array-typed given, not `=`.\n- **The quoted-string and file-level forms are refused at load and no longer exist.** `#(authorize) \"\"` on the `source:` line, in either quote (`'...'` is refused the same way), a file-level `##(authorize) \"\"` applying to every source in the file, and the earlier `internal dimension: authorized is ` form are all retired; the load names the rewrite. Every `.malloy` file in a package compiles at load and any failure aborts the package, so a retired-form gate anywhere in the package is refused. Only a declaring file *outside* the package escapes that: it loads and denies every request instead, with no compile-time hint. See your deployment's reference documentation.\n- **A gated source can be persisted, but the gating column freezes.** `storage=` and `#@ preaggregate` refuse a gated source outright; a colocated `#@ persist` is admitted when the gate is provably the entry point's own row filter. The gate still runs live on every query, so rows come back filtered - but the column it filters ON is frozen at build time, so a row whose access decision changes keeps being served under the old decision until the next rebuild. Pair `#@ persist` on a gated source with a freshness window (`fallback=\"live\"`), which is the only control that bounds that - and read `skill:malloy-materialization` for where that window binds, because on a standalone Publisher it does not.\n\n> **Trust caveat.** Givens are **caller-asserted**, anyone who can reach the query API can claim a favorable given, e.g. `{\"ROLE\":\"admin\"}`. `#(authorize)` is only a real boundary when it sits behind a trusted tier that sets givens from its own verified context, never directly from an untrusted caller. It is not, on its own, end-user authentication.\n>\n> **Forward direction.** Givens are how access control is built here, and the planned next step is **identity-bound (\"secure\") givens** - reserved values a trusted tier populates from a verified token or proxy header, which the caller cannot override - turning `#(authorize)` into a standalone boundary. Model access on `given:` + `#(authorize)` now; it is the surface that carries forward.\n\nFull syntax, inheritance rules, validation, and the error contract are covered in your deployment's `#(authorize)` reference documentation.\n\n## Join Syntax\n\n- Simple join: `join_one: users with user_id`\n- Expression join: `join_one: origin is airports on origin_code = origin.code`\n- Composite key: `join_one: items on order_id = items.order_id and product_id = items.product_id`\n- Multiple joins to same table: `join_one: origin_airport is airports with origin`\n\n**Join Types:** `join_one:` (many-to-one, efficient) | `join_many:` (one-to-many, always safe) | `join_cross:` (many-to-many)\n\n**Verify cardinality** before writing joins: `run: target -> { group_by: fk_col, aggregate: n is count(), having: n > 1, limit: 5 }`. 0 results → `join_one`. Any results → `join_many`.\n\n## After Writing: Check & Review\n\nCheck diagnostics after writing. Errors cascade, fix the FIRST error only, then re-check. If errors persist, use the debugging strategy: look at first error, search docs if unsure, fix, repeat.\n\n**Validate with `execute_query`:** Run queries, check distributions, verify measures, confirm joins (no fan-out).\n\nTo inspect the sources and fields a model already defines, ground yourself with `get_context`. It returns the package's sources, views, and fields, so there is no separate schema-search step. When you're unsure of Malloy syntax, call `search_malloy_docs` rather than guessing.\n\n## Advanced Patterns\n\nLoad the relevant reference file when you encounter these scenarios:\n\n| Scenario | Read |\n|----------|------|\n| Need pre-aggregated or windowed source | `reference/query-sources.md` |\n| Curating access modifiers | `reference/access-modifiers.md` |\n| Normalized/ER-style schema (4+ tables, no clear fact table) | `reference/normalized-schemas.md` |\n| Formalizing analysis into a model | `reference/analysis-to-model.md` |\n| Many-to-many / bridge tables / composite keys | `reference/bridge-tables.md` |\n\n## Done\n\nStep complete. Output: base source files (`.malloy`, one per table) and joined source files (`.malloy`, one per analytical domain).\n\n**Suggest next steps to the user:**\n\n- Open the model in the browser to see it live: `http://localhost:4000//` for the package, or `http://localhost:4000///` for a single model file. First confirm the running server actually serves this package (it is in the loaded `publisher.config.json`, or mounted live with `--server_root . --watch-env `); a package the server has not loaded returns a 404, so do not hand over a link to a package that was just authored but never loaded.\n- Build a notebook with interactive filters over the model (see `skill:malloy-notebooks`).\n- Run analysis questions against the model (see `skill:malloy-analysis`).\n- When you're ready to serve the model, publishing is out of scope for open-source Publisher v1: self-hosters commit the package to git and use their host's publish path.\n\n## Reference files over MCP\n\nThis skill's `reference/` files are served as separate prompts, one per file, fetched only when you ask for them. Where the text above says to read `reference/.md`, get the prompt named `malloy-model/` instead.\n\nAvailable: access-modifiers, analysis-to-model, bridge-tables, normalized-schemas, query-sources." + }, + { + "name": "malloy-model-as-you-go", + "description": "After answering a data question, write down what the answer assumed so the next reader can trust the number. A field with a #(doc) in the model when you can edit it, an extend in the notebook when you can only author reports, or a stated assumption plus a Malloy snippet when you can only chat. Use after every answered question that rested on a judgment call, and whenever a question is asked against tables that have no model yet.", + "body": "# Model as you go\n\nAn analysis that lives in a chat transcript is not reproducible. The numbers were right, and\nsix weeks later nobody can say what \"revenue\" excluded, which of four timestamps was the order\ndate, or whether the last period was complete. The work is unauditable, so it gets redone.\n\nThe fix is to write the assumptions down while the query is still in front of you, in the\nmost durable place your session can write. Answer the question, codify what the answer\nassumed, answer the next one. After a handful of questions there is a model, or a notebook,\nwhere every definition exists because a real question needed it, and every judgment call is\non the record.\n\nThis skill is the codify step. `skill:malloy-analysis` answers the question; this skill\ndecides what to write down afterwards, and where.\n\n> **Tool names** are bare here - `get_context`, `execute_query`, `search_database_schema`. The\n> exact prefixed name depends on the host; match against the tools you actually have.\n\n## The loop\n\n```\nQUESTION → ANSWER → CODIFY ⟲\n ↑___________|\n```\n\n**The user gets a real answer on question one.** If three tool calls have gone by without\nproducing an insight they can read, you have drifted into modelling for its own sake. Stop and\nanswer something.\n\n## 1. Answer the question, with `skill:malloy-analysis`\n\nThis skill starts when someone asks a data question. **That question is the unit of work, and\nit is theirs.** Do not widen it into a modelling project, and do not swap it for a more\ninteresting one you found on the way.\n\n**A broad ask is still an ask.** \"Analyse the sales data\" is a question whose subject is given\nand whose metric is not. Pick the most obvious question about that subject, say in one line\nwhich one you picked, and answer it. Never open with a row-count table and a menu of\n\"analytical domains\"; the menu is worth less than the first real answer would have been.\n\nLoad `skill:malloy-analysis` and follow it: discover the model, construct the query, run it,\nverify it, present it. Two of its rules matter most here:\n\n- **Every number you present comes out of a query.** Adding up the rows of a `limit: 15` table\n by hand drops everything below the cut, and nobody can re-run it.\n- **Your first result is a draft.** Work `skill:malloy-analysis-pitfalls` before presenting.\n\n### Name the decisions the answer rests on\n\n**This is what makes the analysis auditable, and it is the step most easily skipped.**\n\nAlmost every query needs at least one judgment call the data cannot settle. Which of four\ntimestamps is \"the order date\". Whether returns count as revenue. Whether to count lines or\norders. Whether a partial final period belongs in a trend.\n\nA judgment call made silently becomes a hidden assumption. It will be wrong for someone\neventually, and by then nobody remembers it was a choice rather than a fact. So make the call,\nrun with it, and **say what you chose and what you rejected**, with the other number attached\nwhenever it is cheap to get:\n\n> Revenue here excludes cancelled **and** returned orders - that's $8.10M. Counting returns as\n> revenue and netting them separately gives $9.18M. I went with the stricter one; say if your\n> reporting does it the other way.\n\nNot every choice rises to this. Raise it when **any** of these holds:\n\n| Raise it when | Because |\n|---|---|\n| The alternative changes the number materially | The user would answer differently depending on which they meant |\n| You are about to codify it | It stops being your choice and becomes everyone's definition |\n| A reasonable analyst would pick the other one | It is a convention, not a fact |\n\nOtherwise state it in a clause and move on. **Asking about everything is as bad as asking about\nnothing**; it turns the loop into a form and trains the reader to skim past it. And **give the\nnumber under each option, not an abstract question**: \"$8.10M excluding them, $9.18M\nincluding\" lets the user answer in one word.\n\n## 2. When there is no model yet\n\nIf the tables have no model, define only enough to run the first query. One line is normal:\n\n```malloy\nsource: order_items is my_conn.table('ecommerce.order_items')\n```\n\n**Read the columns with `run: source -> { select: * limit: 3 }`.** Prefer it to a schema\nlisting: it returns the columns *and* real values, and the values are what catch the surprises\na column list hides - a date stored as a string, a `total` that excludes tax, a metric column\nthat is null on every row.\n\nIf a schema tool returns something you cannot explain (no columns, or no tables for a filter\nyou can see matches), that is a bug in the tool. Report it. Do not write the workaround into\nyour model or your notes as though it were a property of the data.\n\nNo `primary_key:`, no dimensions, no measures, no joins, **not yet**. Those arrive in CODIFY,\neach one paid for by a question that needed it. Adding fields because the table has them is the\nhabit this skill exists to break.\n\nAdd a join only when *this* question cannot be answered without it, then verify its cardinality\nbefore trusting any aggregate (`group_by: fk, aggregate: n is count(), having: n > 1`).\n\nIf a model already exists, ground yourself in it with `get_context` and reuse what is there.\n\n## 3. CODIFY: write the answer's assumptions down\n\nRun this after *every* answered question, while the query is still in front of you. Skipping it\nis how a session ends with a transcript and nothing else.\n\n### Where it goes depends on what you can write\n\nThe same assumption lands in a different place depending on the session. Pick the **highest\nrung your tools allow**, and never skip codifying because the top rung is out of reach.\n\n| You can | Codify as | Who inherits it |\n|---|---|---|\n| **Edit the model files** (a local package, a draft package, a compile or reload tool) | A `dimension:`, `measure:`, `join_*:`, or `view:` with a `#(doc)`, in the `.malloy` file | Everyone who queries the model. **Confirm binding decisions first** (below) |\n| **Author notebooks or reports, but not the model** (a viewer of a published package) | In the notebook: `source: orders_q is orders extend { measure: ... }` with the `#(doc)` above it, plus a markdown cell stating the assumption. A question worth re-asking becomes a cell | Readers of the report. State the decision in the cell, and say which definitions the model should adopt |\n| **Only answer in chat** (no file or report tools) | The assumption stated in the answer, plus the Malloy snippet a modeler could paste: the `measure:` with its `#(doc)` | Nobody, until someone acts on it. That is why the snippet matters |\n\nTell the rungs apart by the tools you have, not by guessing at the user's role: a file-write or\ncompile tool means the top rung; a report or notebook tool without model edits means the middle\none; neither means the bottom.\n\n### What to codify\n\nLook at the query you just ran. For each dimension, measure, join, filter, **and for the shape\nof the query itself**, ask **in this order**:\n\n| Codify it when | Example |\n|---|---|\n| **A reader needs it to trust the number** | the cancelled/returned exclusion - without it every revenue figure reads high |\n| **It encodes a business rule** | a regex parsing a messy column, a status mapping, tier cutoffs |\n| **Another question would reuse it** | `revenue is sum(sale_price)` - everything about orders needs it |\n| **It was hard to get right** | a window function, a multi-step derivation, a verified join |\n\nThe first row is the one that matters. A definition is worth keeping less because it saves\ntyping than because it stops the next reader misreading the number.\n\n**Codify thin, and leave the ad-hoc behind**: filters tied to one finding, calculations that\nanswered exactly one question. A question that needed one measure codifies one measure, not the\nneighbouring columns as well. An empty CODIFY is a fine outcome; say so and move on.\n\n**Then say what you did**, in one line, every time:\n\n> Codified: `revenue is sum(sale_price)` excluding cancelled and returned, `category` via the\n> products join. Left ad-hoc: the `where: created_at > @2023` - just this question's window.\n\nThat line shows the model growing and gives the reader a place to object. Never codify\nsilently.\n\n### A decision becomes binding the moment it enters the model\n\nWhile a judgment call lives in one ad-hoc query it is yours, and stating it is enough. Once it\nis in the model file it is the definition everyone inherits, and nobody downstream sees the\nreasoning. So on the top rung, **stop, ask, and wait for an answer** before writing it down.\nReporting it afterwards (\"I used the midpoint, say if you'd rather have the floor\") is not\nconfirming it; by then it is already the default. On the middle rung, confirm when the report\nwill be shared; a private notebook is still yours.\n\nThis is not only about measures. Confirm anything that changes what later questions return:\n\n| Confirm before codifying | Because it silently sets |\n|---|---|\n| A source-level `where:` | the scope of *every* later question against that source |\n| A `measure:` definition | the default value of that metric for everyone |\n| A `dimension:` that buckets, maps, or parses | which rows land in which group |\n| A `join_one:`/`join_many:` and its grain | whether aggregates fan out |\n| A saved `view:` | the shape people will re-run and cite |\n\nAsk with the numbers attached:\n\n> I want to make `net_revenue` exclude cancelled and returned. That makes it the default revenue\n> number for every later question - $8.10M rather than $10.81M gross. Good, or does your\n> reporting treat returns differently?\n\n**One confirmation per decision, not per question.** Once the user has settled how revenue\ntreats returns, it is settled: reuse it and stop asking. And if the user has told you to stop\nchecking in, believe them; state each decision in a clause and keep going.\n\n### Write it down twice\n\n**In the model (or the notebook), as a `#(doc)` on the definition.** This is the part that\nsurvives. Use `#(doc)`, never a `//` comment, for anything a consumer needs: `#(doc)` is\nmachine-readable, so it renders in the UI and retrieval tools read it, while a `//` comment\nreaches nobody but whoever opens the file. Keep `//` for maintainer-only notes. Tag formats on\nthe definition too: bare `# currency` on the measure, any scale (`# currency=usd0m`) only in\nviews.\n\n```malloy\n#(doc) Order line items joined to products. Grain is one row per line, not per order.\nsource: order_items is my_conn.table('ecommerce.order_items') extend {\n join_one: products is my_conn.table('ecommerce.products') on product_id = products.id\n\n dimension:\n #(doc) Canonical order date. Data runs 2019-01-05 to 2026-03-14, so the final year is PARTIAL - never present it as a full period.\n order_date is created_at::date\n\n measure:\n #(doc) Revenue in USD, excluding cancelled and returned orders. Those are 25% of gross, so excluding them is not optional.\n # currency\n net_revenue is sum(sale_price) { where: status != 'Cancelled' and status != 'Returned' }\n\n #(doc) Monthly revenue trend. The question asked on 2026-03-02.\n # line_chart\n view: revenue_trend is {\n group_by: order_month is order_date.month\n aggregate:\n # currency=usd0m\n net_revenue\n order_by: order_month\n }\n}\n```\n\nOne file per analytical domain, named for it: `order_revenue.malloy`, not `model.malloy`. It\ngrows monotonically across the session.\n\n**In a notes file, as the reasoning.** Keep an `analysis-notes.md` beside the model (or a\nmarkdown cell in the notebook) recording each question, what it found, what got codified and\nwhat was left ad-hoc, and every verification finding. The model carries the *what*; the notes\ncarry the *why* and the evidence. It is also what lets you resume after losing context.\n\n### Save the view when a question is worth re-asking\n\nA saved `view:` turns \"we answered that once\" into \"re-run it\". A trend wanted again next\nmonth belongs in the file as a `view:` with its chart tag (`skill:malloy-charts`); views wanted\nside by side belong in a notebook (`skill:malloy-notebooks`), or in a dashboard surface if your\nhost has one. A genuine one-off does not.\n\n> **This departs from `skill:malloy-model` on purpose.** Its \"no views in source files\" rule\n> assumes a schema-first model, written before anyone asked a question, so its views would be\n> guesses. Here every view is a question that was asked and verified. Two more of its rules do\n> not apply either: skip access modifiers and curation (there is no discovery surface to curate\n> when every field was paid for by a question), and keep one domain file rather than one file\n> per table until it genuinely gets unwieldy. Everything else in `skill:malloy-model` applies:\n> `#(doc)` on every field, verified join cardinality, `nullif` on division, a `given:` for a\n> runtime parameter.\n\n### Then loop\n\nGo back to the question, and let what you just found sharpen the next one:\n\n> Revenue is concentrated in three categories. Worth asking whether that's new - want the same\n> cut by year?\n\n## When the analysis is going to be shared\n\nDocs are not on this list; they happen in CODIFY, as you go. When the user wants to hand it\nover:\n\n1. **Names.** Rename anything whose name only made sense inside one question.\n2. **Doc pass.** Re-read the `#(doc)` lines for anything that drifted as the model grew. Check\n grain, units, and null handling are each stated somewhere.\n3. **Re-verify.** Re-run the session's key queries against the finished sources. If a number\n moved, something was codified wrong. This is the check that proves reproducibility; do not\n skip it.\n4. **Structure**, only if one file has genuinely become unwieldy: base sources per table plus a\n joined source per domain, per `skill:malloy-model`. Being shared is not itself a reason.\n\n## When to do something else\n\n| Situation | Go to |\n|---|---|\n| Porting prior art (LookML, dbt, a metrics doc): the definitions exist and are agreed, the job is translation | `skill:malloy-lookml-review`, then `skill:malloy-model` |\n| The user names the sources they want built outright, before any question | `skill:malloy-model` |\n| A model already exists, the question rests on no judgment call, and nothing is worth keeping | `skill:malloy-analysis` alone |\n| Open-ended exploration with no intent to keep anything | `skill:malloy-analyze` |\n\n## Anti-patterns\n\n```\nWRONG Turn \"what's our default rate?\" into a modelling project\nRIGHT Answer it, then codify what the answer assumed\n\nWRONG Propose every dimension and measure for each table in scope\nRIGHT Codify the two fields this question actually needed\n\nWRONG Quietly pick one of four timestamps as \"the order date\"\nRIGHT \"Using created_at; shipped_at would drop 38% as nulls. OK?\"\n\nWRONG Skip codifying because you cannot edit the model\nRIGHT Put the extend and its #(doc) in the notebook, or hand over the snippet\n\nWRONG Codify the source-level where:, then mention it in the write-up\nRIGHT Stop and confirm it; it scopes every question anyone asks later\n\nWRONG The assumption lives in the chat transcript, or in a // comment\nRIGHT #(doc) on the definition for what a consumer needs, plus a line in the notes\n\nWRONG Leave the answered question as a transcript table and move on\nRIGHT Save it as a view; a question worth answering is usually worth re-asking\n\nWRONG Curate access modifiers and split one file per table to \"do it properly\"\nRIGHT Skip both; they solve a schema-first problem this model does not have\n```" }, { "name": "malloy-model/access-modifiers", @@ -188,7 +193,7 @@ { "name": "malloy-modeling", "description": "Build semantic models with Malloy for the Malloy Publisher. Read this skill whenever the user asks about modeling data or specifically mentions Malloy.", - "body": "# STOP - READ BEFORE WRITING ANY MALLOY CODE\n\n> **AI AGENTS: You MUST review this file before writing Malloy code.** Cross-skill references below use logical `skill:` names; load the referenced skill before acting. Before writing code, also read the gotcha skills: `skill:malloy-gotchas-modeling`, `skill:malloy-gotchas-queries`, and `skill:malloy-gotchas-rendering`.\n\n## Pre-Flight Checklist\n\n1. **Discover first**: ground yourself before writing ANY code, with the tool that matches what you are modelling.\n - Modelling data **already in a package**: `malloy_getContext` returns that package's sources, views, and fields (with their docs).\n - Modelling **a database with no package yet**: `malloy_getContext` has nothing to return, so use `malloy_searchDatabaseSchema` instead. It walks the connection's schemas and tables, ranks them against a plain-English description, and gives you each table's columns plus the `source:` line to start from. Take those names verbatim into step 5.\n Never guess field names either way.\n2. **Search docs proactively**: call `malloy_searchDocs` BEFORE writing unfamiliar patterns (window functions, query-based sources, pipelines). Don't guess. Malloy syntax is specific and SQL intuition is often wrong.\n3. **Use `skill:malloy-patterns`** to discover available doc topics (YoY, cohorts, rendering, window functions).\n4. **Check diagnostics** after writing: fix the FIRST error first, errors cascade.\n5. **Read the gotcha skills**: `skill:malloy-gotchas-modeling`, `skill:malloy-gotchas-queries`, and `skill:malloy-gotchas-rendering` prevent the most common mistakes.\n\n**Quick syntax reminders:**\n1. **Backtick reserved words:** `` `Date` ``, `` `Hour` ``, `` `Timestamp` ``, `` `Type` ``, `` `number` ``, `` `source` ``\n2. **Use `having:` for aggregate filters**: not `where:` on measures\n3. **Alias joined fields in `group_by`** if using them in `order_by`\n4. **`count()` counts rows; `count(x)` counts distinct values of `x`**: `count(distinct x)` is deprecated, write `count(x)`\n5. **One tag per line**: `# label=\"Revenue\"` and `# currency` on separate lines\n6. **No fixed scale on measures**: use `# currency` not `# currency=usd0m`\n7. **Cast strings for aggregates:** `avg(score::number)` not `avg(score)`\n8. **Boolean columns:** use `= true` not `= 'true'` (no quotes!)\n9. **Read data files in place:** `.csv`, `.parquet`, `.json`, `.ndjson`, and `.xlsx` all work as-is through `duckdb.table('data/file.ext')`. Never convert a file to another format first, and never read one with python or jq to \"have a look\" first: query it. For `.xlsx`, check the row count before trusting it: a workbook with a title row or a blank spacer reads short and reports no error. (Per-format quirks: `skill:malloy-gotchas-modeling`)\n\n## Planning and `modeling-notes.md`\n\nIf the IDE has a native plan mode, use it for the high-level approach: do data exploration during planning, then present a concrete plan for user approval before writing any files.\n\n`modeling-notes.md` is an expected output of the workflow, not an optional extra. Start it at step 2 (Propose Scope) and grow it as you work: it persists alongside the model, and its value is as the thing the user argues with at step 3, before source files exist; written after the build it can only document decisions already baked in. Record findings and problems as they are found during discovery (`skill:malloy-discover`), and every unconfirmed decision as an open item. Only when there is no writable workspace do the notes live in the conversation instead.\n\nKeep it compact, with these sections:\n\n```markdown\n# Modeling notes - \n## Scope what was confirmed, what the model is FOR, skip list with reasons\n## Grain and keys proven by query, not by column name\n## Coverage coverage cliffs; columns excluded for nullity\n## Decisions each with its evidence\n## Open decisions ASSUMPTIONS, NOT CONFIRMED: every threshold or definition the user\n has not settled, one entry each, mirrored by a hedge in its #(doc)\n## Validation reconciliation checks performed, and their results\n```\n\n## 8-Step Modeling Workflow\n\nThe agent orchestrates all steps. Steps marked **(user)** pause for input. Each step has a dedicated skill with full instructions. Read each step's skill **before starting that step**, including the decision skills for steps 1–4 (`skill:malloy-discover`, `skill:malloy-scope`, `skill:malloy-define`). They govern what the model says; skipping them to reach the build skills is how unreviewed business logic ships.\n\n**A field is not complete until it has its definition, `#(doc)` tag, and rendering tags, and any threshold or business convention in it is user-confirmed, distribution-derived, or explicitly flagged in its `#(doc)`** (see `skill:malloy-document` § Mark conventions as conventions). Documentation is part of defining a field, not a separate activity. Read `skill:malloy-document` for full documentation standards (doc string writing, tag ordering).\n\n```\nDISCOVER → SCOPE → SOURCES → DEFINITIONS → BUILD BASE → BUILD JOINED → REVIEW → CURATE\n (silent) (user) (user) (user) (agent) (agent) (user) (user)\n```\n\n| Step | Skill | What Happens |\n|------|-------|-------------|\n| 1. Discover | `skill:malloy-discover` | Read the model and data; scan sources, fields, distributions; detect prior art. With no package yet, start from `malloy_searchDatabaseSchema` to find the tables in the connection |\n| 2. Propose Scope | `skill:malloy-scope` | Present findings, user selects focus |\n| 3. Propose Sources | `skill:malloy-define` | Propose source plan, user confirms architecture |\n| 4. Propose Definitions | `skill:malloy-define` | Propose fields per base source, user confirms logic |\n| 5. Build Base Sources | `skill:malloy-model` | Write fully documented base source files (one per table), check diagnostics. Read `skill:malloy-document` for doc standards. |\n| 6. Build Joined Sources | `skill:malloy-model` | Write fully documented joined source files, validate. Read `skill:malloy-document` for doc standards. |\n| 7. Review | (none) | Present the review checklist below; user confirms or corrects |\n| 8. Curate | `skill:malloy-model` | Propose access controls (`explores`, `queryableSources`, access modifiers); always propose, the user decides whether to apply |\n\n### The pauses are the point\n\nThese are governed semantic models: the business decisions in them must be confirmed by a human subject-matter expert, and the **(user)** steps exist to collect that confirmation. They are real stops, not progress reports. A model can be complete, compiling, and fully documented and still be wrong everywhere it guessed; a capable agent can build the whole thing without pausing once, which is exactly the failure mode this workflow exists to prevent.\n\nWhen a decision goes unanswered (the user explicitly declines to decide, or nobody is there to ask), do not silently proceed as if it were settled. Take your best-supported position, label it an assumption in the field's own `#(doc)` (see `skill:malloy-document` § Mark conventions as conventions), record it under \"Open decisions\" in `modeling-notes.md`, and raise it again at Review. An unlabeled assumption is indistinguishable from a confirmed fact, and misleads everyone downstream.\n\n### Step 7 Review is a checklist, not a summary\n\nPresent these to the user, with answers:\n\n- **Which definitions did the user actually confirm?** List them; everything else is an assumption.\n- **Which thresholds and bucket boundaries did you choose?** For each: the evidence (distribution query, metadata, prior art) and the `#(doc)` hedge that marks it.\n- **Which questions were left unanswered?** Each must already carry a labeled assumption and an \"Open decisions\" entry.\n- **Does the headline metric have more than one defensible definition?** If yes, that is a blocking question: put the candidate definitions to the user with their counts side by side, not in a footnote.\n\nThe user confirming this checklist is what makes the model governed. A summary of what you built is not a checkpoint.\n\nPublishing is out of scope for open-source v1. Self-hosters move a finished model into a served package via git and the host's publish path; see `skill:malloy-publish` for the local-to-served handoff.\n\n**Two paths to a model: both produce the same fully documented result:**\n- **Schema-first:** \"Model my data\" → 8-step workflow above using the relevant skills\n- **Analysis-first:** \"Explore this data\" → `skill:malloy-analyze` → formalize via `skill:malloy-model` (`reference/analysis-to-model.md`)\n\nAfter analysis completes, **always recommend formalizing into a model.**\n\n## Agent Behavior\n\n**Research before asking.** Present proposals with evidence. Never ask open-ended questions: propose with data and let the user confirm.\n\n**Use business language.** Say \"I simplified the column name\" not \"reserved word replaced.\" Don't expose Malloy internals unless the user asks.\n\n**Describe what you're doing, not which step you're on.** The user doesn't have the skill files open. Say \"I'll propose which tables to include and how they relate\" not \"Steps 3 and 4.\" Say \"Now I'll write the source files\" not \"Moving to Step 5.\" Explain the purpose of each phase in plain language before doing it.\n\n**Present choices as A/B/C.** When asking the user to choose, use lettered options with one-line descriptions. Mark your recommendation.\n\n**Complete all workflow steps.** Once modeling begins, complete through Review and propose Curate. A field without documentation is not finished. If you lose track, re-read the model and your notes. Suggest notebooks at the end.\n\n## Route by Intent\n\n| User says... | Route to |\n|-------------|----------|\n| \"Model my data\", \"create a model\" | 8-step workflow (`skill:malloy-discover`) |\n| \"Model from LookML\" | 8-step with prior art via `skill:malloy-lookml-review` |\n| \"Explore this data\", \"what's interesting?\", \"show me the top X\" | `skill:malloy-analyze` (EDA) |\n| \"Build a dashboard\", \"create views\" on existing model | `skill:malloy-analyze` (views), plus `skill:malloy-charts` or `skill:malloy-notebooks` as needed |\n| \"Build a model but not sure what metrics\" | `skill:malloy-analyze` first, then formalize via `skill:malloy-model` |\n\n**If the user's first message is a data question** (not \"build me a model\"), route to `skill:malloy-analyze`. After analysis completes, **always recommend formalizing via the analysis-to-model workflow** (`skill:malloy-model` → `reference/analysis-to-model.md`).\n\n## Additional Support Skills\n\nThese supplemental skills may also be loaded as needed:\n\n- **`skill:malloy`**: Index of Malloy skills and routing guide\n- **`skill:malloy-debug`**: Fix compile errors and interpret diagnostics\n\n## Publisher MCP Tools\n\nEnsure the Publisher MCP tools are configured before modeling. No server yet? `skill:malloy-getting-started` covers setup, including the one-command scaffolder (`npm create @malloy-publisher/malloy-package@latest `) and why local authoring needs `--watch-env `: start the server without it and your saved edits are never read.\n\n| Tool | Purpose |\n|------|---------|\n| `malloy_getContext` | Ground yourself in a package: its sources, views, and fields |\n| `malloy_executeQuery` | Run ad-hoc queries for validation |\n| `malloy_compile` | Compile-check a change and get diagnostics back without running a query |\n| `malloy_reloadPackage` | Recompile a package from disk so a saved edit becomes queryable by name |\n| `malloy_searchDocs` | Search Malloy docs (call BEFORE unfamiliar patterns) |\n| `malloy_searchDatabaseSchema` | Find the tables in a database connection by plain-English description, when modelling data that is not in a package yet. Returns each table's columns and the `source:` line to start from. Names and types only: no row value is returned |\n\nNever guess field names. Ground yourself with `malloy_getContext` to see the sources and fields a package defines.\n\n### The edit-and-run loop\n\nPublisher compiles each configured package at boot and serves that cached model, so a source or view you add afterwards is not queryable by name until you reload the package. The loop is:\n\n1. **Validate** the change with `malloy_compile`, picking the scope that matches what you are doing:\n - Adding a new definition or query: the default (`scope: \"append\"`) compiles your text in the model's namespace. Note its diagnostic positions land in the model-plus-your-text concatenation.\n - **Editing an existing definition: `scope: \"file\"`**, with the whole edited file as `source`. It compiles your text AS the file (append would collide with \"Cannot redefine\"), and diagnostics land at the true line numbers of your text.\n - Before saving a change other files import: `scope: \"package\"` with the edited file as `source` runs reload's worker compiler over every `.malloy` and `.malloynb` file against your edit, so a rename that breaks an importer surfaces now instead of at reload. Each diagnostic carries `model`, the file it points at; files hidden from discovery can appear. If `modelPath` does not exactly match an existing file, a warning says the source was treated as new.\n2. **Save** it to the package's model file.\n3. **Reload** with `malloy_reloadPackage`.\n4. **Run** the new view with `malloy_executeQuery`.\n\nA reload that fails to compile is safe: your files are left alone and the previously compiled model keeps serving, with the compile errors returned to you. Compile first anyway for faster feedback, and a `scope: \"package\"` dry-run with no `source` uses reload's compiler and file selection (imports across files, every `.malloy` and `.malloynb` file as saved) without touching the served model. Keep the source of truth outside `publisher_data/`, which is not version-controlled and is wiped by a `--init` restart. If these tools are missing, the Publisher you are connected to predates them; fall back to validating with a throwaway `malloy_executeQuery`. An older Publisher that has `malloy_compile` but rejects `scope` supports only the append behavior.\n\n## SQL-to-Malloy Quick Reference\n\n| SQL | Malloy |\n|-----|--------|\n| `COUNT(*)` | `count()` |\n| `COUNT(DISTINCT x)` | `count(x)` |\n| `NOW()` | `now` |\n| `CASE WHEN...END` | `pick...when...else` |\n| `col IN ('a','b')` | `col ? 'a' \\| 'b'` |\n| `COALESCE(a,b)` | `a ?? b` |\n| `CAST(x AS type)` | `x::type` |\n| `DATEDIFF(day, a, b)` | `days(a to b)` |\n| `CONCAT(a, b)` or `a \\|\\| b` | `concat(a, b)` |\n| `TIMESTAMP_DIFF(a, b, SECOND)` | `seconds(b to a)` |\n\n## Critical Rules\n\n1. **All keywords require colons**: `source:`, `dimension:`, `measure:`, `view:`\n2. **Use `is` not `as`**: `dimension: name is expression`\n3. **Arrow operator required**: `run: source -> { operations }`\n4. **Specify join type**: `join_one:`, `join_many:`, `join_cross:`\n5. **Safe division**: `revenue / nullif(count, 0)`\n6. **Group definitions under one keyword**: `measure:` then indent fields beneath\n\n## Common Anti-Patterns\n\n```\nWRONG: source flights is ... RIGHT: source: flights is ...\nWRONG: dimension: x as y RIGHT: dimension: y is x\nWRONG: count(*) RIGHT: count()\nWRONG: count(distinct x) RIGHT: count(x)\nWRONG: revenue / order_count RIGHT: revenue / nullif(order_count, 0)\nWRONG: run: src { ... } RIGHT: run: src -> { ... }\n```\n\n## Reserved Words: Scan Schema First\n\n**Malloy has many reserved words. When in doubt, backtick it.** Most likely to appear as column names:\n\n```\ndate, time, day, month, year, quarter, week, hour, minute, second,\nnumber, string, boolean, type, table, source, index, count, sum, avg, min, max,\ntrue, false, null, is, on, with, all, from, by, in, to, for, select, order_by,\ntop, bottom, desc, asc, row, range, current, window, rank\n```\n\n- `number`: only the bare word needs backticking; `account_number` is fine\n- `source`: reserved; use a different alias like `traffic_source`\n- `string`, `boolean`, `true`, `false`: backtick any column with these exact names\n\n## Gotcha Skills: Read Before Writing Code\n\nThe following skills contain detailed WRONG/RIGHT patterns that prevent the most common Malloy errors. **Read them before writing code:**\n\n- **`skill:malloy-gotchas-modeling`**: Reserved words, NULL checks, date functions, type casts, rename pitfalls, query-based source gotchas, `conn.sql()` anti-pattern\n- **`skill:malloy-gotchas-queries`**: Chart constraints, aggregate filters, joined field aliasing, time truncation vs extraction\n- **`skill:malloy-gotchas-rendering`**: Tag syntax, scale rules, sparkline setup, big_value patterns" + "body": "# STOP - READ BEFORE WRITING ANY MALLOY CODE\n\n> **AI AGENTS: You MUST review this file before writing Malloy code.** Cross-skill references below use logical `skill:` names; load the referenced skill before acting. Before writing code, also read the gotcha skills: `skill:malloy-gotchas-modeling`, `skill:malloy-gotchas-queries`, and `skill:malloy-gotchas-rendering`.\n\n## Pre-Flight Checklist\n\n1. **Discover first**: ground yourself before writing ANY code, with the tool that matches what you are modelling.\n - Modelling data **already in a package**: `malloy_getContext` returns that package's sources, views, and fields (with their docs).\n - Modelling **a database with no package yet**: `malloy_getContext` has nothing to return, so use `malloy_searchDatabaseSchema` instead. It walks the connection's schemas and tables, ranks them against a plain-English description, and gives you each table's columns plus the `source:` line to start from. Take those names verbatim into step 5.\n Never guess field names either way.\n2. **Search docs proactively**: call `malloy_searchDocs` BEFORE writing unfamiliar patterns (window functions, query-based sources, pipelines). Don't guess. Malloy syntax is specific and SQL intuition is often wrong.\n3. **Use `skill:malloy-patterns`** to discover available doc topics (YoY, cohorts, rendering, window functions).\n4. **Check diagnostics** after writing: fix the FIRST error first, errors cascade.\n5. **Read the gotcha skills**: `skill:malloy-gotchas-modeling`, `skill:malloy-gotchas-queries`, and `skill:malloy-gotchas-rendering` prevent the most common mistakes.\n\n**Quick syntax reminders:**\n1. **Backtick reserved words:** `` `Date` ``, `` `Hour` ``, `` `Timestamp` ``, `` `Type` ``, `` `number` ``, `` `source` ``\n2. **Use `having:` for aggregate filters**: not `where:` on measures\n3. **Alias joined fields in `group_by`** if using them in `order_by`\n4. **`count()` counts rows; `count(x)` counts distinct values of `x`**: `count(distinct x)` is deprecated, write `count(x)`\n5. **One tag per line**: `# label=\"Revenue\"` and `# currency` on separate lines\n6. **No fixed scale on measures**: use `# currency` not `# currency=usd0m`\n7. **Cast strings for aggregates:** `avg(score::number)` not `avg(score)`\n8. **Boolean columns:** use `= true` not `= 'true'` (no quotes!)\n9. **Read data files in place:** `.csv`, `.parquet`, `.json`, `.ndjson`, and `.xlsx` all work as-is through `duckdb.table('data/file.ext')`. Never convert a file to another format first, and never read one with python or jq to \"have a look\" first: query it. For `.xlsx`, check the row count before trusting it: a workbook with a title row or a blank spacer reads short and reports no error. (Per-format quirks: `skill:malloy-gotchas-modeling`)\n\n## Planning and `modeling-notes.md`\n\nIf the IDE has a native plan mode, use it for the high-level approach: do data exploration during planning, then present a concrete plan for user approval before writing any files.\n\n`modeling-notes.md` is an expected output of the workflow, not an optional extra. Start it at step 2 (Propose Scope) and grow it as you work: it persists alongside the model, and its value is as the thing the user argues with at step 3, before source files exist; written after the build it can only document decisions already baked in. Record findings and problems as they are found during discovery (`skill:malloy-discover`), and every unconfirmed decision as an open item. Only when there is no writable workspace do the notes live in the conversation instead.\n\nKeep it compact, with these sections:\n\n```markdown\n# Modeling notes - \n## Scope what was confirmed, what the model is FOR, skip list with reasons\n## Grain and keys proven by query, not by column name\n## Coverage coverage cliffs; columns excluded for nullity\n## Decisions each with its evidence\n## Open decisions ASSUMPTIONS, NOT CONFIRMED: every threshold or definition the user\n has not settled, one entry each, mirrored by a hedge in its #(doc)\n## Validation reconciliation checks performed, and their results\n```\n\n## 8-Step Modeling Workflow\n\nThe agent orchestrates all steps. Steps marked **(user)** pause for input. Each step has a dedicated skill with full instructions. Read each step's skill **before starting that step**, including the decision skills for steps 1–4 (`skill:malloy-discover`, `skill:malloy-scope`, `skill:malloy-define`). They govern what the model says; skipping them to reach the build skills is how unreviewed business logic ships.\n\n**A field is not complete until it has its definition, `#(doc)` tag, and rendering tags, and any threshold or business convention in it is user-confirmed, distribution-derived, or explicitly flagged in its `#(doc)`** (see `skill:malloy-document` § Mark conventions as conventions). Documentation is part of defining a field, not a separate activity. Read `skill:malloy-document` for full documentation standards (doc string writing, tag ordering).\n\n```\nDISCOVER → SCOPE → SOURCES → DEFINITIONS → BUILD BASE → BUILD JOINED → REVIEW → CURATE\n (silent) (user) (user) (user) (agent) (agent) (user) (user)\n```\n\n| Step | Skill | What Happens |\n|------|-------|-------------|\n| 1. Discover | `skill:malloy-discover` | Read the model and data; scan sources, fields, distributions; detect prior art. With no package yet, start from `malloy_searchDatabaseSchema` to find the tables in the connection |\n| 2. Propose Scope | `skill:malloy-scope` | Present findings, user selects focus |\n| 3. Propose Sources | `skill:malloy-define` | Propose source plan, user confirms architecture |\n| 4. Propose Definitions | `skill:malloy-define` | Propose fields per base source, user confirms logic |\n| 5. Build Base Sources | `skill:malloy-model` | Write fully documented base source files (one per table), check diagnostics. Read `skill:malloy-document` for doc standards. |\n| 6. Build Joined Sources | `skill:malloy-model` | Write fully documented joined source files, validate. Read `skill:malloy-document` for doc standards. |\n| 7. Review | (none) | Present the review checklist below; user confirms or corrects |\n| 8. Curate | `skill:malloy-model` | Propose access controls (`explores`, `queryableSources`, access modifiers); always propose, the user decides whether to apply |\n\n### The pauses are the point\n\nThese are governed semantic models: the business decisions in them must be confirmed by a human subject-matter expert, and the **(user)** steps exist to collect that confirmation. They are real stops, not progress reports. A model can be complete, compiling, and fully documented and still be wrong everywhere it guessed; a capable agent can build the whole thing without pausing once, which is exactly the failure mode this workflow exists to prevent.\n\nWhen a decision goes unanswered (the user explicitly declines to decide, or nobody is there to ask), do not silently proceed as if it were settled. Take your best-supported position, label it an assumption in the field's own `#(doc)` (see `skill:malloy-document` § Mark conventions as conventions), record it under \"Open decisions\" in `modeling-notes.md`, and raise it again at Review. An unlabeled assumption is indistinguishable from a confirmed fact, and misleads everyone downstream.\n\n### Step 7 Review is a checklist, not a summary\n\nPresent these to the user, with answers:\n\n- **Which definitions did the user actually confirm?** List them; everything else is an assumption.\n- **Which thresholds and bucket boundaries did you choose?** For each: the evidence (distribution query, metadata, prior art) and the `#(doc)` hedge that marks it.\n- **Which questions were left unanswered?** Each must already carry a labeled assumption and an \"Open decisions\" entry.\n- **Does the headline metric have more than one defensible definition?** If yes, that is a blocking question: put the candidate definitions to the user with their counts side by side, not in a footnote.\n\nThe user confirming this checklist is what makes the model governed. A summary of what you built is not a checkpoint.\n\nPublishing is out of scope for open-source v1. Self-hosters move a finished model into a served package via git and the host's publish path; see `skill:malloy-publish` for the local-to-served handoff.\n\n**Two paths to a model: both produce the same fully documented result:**\n- **Schema-first:** \"Model my data\" → 8-step workflow above using the relevant skills\n- **Analysis-first:** a data question arrives before any model exists → `skill:malloy-model-as-you-go`. It answers the question with `skill:malloy-analysis`, then codifies what the answer assumed into the model, one question at a time, confirming binding decisions first. The model exists by the end; there is no separate formalize step.\n- **Open-ended exploration** with no intent to keep anything: `skill:malloy-analyze`. If it turns into something worth keeping, formalize via `skill:malloy-model` (`reference/analysis-to-model.md`).\n\n## Agent Behavior\n\n**Research before asking.** Present proposals with evidence. Never ask open-ended questions: propose with data and let the user confirm.\n\n**Use business language.** Say \"I simplified the column name\" not \"reserved word replaced.\" Don't expose Malloy internals unless the user asks.\n\n**Describe what you're doing, not which step you're on.** The user doesn't have the skill files open. Say \"I'll propose which tables to include and how they relate\" not \"Steps 3 and 4.\" Say \"Now I'll write the source files\" not \"Moving to Step 5.\" Explain the purpose of each phase in plain language before doing it.\n\n**Present choices as A/B/C.** When asking the user to choose, use lettered options with one-line descriptions. Mark your recommendation.\n\n**Complete all workflow steps.** Once modeling begins, complete through Review and propose Curate. A field without documentation is not finished. If you lose track, re-read the model and your notes. Suggest notebooks at the end.\n\n## Route by Intent\n\n| User says... | Route to |\n|-------------|----------|\n| \"Model my data\", \"create a model\" | 8-step workflow (`skill:malloy-discover`) |\n| \"Model from LookML\" | 8-step with prior art via `skill:malloy-lookml-review` |\n| \"Explore this data\", \"what's interesting?\", \"show me the top X\" | `skill:malloy-analyze` (EDA) |\n| \"Build a dashboard\", \"create views\" on existing model | `skill:malloy-analyze` (views), plus `skill:malloy-charts` or `skill:malloy-notebooks` as needed |\n| \"Build a model but not sure what metrics\" | `skill:malloy-model-as-you-go`: answer their first real question, codify what it assumed, repeat |\n\n**If the user's first message is a data question** (not \"build me a model\"), route to `skill:malloy-model-as-you-go`. It answers with `skill:malloy-analysis` and grows the model from what each answer assumed, so there is nothing to formalize afterwards.\n\n## Additional Support Skills\n\nThese supplemental skills may also be loaded as needed:\n\n- **`skill:malloy`**: Index of Malloy skills and routing guide\n- **`skill:malloy-debug`**: Fix compile errors and interpret diagnostics\n\n## Publisher MCP Tools\n\nEnsure the Publisher MCP tools are configured before modeling. No server yet? `skill:malloy-getting-started` covers setup, including the one-command scaffolder (`npm create @malloy-publisher/malloy-package@latest `) and why local authoring needs `--watch-env `: start the server without it and your saved edits are never read.\n\n| Tool | Purpose |\n|------|---------|\n| `malloy_getContext` | Ground yourself in a package: its sources, views, and fields |\n| `malloy_executeQuery` | Run ad-hoc queries for validation |\n| `malloy_compile` | Compile-check a change and get diagnostics back without running a query |\n| `malloy_reloadPackage` | Recompile a package from disk so a saved edit becomes queryable by name |\n| `malloy_searchDocs` | Search Malloy docs (call BEFORE unfamiliar patterns) |\n| `malloy_searchDatabaseSchema` | Find the tables in a database connection by plain-English description, when modelling data that is not in a package yet. Returns each table's columns and the `source:` line to start from. Names and types only: no row value is returned |\n\nNever guess field names. Ground yourself with `malloy_getContext` to see the sources and fields a package defines.\n\n### The edit-and-run loop\n\nPublisher compiles each configured package at boot and serves that cached model, so a source or view you add afterwards is not queryable by name until you reload the package. The loop is:\n\n1. **Validate** the change with `malloy_compile`, picking the scope that matches what you are doing:\n - Adding a new definition or query: the default (`scope: \"append\"`) compiles your text in the model's namespace. Note its diagnostic positions land in the model-plus-your-text concatenation.\n - **Editing an existing definition: `scope: \"file\"`**, with the whole edited file as `source`. It compiles your text AS the file (append would collide with \"Cannot redefine\"), and diagnostics land at the true line numbers of your text.\n - Before saving a change other files import: `scope: \"package\"` with the edited file as `source` runs reload's worker compiler over every `.malloy` and `.malloynb` file against your edit, so a rename that breaks an importer surfaces now instead of at reload. Each diagnostic carries `model`, the file it points at; files hidden from discovery can appear. If `modelPath` does not exactly match an existing file, a warning says the source was treated as new.\n2. **Save** it to the package's model file.\n3. **Reload** with `malloy_reloadPackage`.\n4. **Run** the new view with `malloy_executeQuery`.\n\nA reload that fails to compile is safe: your files are left alone and the previously compiled model keeps serving, with the compile errors returned to you. Compile first anyway for faster feedback, and a `scope: \"package\"` dry-run with no `source` uses reload's compiler and file selection (imports across files, every `.malloy` and `.malloynb` file as saved) without touching the served model. Keep the source of truth outside `publisher_data/`, which is not version-controlled and is wiped by a `--init` restart. If these tools are missing, the Publisher you are connected to predates them; fall back to validating with a throwaway `malloy_executeQuery`. An older Publisher that has `malloy_compile` but rejects `scope` supports only the append behavior.\n\n## SQL-to-Malloy Quick Reference\n\n| SQL | Malloy |\n|-----|--------|\n| `COUNT(*)` | `count()` |\n| `COUNT(DISTINCT x)` | `count(x)` |\n| `NOW()` | `now` |\n| `CASE WHEN...END` | `pick...when...else` |\n| `col IN ('a','b')` | `col ? 'a' \\| 'b'` |\n| `COALESCE(a,b)` | `a ?? b` |\n| `CAST(x AS type)` | `x::type` |\n| `DATEDIFF(day, a, b)` | `days(a to b)` |\n| `CONCAT(a, b)` or `a \\|\\| b` | `concat(a, b)` |\n| `TIMESTAMP_DIFF(a, b, SECOND)` | `seconds(b to a)` |\n\n## Critical Rules\n\n1. **All keywords require colons**: `source:`, `dimension:`, `measure:`, `view:`\n2. **Use `is` not `as`**: `dimension: name is expression`\n3. **Arrow operator required**: `run: source -> { operations }`\n4. **Specify join type**: `join_one:`, `join_many:`, `join_cross:`\n5. **Safe division**: `revenue / nullif(count, 0)`\n6. **Group definitions under one keyword**: `measure:` then indent fields beneath\n\n## Common Anti-Patterns\n\n```\nWRONG: source flights is ... RIGHT: source: flights is ...\nWRONG: dimension: x as y RIGHT: dimension: y is x\nWRONG: count(*) RIGHT: count()\nWRONG: count(distinct x) RIGHT: count(x)\nWRONG: revenue / order_count RIGHT: revenue / nullif(order_count, 0)\nWRONG: run: src { ... } RIGHT: run: src -> { ... }\n```\n\n## Reserved Words: Scan Schema First\n\n**Malloy has many reserved words. When in doubt, backtick it.** Most likely to appear as column names:\n\n```\ndate, time, day, month, year, quarter, week, hour, minute, second,\nnumber, string, boolean, type, table, source, index, count, sum, avg, min, max,\ntrue, false, null, is, on, with, all, from, by, in, to, for, select, order_by,\ntop, bottom, desc, asc, row, range, current, window, rank\n```\n\n- `number`: only the bare word needs backticking; `account_number` is fine\n- `source`: reserved; use a different alias like `traffic_source`\n- `string`, `boolean`, `true`, `false`: backtick any column with these exact names\n\n## Gotcha Skills: Read Before Writing Code\n\nThe following skills contain detailed WRONG/RIGHT patterns that prevent the most common Malloy errors. **Read them before writing code:**\n\n- **`skill:malloy-gotchas-modeling`**: Reserved words, NULL checks, date functions, type casts, rename pitfalls, query-based source gotchas, `conn.sql()` anti-pattern\n- **`skill:malloy-gotchas-queries`**: Chart constraints, aggregate filters, joined field aliasing, time truncation vs extraction\n- **`skill:malloy-gotchas-rendering`**: Tag syntax, scale rules, sparkline setup, big_value patterns" }, { "name": "malloy-notebook-chat", diff --git a/packages/skills/package.json b/packages/skills/package.json index c68925ee6..d57de1a8e 100644 --- a/packages/skills/package.json +++ b/packages/skills/package.json @@ -1,7 +1,7 @@ { "name": "@malloy-publisher/skills", "description": "Agent skills for Malloy and the Malloy Publisher", - "version": "0.1.9", + "version": "0.1.10", "license": "MIT", "type": "module", "main": "dist/index.js", diff --git a/skills/README.md b/skills/README.md index b8cbf0ccc..ba76f1629 100644 --- a/skills/README.md +++ b/skills/README.md @@ -18,7 +18,7 @@ Most of these skills are **shared, open-source Malloy skills** kept in sync with ## Shared vs Publisher-specific -- **Shared engine skills** (identical to upstream): `malloy-model`, `malloy-materialization`, `malloy-analyze`, `malloy-analysis`, `malloy-charts`, `malloy-queries`, `malloy-debug`, `malloy-define`, `malloy-discover`, `malloy-notebooks`, `malloy-review`, `malloy-scope`, `malloy-gotchas-*`, `malloy-notebook-chat`, `malloy-phrase-detection`, `malloy-analysis-pitfalls`, `malloy-analysis-report`, `malloy-html-data-app*`, `malloy-lookml-review`, `malloy-patterns`. +- **Shared engine skills** (identical to upstream): `malloy-model`, `malloy-model-as-you-go`, `malloy-materialization`, `malloy-analyze`, `malloy-analysis`, `malloy-charts`, `malloy-queries`, `malloy-debug`, `malloy-define`, `malloy-discover`, `malloy-notebooks`, `malloy-review`, `malloy-scope`, `malloy-gotchas-*`, `malloy-notebook-chat`, `malloy-phrase-detection`, `malloy-analysis-pitfalls`, `malloy-analysis-report`, `malloy-html-data-app*`, `malloy-lookml-review`, `malloy-patterns`. - **Publisher-specific skills** (not shared): `malloy-modeling`, `malloy-publish`, `malloy-document`, `malloy-getting-started`, and the root `malloy` index (Publisher's own host/router entry points), plus `malloy-materialization-tuning` (a tuning skill built on the `malloy-pub` CLI) and `malloy-dashboards` (dashboards are a Publisher surface). These name Publisher's own tools directly and are never synced upstream to `ms2data/agent-skills`. ## Tool names in shared skills @@ -27,7 +27,7 @@ Shared skills refer to MCP tools by **bare name** (`get_context`, `execute_query ## Adding or updating a skill -- Update a shared skill **upstream first** (in `ms2data/agent-skills`), then copy it here, which keeps the two byte-identical. Editing only this copy makes the next sync a conflict. +- Update a shared skill **upstream first** (in `ms2data/agent-skills`), then copy it here, which keeps the two byte-identical. Editing only this copy makes the next sync a conflict. **This direction is being reversed**: under the consolidation proposed in `ms2data/service#6177`, this repo becomes the source of truth and `agent-skills` vendors from a Publisher tag with a CI drift check, so the hand-carry goes away. `malloy-model-as-you-go` and the `malloy-model` hunk that scopes its "no views" rule to schema-first were authored here on that basis; mirror them upstream by hand until the vendoring script lands. - **Any edit under `skills/` means regenerating the MCP bundle** (`cd packages/server && bun run src/mcp/skills/build_skills_bundle.ts ../../skills`) and committing the resulting `src/mcp/skills/skills_bundle.json`. It is a committed generated asset, and `skills_bundle.spec.ts` fails the build when it drifts from this tree. The bundle is committed indented so that two PRs touching different skills merge cleanly; if you do hit a conflict in it, resolve it by regenerating from the merged `skills/` tree, never by editing the JSON by hand. - A new skill directory needs a `.claude/skills/` symlink (`ln -s ../../skills/ .claude/skills/`) so Claude Code discovers it. - A shared skill may only `skill:`-reference other shared skills; refer to a host wrapper in neutral prose so a verbatim copy never leaves a dangling reference. diff --git a/skills/malloy-analysis-report/SKILL.md b/skills/malloy-analysis-report/SKILL.md index 09332b63d..afe100d13 100644 --- a/skills/malloy-analysis-report/SKILL.md +++ b/skills/malloy-analysis-report/SKILL.md @@ -23,7 +23,7 @@ Do NOT build the notebook in the same turn as `execute_query`. Explain first, th ## Filters are inherited from the model, don't declare them in the report -Reports do not (and cannot) define their own filters. If the source has `#(filter)` annotations, Publisher renders the filter widgets, parses caller parameters, and injects `where:` clauses server-side automatically: the report inherits and displays those filters with no extra work. If the analysis needs a knob the source doesn't expose, the right move is to add a `#(filter)` to the source itself (see your modeling workflow's parameterizable-filter guidance for `#(filter)`), not to wedge a filter widget into the report. For curated notebooks with their own per-notebook filter UI on top of the model, see `skill:malloy-notebooks` instead. +Reports do not (and cannot) define their own filters. If the source declares `given:` parameters (or legacy `#(filter)` annotations), Publisher renders the controls, parses caller parameters, and applies them server-side automatically: the report inherits and displays them with no extra work. If the analysis needs a knob the source doesn't expose, the right move is to add a `given:` to the source itself, not to wedge a filter widget into the report. `#(filter)` is deprecated in favour of native Malloy `given:` parameters. Do not add new `#(filter)` annotations; the two exceptions are `required` and `implicit`, which `given:` cannot cover yet. See `skill:malloy-model` § Legacy: Parameterizable Filters. For curated notebooks with their own per-notebook filter UI on top of the model, see `skill:malloy-notebooks` instead. ## What goes in the report diff --git a/skills/malloy-analyze/SKILL.md b/skills/malloy-analyze/SKILL.md index a5ddf6142..2f3b4fdfd 100644 --- a/skills/malloy-analyze/SKILL.md +++ b/skills/malloy-analyze/SKILL.md @@ -1,6 +1,6 @@ --- name: malloy-analyze -description: Explore data for insights and build views/dashboards/notebooks. Use when user asks to "analyze this data", "find insights", "explore for patterns", "what's interesting", "what's driving X", "build a dashboard", "create views", or any analysis task. For EDA exploration, start at Step 1. For building views on an existing model, jump to View Patterns. +description: Open-ended exploration with no specific question to answer, and building views/dashboards/notebooks on an existing model. Use when the user asks "what's interesting?", "explore this data", "find insights", "look for patterns", "build a dashboard", or "create views". Not for a specific data question - answering one is `malloy-analysis`, and writing down what the answer assumed is `malloy-model-as-you-go`. For exploration start at Step 1; for views on an existing model jump to View Patterns. --- + +# Model as you go + +An analysis that lives in a chat transcript is not reproducible. The numbers were right, and +six weeks later nobody can say what "revenue" excluded, which of four timestamps was the order +date, or whether the last period was complete. The work is unauditable, so it gets redone. + +The fix is to write the assumptions down while the query is still in front of you, in the +most durable place your session can write. Answer the question, codify what the answer +assumed, answer the next one. After a handful of questions there is a model, or a notebook, +where every definition exists because a real question needed it, and every judgment call is +on the record. + +This skill is the codify step. `skill:malloy-analysis` answers the question; this skill +decides what to write down afterwards, and where. + +> **Tool names** are bare here - `get_context`, `execute_query`, `search_database_schema`. The +> exact prefixed name depends on the host; match against the tools you actually have. + +## The loop + +``` +QUESTION → ANSWER → CODIFY ⟲ + ↑___________| +``` + +**The user gets a real answer on question one.** If three tool calls have gone by without +producing an insight they can read, you have drifted into modelling for its own sake. Stop and +answer something. + +## 1. Answer the question, with `skill:malloy-analysis` + +This skill starts when someone asks a data question. **That question is the unit of work, and +it is theirs.** Do not widen it into a modelling project, and do not swap it for a more +interesting one you found on the way. + +**A broad ask is still an ask.** "Analyse the sales data" is a question whose subject is given +and whose metric is not. Pick the most obvious question about that subject, say in one line +which one you picked, and answer it. Never open with a row-count table and a menu of +"analytical domains"; the menu is worth less than the first real answer would have been. + +Load `skill:malloy-analysis` and follow it: discover the model, construct the query, run it, +verify it, present it. Two of its rules matter most here: + +- **Every number you present comes out of a query.** Adding up the rows of a `limit: 15` table + by hand drops everything below the cut, and nobody can re-run it. +- **Your first result is a draft.** Work `skill:malloy-analysis-pitfalls` before presenting. + +### Name the decisions the answer rests on + +**This is what makes the analysis auditable, and it is the step most easily skipped.** + +Almost every query needs at least one judgment call the data cannot settle. Which of four +timestamps is "the order date". Whether returns count as revenue. Whether to count lines or +orders. Whether a partial final period belongs in a trend. + +A judgment call made silently becomes a hidden assumption. It will be wrong for someone +eventually, and by then nobody remembers it was a choice rather than a fact. So make the call, +run with it, and **say what you chose and what you rejected**, with the other number attached +whenever it is cheap to get: + +> Revenue here excludes cancelled **and** returned orders - that's $8.10M. Counting returns as +> revenue and netting them separately gives $9.18M. I went with the stricter one; say if your +> reporting does it the other way. + +Not every choice rises to this. Raise it when **any** of these holds: + +| Raise it when | Because | +|---|---| +| The alternative changes the number materially | The user would answer differently depending on which they meant | +| You are about to codify it | It stops being your choice and becomes everyone's definition | +| A reasonable analyst would pick the other one | It is a convention, not a fact | + +Otherwise state it in a clause and move on. **Asking about everything is as bad as asking about +nothing**; it turns the loop into a form and trains the reader to skim past it. And **give the +number under each option, not an abstract question**: "$8.10M excluding them, $9.18M +including" lets the user answer in one word. + +## 2. When there is no model yet + +If the tables have no model, define only enough to run the first query. One line is normal: + +```malloy +source: order_items is my_conn.table('ecommerce.order_items') +``` + +**Read the columns with `run: source -> { select: * limit: 3 }`.** Prefer it to a schema +listing: it returns the columns *and* real values, and the values are what catch the surprises +a column list hides - a date stored as a string, a `total` that excludes tax, a metric column +that is null on every row. + +If a schema tool returns something you cannot explain (no columns, or no tables for a filter +you can see matches), that is a bug in the tool. Report it. Do not write the workaround into +your model or your notes as though it were a property of the data. + +No `primary_key:`, no dimensions, no measures, no joins, **not yet**. Those arrive in CODIFY, +each one paid for by a question that needed it. Adding fields because the table has them is the +habit this skill exists to break. + +Add a join only when *this* question cannot be answered without it, then verify its cardinality +before trusting any aggregate (`group_by: fk, aggregate: n is count(), having: n > 1`). + +If a model already exists, ground yourself in it with `get_context` and reuse what is there. + +## 3. CODIFY: write the answer's assumptions down + +Run this after *every* answered question, while the query is still in front of you. Skipping it +is how a session ends with a transcript and nothing else. + +### Where it goes depends on what you can write + +The same assumption lands in a different place depending on the session. Pick the **highest +rung your tools allow**, and never skip codifying because the top rung is out of reach. + +| You can | Codify as | Who inherits it | +|---|---|---| +| **Edit the model files** (a local package, a draft package, a compile or reload tool) | A `dimension:`, `measure:`, `join_*:`, or `view:` with a `#(doc)`, in the `.malloy` file | Everyone who queries the model. **Confirm binding decisions first** (below) | +| **Author notebooks or reports, but not the model** (a viewer of a published package) | In the notebook: `source: orders_q is orders extend { measure: ... }` with the `#(doc)` above it, plus a markdown cell stating the assumption. A question worth re-asking becomes a cell | Readers of the report. State the decision in the cell, and say which definitions the model should adopt | +| **Only answer in chat** (no file or report tools) | The assumption stated in the answer, plus the Malloy snippet a modeler could paste: the `measure:` with its `#(doc)` | Nobody, until someone acts on it. That is why the snippet matters | + +Tell the rungs apart by the tools you have, not by guessing at the user's role: a file-write or +compile tool means the top rung; a report or notebook tool without model edits means the middle +one; neither means the bottom. + +### What to codify + +Look at the query you just ran. For each dimension, measure, join, filter, **and for the shape +of the query itself**, ask **in this order**: + +| Codify it when | Example | +|---|---| +| **A reader needs it to trust the number** | the cancelled/returned exclusion - without it every revenue figure reads high | +| **It encodes a business rule** | a regex parsing a messy column, a status mapping, tier cutoffs | +| **Another question would reuse it** | `revenue is sum(sale_price)` - everything about orders needs it | +| **It was hard to get right** | a window function, a multi-step derivation, a verified join | + +The first row is the one that matters. A definition is worth keeping less because it saves +typing than because it stops the next reader misreading the number. + +**Codify thin, and leave the ad-hoc behind**: filters tied to one finding, calculations that +answered exactly one question. A question that needed one measure codifies one measure, not the +neighbouring columns as well. An empty CODIFY is a fine outcome; say so and move on. + +**Then say what you did**, in one line, every time: + +> Codified: `revenue is sum(sale_price)` excluding cancelled and returned, `category` via the +> products join. Left ad-hoc: the `where: created_at > @2023` - just this question's window. + +That line shows the model growing and gives the reader a place to object. Never codify +silently. + +### A decision becomes binding the moment it enters the model + +While a judgment call lives in one ad-hoc query it is yours, and stating it is enough. Once it +is in the model file it is the definition everyone inherits, and nobody downstream sees the +reasoning. So on the top rung, **stop, ask, and wait for an answer** before writing it down. +Reporting it afterwards ("I used the midpoint, say if you'd rather have the floor") is not +confirming it; by then it is already the default. On the middle rung, confirm when the report +will be shared; a private notebook is still yours. + +This is not only about measures. Confirm anything that changes what later questions return: + +| Confirm before codifying | Because it silently sets | +|---|---| +| A source-level `where:` | the scope of *every* later question against that source | +| A `measure:` definition | the default value of that metric for everyone | +| A `dimension:` that buckets, maps, or parses | which rows land in which group | +| A `join_one:`/`join_many:` and its grain | whether aggregates fan out | +| A saved `view:` | the shape people will re-run and cite | + +Ask with the numbers attached: + +> I want to make `net_revenue` exclude cancelled and returned. That makes it the default revenue +> number for every later question - $8.10M rather than $10.81M gross. Good, or does your +> reporting treat returns differently? + +**One confirmation per decision, not per question.** Once the user has settled how revenue +treats returns, it is settled: reuse it and stop asking. And if the user has told you to stop +checking in, believe them; state each decision in a clause and keep going. + +### Write it down twice + +**In the model (or the notebook), as a `#(doc)` on the definition.** This is the part that +survives. Use `#(doc)`, never a `//` comment, for anything a consumer needs: `#(doc)` is +machine-readable, so it renders in the UI and retrieval tools read it, while a `//` comment +reaches nobody but whoever opens the file. Keep `//` for maintainer-only notes. Tag formats on +the definition too: bare `# currency` on the measure, any scale (`# currency=usd0m`) only in +views. + +```malloy +#(doc) Order line items joined to products. Grain is one row per line, not per order. +source: order_items is my_conn.table('ecommerce.order_items') extend { + join_one: products is my_conn.table('ecommerce.products') on product_id = products.id + + dimension: + #(doc) Canonical order date. Data runs 2019-01-05 to 2026-03-14, so the final year is PARTIAL - never present it as a full period. + order_date is created_at::date + + measure: + #(doc) Revenue in USD, excluding cancelled and returned orders. Those are 25% of gross, so excluding them is not optional. + # currency + net_revenue is sum(sale_price) { where: status != 'Cancelled' and status != 'Returned' } + + #(doc) Monthly revenue trend. The question asked on 2026-03-02. + # line_chart + view: revenue_trend is { + group_by: order_month is order_date.month + aggregate: + # currency=usd0m + net_revenue + order_by: order_month + } +} +``` + +One file per analytical domain, named for it: `order_revenue.malloy`, not `model.malloy`. It +grows monotonically across the session. + +**In a notes file, as the reasoning.** Keep an `analysis-notes.md` beside the model (or a +markdown cell in the notebook) recording each question, what it found, what got codified and +what was left ad-hoc, and every verification finding. The model carries the *what*; the notes +carry the *why* and the evidence. It is also what lets you resume after losing context. + +### Save the view when a question is worth re-asking + +A saved `view:` turns "we answered that once" into "re-run it". A trend wanted again next +month belongs in the file as a `view:` with its chart tag (`skill:malloy-charts`); views wanted +side by side belong in a notebook (`skill:malloy-notebooks`), or in a dashboard surface if your +host has one. A genuine one-off does not. + +> **This departs from `skill:malloy-model` on purpose.** Its "no views in source files" rule +> assumes a schema-first model, written before anyone asked a question, so its views would be +> guesses. Here every view is a question that was asked and verified. Two more of its rules do +> not apply either: skip access modifiers and curation (there is no discovery surface to curate +> when every field was paid for by a question), and keep one domain file rather than one file +> per table until it genuinely gets unwieldy. Everything else in `skill:malloy-model` applies: +> `#(doc)` on every field, verified join cardinality, `nullif` on division, a `given:` for a +> runtime parameter. + +### Then loop + +Go back to the question, and let what you just found sharpen the next one: + +> Revenue is concentrated in three categories. Worth asking whether that's new - want the same +> cut by year? + +## When the analysis is going to be shared + +Docs are not on this list; they happen in CODIFY, as you go. When the user wants to hand it +over: + +1. **Names.** Rename anything whose name only made sense inside one question. +2. **Doc pass.** Re-read the `#(doc)` lines for anything that drifted as the model grew. Check + grain, units, and null handling are each stated somewhere. +3. **Re-verify.** Re-run the session's key queries against the finished sources. If a number + moved, something was codified wrong. This is the check that proves reproducibility; do not + skip it. +4. **Structure**, only if one file has genuinely become unwieldy: base sources per table plus a + joined source per domain, per `skill:malloy-model`. Being shared is not itself a reason. + +## When to do something else + +| Situation | Go to | +|---|---| +| Porting prior art (LookML, dbt, a metrics doc): the definitions exist and are agreed, the job is translation | `skill:malloy-lookml-review`, then `skill:malloy-model` | +| The user names the sources they want built outright, before any question | `skill:malloy-model` | +| A model already exists, the question rests on no judgment call, and nothing is worth keeping | `skill:malloy-analysis` alone | +| Open-ended exploration with no intent to keep anything | `skill:malloy-analyze` | + +## Anti-patterns + +``` +WRONG Turn "what's our default rate?" into a modelling project +RIGHT Answer it, then codify what the answer assumed + +WRONG Propose every dimension and measure for each table in scope +RIGHT Codify the two fields this question actually needed + +WRONG Quietly pick one of four timestamps as "the order date" +RIGHT "Using created_at; shipped_at would drop 38% as nulls. OK?" + +WRONG Skip codifying because you cannot edit the model +RIGHT Put the extend and its #(doc) in the notebook, or hand over the snippet + +WRONG Codify the source-level where:, then mention it in the write-up +RIGHT Stop and confirm it; it scopes every question anyone asks later + +WRONG The assumption lives in the chat transcript, or in a // comment +RIGHT #(doc) on the definition for what a consumer needs, plus a line in the notes + +WRONG Leave the answered question as a transcript table and move on +RIGHT Save it as a view; a question worth answering is usually worth re-asking + +WRONG Curate access modifiers and split one file per table to "do it properly" +RIGHT Skip both; they solve a schema-first problem this model does not have +``` diff --git a/skills/malloy-model/SKILL.md b/skills/malloy-model/SKILL.md index 14fbdfba9..3caddc48c 100644 --- a/skills/malloy-model/SKILL.md +++ b/skills/malloy-model/SKILL.md @@ -168,9 +168,21 @@ source: customer_health is customers extend { | **Dimensions** | Intrinsic to this table only | Cross-source (require joins) | | **Measures** | Single-table aggregations | Cross-source aggregations | | **Joins** | None (or only lookup joins intrinsic to the source) | Defines relationships between base sources | -| **Views** | None (views belong in analysis) | None | +| **Views** | None, in schema-first (see below) | None, in schema-first (see below) | | **One per** | Physical table or computed source | Analytical domain | +**The "no views" rule is schema-first only.** A schema-first model is built before anyone +has asked a question, so any view in it is a guess. It does **not** apply to the +analysis-first workflow (`skill:malloy-model-as-you-go`), where every view is a +question that was asked and verified - there, **saving a view or a dashboard is the right +call**, in the model file next to the measures it uses. Analysis-first still models +everything else properly: documented dimensions, measures, and joins. + +**Two more rules here are schema-first only.** Analysis-first should skip access modifiers +and curation - there is no discovery surface to curate when every field was paid for by a +question - and skip one-file-per-table, keeping a single domain file until it genuinely +gets unwieldy. + ## Key Rules - **Every `dimension:` needs `name is expr`**: a bare `dimension: species` is a parse error. Raw columns are queryable directly in `group_by` / `select`; only declare a `dimension:` to derive or rename a field. diff --git a/skills/malloy-modeling/SKILL.md b/skills/malloy-modeling/SKILL.md index cd7072abc..3f6889df0 100644 --- a/skills/malloy-modeling/SKILL.md +++ b/skills/malloy-modeling/SKILL.md @@ -95,9 +95,8 @@ Publishing is out of scope for open-source v1. Self-hosters move a finished mode **Two paths to a model: both produce the same fully documented result:** - **Schema-first:** "Model my data" → 8-step workflow above using the relevant skills -- **Analysis-first:** "Explore this data" → `skill:malloy-analyze` → formalize via `skill:malloy-model` (`reference/analysis-to-model.md`) - -After analysis completes, **always recommend formalizing into a model.** +- **Analysis-first:** a data question arrives before any model exists → `skill:malloy-model-as-you-go`. It answers the question with `skill:malloy-analysis`, then codifies what the answer assumed into the model, one question at a time, confirming binding decisions first. The model exists by the end; there is no separate formalize step. +- **Open-ended exploration** with no intent to keep anything: `skill:malloy-analyze`. If it turns into something worth keeping, formalize via `skill:malloy-model` (`reference/analysis-to-model.md`). ## Agent Behavior @@ -119,9 +118,9 @@ After analysis completes, **always recommend formalizing into a model.** | "Model from LookML" | 8-step with prior art via `skill:malloy-lookml-review` | | "Explore this data", "what's interesting?", "show me the top X" | `skill:malloy-analyze` (EDA) | | "Build a dashboard", "create views" on existing model | `skill:malloy-analyze` (views), plus `skill:malloy-charts` or `skill:malloy-notebooks` as needed | -| "Build a model but not sure what metrics" | `skill:malloy-analyze` first, then formalize via `skill:malloy-model` | +| "Build a model but not sure what metrics" | `skill:malloy-model-as-you-go`: answer their first real question, codify what it assumed, repeat | -**If the user's first message is a data question** (not "build me a model"), route to `skill:malloy-analyze`. After analysis completes, **always recommend formalizing via the analysis-to-model workflow** (`skill:malloy-model` → `reference/analysis-to-model.md`). +**If the user's first message is a data question** (not "build me a model"), route to `skill:malloy-model-as-you-go`. It answers with `skill:malloy-analysis` and grows the model from what each answer assumed, so there is nothing to formalize afterwards. ## Additional Support Skills diff --git a/skills/malloy/SKILL.md b/skills/malloy/SKILL.md index b7a837905..8989770c3 100644 --- a/skills/malloy/SKILL.md +++ b/skills/malloy/SKILL.md @@ -41,7 +41,8 @@ Every skill in this deployment, by what it is for. Start at a driver; it routes | Skill | Use when... | |-------|-------------| -| `skill:malloy-analyze` | Exploratory data analysis: profiling, building views and dashboards | +| `skill:malloy-model-as-you-go` | After answering a question, writing down what it assumed: a `#(doc)`'d field in the model, an `extend` in the notebook, or a stated assumption plus snippet, depending on what the session can write | +| `skill:malloy-analyze` | Open-ended exploration with no intent to keep anything: profiling, hypotheses, views | | `skill:malloy-charts` | Chart selection and renderer reference for Malloy visualizations | | `skill:malloy-notebooks` | Building Malloy notebooks (.malloynb) | | `skill:malloy-analysis-report` | Combining validated queries into a notebook report or dashboard |