docs(skills): tell agents how malloy-config.json must be set for local data - #1052
Closed
jswir wants to merge 1 commit into
Closed
docs(skills): tell agents how malloy-config.json must be set for local data#1052jswir wants to merge 1 commit into
jswir wants to merge 1 commit into
Conversation
…l data
A package that Publisher serves correctly can fail to compile in the VS Code
extension or the CLI, on the same machine, from the same files. Publisher
resolves a relative `duckdb.table('data/x.csv')` against the package root
itself; the other two hosts resolve it against the DuckDB connection's
`workingDirectory` from `malloy-config.json`, which nothing was telling anyone
to set.
Worse, a *relative* `workingDirectory` is resolved against the process's cwd:
`canonicalizeConfigPath` (malloy-db-duckdb/src/duckdb_config.ts) calls
`canonicalizePath` with no `baseDirectory`, so it is a bare
`path.resolve(input)` - not the config file's directory, and not the VS Code
workspace root. The same config therefore compiles from one directory and fails
from another, which is what makes this look intermittent and look like a model
bug.
Verified with malloy-cli, which reads the same config file, against a package
whose model lives in `pkg/` and data in `pkg/data/`:
relative "malloy", cwd = repo root -> compiles, returns rows
relative "malloy", cwd = /tmp -> IO Error: No files found that match
the pattern "data/product_usage.csv",
then a "not defined" error per field
absolute path, cwd = /tmp -> compiles, returns rows
Three edits, one concept, no duplicated definition:
- malloy-getting-started gains a setup step that has the agent ASK whether the
CLI or the extension is also in use, then gives the config. It also records
that a remote Credible connection is not this case: that arrives as a
`publisher` proxy connection the extension supplies, where no
`workingDirectory` applies, and whose tokens are short-lived and unrefreshed.
- malloy-gotchas-modeling carries the mechanism, next to the other
read-a-file-in-place sections, since that is what an agent reads before
writing `duckdb.table(...)`.
- malloy-debug gets one row mapping the error to the fix, and says the field
errors under it are cascade - so nobody starts renaming columns.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
A package Publisher serves correctly can fail to compile in the VS Code extension or the Malloy CLI, on the same machine, from the same files. Nothing in the skills said why, so an agent hitting it reads the cascade of
'org_slug' is not definederrors and starts editing the model.Two things combine:
duckdb.table('data/x.csv')against the package root on its own. The CLI and the extension resolve it against the DuckDB connection'sworkingDirectory, read frommalloy-config.json- a file no skill mentioned.workingDirectoryresolves against the process cwd.canonicalizeConfigPath(malloy-db-duckdb/src/duckdb_config.ts) callscanonicalizePathwith nobaseDirectory, so it is a barepath.resolve(input). Not the config file's directory, and not the VS Code workspace root.So the same config compiles from one directory and fails from another. That is what makes this feel intermittent, and what makes it look like a model bug rather than a config one.
Verified, not inferred
malloy-clireads the samemalloy-config.jsonthe extension does, which makes it a real oracle. Against a package with the model inpkg/and data inpkg/data/:workingDirectory"malloy"(relative)"malloy"(relative)/tmpIO Error: No files found that match the pattern "data/product_usage.csv", then one "not defined" error per field/tmpThe middle row is the reported symptom, reproduced.
The change
Three files, one concept, no definition duplicated - the mechanism lives in one place and the other two point at it.
malloy-getting-startedgains a setup step that has the agent ask whether the CLI or the VS Code extension is also in use, then gives the config to write. Setup-time prevention, which is the only point where this is cheap.malloy-gotchas-modelingcarries the mechanism, beside the existing read-a-file-in-place sections, because that is the file an agent reads before writingduckdb.table(...).malloy-debuggets one quick-fix row mapping the error to the fix, and states that the field errors beneath it are cascade - so nobody starts renaming columns.It also records what is not this case: a remote Credible connection arrives as a
publisherproxy connection that the extension supplies, where noworkingDirectoryapplies at all. Since that came up in the same breath, the note also mentions the documented token caveat - user-scoped, short-lived, unrefreshed - so auth failures late in a long session are not misread as a broken model.Notes for the reviewer
reference/files, sotest:skills(frontmatter stamping, name-matches-directory, reference shipping) is untouched.-and no em-dashes.rootDirectorydefault meant to cover this (src/common/connections/CONTEXT.mdmentionsworkingDirectory: {default: {config: 'rootDirectory'}}), then the better fix is in the extension and this doc should say so instead - happy to change tack.