Skip to content

Commit 3977af4

Browse files
rogpepperogpeppe
committed
designs: add cue/load io.FS proposal
This is to address https://cuelang.org/issues/607. Signed-off-by: Roger Peppe <rogpeppe@gmail.com> Change-Id: I79809c983e532a1cfb2d42917511a7601b4c1de3 Reviewed-on: https://cue.gerrithub.io/c/cue-lang/proposal/+/1231971 TryBot-Result: CUEcueckoo <cueckoo@cuelang.org> Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
1 parent ff5b5c1 commit 3977af4

1 file changed

Lines changed: 262 additions & 0 deletions

File tree

‎designs/4285-load-io-fs.md‎

Lines changed: 262 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,262 @@
1+
# Proposal: Support `io/fs.FS` in `cue/load`
2+
3+
Roger Peppe
4+
5+
Date: 2026-02-18
6+
7+
Discussion at [https://github.com/cue-lang/cue/discussions/4285](https://github.com/cue-lang/cue/discussions/4285).
8+
9+
## Abstract
10+
11+
We propose adding an `FS` field of type `io/fs.FS` to
12+
`cue/load.Config`, allowing the CUE loader to read packages and
13+
modules from virtual filesystems rather than the host operating
14+
system. This enables use cases such as embedding CUE modules in Go
15+
binaries via `embed.FS`, running CUE evaluation in sandboxed or server
16+
environments, and testing loaders against in-memory filesystems.
17+
18+
## Background
19+
20+
The CUE loader (`cue/load`) currently reads all source files from the
21+
host filesystem. An `Overlay` mechanism exists that maps absolute file
22+
paths to in-memory contents, layered on top of the real filesystem,
23+
but this has significant limitations. Since overlay paths must be
24+
absolute and are resolved relative to directories that must exist (or
25+
be themselves overlaid), users who want a fully virtual filesystem
26+
must construct plausible absolute paths and ensure directory entries
27+
are consistent. This is fragile and unintuitive, as the [discussion on
28+
\#607](https://github.com/cue-lang/cue/issues/607) demonstrates. See
29+
also [this thread](https://github.com/cue-lang/cue/discussions/1145#discussioncomment-1082053)
30+
which discusses how to load CUE files that have been embedded in a Go
31+
binary.
32+
33+
Go 1.16 introduced the `io/fs.FS` interface, which has since become
34+
the standard abstraction for read-only filesystems in the Go
35+
ecosystem. Types such as `embed.FS`, `fstest.MapFS`, and `zip.Reader`
36+
all implement it. Supporting `io/fs.FS` directly in the loader would
37+
give CUE users access to this entire ecosystem without the friction of
38+
the overlay mechanism.
39+
40+
The original issue for this feature is at
41+
[https://github.com/cue-lang/cue/issues/607](https://github.com/cue-lang/cue/issues/607).
42+
43+
[PR \#4222](https://github.com/cue-lang/cue/pull/4222) by @pskry
44+
proposed an implementation of this feature. We are grateful for that
45+
work and the thinking behind it. The design below builds on the same
46+
core idea but simplifies the approach, in particular by avoiding a
47+
synthetic path prefix and by keeping the change surface small.
48+
49+
## Proposal
50+
51+
We add two new fields to `cue/load.Config`:
52+
53+
```go
54+
type Config struct {
55+
// ...existing fields...
56+
57+
// FS, if non-nil, provides the filesystem used by the loader
58+
// for discovering packages, resolving modules, and reading
59+
// files. It is mutually exclusive with [Config.Overlay]; it is
60+
// an error to set both.
61+
//
62+
// When FS is nil, the loader uses the host operating system
63+
// filesystem ([os.Stat], [os.ReadDir], os.Open), which is the
64+
// default and preserves the existing behavior. Overlay can
65+
// be used in that case.
66+
//
67+
// When FS is set, all paths — including [Config.Dir],
68+
// [Config.ModuleRoot], and the arguments to [Instances] — are
69+
// interpreted as forward-slash-separated paths within FS.
70+
// Absolute paths (those starting with "/") are permitted and
71+
// are interpreted relative to the root of FS. Dir defaults
72+
// to "/" when FS is set and Dir is empty.
73+
//
74+
// FS enables loading CUE packages and modules from virtual or
75+
// embedded filesystems (for example, [embed.FS] or
76+
// [fstest.MapFS) without accessing the host filesystem.
77+
FS fs.FS
78+
79+
// FromFSPath maps file names as they appear inside [Config.FS]
80+
// to file names as they should appear in error messages and
81+
// position information. It is ignored when FS is nil. When FS
82+
// is set and FromFSPath is nil, paths are left unchanged.
83+
FromFSPath func(path string) string
84+
}
85+
```
86+
87+
### Path semantics
88+
89+
When `FS` is set, all file and directory paths are slash-separated,
90+
following `io/fs` convention. Unlike `io/fs.FS` itself, we allow paths
91+
that start with `/`. Such paths are treated as rooted at the FS — that
92+
is, `/foo/bar` is looked up as `foo/bar` in the FS. This preserves the
93+
existing convention that `Config.Dir` and `Config.ModuleRoot` can be
94+
absolute, and avoids requiring callers to reason about a fundamentally
95+
different path model.
96+
97+
When `FS` is set and `Dir` is empty, it defaults to `/` rather than
98+
the process working directory (which has no meaning relative to a
99+
virtual filesystem).
100+
101+
### Mutual exclusion with Overlay
102+
103+
Setting both `FS` and `Overlay` is an error. The two mechanisms serve
104+
overlapping purposes: `Overlay` patches the host filesystem while `FS`
105+
replaces it entirely. Supporting the combination would add complexity
106+
for a use case that `FS` alone covers. If a caller has an `io/fs.FS`
107+
and wants to overlay additional files, they can compose FS
108+
implementations in user code (for example, using `io/fs.Sub` or a
109+
small merging wrapper), and this can also combine OS files with
110+
os.DirFS.
111+
112+
### Error path mapping
113+
114+
Errors and source positions produced by the loader contain file paths.
115+
When the underlying filesystem is virtual, these paths may not
116+
correspond to anything meaningful on the host. The `FromFSPath`
117+
function, when provided, gives the caller control over how paths
118+
appear in diagnostics.
119+
120+
A typical use is mapping back to the original source location:
121+
122+
```go
123+
cfg := &load.Config{
124+
FS: moduleFS,
125+
FromFSPath: func(p string) string {
126+
return filepath.Join("/real/source/root", p)
127+
},
128+
}
129+
```
130+
131+
When `FromFSPath` is nil and `FS` is set, paths are left as-is, which
132+
is reasonable for testing and contexts where the slash-separated paths
133+
are already suitable for display.
134+
135+
### Relation to cue/load v2
136+
137+
[Issue 3911](https://github.com/cue-lang/cue/issues/3911) tracks the
138+
possibility of making a new API for cue/load. Although that API would
139+
include support for fs.FS, the API described here is specifically made
140+
for compatibility with the existing API and any new API would not
141+
necessarily take this form.
142+
143+
### Usage example
144+
145+
Loading a CUE module from an embedded filesystem:
146+
147+
```go
148+
//go:embed mymodule
149+
var moduleFS embed.FS
150+
151+
func loadEmbeddedModule() ([]*build.Instance, error) {
152+
cfg := &load.Config{
153+
FS: moduleFS,
154+
Dir: "/mymodule",
155+
ModuleRoot: "/mymodule",
156+
}
157+
insts := load.Instances([]string{"."}, cfg)
158+
// ...
159+
return insts, nil
160+
}
161+
```
162+
163+
Loading from an in-memory filesystem in tests:
164+
165+
```go
166+
func TestLoadFromFS(t *testing.T) {
167+
fs := fstest.MapFS{
168+
"cue.mod/module.cue": &fstest.MapFile{
169+
Data: []byte(`module: "example.com/test"`),
170+
},
171+
"x.cue": &fstest.MapFile{
172+
Data: []byte(`package x; a: 1`),
173+
},
174+
}
175+
cfg := &load.Config{FS: fs}
176+
insts := load.Instances([]string{"."}, cfg)
177+
if insts[0].Err != nil {
178+
t.Fatal(insts[0].Err)
179+
}
180+
}
181+
```
182+
183+
## Rationale
184+
185+
### Why not extend Overlay?
186+
187+
The overlay mechanism could in principle be extended to work without a
188+
host filesystem backing. However, overlay was designed as a patch
189+
layer — it assumes an underlying real filesystem and uses absolute OS
190+
paths as keys. Bending it to serve as a standalone virtual filesystem
191+
would require either significant internal reworking or awkward API
192+
conventions (such as requiring synthetic absolute paths). A dedicated
193+
`FS` field is a cleaner fit for the use case and aligns with the
194+
broader Go ecosystem.
195+
196+
### Why not allow FS and Overlay together?
197+
198+
Supporting both simultaneously would require defining precedence rules
199+
and reconciling two different path models (OS-absolute for overlays,
200+
slash-separated for FS). The practical benefit is small: any layering
201+
that a caller needs can be achieved by composing `io/fs.FS`
202+
implementations. Keeping the two mutually exclusive simplifies both
203+
the implementation and the mental model for users.
204+
205+
### Why allow absolute paths in FS mode?
206+
207+
The `io/fs.FS` specification requires unrooted, slash-separated paths
208+
and does not permit a leading `/`. We relax this constraint at the
209+
`cue/load` API level so that `Config.Dir`, `Config.ModuleRoot`, and
210+
package arguments can continue to use absolute paths, as they do today
211+
with the host filesystem. Internally, a leading `/` is simply stripped
212+
before calling into the `FS`. This keeps the API transition minimal
213+
for callers porting code from host-filesystem loading to
214+
virtual-filesystem loading.
215+
216+
### Why FromFSPath rather than ToFSPath?
217+
218+
Errors flow outward from the loader to the caller. The mapping that
219+
matters is from the internal FS path to the display path. A `ToFSPath`
220+
function (mapping display paths into FS paths) would only be needed if
221+
we accepted external paths as input, but `Config.Dir` and friends
222+
already use FS paths directly when `FS` is set. The outward direction
223+
is sufficient.
224+
225+
## Compatibility
226+
227+
This proposal is fully backward compatible. When `Config.FS` is nil —
228+
the default — behavior is unchanged. The `Overlay` field continues to
229+
work as before. The only new error condition is the case where both
230+
`FS` and `Overlay` are set, which was previously impossible since `FS`
231+
did not exist.
232+
233+
## Implementation
234+
235+
The implementation requires modifying the internal `fileSystem` type
236+
in `cue/load` to route its `stat`, `readDir`, `openFile`, and `walk`
237+
operations through the provided `io/fs.FS` when one is configured. The
238+
overlay logic can be bypassed entirely in that case, since `FS` and
239+
`Overlay` are mutually exclusive.
240+
241+
The `Config.complete` method will need adjustment: when `FS` is set,
242+
it should default `Dir` to `"/"` instead of calling `os.Getwd`, and
243+
path resolution should use slash-separated logic rather than
244+
`filepath` functions.
245+
246+
Error wrapping and position reporting will apply `FromFSPath` (when
247+
non-nil) at the points where file paths are embedded in `token.Pos`
248+
values and error messages.
249+
250+
For now, even though symbolic links are supported since Go 1.25 via
251+
[io/fs.ReadLinkFS](https://pkg.go.dev/io/fs#ReadLinkFS), we will not
252+
use this interface for now. That is, interpretation of symbolic links
253+
is left up to the underlying implementation to decide, conventionally
254+
by treating the symbolic link as the file it points to as
255+
[os.DirFS](https://pkg.go.dev/os#DirFS) does. This is the same
256+
behavior used in the APIs exposed in the `mod` packages.
257+
258+
## Open issues
259+
260+
- Should `FromFSPath` be applied lazily (at error-formatting time) or
261+
eagerly (when positions are first recorded)? Eager application is
262+
simpler but means the original FS path is lost.

0 commit comments

Comments
 (0)