A FUSE filesystem that stores data as append-only packed chunk objects in an object store (e.g. S3), designed for use cases where thousands of independent filesystem instances must be resumed quickly by attaching to a warm process rather than waiting for a cold server start.
Fusey was designed to solve a specific warm-pool problem:
- You have thousands of Kubernetes deployments, each with 0 or 1 pods and persistent storage.
- Scaling to 0 pods pauses the deployment; scaling to 1 resumes it.
- A pool of warm pods exists with empty storage, giving fast cold starts.
- Resume is slow because you must wait for a pod to start (~15 s).
The goal is to make resume just as fast as a cold start: assign a warm pod and mount its data in-place, without copying gigabytes before the server can serve requests.
Fusey achieves this by:
- Storing all file data in an object store as append-only chunk objects.
- Keeping a compact index (inode metadata + extent map) that can be pulled and mounted in under a second, even for a 25 GB filesystem with tens of thousands of files.
- Serving file reads as HTTP range-GETs against chunk objects — the filesystem is immediately usable while data streams in on demand.
- On-demand compaction (
fusey compact) that removes orphaned bytes from chunks after files are overwritten or deleted — designed to run as a Kubernetes CronJob rather than a background goroutine inside the server.
┌─────────────────────────────────────────────────────────────────┐
│ Linux kernel (VFS) │
└──────────────────────────────┬──────────────────────────────────┘
│ FUSE protocol (/dev/fuse)
┌──────────────────────────────▼──────────────────────────────────┐
│ fusey FUSE daemon │
│ │
│ ┌─────────────────────┐ ┌───────────────────────────────┐ │
│ │ Index (in memory) │ │ Chunk store │ │
│ │ │ │ │ │
│ │ inodes │ │ active chunk (write buffer) │ │
│ │ dirEntries │ │ sealed chunks → S3 │ │
│ │ extentMap │ │ │ │
│ │ symlinkTargets │ │ reads → HTTP range-GET │ │
│ │ xattrs │ │ writes → append to active │ │
│ └──────────┬──────────┘ └───────────────────────────────┘ │
│ │ │
│ ┌──────────▼──────────┐ │
│ │ On-disk cache │ fusey compact (run as a CronJob) │
│ │ (FUSEY_CACHE_DIR) │ removes orphaned bytes from chunks │
│ └─────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
File data is stored as a sequence of append-only chunk objects in the backing store. Each chunk is a flat binary blob. The index records where in a chunk each file's bytes live (an extent):
chunk-00000000: [file A bytes 0-4095][file B bytes 0-8191][file A bytes 4096-8191]...
chunk-00000001: [file C bytes 0-16383]...
index:
inode 42 (file A) → extents:
{ chunk: chunk-00000000, chunkOffset: 0, length: 4096, fileOffset: 0 }
{ chunk: chunk-00000000, chunkOffset: 12288, length: 4096, fileOffset: 4096 }
inode 43 (file B) → extents:
{ chunk: chunk-00000000, chunkOffset: 4096, length: 8192, fileOffset: 0 }
Writes append to the active chunk. When the active chunk reaches
FUSEY_CHUNK_SIZE bytes it is sealed (immutable) and a fresh active chunk is
opened. A read for any byte range in a file is satisfied by one or more HTTP
range-GET requests, one per extent that covers the requested range.
The index is stored:
- In memory — the primary working copy.
- On disk (
FUSEY_CACHE_DIR) — a serialised snapshot written periodically and at unmount. Used for fast recovery without hitting the backing store. - In the backing store — the authoritative copy written at unmount (or on an explicit flush). Used when the disk cache is absent (first mount on a new node).
Over time, overwrites and deletes leave orphaned bytes in sealed chunks.
fusey compact reclaims this space:
- Identifies sealed chunks above the orphan-fraction threshold (
FUSEY_COMPACTION_THRESHOLD). - Reads all live extents from those chunks.
- Packs them into new compacted chunks, each no larger than
FUSEY_CHUNK_SIZE. - Updates the index to point at the new locations.
- Persists the index (to both local disk and S3).
- Deletes the old chunk objects.
The index is always persisted before old chunks are deleted, so the process is crash-safe: a crash between steps 5 and 6 leaves duplicate data that is cleaned up on the next compaction run.
Compaction is intentionally not run automatically inside the filesystem process — it generates a burst of S3 API calls and is best scheduled as a Kubernetes CronJob at a time and frequency that suits the workload:
# Example: run compaction nightly at 02:00
apiVersion: batch/v1
kind: CronJob
metadata:
name: fusey-compact
spec:
schedule: "0 2 * * *"
jobTemplate:
spec:
template:
spec:
containers:
- name: compact
image: your-registry/fusey:latest
command: ["fusey", "compact"]
env:
- name: FUSEY_BUCKET
value: my-bucket
- name: FUSEY_PREFIX
value: pod-abc/
restartPolicy: OnFailureFusey can optionally record every state-mutating operation to an append-only edit log in the object store. This enables two additional features:
fusey journal-dump— a debug subcommand that prints every recorded edit (with timestamp, sequence number, and full op fields) for auditing.fusey mount <mp> --as-of=<ts>— a read-only mount of the filesystem as it existed at wall-clock timets(RFC 3339, with optional sub-second precision). Useful for "what was the state at 10:00?" style queries and for recovering from a bad write bycp-ing the historical state out.
The journal is enabled by setting FUSEY_JOURNAL_ENABLED=true on the
mount/daemon. When disabled, no journal entries are recorded and as-of
mounts fail with a clear error.
Design choices (set during design review, see AGENTS.md):
- TouchAtime is NOT journaled. Read access is too high-frequency and has no replay value. The journal only records writes (create, delete, rename, setattr, write, truncate, xattr, symlink).
- Sharded, append-only. Each flush writes a new shard named
{prefix}journal-NNNNNNNNN.cborwhere N is the starting sequence number of the batch. Existing shards are never overwritten — recovery is the same shape as the chunk store'sRecoverNextSeq. - Wiped on compact.
fusey compactdeletes the journal at the START of its cycle, before any chunk remap. If compact crashes mid-cycle, the next mount has an empty journal but the index and chunks are in a consistent post-remap state; as-of mounts simply error during that window. A fresh base snapshot is written at the end of compact so subsequent as-of mounts can replay against the post-compact state. - As-of mounts are read-only. The FUSE mount is opened with
-o ro, so the kernel rejects all writes withEROFS. Recovery from a historical state iscp -a <as-of-mp>/path /live-mp/pathto copy the desired files into a live mount. - Storage cost. The journal grows with write activity between compactions. Operators who enable it should size their object store budget accordingly (typical: < 1% of total data size for write-light workloads; can dominate for write-heavy ones).
Examples:
# Mount with the journal enabled (live mode)
export FUSEY_JOURNAL_ENABLED=true
./fusey mount /mnt/fusey
# Inspect the journal
./fusey journal-dump
./fusey journal-dump --json | jq '.[].op.kind' | sort | uniq -c
# Mount a historical snapshot read-only
./fusey mount /mnt/fusey-asof --as-of=2026-01-15T10:00:00Z
ls /mnt/fusey-asof/ # shows the state at that timestamp
diff -r /mnt/fusey-asof/ /mnt/fusey/ # shows what changed sinceThe journal requires a base snapshot to be useful for as-of replay. The
first time you mount with FUSEY_JOURNAL_ENABLED=true, the next run of
fusey compact will write a base snapshot so as-of mounts work.
The specs/ directory contains a Quint formal
specification of the system before any Go code is written. Quint is a
model-checking language (a modern successor to TLA+) that lets us verify
invariants and temporal properties of the design.
| File | What it specifies |
|---|---|
specs/types.qnt |
Shared data types (Inode, Extent, DirEntry, Chunk, JournalEvent, …) |
specs/index.qnt |
Index state machine: mutations, persistence, invariants |
specs/chunks.qnt |
Chunk store: append, seal, rotate, read-range |
specs/compaction.qnt |
Compaction: target selection, remap, commit, safety |
specs/journal.qnt |
Optional edit log: append, flush, clear, as-of replay |
specs/filesystem.qnt |
Full POSIX operations composing index + chunks + journal |
Install Quint:
npm install -g @informalsystems/quintType-check a spec:
quint specs/filesystem.qntRun the model checker against an invariant:
quint verify specs/filesystem.qnt --invariant rootInvariant
quint verify specs/filesystem.qnt --invariant nlinkConsistency
quint verify specs/filesystem.qnt --invariant extentSizeConsistencySimulate a random execution:
quint run specs/filesystem.qntAll configuration is via environment variables so the binary can be deployed as a Kubernetes container without config files.
| Variable | Default | Description |
|---|---|---|
FUSEY_CHUNK_SIZE |
67108864 (64 MiB) |
Maximum size of a single chunk object |
FUSEY_BLOCK_SIZE |
4096 |
Preferred I/O block size reported to the kernel |
FUSEY_MAX_SIZE |
1099511627776 (1 TiB) |
Total filesystem capacity in bytes reported to the kernel via statfs. Set to the expected maximum data size per instance (e.g. 26843545600 for 25 GiB). |
FUSEY_CACHE_DIR |
/var/cache/fusey |
Directory for the on-disk index cache (fast path on restart) |
FUSEY_BUCKET |
(required) | S3 bucket name |
FUSEY_ENDPOINT |
(AWS default) | S3-compatible endpoint URL (e.g. http://minio:9000 for MinIO) |
FUSEY_REGION |
us-east-1 |
S3 region |
FUSEY_ACCESS_KEY |
(ambient creds) | S3 access key ID. Omit to use the ambient credential chain (IAM role, AWS_* env vars, ~/.aws) |
FUSEY_SECRET_KEY |
(ambient creds) | S3 secret access key |
FUSEY_FORCE_PATH_STYLE |
false |
Set true for MinIO and most self-hosted S3-compatible stores that require path-style URLs |
FUSEY_PREFIX |
(none) | Key prefix for all objects in the bucket (e.g. pod-abc/). Use this to share one bucket across multiple Fusey instances |
FUSEY_COMPACTION_THRESHOLD |
0.3 |
Orphan fraction above which a chunk is targeted by fusey compact |
FUSEY_PERSIST_INTERVAL |
30s |
How often the index is flushed to disk and the object store |
FUSEY_JOURNAL_ENABLED |
false |
Set to true to record every state-mutating operation to the optional edit log. Required for fusey journal-dump and fusey mount <mp> --as-of=<ts>. |
FUSEY_ALLOW_OTHER |
true |
When true, fusey mounts with the FUSE allow_other option so any user on the host can access the mountpoint. When false, only the UID that called mount(2) (the fusey daemon) can access it (the FUSE default). The default is true because fusey's typical deployment runs the daemon as root while the consumer (e.g. an agent-server) runs as a non-root user; without allow_other the kernel rejects the consumer's VFS operations before fusey's permission code runs. Set FUSEY_ALLOW_OTHER=false to restore the FUSE default if you need to restrict access to the mounter's UID. |
When FUSEY_BROKER_URL is set, fusey uses a BrokerStore instead of talking directly to S3. The broker is a trusted HTTP service that holds object-store credentials; fusey authenticates with a single bearer token and never contacts S3 directly. This is the recommended model for multi-tenant deployments where app containers are untrusted.
| Variable | Default | Description |
|---|---|---|
FUSEY_BROKER_URL |
(none) | Base URL of the broker service. When set, overrides all S3 config |
FUSEY_BROKER_AUTH_HEADER |
X-SESSION-API-KEY |
HTTP header name sent on every broker request |
FUSEY_BROKER_AUTH_VALUE |
(none) | Auth token value — required when FUSEY_BROKER_URL is set |
A reference broker implementation is in broker/. It stores objects in a local directory and is suitable for local testing and development. See broker/main.py for the full HTTP API contract.
- Formal specification (Quint)
- Go implementation — index, chunk store, compaction, FUSE layer
- Index persistence (local disk cache + S3)
-
statfssupport (FUSEY_MAX_SIZE) - S3 chunk store backend (
aws-sdk-go-v2, S3-compatible) - Cold-start recovery (local cache → S3 → fresh)
- Optional edit log (
FUSEY_JOURNAL_ENABLED) with as-of replay andjournal-dump - Kubernetes deployment guide
- Go 1.22+
- Quint (for spec verification)
- A FUSE-capable Linux host
Fusey uses Linux FUSE and is not tested against macFUSE or FUSE-T. For local development and testing on macOS, use Lima to run a lightweight Linux VM:
brew install lima
limactl start # starts a default Ubuntu VM
limactl shell default # open a shell inside the VMLima mounts your home directory into the VM by default, so you can edit files with your normal macOS tools and build/run inside the VM.
go build ./cmd/fusey# Mount a filesystem
export FUSEY_BUCKET=my-bucket FUSEY_ENDPOINT=http://localhost:9000
export FUSEY_ACCESS_KEY=... FUSEY_SECRET_KEY=... FUSEY_FORCE_PATH_STYLE=true
./fusey mount /mnt/fusey
# Run a compaction cycle (e.g. from a CronJob)
./fusey compact
# Print the fusey version and exit. This is the recommended way to
# identify which fusey release is running in a container
# (e.g. `kubectl exec $pod -- fusey version`).
./fusey version
# fusey v0.0.12 # the v0.0.12 release is installed
# fusey dev # built from `go build` / `go run` without release ldflags# Unit tests (no external dependencies)
go test ./...
# S3 integration tests (requires a running MinIO instance)
docker run -d -p 9000:9000 -e MINIO_ROOT_USER=test -e MINIO_ROOT_PASSWORD=testtest \
minio/minio server /data
FUSEY_TEST_ENDPOINT=http://localhost:9000 \
FUSEY_ACCESS_KEY=test FUSEY_SECRET_KEY=testtest \
go test ./internal/chunks/ -run TestS3Apache 2.0 — see LICENSE.