rfc: FilOne Appliance Deployment Strategy

[bajtos]

📖 Preview

Summary

FilOne Appliance is a set of services operating FilOne node on infrastructure provided by a regional provider. This RFC proposes how to deploy and operate the appliance - first an MPV to get us off the ground quickly, then a direction towards a more robust & scalable solution.

[Copilot]

Copilot wasn't able to review any files in this pull request.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[pyropy]

Given the new routing policy RFC, would ingot nodes be able to re-route the requests during the 'draining' process?

[bajtos]

I don't know, I guess it's something we can investigate later.

[pyropy]

Are we going to aggregate logs in somewhere?

[bajtos]

For appliances operated by us, we should probably forward all logs to Grafana.

For appliances operated by region providers, it's up to them to decide.

[hannahhoward]

Broadly, this is getting close to GTG. (probably drop the raw notes before merge)

I do want to talk about zero downtime. Agree that it's an unrealistic goal for v1.

However, I think we should be clear about our goals and evolution:

1. I think it's important to get to zero downtime as quickly as possible for Ingot. My believe is this simply needs an audit for where Ingot is storing state that isn't fundamentally multiprocess compatible. I believe that is actually very few places. The goal is to clean up any surfaces and then move quickly to blue-green deployments via Caddy as a load balancer.

2. Next comes Piri -- my first thought is we can seperate the web interface of Piri which is near stateless from the task scheduling parts. This may not be as hard as we think as I think the Harmony task scheduler is pretty multiprocess friendly.

3. Postgres/OpenBao are really future work for super large providers.

I just want to capture this knowledge -- if we can keep Ingot up during upgrades, I think we everything else becomes a much later/downstream problem, especially since Ingot has a cache of most of the requested data and can continue serving many requests without Piri.

[hannahhoward]

My only blocker with the current design is we should clear the raw notes before merge. I'd like @frrist to sign off first though.

[frrist]

Thanks for putting this together. The research went deep and much of it I suspect will be valuable to reflect on as we build. But as it stands I think this RFC is doing too much: architecture doc, MVP plan, operating model, post-mvp plan, runbook, and research archive, for several environments at once. For me, that makes it hard to extract the actual decisions being made and why they are being made.

Given that, I'd propose a rescope to just a CD dev environment in the rough shape we want. The appliance material (Podman, immutable OS, partner trust boundary, pdp-gated windows, detailed runbooks, etc.) can then land as follow-up RFCs where each can be properly motivated, discussed, agreed upon, and land as a smaller scoped change.

A few thoughts on this as it stands in addition to the comments I've left in review:

• Podman + Quadlet needs to be motivated or deferred.: The proposal asserts it but never sells it. The alternatives compare against Portainer/Komodo/Kamal but never against the current baseline: plain docker compose. This deserves its own RFC and is a good thing to consider, but I don't believe we need to make a decision on this point yet.
• Provisioning is a gap: For dev and staging we own the whole stack, so the doc should say how the infra itself gets deployed: Terraform/Tofu for the machine, volumes, DNS, firewall, hardware requirements spec (disk size, CPU, memory, location, cloud provider), and splitting the services across different VMs (e.g. SP instance and Forge instance).
• Dev must be CD: The MVP section is currently waffling on this, but I think we are all in agreement now that CD is needed for Dev per your last comment there. Fortunetly the machinery in this RFC already supports this: the git-pull reconciler on a short timer is a CD engine and CI already auto-publishes new images on every merge to main. Policy (imho) being the differentiating factor across environments (How we promote release from dev -> staging -> prod, and how we validate them also deserves its own RFC).
• Secrets: defer to RFC rfc: regional security principles and key management deployment proposal #21 as the authority: Dev then takes the simplest path consistent with that design, which for now roughly looks like a Vault container running in Dev.

[frrist]

far as I am aware, Piri and Ingot don't store sensitive information in config files. All sensitive information is already kept external files.

[bajtos]

You are right that Piri config file does not contain any sensitive information 👍🏻

Based on fil-forge/smelt#11, Ingot config file contains the following secrets:

• Root S3 account key & secret
• postgres_dsn

See environments/staging/piri/config/ingot/config.yaml.tpl

[bajtos]

Actually, I think Piri will need Postgres connection string in its config file once we switch from SQLite to Postgres for storage.

https://github.com/fil-forge/piri/blob/93b85a6d5bf44ce299d3668674bb7d958cab2617/pkg/config/app/storage.go#L38-L41

type PostgresConfig struct {
	// URL is the PostgreSQL connection string (only used when Type is "postgres").
	// Format: postgres://user:password@host:port/dbname?sslmode=disable
	URL url.URL

[frrist]

I don't want to block this, so approved!

Your call if you want to incorperate any of the comments I left, they're mainly nits. Happy for a doc on a dev environment to land later, if at all.

Only blocking comment here is Hannah's about the notes - which I'd also like cut

[bajtos]

Responding to @frrist's comment:

Podman + Quadlet needs to be motivated or deferred.: The proposal asserts it but never sells it. The alternatives compare against Portainer/Komodo/Kamal but never against the current baseline: plain docker compose. This deserves its own RFC and is a good thing to consider, but I don't believe we need to make a decision on this point yet.

TL;DR: I need to reconsider this decision.

The long answer written by Claude and lightly edited by me:

Arguments for Podman + Quadlet

• Per-service unit granularity fits the mixed-cadence policy. Each service is its own systemd unit, so the reconciler restarts exactly one thing and leaves the rest untouched — directly serving "Postgres and OpenBao must never be auto-touched." Compose reconciles at the project level, and there are documented cases (Container needlessly recreated and then won't start on docker-compose up docker/compose#9357, [BUG] up Incorrectly recreates containers with no changes docker/compose#10259) of it needlessly recreating containers, including Postgres.
• Native on the immutable-OS path. bootc/FCOS ships Podman and does not ship Docker Engine; getting Docker there means rpm-ostree layering, a reboot, and ongoing maintenance against read-only /usr.
• Notify=healthy (Podman ≥5.0) makes a restart genuinely fail if the healthcheck never passes; OnFailure= gives alerting; ordering handles OpenBao-before-Ingot. Compose's equivalent (up -d --wait) is one project-wide gate, not per-unit.
• Rootless, daemonless on semi-trusted partner hardware. No root daemon socket to steal — a well-known privilege-escalation path in Docker's default posture. Fewer long-running moving parts overall.
• Better secrets story. systemd-creds / LoadCredentialEncrypted= gives host-bound, optionally TPM-sealed blobs, versus SOPS+age decrypting into the daemon's environment on the Compose side.

Downsides

• Distro version sensitivity. Ubuntu LTS ships Podman too old for Notify=healthy (24.04 has 4.9.3); Debian 13's 5.4.2 works but predates ReloadCmd= and podman quadlet (5.6.0), needing workarounds. Compose installs trivially anywhere via Docker's apt repo.
• The reconciler needs custom diff/restart logic. Compose's up -d --wait is itself the reconciler and only recreates changed services; with Quadlet you copy changed units, daemon-reload, and selectively restart yourself — daemon-reload alone won't restart anything.
• Weaker newcomer ergonomics and smaller ecosystem. "One YAML, docker compose up -d" is a friendlier mental model, and most GitOps-lite tooling (Portainer, Komodo) is Compose-only, so those options are foreclosed.
• No native zero-downtime story either way, but Quadlet's is more manual. Compose at least has the docker rollout plugin pattern; Quadlet requires hand-rolled blue/green with templated units.
• Live sharp edges. The Notify=healthy + TimeoutStartSec footgun (podman #27290) must be explicitly configured given Piri's up-to-60-minute drain, and the BoltDB→SQLite transition (#28216) is a known concurrency hazard for Quadlet-at-boot topologies.

Note: I rejected podman auto-update as the CD mechanism because AutoUpdate=registry never fires on digest-pinned images — it would make the registry, not git, the source of truth.

[alanshaw]

Seems as though we're replicating a systems package manager with auto updates enabled. Why don't we just ship an apt package (host our own apt source) and use unattended-upgrades (https://wiki.debian.org/PeriodicUpdates) (or alternative e.g. cron-apt) to get auto upgrades?

The package gets to administer service starts/restarts on install/upgrade via maintenance scripts and we get a really nice, easy, way to distribute the appliance.

Installing the appliance is a case of adding the aptitude source:

sudo add-apt-repository "deb http://deb.fil-forge.com/forge <codename> main"

...and install:

sudo apt install fil-one-appliance

...and adding auto updates:

sudo tee /etc/apt/apt.conf.d/51unattended-upgrades-custom << 'EOF'
Unattended-Upgrade::Allowed-Origins {
    "FilForge:stable";
};
EOF

[bajtos]

Interesting idea!

How do you propose to deal with the dependencies (Postgres, OpenBao, etc.)?

[alanshaw]

They are either dependencies of the package (so get installed automatically by apt when installing the appliance) or they just need to be installed out of band or like sudo apt install postgres openbao fil-one-appliance.

[alanshaw]

That said, I'm not sure these have to be local-local, as in on the same machine. As I understand it they need to be available on the local network...

[bajtos]

I love this idea as a solution for distributing the appliance updates for region operators not using our automation, I added a section to the proposal.

I also added an entry to "alternatives considered" to explain why I think this is not a good fit for our continuosly updated dev node.

[Copilot]

🟡 Changes recommended

The RFC contains multiple clarity/typo issues (including an inconsistent repository name reference) that should be corrected before merging.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (5)

rfcs/2026-08-fil-one-appliance-deployment-strategy.md:293

• This sentence repeats “requires” and is hard to read; rephrase to avoid the duplication.

Upgrading OpenBao requires restart that seals the vault requires the unseal round-trip to central —

rfcs/2026-08-fil-one-appliance-deployment-strategy.md:311

• There’s an extra space before the period in “restart .”, which is a typo.

Postgres is able to pick up most configuration changes via SIGHUP without restart . We should

rfcs/2026-08-fil-one-appliance-deployment-strategy.md:420

• “Trusted Executed Environment” is a typo; the common term is “Trusted Execution Environment (TEE)”.

Alternative to consider: Trusted Executed Environment, e.g. [Confidential Metal](https://confidential.ai/products#confidential-metal).

rfcs/2026-08-fil-one-appliance-deployment-strategy.md:465

• Typo at end of bullet (“directiveso”).

- Loses the systemd `OnFailure=` alerting path and per-unit cgroup directiveso

rfcs/2026-08-fil-one-appliance-deployment-strategy.md:480

• This bullet has a grammar error (“it not relevant”) and likely an incorrect acronym (“FOCS” vs “FCOS” for Fedora CoreOS).

- The biggest benefit - first-class support on FOCS/bootc - it not relevant, since we decided to use mutable OS in MVP for simplicity.

• Files reviewed: 1/1 changed files
• Comments generated: 4
• Review effort level: Lite

[alanshaw]

You have "We should aim for zero-downtime upgrades." in the list above.

[alanshaw]

I think _we_should run one of these for providers to use. Relying on chain.love or Gilf is ricky as we have no control over their operations.

[alanshaw]

Maybe worth mentioning which of these services must be run on the machine running the app. (Caddy, OpenBao(?)), which should be on the same network or machine (Postgres) and which can be external services (Lotus).

[alanshaw]

Does this generate the region KEK?

[alanshaw]

👌

[alanshaw]

We would instruct them to enable automatic updates for those packages.