Repository navigation
Conversation
Proposes RFC `610.0000` (`Pachunk`), an in-place content-addressed installer and delta updater for the Flutter SDK.
|
I'm currently working on getting the prototype in a github repo to share. |
|
|
||
| --- | ||
|
|
||
| ### 3.5 The Universal Packfile, FlatBuffer Manifest, & `SHA-256` Verification |
There was a problem hiding this comment.
I am re-working my prototype and trying to remove the techdebt. I expect this section to change, but it does not change the RFC's goals and measures.
| 4. Host each release from a single static `<version>.pack` file on GCS/CDN for serverless delivery over HTTP/2 `Range` requests. | ||
| 5. Update in place (Zero-CAS) using the local files in `~/flutter` as the chunk cache instead of maintaining a duplicate object store on disk, and verify every reconstructed file against its authoritative `SHA-256` digest. | ||
|
|
||
| ### Critical User Journeys (CUJs) |
There was a problem hiding this comment.
What happens in terms of signing or validating hashes?
There was a problem hiding this comment.
For the packfiles? Nothing. We don't sign our zip or tar.xz files today.
The files that are signed (mac/ios binaries) remain signed and bit-for-bit equal.
What we should do: sha256 sum the packfile and toc on luci and upload to the same "release_*.json" files we use today. The tool can verify the toc hash quickly. We want to avoid downloading the entire 1.8GB file just to verify it and download it for the 50MB we need.
If I misunderstood the question, let me know.
There was a problem hiding this comment.
Today, the "Attestation bundles" linked from https://docs.flutter.dev/install/archive are .intoto.jsonl files whose payload contains the sha256 of the zip or tar.xz files. (This is for our SLSA compliance). We probably need to continue publishing the attestation bundle, but it could be for the packfiles instead of for the zip/tar.xz?
There was a problem hiding this comment.
Ah, I see. The Flutter Tool doesn't care about these in-toto files as far as I can tell (git grep for "in-toto" and "intoto" show nothing). I believe these intoto files are only for releases and not for commits on main.
The solution here would be produce these files for the packfiles as well. The packer could run at the same time as the tar.xz and .zip files are being generated, since it only relies on the zipfiles and flutter tool right now.
| ### 5.2 Open Questions for Reviewers | ||
|
|
||
| 1. How should `pachunk` integrate with `flutter_tools` and the official install/upgrade flow? | ||
| * Option A (Recommended): Make `pachunk` the primary installation method on `flutter.dev` so `flutter` "Just Works" out of the box. Because `flutter_tools` already uses `flutter precache` to download engine and tool artifacts today, have `flutter precache`, `flutter upgrade`, and `flutter downgrade` delegate to `pachunk` under the hood (migrating artifact download responsibility into `pachunk` over time) with an automatic fallback to legacy `.zip` archive downloads during transition. |
There was a problem hiding this comment.
For both of these options it's not clear to me from the descriptions:
- What does the user download from flutter.dev initially by clicking on a link.
- What are they missing at that point, and how do they get the rest.
Asking the user to copy-paste a command is not my favorite when others do it, but would be an option as well if it is technically necessary.
Also worth noting here that we currently document an "Install with VSCode" flow that would also need to be modified, I think? https://docs.flutter.dev/install/with-vs-code
There was a problem hiding this comment.
Good questions and worth arguing here - maybe I should leave off the recommended for now and just have it as an opinion.
- What does the user download from flutter.dev initially by clicking on a link.
I think a script or command much like https://brew.sh/ or https://github.com/jtmcdole/flutter_worktree#unattended-installation.
- What are they missing at that point, and how do they get the rest
If its the script, we could have it automatically install for the host/arch. If its just a binary, they would need to run it to download and install a version.
- We could consult
release_<host>.jsonfor latest stable / beta if they use--stableor--beta - We could offer code block copy-paste like brew for the different components.
VSCode: noted! I... assume devtools just downloads the sdk and didn't invent a new method of downloading Flutter? @kenzieschmoll
| * Recommended: Record the `sha256` of both `<version>.pack` and `manifest_<version>.fb.zst` in `releases_<platform>.json` and verify `manifest_<version>.fb.zst` before assembly. Is that sufficient, or do we need anything further for third-party mirrors (`FLUTTER_STORAGE_BASE_URL`)? | ||
| 5. Should we retire the `bin/internal/` bootstrap scripts (`.sh`, `.bat`, `.ps1`) and compile `flutter_tools` at install time? | ||
| * Today, every `flutter` command runs `bin/internal/` scripts (`shared.*`, `update_dart_sdk.*`, `update_engine_version.*`) to check locks, query Git, fetch the Dart SDK, and build snapshots. Because `pachunk` verifies the Dart SDK, engine files, and `flutter.version.json` during install or upgrade, these runtime scripts are obsolete. | ||
| * Recommended: Compile a standalone AOT native binary (`dart compile exe` or `aot-snapshot`) into `bin/cache/` during install and upgrade. Retire the `bin/internal/` scripts in `pachunk` installs, and remove them upstream once `pachunk` becomes the default installer. Keep `bin/flutter*` and `bin/dart*` (`sh`, `.bat`, `.ps1`) only as thin pointer scripts that `exec` the `bin/cache/` binaries directly. This lets users run `flutter` and `dart` without typing `.exe` across POSIX shells, CMD, PowerShell, and Git Bash. |
There was a problem hiding this comment.
The bootstrapping scripts currently download a Dart SDK as an early (first?) step. Where do they get the Dart SDK to run the rest of the code without the boostrapping scripts? (I think the answer here is "pachunk", but how do they get pachunk?)
Overall, let's say we take the recommended answers to all of these questions, I'm having trouble seeing exactly how the initial download/bootstrap/run flow works at a level of detail that gives me confidence that it will work. Can this doc dive into that a bit more?
There was a problem hiding this comment.
Yeah, I can go into that flow in the appendix if you like. "Expected workflows". But to quickly answer:
- Download pachunk
- pachunk downloads all the components (dart, engine artifacts, packages)
- pachunk compiles the flutter tool for your host-arch
- fin.
"flutter.bat" and "flutter" (bash) are only there as pointers to the compiled executables so the developer continues to see flutter run -d chorme.
I was chatting a little with @bkonyi about this and I think we'd like the simplicity of dropping some of these scripts.
There was a problem hiding this comment.
100% in favor of dropping gnarly shell scripts. Just want to understand how the design is intended to work =)
There was a problem hiding this comment.
Yeah, I think that's also "we can choose to do this in the future" incremental improvement and leave the scripts as-is today. There are some "please don't check git for tags or version" changes... "content aware hashing" continues to work for Contributors since they are working in git and need to solve for that.
- Remove "exact" numbers, e.g. 10,183 files, from all over the RFC - use relative numbers bigger: - address prototype changes (format of packfile, techdebt, pointer plane)
| - '"John McDole" <codefu@google.com>' | ||
| --- | ||
|
|
||
| # RFC 610.0000: Pachunk: Content-Addressed Packing & In-Place Delta Patching |
There was a problem hiding this comment.
The goals sound good and the overview of how we'll do it sounds good.
Skimming through it, how does this play with the .git directory? What will that look like after I checkout? How do I switch back to a git repo? I think some of these answers are scattered around but a clear declaration of that would be helpful.
Relying on git we get the advantage of git's testing. What are our plans for integration testing this to make sure it's correct in all situations. Running through typical CI workflows won't be enough since they won't duplicate situations developers will run into by default.
Some testing ideas from gemini:
• Messy local states: files the user edited or deleted, a half-finished update (killed process, Ctrl+C), legacy flutter precache mixed with pachunk, two flutter processes running
at once, and a stale .fast_cache.fb.
• Filesystem and OS cases: Windows file locks and Defender, symlinks and junctions, case-insensitive filesystems, permissions or read-only installs, a full disk, and long paths on Windows.
• Network cases: proxies or mirrors (FLUTTER_STORAGE_BASE_URL) that don't support Range or HTTP/2, dropped connections, and truncated responses.
• Version pairs: upgrade and downgrade tests across many pairs of backfilled releases, not just N→N+1. Also check that old clients reject newer PINF formats cleanly.
• Fuzz and property tests: round-trip zucchini split/merge and ZIP repacking on every binary in every release, plus fuzzing the manifest and pack parsers.
• Rollout data: telemetry on SHA-256 mismatches and how often fallbacks happen during the opt-in period, with a threshold that must be met before pachunk becomes the default.
There was a problem hiding this comment.
You're specifically wanting the view as a Contributor right (someone who has .git folders, the engine and dev folder?
There was a problem hiding this comment.
Yea, or an app developer that wants to try out a patch.
justinmc
left a comment
justinmc
left a comment
There was a problem hiding this comment.
This would save a lot of people a lot of time! Sounds great if we can pull it off.
One thing we should keep an eye on as we change this tooling is reliability, like the ability to recover from failed partial downloads, weird states, etc. Even the current tooling has bitten me many times here.
Also FYI the PR description says "610.0001" but everything else says "610.0000".
|
|
||
| We previously considered (and prototyped) splitting Flutter's monolithic `artifacts.zip` and `flutter.jar` packages into finer-grained per-ABI and per-tool `.zip` archives so `flutter_tools` could download smaller zips. We rejected stopping there for two reasons: | ||
|
|
||
| 1. You still have to download duplicate data on every update: even if `android-arm64` or `dart-sdk` is split into its own archive, any Engine or Dart commit still invalidates the entire archive (`~171 MB` for `android-arm64/artifacts.zip`, `~216 MB` for `dart-sdk-darwin-arm64.zip`, `~157 MB` for `darwin-x64-release/framework.zip`), forcing developers to re-download 95%+ unchanged bytes on every point release or `git bisect` step. |
There was a problem hiding this comment.
Would you mind expanding on how you get the 95% number? I think this is a key point that I want to understand for this RFC. There is an alternative that sounds much simpler than the proposal, so I want to make sure that we are in agreement that the more complex solution is worth it.
| We previously considered (and prototyped) splitting Flutter's monolithic `artifacts.zip` and `flutter.jar` packages into finer-grained per-ABI and per-tool `.zip` archives so `flutter_tools` could download smaller zips. We rejected stopping there for two reasons: | ||
|
|
||
| 1. You still have to download duplicate data on every update: even if `android-arm64` or `dart-sdk` is split into its own archive, any Engine or Dart commit still invalidates the entire archive (`~171 MB` for `android-arm64/artifacts.zip`, `~216 MB` for `dart-sdk-darwin-arm64.zip`, `~157 MB` for `darwin-x64-release/framework.zip`), forcing developers to re-download 95%+ unchanged bytes on every point release or `git bisect` step. | ||
| 2. Added complexity for LUCI recipe tooling and `flutter_tools`: splitting every platform, ABI, build mode, and host tool into dozens of separate `.zip` archives multiplies the artifact matrix that LUCI packaging recipes must build and upload, and that `flutter_tools` must track, download, and stamp. |
There was a problem hiding this comment.
Ok dozens of zip files is pretty excessive, but to be thorough, how many of those zip files change between typical releases?
|
|
||
| ### 4.2 Pairwise Binary Diffs (`bsdiff`, `xdelta3`, Upstream `Zucchini`, and Omaha) | ||
|
|
||
| Pairwise diff tools generate a directional patch from a specific old version ($V_A$) to a specific new version ($V_B$). Supporting arbitrary developer upgrades, downgrades ($N \to N-3$), channel switches, and contributor `git bisect` runs across thousands of Engine commits (`content_aware_hash`) would require precomputing and storing an $O(N \times M)$ matrix of pairwise patches on GCS. Upstream Chromium `Zucchini` also only parses basic `ELF` and `PE` binaries (lacking `Mach-O`, Universal Fat binaries, `DWARF`, `PDB`, and `DEX`), and its sequential `#ref_N` tokens shift whenever a function is inserted, defeating content-defined chunking. |
There was a problem hiding this comment.
What about supporting only adjacent version upgrades rather than arbitrary upgrades? If you are upgrading 3.47->3.50 then you get the fast path. If you are upgrading 3.47->3.53 then you either have to upgrade sequentially, or you redownload everything as you do today.
This misses lots of the benefits of the full solution obviously, but I think it's worth making the comparison.
|
|
||
| ## 5. Rollout Plan & Open Questions | ||
|
|
||
| ### 5.1 Phased Rollout Plan |
There was a problem hiding this comment.
It's good to see this can be out in phases like this. Are any of the intermediate steps below useful to contributors or app devs before the end (or I guess step 4 is done)? At first glance it seems like some of the pieces of this design might be pretty impactful on their own.
| 3. Is the Android team open to stripping non-alloc DWARF `.debug_*` sections from `libflutter.so` inside default `flutter.jar` builds? | ||
| * Today, unstripped DWARF `.debug_*` sections make up over 90% of `libflutter.so` inside `flutter.jar` (`android-arm64/artifacts.zip` is `~170 MB` compressed), even though `symbols.zip` already archives debug symbols separately. | ||
| * Recommended: Strip `.debug_*` from default `flutter.jar` artifacts and expose granular per-platform/per-ABI `--include-debug-symbols` flags in `pachunk`. |
| 4. Is recording `sha256` digests in `releases_<platform>.json` sufficient for `manifest_<version>.fb.zst` verification? | ||
| * We don't sign our `.zip` or `.tar.xz` archives today (the only signed binaries are for macOS and iOS); instead, we publish `sha256` values in `releases_linux.json` and its macOS/Windows equivalents. Because a client only fetches HTTP `Range` slices of `<version>.pack`, verifying a whole-packfile `sha256` would require downloading the entire `~1.8 GB` `.pack` file. Instead, `pachunk` verifies `manifest_<version>.fb.zst` against `manifest_sha256` in `releases_<platform>.json` before decompressing or parsing it. This check anchors the chain of trust before `pachunk` trusts any chunk `XXH3-128` or file `SHA-256` digest in the manifest. | ||
| * Recommended: Record the `sha256` of both `<version>.pack` and `manifest_<version>.fb.zst` in `releases_<platform>.json`. Is that sufficient, or do we need anything further for third-party mirrors (`FLUTTER_STORAGE_BASE_URL`)? |
There was a problem hiding this comment.
Would it be possible to publish a hash for the final local file(s) post-patching? Is there something that's always the same for everyone despite how they upgraded/downgraded to get there?
As written it seems like this might be vulnerable to a middle man modifying a chunk, but I'm no expert here. I wonder if Google has some standards or guidelines.
|
|
||
| * Minimize network bandwidth: cut fresh install downloads by **>50%** and point-release upgrades by **~90%**. | ||
| * Minimize local storage space: cut installed disk footprint by **55%–67%** with zero extra local CAS cache overhead on disk. | ||
| * Minimize time to update in any direction ($V_X \leftrightarrow V_Y$): support arbitrary upgrades, downgrades, channel switches, and contributor `git bisect` runs in **under 25 seconds** without precomputing an $N \times M$ patch matrix on the server. |
| * ***As a release/infra engineer***, I should be able to publish a single static `<version>.pack` artifact per build to GCS/CDN without precomputing an $N \times M$ matrix of pairwise patches. | ||
| * ***As an engineer rolling Flutter into Google***, I should be able to continue rolling Flutter and its engine artifacts into Google without disruption. | ||
|
|
||
| ### 1.1 Goals |
|
|
||
| --- | ||
|
|
||
| ## 4. Alternatives Considered |
There was a problem hiding this comment.
Nit: In addition to the big alternative approaches given below, I wonder if you did any analysis when choosing the specific tools that are involved here? I'm no expert in any of this, but it might be worth explaining why you chose FastCDC, zucchini, etc.
| 3. Integrate with `flutter_tools`: Land native gitless metadata support (`flutter.version.json`) in `flutter_tools` so `pachunk` no longer needs `bin/internal/` script patches (`trim_mutations`). | ||
| 4. Backfill historical releases: Backfilling is cheap (~2 minutes of machine time per release). We can start with `stable - 2` and use machines over a weekend to backfill several versioned releases so historical downgrades and release bisects work immediately. | ||
| 5. Add LUCI recipe testing: Add continuous LUCI recipe tests verifying fresh installs, upgrades, downgrades, and `--cache-only` syncs from `.pack` files. | ||
| 6. Gather community feedback: Collect telemetry and early-adopter feedback before making `pachunk` the default install and upgrade path on `flutter.dev`. |
There was a problem hiding this comment.
Even if/when it's the default, will people be able to opt out and do things the old way? Any reason they might want to? I'm mainly just thinking of the many times that switching flutter versions has gone wrong for me and I've ended up in a weird state.
4 participants
Proposes RFC
610.0001pachunk(from "patch + chunk"), an in-place content-addressed installer and delta updater for the Flutter SDK.Flutter distributes SDK releases and engine artifacts through full-history
gitclones and monolithic archives (.zipand.tar.xz) on GCS. A fresh release archive unpacks at least 2.5 GB on Linux (1.5 GBarchive), 3.3 GB on Windows (1.9 GBarchive), and 4.1 GB on macOS ARM64 (2.2 GBarchive; 6.24 GB onceflutter precachepopulatesbin/cache/)...What if delta-upgrades (3.47.5 -> 3.47.6) were only 50MB and 20 seconds?
What if you could upgrade/downgrade from any version to any version?
What if Flutter had a dedicated installer?
Tracking issue: flutter/flutter#194032
Pre-launch Checklist
design doc.dart run bin/assign_rfc_number.dartlocally when instructed).