Support for early cutoff

[ranger-ross]

Problem

Currently when updating a leaf dependency all crates that depend on it are rebuilt (both directly and transitively).
This means that a simple dependency bump can force large part of the dependency tree to recompile when it otherwise wouldn't need to.

Proposed Solution

Build systems like Bazel only rebuild a unit if the of tasks inputs have changed.
This is known as "Early Cutoff". ¹

This differs from Cargo's implementation in that the dependencies for a build unit are the dependents outputs.
Cargo only uses the inputs of dependency units, so if the output (rlib) does not change it still rebuilds dependent units.

The proposed solution is to modify Cargo's fingerprinting system to use build unit outputs (instead of inputs) when determining a units fingerprint. Avoiding rebuilds of the inputs of a previous build unit did not changes it output.

Notes

This may be a good opportunity to re-evaluate at a high level how fingerprinting works in cargo in general

Footnotes

1. Mentioned in Build systems à la carte. I don't know the origin. ↩

[abhinav-phi]

Hi! I'd like to work on this.

Plan: I'll design the feature against the described behavior, implement it with the project's conventions, add unit/integration tests, and update the docs.

Could you please assign this issue to me? Thanks!

/assign take

[weihanglo]

FWIW, when interoperating with mtime-based external build systems e.g, GNU Make, they usually require updating mtime for output artifacts. I doubt this is a "hard" requirement but a thing worth keeping in mind

See also

• fix(fingerprint): force update mtime of cargo-check artifacts #16262 (comment)
• rustc does not always update the mtime of all its outputs rust#148948

[epage]

I want to first establish some made up terms to try to reduce confusing in discussion.

Cargo's current hashing system for detecting build changes is to hash source inputs:

let unit_source_input_hash = hash(unit_build_flags, deps.map(|d| d.unit_source_input_hash));

• This can be calculated before the build starts.
• Source files aren't included in this, but instead in a fingerprint, so we can reuse the the source code enumeration from a previous build of unit_source_input_hash. This also acts as a cheap GC

The proposed system is

let unit_build_input_hash = hash(unit_build_flags,. deps.map(|d| d.unit_output_hash));
// ...
let unit_output_hash = hash(read_to_string(rmeta_path));
// or for build+link
let unit_output_hash = hash(read_to_string(rlib_path));
// or for build+link with -Zembedded-metadata=else
let unit_output_hash = hash(read_to_string(rmeta_path), read_to_string(rlib_path));

• If we implement this with a CAS and apply it to local builds, we'll need an aggressive GC
• This can only be calculated during a build.
  • iirc all of the hashes are immutable during a build so this will require some re-work

We would need to decide how fingerprinting fits into this. Ideally, fingerprinting still happens before the build so we have more flexibility in speeding it up. That would require us to still have a unit_source_input_hash since fingerprints are looked up by that.

This is tightly related to relink-dont-rebuild (#14604). That too involves early cut off but only for build-only crates (not build-and-link or buildscripts) and it is unclear if the hash will be reported by rustc through a json message or by having a separate file (#t-compiler/relink-dont-rebuild > David's RDR experiment / relationship to LLM policy @ 💬).

If we assume relink-dont-rebuild uses a separate file that gets hashed, this issue becomes a pure subset. This issue will help avoid unnecessary links as well as changes that don't impact the rmeta in any way.

Like with relink-dont-rebuild, stable DefIds is required. rustc build outputs are bit-for-bit stable for the same build inputs but not if the build inputs change in uninteresting ways. Technically, we aren't blocked on them but won't see much or any benefit.

I also think we should push for -Zlink-only to be done first so we split build+link jobs into build and link,

• allowing for better early cut off (don't rebuild if only a relink is needed)
• simplify our calculations

I would say that build scripts could be a way for us to experiment with all of this without waiting on rustc. However, the outputs of a build script are hash(directives, out_dir.files().read()) and OUT_DIR tends to be used as a scratch pad. We may need to get a scratch pad (#t-cargo/build-script > Providing a dedicated scratchpad, instead of using `OUT_DIR`) and work to transition the ecosystem to use it before we can meaningfully (and in a performant way) has OUT_DIR.

[epage]

Before, I mentioned that rustc is meant to be bit-for-bit compatible based on build inputs. Proc-macros and build scripts can sabotage this.

Option for reproducible proc-macros

• wasm sandboxing
  • likely slower
  • removes de-duplication of build jobs between host and target
  • many people assume you can do pre-built binaries with these but that isn't the case without opaque deps (How should dynamic linking work in Rust? #3573 (comment))
• linting for HashMap::iter
  • shallow
  • incomplete
• alloc-only proc-macros (determined through build-std putting std deps in manifests)
  • technically ffi could interfere still
    • also playing with having cargo pass --forbid unsafe unless rust:unsafe is activated by a feature
  • serde would likely be fine
  • clap reads env variables

Option for reproducible build scripts

• Reduce the need for users to write build scripts #14948
• Static build script fallback when disallowd (https://rust-lang.zulipchat.com/#narrow/channel/246057-t-cargo/topic/build.20script.20and.20proc.20macro.20allow.20list/near/595506317)
• wasm sandboxing
• OS native sandboxig
• For sys crates, find a way for cargo to reason about what is happening
  • if linking against system crate and doing no special work, this only makes the link step non-reproducible across systems
  • if building against source, some way to assert to cargo that that process is reproducible

What do we do when there are non-reproducible proc-macros and build scripts?

[weihanglo]

If we implement this with a CAS and apply it to local builds, we'll need an aggressive GC

CAS for unit output may not work well with a shared/user-wide cache without trim-paths. And even with trim-paths, there is a limitation in --remap-path-prefix that it doesn't remap absolute paths in metadata section in rlib/rmeta if the --remap-path-scope != all.

Bazel and Buck2 avoid similar problems by giving some stable paths and materializing outputs from a centralized store (via some hardlink store IIRC).

[epage]

Trim-paths won't affect early cut off on its own, but shared caches. Even then it just reduces cache reuse but doesn't poison it.