GHA caching of cargo dependencies

[thomasp85]

Quick compilation is not Rusts strength and for packages with heavy dependencies check times becomes prohibitively long as the dependency tree needs to be rebuilt in every action. Can you provide guidance on how to best cache compiled crates between workflow runs?

[CGMossa]

Hello Thomas. Nice to see you around here.

We are actively working on this area, as there are many things that need to be aligned.
First, CRAN-check necessitates removing build artefacts within every build call. This sets the bar, that if your CI has to do CRAN-check, you will atleast need to have that part of your process.

Then, devtools/pkgbuild calls to rebuild when any Cargo.toml or *.rs file has changed in src/. This means that you need to add

Config/build/extra-sources: src/rust/Cargo.lock

to ensure that the lockfile is also part of this awareness.

Then, you need to use this

Config/build/never-clean: true

to avoid the flushing of build artefacts.

The target directory has to be outside of src/.

I am actively trying to vibe code a re-imagining of cargo vendor such that it
can not re-vendor unnecessarily, as cargo vendor does, and such that it points to
the vendor directory properly. See progress on this here https://github.com/A2-ai/miniextendr/tree/main/cargo-revendor

There are overlapping issues; In dev-mode (NOT_CRAN unset), we wish not to flush dependencies, and yet we wish for the R build process to be invoked whenever rust dependencies change. Then you've got the vendoring issue. My goal is to have the vendor-tarball be the thing that ought to be cached and have that alleviate CI run time.

Sorry, I might have forgot other things. Unfortunately it is hard to keep track of.

[CGMossa]

I forgot to say, that once I have figured all this out, we are definitely prioritizing getting this into extendr. However, despite relentless effort, I keep finding bugs and issues to resolve. I am close however.

[JosiahParry]

@thomasp85 we're in a bit of a pickle here, to be honest.

There's 3 things going on.

1. Makevars.in uses the vendored deps by default which is required to not get Warning: Downloads Rust crates. I've asked in r-devel to make this a check under --as-cran and was vilified
2. Caching requires us to persist the targets directory however if we do we get yet another CRAN warning: Warning: File ‘arcgisutils/libs/arcgisutils.so’: Found ‘abort’, possibly from ‘abort’ (C)
   • to get rid of this we need to remove intermediary objects. That's how its handled in here, hellorust, and savvy
3. If address the second warning then we cannot cache.

You can see the process i was making at https://github.com/R-ArcGIS/arcgisutils/actions/runs/24634401909/job/72027175673

Actions we can take:

• Contact r-devel and ask if we can put the download warning behind an env var flag like other checks citing this issue
• Explore doing a targeted clean of targets/ to
  • delete only .o files and add keep .rlib files for incremental compilation
  • permitting rust-cache@v2 to actually work
• Update Makevars.in to only use vendored deps when NOT_CRAN != "true"

[CGMossa]

Thank you josiah. I didn't manage to answer the actual question.

[thomasp85]

Thanks for the responses. I understand this is not a simple ask, especially due to the vending requirement from CRAN (which is somewhat ridiculous tbh)...

Is it possible to fix this with the much narrower scope of GHA? Basically provide some glue that pulls out compiled crates and caches them, and then inserts them in the next run?

[CGMossa]

I think there is. We always aim for the comprehensive fixes, because we are somewhat idealists. But I think for this we should definitely consider things.. If you have ideas, let us know. I need to grog this.

[JosiahParry]

@thomasp85 is your use case strictly R CMD check in GHA? Or ensuring that the crate compiles?

[JosiahParry]

I've made a good amount of progress in my own R package as a proof of concept. See this latest run.

This gets rid of both warning mention. There's a few major things that have to happen for when NOT_CRAN=true:

• Retain incremental build artifacts from /targets
• Set the CARGO_TARGET_DIR via env-var to a persistent location to avoid R building the target in a temp dir
• Build a custom rust-cache using actions/cache and actions/restore

I'm still working on the Makevars.win.in but I suspect this is mostly correct.

Note

The first run will have WARNING: Downloads rust crates.
This run is necessary to build the cache. Subsequent runs will not need to download crates.
That is unless the dependency tree (Cargo.lock file) changes and requires downloading.

AI Summary

src/Makevars.in:

1. TARGET_DIR = @TARGET_DIR@ — makes target dir configurable via config.R
2. Vendor extraction gated on NOT_CRAN != "true" — skips vendor tarball in CI
3. CARGO_HOME only set when using vendor tarball — allows ~/.cargo to be used in
   CI
4. rust_clean preserves target dir when NOT_CRAN=true, only deletes
   arcgisutils-.rcgu.o, arcgisutils-.d, and libarcgisutils.a

tools/config.R:

1. Reads CARGO_TARGET_DIR env var, falls back to ./rust/target — sets
   @TARGET_DIR@
2. @CLEAN_TARGET@ expands to "" when NOT_CRAN=true or debug — preserves target
   dir

.Rbuildignore:

1. Added ^rust-target$ — excludes the CI target dir from the package tarball

.github/workflows/R-CMD-check.yaml:

1. NOT_CRAN: true in job env
2. CARGO_TARGET_DIR: ${{ github.workspace }}/rust-target — stable path outside
   check sandbox
3. actions/cache/restore@v5 + actions/cache/save@v5 with if: always() caching
   ~/.cargo/registry, rust-target/release/deps, rust-target/release/build

[hansvancalster]

@thomasp85 is your use case strictly R CMD check in GHA? Or ensuring that the crate compiles?

For the R CMD check in GHA use case, I have been using rust-cache and rust-toolchain actions and they seem to work fine. See the workflow file.

EDIT: I should have read this thread more carefully: "permitting rust-cache@v2 to actually work". I was under the impression that it worked, but on closer inspection the cache size was too small. A LLM told me it only cached the downloaded source code of the dependencies, but not the compiled binaries. So, keep up the good work of finding a better solution!

[eitsupi]

Ultimately, it depends on what you want to do, but I think prqlr has been caching Rust builds on GitHub Actions for years.
https://github.com/PRQL/prqlc-r/blob/main/.github/workflows/check.yml