Skip to content

Create Lean projects atomically with catalog-backed defaults - #96

Merged
Deicyde merged 19 commits into
mainfrom
project-new-flexible-release
Oct 6, 2026
Merged

Deicyde merged 19 commits into
mainfrom
project-new-flexible-release

Conversation

@Deicyde

@Deicyde Deicyde commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Supersedes #86 and the closed #26.

Summary

  • add autoform project new <TARGET> --package <Name> for a complete Lean + Autoform repository: Lean shell, exact catalog lock, blueprint vault, site configuration, ignore rules, and optionally pinned CI
  • default to the bundled recommended Lean/Mathlib release; advanced release-tag inputs remain unlocked until lake update and are reported explicitly
  • stage and verify the complete tree in a private sibling, then publish it with an atomic no-replace rename; existing targets are never overwritten
  • expose stable autoform-project-creation/v1 JSON and make Setup use the copied plugin through uv run --project, with no second init after project new
  • make init safely append missing Autoform ignore rules to an existing single-link regular .gitignore

Safety and reproducibility

  • retain and reverify the caller's exact parent spelling; every symlink component is refused, including macOS /tmp (use /private/tmp)
  • bind the parent and staged tree through descriptors, verify bytes, modes, links, ownership and directory generations, sync before/after publication, and report uncertain commits without retry advice
  • validate catalog manifests, module-root collisions, Git revision spellings, and every derived Lake artifact filename before writing
  • read scaffold templates as bounded, stable, regular non-link snapshots and canonicalize generated modes to 0644/0755
  • infer workflow pins only when retained bytes match the committed tree and a cached tracking ref for a safe remote contains HEAD; prefer the canonical facebookresearch source and ignore local Git replacement objects
  • append .gitignore rules through the retained descriptor so a concurrent replacement is never overwritten; hard links are refused and partial appends are reported for inspection
  • fail closed where the required POSIX descriptor, locking, sync, or no-replace primitives are unavailable, including Windows

Validation

  • Ruff, diff checks, and make check-example pass
  • local current-main integration runs pass 348 creation/scaffold tests and 269 additional copied-plugin, inspection, skill, and work-frontier tests
  • both GitHub runs pass the complete Python 3.10 and 3.13 suites, Windows hardening, CLA, and the real-Lean job that creates and builds the bundled release

Add `autoform project new`, which stages a complete Lean project (toolchain,
lakefile, the release's bundled lake-manifest.json, a root module, and the
`autoform init` scaffold) in a private sibling directory, verifies it, and
publishes it with a no-replace rename. It never overwrites an existing target
and runs no Lake or network operations.

This carries #86's creation fixes over to main without the plugin-provenance
dependency. Template modes are normalized, so installs under umask 002 work.
project new now defaults --release to the catalog recommended release.
--lean-toolchain, with an optional --mathlib-rev that defaults to the
tag of the same name, creates a project for a pair the catalog does not
list: it writes no lake-manifest.json and warns that lake update must
resolve and lock Mathlib. A pair that matches a catalog release, by tag
or commit, still gets the bundled lock. Toolchains below Lean v4.27.0
also warn, since the skeleton probe and the generated CI audit need it.

Package names may no longer shadow the Mathlib roots docs, LongestPole,
or Wanted, which no creation descriptor lists. Human output prints
warnings on stderr; JSON adds a warnings array, and release is null for
an unlisted pair.
The docstring said Lake reads manifest versions up to 2.0.0, but Lake
and the inspector both reject 2.0.0 and later; they read any 1.x. Test
that 2.0.0-rc1 is refused and that 1.3.0, which Lake v4.35 writes, and
later 1.x versions are read.
Parent problems now have their own codes. A missing parent or ancestor is
project-parent-missing, one that cannot be read or searched is
project-parent-inaccessible, and the project-parent-unsafe message names
the chmod that fixes a group- or world-writable parent. macOS reports a
symbolic link opened with O_NOFOLLOW as ENOTDIR, which is now classified
as a link. Waiting for another creator's lock gives up after 30 seconds
with project-parent-busy instead of blocking indefinitely.

Toolchain version components are limited to nine digits, so an oversized
tag is project-version-invalid rather than a slow int() conversion. The
unlisted warning says that a catalog pair is recognized by its tag or full
commit.

On the CLI, missing target and --package errors say what is required, an
interrupt exits 130 with project-create-interrupted and mentions the
hidden stage it may leave, and the omitted-workflows warning gives the
init command that adds them. Only project new escapes its report to ASCII;
inspect keeps printable Unicode, as on main.
A parent that can be read but not searched now fails its first lookup
as project-parent-inaccessible, and a parent that cannot be written
fails stage creation with the same code and a message naming the
missing write permission; both were project-create-failed. A regular
file swapped into the parent path after validation is reported as
project-parent-invalid instead of project-path-is-symlink. macOS
reports links and other non-directories alike as ENOTDIR, so the
component is examined before it is called a link.

An interrupt during create_project's final cleanup can follow a
successful publication. When the target did not exist beforehand and
does now, the CLI says it may be this run's complete project instead of
saying that nothing was published. The omitted-workflows hint keeps an
explicit --autoform-source, which init would otherwise replace with
this checkout's origin.

New tests pin project-parent-missing, the not-a-directory parent, the
_open_parent errno routing, and the reworded unsafe-parent and unlisted
messages. The missing-option CLI test no longer depends on the mode of
the working directory.
@Deicyde Deicyde changed the title Create Lean projects atomically for any Lean release Create Lean projects atomically with catalog-backed defaults Oct 5, 2026
@Deicyde
Deicyde marked this pull request as ready for review October 5, 2026 17:25
@Deicyde Deicyde added the review: ready Review complete with no known merge blockers label Oct 5, 2026
@Deicyde

Deicyde commented Oct 5, 2026

Copy link
Copy Markdown
Contributor Author

Exact-head adversarial review is complete at 6e43264f. The branch is merged with current main and now closes the parent-alias publication race, unreachable/fork workflow pins, Git replacement-object spoofing, template snapshot/mode drift, unsafe Git ref and filename inputs, concurrent .gitignore overwrite, copied-plugin PATH assumptions, and the stale Setup helper references. Both GitHub runs pass Python 3.10/3.13, Windows hardening, real Lean project creation/build, and CLA. No known code blocker remains; this is ready for human review.

@Deicyde

Deicyde commented Oct 5, 2026

Copy link
Copy Markdown
Contributor Author

Exact-head review is refreshed at 6f2d4150. This is the reviewed project-creation change restacked onto current main, with the one-line README contract correction included. The PR is mergeable.

Across the exact-head runs, Python 3.10, Python 3.13, Windows, real Lean project creation/build, and CLA each have a successful check. The duplicate jobs shown as cancelled were runner shutdowns before steps; the dedicated real-Lean rerun completed successfully. No known code blocker remains, and #137 becomes redundant when this lands.

@Deicyde

Deicyde commented Oct 5, 2026

Copy link
Copy Markdown
Contributor Author

Merge conflicts are resolved and published at exact head eeb59260, which now contains current main through merged #95, #118, #125, #126, #135, and #137. The resolution preserves both project-creation release generation and #95's descriptor/source-evidence rules; the development skill remains within its 220-word contract and both assertion sets pass.

Focused skill tests pass (23), Ruff and diff checks are clean, and the executable example passes. The broad local integration run passed 601 tests; its two failures were host-load timeouts (a real-Lake build and the 15-second daemon startup check). The wheel/daemon check passes with a longer diagnostic startup budget, and the committed 15-second contract is unchanged. GitHub's isolated exact-head Python, Windows, and real-Lean runs are now in progress; the PR is mergeable with no unresolved files.

@Deicyde

Deicyde commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

Final exact-head gate at eeb59260: mergeable/CLEAN, non-draft, and every check passes. Both workflow runs pass Python 3.10, Python 3.13, Windows, and the real-Lean project-creation/build job; CLA passes as well. The branch includes current main through #95/#118/#125/#126/#135/#137, and there are no unresolved conflicts or known code blockers.

@Deicyde
Deicyde merged commit e5dd05a into main Oct 6, 2026
9 checks passed
Deicyde added a commit that referenced this pull request Oct 6, 2026
Conflicts:
- README.md: keep main's Lean v4.27.0 floor and this branch's Intel Mac
  sentence.
- skills/setup/SKILL.md: keep this branch's MathJax paragraph and main's
  longer paragraph on how init pins CI.
- autoform_cli/__main__.py, audit.py, render.py and their tests: take the
  union of the imports. render_site turns a source-index OSError into a
  PublicationError, as main does.
- autoform_cli/lean.py and tests/test_lean_sources.py: main's snapshot model
  (#95) plus this branch's `#exit` handling, review-packet output schema,
  output-stage exclusion and `elsewhere` declarations, with their tests.
- autoform_cli/skeleton.py: main's labelled run_probe (#136), which the
  impact probe shares, plus this branch's compiled helper module. The helper
  is built only when the probe imports it, so the impact probe skips it, and
  the label reaches the lake-manifest and shadowed-module messages.
  extract_skeletons reports an unreadable source tree with
  index_failure_message, as main's other commands do.

Semantic:
- tests/test_impact.py: the impact probe now names declarations with fully
  qualified `Lean.Name.str` terms, through this branch's `_lean_name`.
- autoform_cli/scaffold.py: the required template paths drop
  blueprint/javascripts/mathjax.js, which this branch deleted, and add the
  review marker and the review-gate workflow.
- tests/test_skeleton.py: drop the test that a tagged-process scan skips a
  process it cannot read, which main's #118 process scan contradicts.
- tests/test_scaffold.py: this branch's workflow tests pass an explicit
  autoform_ref, as main's do. Since #96, plugin_pin finds no pin for a
  commit that no remote-tracking ref contains, and on such a commit init
  omits the workflows these tests read.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot. review: ready Review complete with no known merge blockers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant