Skip to content

Use the core script-output constants - #101

Open
TorecLuik wants to merge 46 commits into
masterfrom
use-core-output-constants
Open

TorecLuik wants to merge 46 commits into
masterfrom
use-core-output-constants

Conversation

@TorecLuik

@TorecLuik TorecLuik commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #100
Refs NL-BioImaging/biomero#110

Core: the scripts need the matching core release (it contains biomero#111); scripts and core are released together. No runtime check is added to each script. The VERSION of every script is locked to the core release and is bumped in the release preparation, not in this PR.
Conflicts resolved in this branch (it contains #97, #88, #90 and #95, which merge before it): SLURM_Run_Workflow.py (kept #97's RESULTS_FOLDER_NAME/ZARR_ROOT_FILES, dropped the local CANONICAL_INPUTS_OUTPUT, took #90's stage functions and applied the constants to their new literals) and SLURM_Run_Workflow_Batched.py (took #90's register_detached_batches, MESSAGE_OUTPUT for the launcher message). Dead commented-out setOutput("File_Annotation", ...) lines removed; touched long lines reflowed.

Producers and readers of the outputs Message, URL, File_Annotation and Canonical_Inputs use biomero.constants.results.MESSAGE_OUTPUT, URL_OUTPUT, FILE_ANNOTATION_OUTPUT and CANONICAL_INPUTS_OUTPUT instead of string literals, in SLURM_Run_Workflow.py, SLURM_Run_Workflow_Batched.py, SLURM_CellPose_Segmentation.py, _SLURM_Image_Transfer.py, SLURM_Get_Results.py, SLURM_Import_Results.py, SLURM_Get_Update.py, SLURM_Remote_Conversion.py and _SLURM_File_Transfer.py. The two local CANONICAL_INPUTS_OUTPUT definitions are removed. Results_Path is left to scripts#97. The "type": "URL" of the OMERO URL value and the internal UI_messages keys of the batched script are not output names and stay. The admin scripts only show a message to the user (no reader) and are unchanged.

No behaviour change: the strings are identical.

Tests

Scripts tests live on test-suite: companion PR #102 adds a contract test that fails if a script uses the literal, and gives the existing tests that run script functions in a bare namespace the constants. Harness (test-suite with #102) against this branch: 606 passed, 9 skipped, core 2.9.0b26 from PyPI. The contract test flags every string constant equal to one of the four names (except docstrings, "type": "URL" and the batched runner's UI_messages keys).

Merge order (production and test PRs, interleaved)

Each step keeps master CI green: every (production state, test-suite state) pair on this path was run locally with core 2.9.0b26 from PyPI (482, 484, 484, 513, 524, 588, 594, 597, 606 passed; no failures). The production PRs merge cleanly in this order: each branch already contains the conflict resolutions with the PRs before it.

  1. test PR Tests: install biomero core 2.9.0b26 in the harness #103 (test-suite: biomero>=2.9.0b26,<3 in tests/requirements.txt, so core-dependent tests run in CI)
  2. test PRs Tests: schema imports, metadata view imports and workflow color #89, Tests: detached resume, cancel and registration #91 and Tests: per-directory canonical sidecars and unwritable folders #99 (capability-gated; they pass for the current master and for the new source)
  3. production PR Remove the input marker from results after import on every path #97
  4. test PR Tests: remove the input marker from results after import #98 (fails on master before Remove the input marker from results after import on every path #97, by design)
  5. production PR Import Zarr contracts from biomero-schema and own the workflow color #88
  6. production PR Detached processing: own launcher registration, resume and cancel in the scripts #90
  7. production PR Migration script: move canonical sidecars via importer helpers #95
  8. test PR Tests: shared output names come from core constants #102
  9. production PR Use the core script-output constants #101 (this PR)

A production PR's own check stays red until the test PRs before it in this list are merged (CI runs the shared harness from test-suite, which is not pinned); merge the test PR, then rerun the check.

🤖 Generated with Claude Code

TorecLuik and others added 19 commits November 12, 2026 14:28
- Import the Zarr contracts from biomero_schema.zarr; biomero.zarr_contracts
  is removed from core.
- Import the metadata view functions from biomero.metadata_views (renamed from
  biomero.provenance) and format reduced values with core's
  format_large_field_reference, so core recognises them on refresh.
- Move the Blueprint workflow color to SLURM_Run_Workflow.py, which forwards
  the resolved color to the result scripts.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…lation in the scripts

- Registration builds the launcher params from the script client and hands
  them to WorkflowTracker.add_launcher_task (core only sets the marker).
- Resume reads the task aggregates (completed/failed) instead of a second
  source, raises when the history cannot be read, never re-attaches to a
  failed/cancelled/timed-out Slurm job, and skips a workflow whose results
  were already imported instead of submitting it again.
- The pipeline stops between steps and between polls when a cancel was
  requested, cancels its Slurm jobs, closes its tasks and fails the workflow.
- One reader of BIOMERO_DETACHED_WORKFLOWS: biomero.detached.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
- Fail with a clear message when the installed biomero predates
  biomero.metadata_views; README states the minimum core release.
- Move the colour palette to the module constants area of the runner.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…stages split

- WorkflowCancelled raised inside export, conversion and submission is no
  longer turned into a generic error by those stages' handlers.
- cancel_workflow_run fails every open task of the workflow (not only the
  Slurm-job tasks) and cancels the jobs found in the task history.
- admin/SLURM_Cancel_Workflow.py requests a cancel for a workflow id (owner
  or administrator), so a detached run can be cancelled end to end.
- run_pipeline_stages split into named stages that return their state.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…neRun, monitor split

- Stage handlers re-raise any WorkflowInterrupted without failing tasks; only
  WorkflowCancelled makes the run cancel jobs and fail tasks and workflow.
- execute_workflow_pipeline(stop_check=...) is an explicit hook for the caller
  (a detached worker that lost its claim); stage arguments grouped in
  PipelineRun.
- monitor_slurm_jobs split into per-state handlers; one failed-job path.
- cancel_workflow_run fails tasks and the workflow even when scancel raises.
- Cancel script: finished workflows record nothing, reason bounded, denial
  logged once, plain copyright header.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…, nits

- PipelineRun.raise_if_claim_lost: checked after the poll's progress read and
  before the outcome write (a late cancel still completes); a stop check also
  between conversion and submission.
- cancel_workflow_run: reading the tasks is inside the scancel handling; a
  failure puts 'Slurm jobs [...] could not be cancelled' into the reason.
- Dead handler in run_workflow removed; run parameter renamed pipeline; types
  and docstring for tracker.
- Cancel script message mentions inline batched runs do not honour cancel.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The workflow runner now owns the cleanup of .biomero-input.json: after the
result import succeeded it removes the marker from the imported result folder.
SLURM_Import_Results.py reports that folder as the Results_Path output.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
biomero.zarr_contracts is being removed from core; the marker constant is
defined in biomero_schema.zarr, which the scripts already import.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ogether

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The harness still expects the CanonicalInput import from
biomero.zarr_contracts until #88 and its tests merge; only the new
TRANSFER_INPUT_MARKER import comes from biomero_schema.zarr.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
- The admin migration passes the importer's PROCESSED_DATA_FOLDER explicitly,
  moves sidecars with move_canonical_marker and cleans up with
  remove_external_canonical_markers (no empty .biomero folders left).
- upgrade_reused_canonical no longer fails the workflow when the BIOMERO
  folder is not writable: it logs a warning and keeps the registration
  unverified.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Accept only .analyzed/<workflow>/<timestamp>, never fail the run on a cleanup
error, warn on unreadable folders, drop unreachable parameters, name the
shallow storage condition and document the Results_Path output.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Extracted from runScript so the persisted launcher params, the contract with
the supervisor's batched parent, can be tested. No behaviour change.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Both scripts take RESULTS_PATH_OUTPUT from core (biomero#110), like
WORKFLOW_UUID_OUTPUT, instead of defining it locally.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Producers and readers of the Message, URL, File_Annotation and
Canonical_Inputs outputs use biomero.constants.results instead of their own
string literals. Results_Path is handled by the result-path PR.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
- upgrade_reused_canonical treats a corrupt, uncommitted or unreachable
  sidecar (OSError, InvalidCanonicalStore, ValueError) as absent, with a
  WARNING naming the path and reason.
- The transfer script passes the importer's PROCESSED_DATA_FOLDER to every
  sidecar call, as the admin migration already does.
- _move_canonical_store renames the store back, or raises a clear error, when
  moving its sidecar fails after the rename.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
TorecLuik and others added 7 commits November 13, 2026 16:07
Resolve the gated import block with #97 (the input marker cleanup, which
merges first) to
`from biomero_schema.zarr import CanonicalInput, TRANSFER_INPUT_MARKER`.
Name the released minimum core version (2.9.0b26) in the README and in the
import error messages.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
#97 and #88 merge before this PR; merging them here applies cleanly.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Conflicts with the PRs that merge before this one, resolved by taking their
code and applying the same mechanical replacement of output-name literals
with biomero.constants.results.*_OUTPUT:
- SLURM_Run_Workflow.py: keep RESULTS_FOLDER_NAME and ZARR_ROOT_FILES from #97,
  drop the local CANONICAL_INPUTS_OUTPUT; the export, conversion and import
  stages of #90 use MESSAGE_OUTPUT, URL_OUTPUT and FILE_ANNOTATION_OUTPUT.
- SLURM_Run_Workflow_Batched.py: use #90's register_detached_batches and
  MESSAGE_OUTPUT for the launcher message.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…er>=1.5.0b14

The merge is clean. The scripts declared no importer minimum anywhere, so
the README now names biomero-importer>=1.5.0b14 (it provides the
PROCESSED_DATA_FOLDER and sidecar helpers this PR imports).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The merge is clean.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
TorecLuik and others added 5 commits November 16, 2026 14:23
…e stores

The result folder is the ANALYZED_DATA_FOLDER setting (read_analyzed_data_folder),
not a local constant. The marker sits at the root of each exported store, one
level below the result folder, so the sweep no longer detects Zarr stores.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Scripts and core are released together, so the README and the import
error name the renamed module, not a version floor.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@TorecLuik

Copy link
Copy Markdown
Collaborator Author

Independent review

Scope: only the forward-merge commit ec6b1a9 ("Merge #95 into use-core-output-constants"), compared with both parents (9def6e0 and a19f4d7) via git show --remerge-diff. The PR's original content was not re-reviewed.

Verdict: the merge is correct. No blocking findings.

  • The only conflict is the module-constant block of __workflows/SLURM_Run_Workflow.py. Parent 1 still carried the old RESULTS_FOLDER_NAME / ZARR_ROOT_FILES (removed by Remove the input marker from results after import on every path #97 since), parent 2 carried CANONICAL_INPUTS_OUTPUT = "Canonical_Inputs". The resolution drops both, which is right: Remove the input marker from results after import on every path #97 removed the first two, and this PR moves the last to constants.results.CANONICAL_INPUTS_OUTPUT. All remaining uses in the runner (lines 259-266) and in _SLURM_Image_Transfer.py (line 3358) use the core constant; git grep finds no leftover reference to any of the three names.
  • No conflict markers; the touched scripts parse; no duplicate top-level definitions; read_analyzed_data_folder (3 uses in the runner, 4 in the import script) and constants.results.RESULTS_PATH_OUTPUT are intact; no biomero>= floor.
  • Harness check (local only, not pushed): test-suite (093f974) + test PR Tests: remove the input marker from results after import #98 + origin/use-core-output-constants-tests merged cleanly and run against this head with core from analyzed-data-folder: 611 passed, 9 skipped (symlink, POSIX directory sync, standalone-entrypoint and omero import skips only).

Nothing found in any category for the merge.

🤖 Generated with Claude Code

TorecLuik and others added 5 commits November 16, 2026 14:55
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…constants

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@TorecLuik

Copy link
Copy Markdown
Collaborator Author

Resolved in d34083c: #95 (160938c) merged in, clean.

🤖 Generated with Claude Code

TorecLuik and others added 5 commits November 16, 2026 16:03
…the folder name

Core will not provide an ANALYZED_DATA_FOLDER setting for 1.8 (tracked in
NL-BIOMERO#111). SLURM_Import_Results.py keeps one RESULTS_FOLDER constant,
and the marker sweep accepts only a result folder of this run's workflow:
<...>/<workflow id>/<timestamp>. No general constants remain in the run script.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…constants

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@TorecLuik

Copy link
Copy Markdown
Collaborator Author

Independent review

Reviewer: independent agent. Scope: the latest forward-merge commit of this branch (708e3da) only (conflict resolution), plus a repository-wide search; the PR's own change was reviewed in its original review.

Verdict: approve the forward merge. No blocking, no should fix.

  • Merge commit checked with git show --cc: merge of the sidecar branch into this branch; no conflict, no resolution content. SLURM_Run_Workflow.py here has no general constant left except VERSION (CANONICAL_INPUTS_OUTPUT is gone), so this PR resolves the blocking constants finding on Remove the input marker from results after import on every path #97 once merged. With test-suite + Tests: remove the input marker from results after import #98 + use-core-output-constants-tests merged locally against this head the suite gives 611 passed, 9 skipped.
  • git grep over the branch head finds no biomero.storage, ANALYZED_DATA_FOLDER, read_analyzed_data_folder or analyzed_data_folder. RESULTS_FOLDER = ".analyzed" is the only code use of the folder name in SLURM_Import_Results.py (the remaining .analyzed text is in docstrings and comments).
  • SLURM_Run_Workflow.py and SLURM_Import_Results.py parse at this head.
  • Merge order from Remove the input marker from results after import on every path #97's description still holds: each branch contains the one before it, and no duplicate definitions or changed signatures were introduced by the resolution.

🤖 Generated with Claude Code

TorecLuik and others added 5 commits November 16, 2026 16:42
…d exact scope-check wording

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…constants

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@TorecLuik

Copy link
Copy Markdown
Collaborator Author

Resolved in 963101f: #95 (9573c42) merged in; the only conflict was line wrapping of the same core-constant lines, resolved to the wrapped form.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Use the core script-output constants

1 participant