Skip to content

Detached processing: own launcher registration, resume and cancel in the scripts - #90

Open
TorecLuik wants to merge 29 commits into
masterfrom
detached-ownership
Open

TorecLuik wants to merge 29 commits into
masterfrom
detached-ownership

Conversation

@TorecLuik

@TorecLuik TorecLuik commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #86
Refs NL-BioImaging/NL-BIOMERO#92

This branch contains the merge of #88 (with #97), which merge before it; the merge was clean. After this PR: OMERO.biomero#81, then NL-BioImaging/NL-BIOMERO#94 (after the pin bump).

What

Launcher registration and resume decisions move from core (biomero.detached) into SLURM_Run_Workflow.py; cancellation is honoured by the pipeline.

  • Registration (register_detached_launcher) builds the launcher params from the script client and calls core's WorkflowTracker.add_launcher_task (core only sets the marker). The batched script does the same inline (scripts cannot import each other). What is persisted is unchanged: all script inputs plus derived values, as before.

  • Resume (load_task_history, task_completed, decide_workflow_resume, ...) reads the task aggregates only (completed / failed), no longer the asynchronous execution view. Fixes:

    • an unreadable history raises (RuntimeError), it is never an empty history that restarts the run and resubmits Slurm jobs;
    • a Slurm job that FAILED, was CANCELLED, timed out (or whose task failed) is not re-attached; the workflow is submitted anew;
    • a workflow whose results were already imported is now skipped. The old submitted_job returned "not submitted" in that case, so the caller submitted it again (a double run despite its docstring);
    • job ids are compared as integers; the decision is logged.
  • Cancellation: execute_workflow_pipeline checks raise_if_cancel_requested before export, conversion and each submission, and on every poll. On cancel it calls SlurmClient.cancel_slurm_jobs, fails the open tasks and the workflow with the reason. The stages moved into run_pipeline_stages; execute_workflow_pipeline keeps its signature and wraps them. Applies to inline runs too.

  • One reader of BIOMERO_DETACHED_WORKFLOWS: both scripts import biomero.detached.detached_mode_enabled; the ImportError fallbacks are removed (scripts are version-locked to core).

  • Cancel passes through every stage: WorkflowCancelled raised inside export, conversion and submission is no longer turned into a generic error by those stages' except Exception; cancel_workflow_run fails every open task of the workflow (not only Slurm-job tasks, never the launcher) and cancels the jobs found in the task history.

  • Cancel trigger: new admin script admin/SLURM_Cancel_Workflow.py ("Cancel BIOMERO Workflow"): workflow UUID and reason, allowed for the workflow's owner or an administrator, logged. It calls WorkflowTracker.request_workflow_cancel. An OMERO.biomero button can follow later.

  • run_pipeline_stages is split into export_data_stage, convert_data_stage, submit_workflow_jobs and monitor_slurm_jobs, which return their state.

Behaviour changes

Everything listed above; inline runs additionally do one tracker read per poll for the cancel check. The inline batched script has no cancel check (cancel is supported for single runs and for detached batched runs through the processor); the batched and single launcher registration are two copies of the same few lines because scripts cannot import each other, and both end in add_launcher_task. These scripts require the matching core release (contains biomero#105 and #106; hard dependency; scripts and core are released together); VERSION values are bumped in the release PR. Admin scripts are unchanged (still use biomero.detached / biomero.maintenance, which stay in core; see biomero PR 105).

Tests

On test-suite: NL-BioImaging/biomero-scripts PR (companion, see below). 27 new tests run the real runner functions against a real tracker (resume decisions, imported results, failed/cancelled jobs, unreadable history, cancel before start and during a poll, scancel, registration); existing structural tests accept both layouts. Mutation checks (drop the failed-job filter, swallow the read error, remove the poll check, ignore imported results, skip scancel, fail completed tasks) each turn a test red. Full harness locally: 493 passed. Core 2.9.0b26 (published, contains #105 and #106) is installed by the harness once test PR #103 is merged, so these tests run in CI.

Merge order (production and test PRs, interleaved)

Each step keeps master CI green: every (production state, test-suite state) pair on this path was run locally with core 2.9.0b26 from PyPI (482, 484, 484, 513, 524, 588, 594, 597, 606 passed; no failures). The production PRs merge cleanly in this order: each branch already contains the conflict resolutions with the PRs before it.

  1. test PR Tests: install biomero core 2.9.0b26 in the harness #103 (test-suite: biomero>=2.9.0b26,<3 in tests/requirements.txt, so core-dependent tests run in CI)
  2. test PRs Tests: schema imports, metadata view imports and workflow color #89, Tests: detached resume, cancel and registration #91 and Tests: per-directory canonical sidecars and unwritable folders #99 (capability-gated; they pass for the current master and for the new source)
  3. production PR Remove the input marker from results after import on every path #97
  4. test PR Tests: remove the input marker from results after import #98 (fails on master before Remove the input marker from results after import on every path #97, by design)
  5. production PR Import Zarr contracts from biomero-schema and own the workflow color #88
  6. production PR Detached processing: own launcher registration, resume and cancel in the scripts #90 (this PR)
  7. production PR Migration script: move canonical sidecars via importer helpers #95
  8. test PR Tests: shared output names come from core constants #102
  9. production PR Use the core script-output constants #101

A production PR's own check stays red until the test PRs before it in this list are merged (CI runs the shared harness from test-suite, which is not pinned); merge the test PR, then rerun the check.

🤖 Generated with Claude Code

Round 2 changes

  • Stage handlers re-raise any WorkflowInterrupted without failing tasks; only WorkflowCancelled makes the run cancel jobs and fail tasks and the workflow. execute_workflow_pipeline(..., stop_check=None) is the explicit hook for the caller (the detached worker passes its claim check); stage arguments are grouped in a PipelineRun dataclass.
  • monitor_slurm_jobs split into process_job_state, end_failed_job (one place for 'fail task, keep the log') and import_completed_job.
  • cancel_workflow_run still fails tasks and the workflow when cancel_slurm_jobs raises (logged; the jobs may need cancelling by hand).
  • Cancel script: a finished workflow records nothing and says so; reason cut to 200 characters; a denial is logged once; plain 2026 header.
  • Needs biomero#105 head 07429d1 or later (WorkflowInterrupted, is_workflow_finished).

Round 3 changes

  • PipelineRun.raise_if_claim_lost() (claim only; a late cancel still completes) is checked after the poll's progress read and before the outcome write; a stop check also runs between conversion and submission.
  • cancel_workflow_run: the task read is inside the scancel handling; a failure puts 'Slurm jobs [...] could not be cancelled; cancel them manually' into the failure reason.
  • Dead handler in run_workflow removed (its file-transfer call is outside that try); run parameter renamed pipeline; types and docstring for tracker; the cancel script message says inline batched runs do not honour cancel.

TorecLuik and others added 4 commits November 12, 2026 14:28
- Import the Zarr contracts from biomero_schema.zarr; biomero.zarr_contracts
  is removed from core.
- Import the metadata view functions from biomero.metadata_views (renamed from
  biomero.provenance) and format reduced values with core's
  format_large_field_reference, so core recognises them on refresh.
- Move the Blueprint workflow color to SLURM_Run_Workflow.py, which forwards
  the resolved color to the result scripts.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…lation in the scripts

- Registration builds the launcher params from the script client and hands
  them to WorkflowTracker.add_launcher_task (core only sets the marker).
- Resume reads the task aggregates (completed/failed) instead of a second
  source, raises when the history cannot be read, never re-attaches to a
  failed/cancelled/timed-out Slurm job, and skips a workflow whose results
  were already imported instead of submitting it again.
- The pipeline stops between steps and between polls when a cancel was
  requested, cancels its Slurm jobs, closes its tasks and fails the workflow.
- One reader of BIOMERO_DETACHED_WORKFLOWS: biomero.detached.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
- Fail with a clear message when the installed biomero predates
  biomero.metadata_views; README states the minimum core release.
- Move the colour palette to the module constants area of the runner.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…stages split

- WorkflowCancelled raised inside export, conversion and submission is no
  longer turned into a generic error by those stages' handlers.
- cancel_workflow_run fails every open task of the workflow (not only the
  Slurm-job tasks) and cancels the jobs found in the task history.
- admin/SLURM_Cancel_Workflow.py requests a cancel for a workflow id (owner
  or administrator), so a detached run can be cancelled end to end.
- run_pipeline_stages split into named stages that return their state.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@TorecLuik

Copy link
Copy Markdown
Collaborator Author

Review round 1 (independent reviewer: Sonnet 5.5 subagent, read-only, mutation checks; resolved by the author)

Finding Resolution (e2c6b4c; tests in #91 dea7cea)
B1 WorkflowCancelled swallowed by export/conversion/submission handlers, tasks left started Those handlers re-raise it first; cancel_workflow_run fails every open task (never the launcher) and cancels the jobs found in the task history. Tests: real exportImageToSLURM with a cancelling sub-script (red when the passthrough is removed), plus a structural test that every generic handler around run_omero_script lets a cancel pass
Cancel trigger New admin/SLURM_Cancel_Workflow.py (owner or admin, logged), tests in #91; README row
S1 long run_pipeline_stages Split into export_data_stage, convert_data_stage, submit_workflow_jobs, monitor_slurm_jobs; no mutated task_ids
S2 inline batched cancel Documented as detached-only for batched runs
S3 two registration copies Accepted (scripts cannot import each other); both end in add_launcher_task; stated in the description
S4 hard core dependency, VERSION Stated in the description; bump in the release PR

Open: use biomero#106's SlurmJob helper in slurm_job_failed() once it merges. CI here is red until #91 merges (old harness). Second round pending.

…neRun, monitor split

- Stage handlers re-raise any WorkflowInterrupted without failing tasks; only
  WorkflowCancelled makes the run cancel jobs and fail tasks and workflow.
- execute_workflow_pipeline(stop_check=...) is an explicit hook for the caller
  (a detached worker that lost its claim); stage arguments grouped in
  PipelineRun.
- monitor_slurm_jobs split into per-state handlers; one failed-job path.
- cancel_workflow_run fails tasks and the workflow even when scancel raises.
- Cancel script: finished workflows record nothing, reason bounded, denial
  logged once, plain copyright header.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@TorecLuik

Copy link
Copy Markdown
Collaborator Author

Review round 2 (resolved by the author; head e79cf92, tests in #91 72bf792)

Finding Resolution
S1 cancel_workflow_run when scancel raises Logged; tasks and workflow are failed anyway; test
S2 split monitor_slurm_jobs, dedupe failed-job block, context dataclass process_job_state / end_failed_job / import_completed_job; PipelineRun groups client, connection, Slurm client, workflow id and the stop check; tests per job state
Interruption design Stage handlers re-raise any WorkflowInterrupted without failing tasks; stop_check= argument of execute_workflow_pipeline is the explicit hook; only WorkflowCancelled cancels jobs and fails tasks and workflow
S3 cancel script on a finished workflow Records nothing, says so, logs; reason bounded; denial logged once; plain 2026 header
Dead except in run_workflow Kept, now WorkflowInterrupted: the file-transfer sub-script runs inside that try

Mutations (drop stop_check, scancel handler, handler passthrough) turn tests red. Full harness 517 passed locally, also against current master source. CI red until #91 merges (old harness).

@TorecLuik
TorecLuik marked this pull request as draft November 12, 2026 15:30
TorecLuik and others added 2 commits November 12, 2026 16:33
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…, nits

- PipelineRun.raise_if_claim_lost: checked after the poll's progress read and
  before the outcome write (a late cancel still completes); a stop check also
  between conversion and submission.
- cancel_workflow_run: reading the tasks is inside the scancel handling; a
  failure puts 'Slurm jobs [...] could not be cancelled' into the reason.
- Dead handler in run_workflow removed; run parameter renamed pipeline; types
  and docstring for tracker.
- Cancel script message mentions inline batched runs do not honour cancel.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@TorecLuik

Copy link
Copy Markdown
Collaborator Author

Review round 3 (resolved by the author; head 5dcc347, tests in #91 b226e94)

Finding Resolution
S1 claim check before owner-only writes PipelineRun.raise_if_claim_lost() after the poll's progress read and before the outcome write; a late cancel still completes; stop check also between conversion and submission. Tests flip the claim inside a poll, the import and the conversion stub; mutations red
S4 scancel failure Reason names the jobs to cancel by hand; task read inside the same handling
Nits Dead handler removed; pipeline rename; types and tracker docstring; cancel script message on inline batched runs

Full harness locally 522 passed, 1 failed (the ssh_recovery legacy test, handled in biomero-scripts#92). CI stays red until #91 merges.

TorecLuik and others added 8 commits November 13, 2026 14:02
The workflow runner now owns the cleanup of .biomero-input.json: after the
result import succeeded it removes the marker from the imported result folder.
SLURM_Import_Results.py reports that folder as the Results_Path output.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
biomero.zarr_contracts is being removed from core; the marker constant is
defined in biomero_schema.zarr, which the scripts already import.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ogether

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The harness still expects the CanonicalInput import from
biomero.zarr_contracts until #88 and its tests merge; only the new
TRANSFER_INPUT_MARKER import comes from biomero_schema.zarr.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Accept only .analyzed/<workflow>/<timestamp>, never fail the run on a cleanup
error, warn on unreadable folders, drop unreachable parameters, name the
shallow storage condition and document the Results_Path output.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Extracted from runScript so the persisted launcher params, the contract with
the supervisor's batched parent, can be tested. No behaviour change.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Both scripts take RESULTS_PATH_OUTPUT from core (biomero#110), like
WORKFLOW_UUID_OUTPUT, instead of defining it locally.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
TorecLuik and others added 2 commits November 13, 2026 16:07
Resolve the gated import block with #97 (the input marker cleanup, which
merges first) to
`from biomero_schema.zarr import CanonicalInput, TRANSFER_INPUT_MARKER`.
Name the released minimum core version (2.9.0b26) in the README and in the
import error messages.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
#97 and #88 merge before this PR; merging them here applies cleanly.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
TorecLuik added a commit that referenced this pull request Nov 13, 2026
…er>=1.5.0b14

The merge is clean. The scripts declared no importer minimum anywhere, so
the README now names biomero-importer>=1.5.0b14 (it provides the
PROCESSED_DATA_FOLDER and sidecar helpers this PR imports).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
TorecLuik added a commit that referenced this pull request Nov 13, 2026
Conflicts with the PRs that merge before this one, resolved by taking their
code and applying the same mechanical replacement of output-name literals
with biomero.constants.results.*_OUTPUT:
- SLURM_Run_Workflow.py: keep RESULTS_FOLDER_NAME and ZARR_ROOT_FILES from #97,
  drop the local CANONICAL_INPUTS_OUTPUT; the export, conversion and import
  stages of #90 use MESSAGE_OUTPUT, URL_OUTPUT and FILE_ANNOTATION_OUTPUT.
- SLURM_Run_Workflow_Batched.py: use #90's register_detached_batches and
  MESSAGE_OUTPUT for the launcher message.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
TorecLuik added a commit that referenced this pull request Nov 13, 2026
The merge is clean.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
TorecLuik and others added 3 commits November 16, 2026 14:23
…e stores

The result folder is the ANALYZED_DATA_FOLDER setting (read_analyzed_data_folder),
not a local constant. The marker sits at the root of each exported store, one
level below the result folder, so the sweep no longer detects Zarr stores.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Scripts and core are released together, so the README and the import
error name the renamed module, not a version floor.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
TorecLuik added a commit that referenced this pull request Nov 16, 2026
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@TorecLuik

Copy link
Copy Markdown
Collaborator Author

Independent review

Scope: only the forward-merge commit 50cfa08 ("Merge #88 into detached-ownership"), compared with both parents (c706411 and 8425807) via git show --remerge-diff. The PR's original content was not re-reviewed.

Verdict: the merge is correct. No blocking findings.

Nothing found in any category for the merge.

🤖 Generated with Claude Code

TorecLuik and others added 3 commits November 16, 2026 14:55
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@TorecLuik

Copy link
Copy Markdown
Collaborator Author

Resolved in c7674dc: #88 (5714bb4) merged in, clean.

🤖 Generated with Claude Code

TorecLuik and others added 3 commits November 16, 2026 16:03
…the folder name

Core will not provide an ANALYZED_DATA_FOLDER setting for 1.8 (tracked in
NL-BIOMERO#111). SLURM_Import_Results.py keeps one RESULTS_FOLDER constant,
and the marker sweep accepts only a result folder of this run's workflow:
<...>/<workflow id>/<timestamp>. No general constants remain in the run script.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@TorecLuik

Copy link
Copy Markdown
Collaborator Author

Independent review

Reviewer: independent agent. Scope: the latest forward-merge commit of this branch (0207f8a) only (conflict resolution), plus a repository-wide search; the PR's own change was reviewed in its original review.

Verdict: approve the forward merge. No blocking, no should fix.

  • Merge commit checked with git show --cc: merge of Import Zarr contracts from biomero-schema and own the workflow color #88 into this branch. The one conflict, the import block of SLURM_Run_Workflow.py, keeps the detached-mode imports and drops from biomero.storage import read_analyzed_data_folder.
  • git grep over the branch head finds no biomero.storage, ANALYZED_DATA_FOLDER, read_analyzed_data_folder or analyzed_data_folder. RESULTS_FOLDER = ".analyzed" is the only code use of the folder name in SLURM_Import_Results.py (the remaining .analyzed text is in docstrings and comments).
  • SLURM_Run_Workflow.py and SLURM_Import_Results.py parse at this head.
  • Merge order from Remove the input marker from results after import on every path #97's description still holds: each branch contains the one before it, and no duplicate definitions or changed signatures were introduced by the resolution.

🤖 Generated with Claude Code

TorecLuik and others added 3 commits November 16, 2026 16:42
…d exact scope-check wording

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@TorecLuik

Copy link
Copy Markdown
Collaborator Author

Resolved in f4ecc98: #88 (34d9fa5) merged in, clean.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Detached processing: own launcher registration and resume in the scripts

1 participant