Repository navigation
Conversation
- Import the Zarr contracts from biomero_schema.zarr; biomero.zarr_contracts is removed from core. - Import the metadata view functions from biomero.metadata_views (renamed from biomero.provenance) and format reduced values with core's format_large_field_reference, so core recognises them on refresh. - Move the Blueprint workflow color to SLURM_Run_Workflow.py, which forwards the resolved color to the result scripts. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…lation in the scripts - Registration builds the launcher params from the script client and hands them to WorkflowTracker.add_launcher_task (core only sets the marker). - Resume reads the task aggregates (completed/failed) instead of a second source, raises when the history cannot be read, never re-attaches to a failed/cancelled/timed-out Slurm job, and skips a workflow whose results were already imported instead of submitting it again. - The pipeline stops between steps and between polls when a cancel was requested, cancels its Slurm jobs, closes its tasks and fails the workflow. - One reader of BIOMERO_DETACHED_WORKFLOWS: biomero.detached. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
- Fail with a clear message when the installed biomero predates biomero.metadata_views; README states the minimum core release. - Move the colour palette to the module constants area of the runner. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…stages split - WorkflowCancelled raised inside export, conversion and submission is no longer turned into a generic error by those stages' handlers. - cancel_workflow_run fails every open task of the workflow (not only the Slurm-job tasks) and cancels the jobs found in the task history. - admin/SLURM_Cancel_Workflow.py requests a cancel for a workflow id (owner or administrator), so a detached run can be cancelled end to end. - run_pipeline_stages split into named stages that return their state. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Review round 1 (independent reviewer: Sonnet 5.5 subagent, read-only, mutation checks; resolved by the author)
Open: use biomero#106's |
…neRun, monitor split - Stage handlers re-raise any WorkflowInterrupted without failing tasks; only WorkflowCancelled makes the run cancel jobs and fail tasks and workflow. - execute_workflow_pipeline(stop_check=...) is an explicit hook for the caller (a detached worker that lost its claim); stage arguments grouped in PipelineRun. - monitor_slurm_jobs split into per-state handlers; one failed-job path. - cancel_workflow_run fails tasks and the workflow even when scancel raises. - Cancel script: finished workflows record nothing, reason bounded, denial logged once, plain copyright header. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Review round 2 (resolved by the author; head e79cf92, tests in #91 72bf792)
Mutations (drop |
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…, nits - PipelineRun.raise_if_claim_lost: checked after the poll's progress read and before the outcome write (a late cancel still completes); a stop check also between conversion and submission. - cancel_workflow_run: reading the tasks is inside the scancel handling; a failure puts 'Slurm jobs [...] could not be cancelled' into the reason. - Dead handler in run_workflow removed; run parameter renamed pipeline; types and docstring for tracker. - Cancel script message mentions inline batched runs do not honour cancel. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Review round 3 (resolved by the author; head 5dcc347, tests in #91 b226e94)
Full harness locally 522 passed, 1 failed (the |
The workflow runner now owns the cleanup of .biomero-input.json: after the result import succeeded it removes the marker from the imported result folder. SLURM_Import_Results.py reports that folder as the Results_Path output. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
biomero.zarr_contracts is being removed from core; the marker constant is defined in biomero_schema.zarr, which the scripts already import. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ogether Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The harness still expects the CanonicalInput import from biomero.zarr_contracts until #88 and its tests merge; only the new TRANSFER_INPUT_MARKER import comes from biomero_schema.zarr. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Accept only .analyzed/<workflow>/<timestamp>, never fail the run on a cleanup error, warn on unreadable folders, drop unreachable parameters, name the shallow storage condition and document the Results_Path output. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Extracted from runScript so the persisted launcher params, the contract with the supervisor's batched parent, can be tested. No behaviour change. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Both scripts take RESULTS_PATH_OUTPUT from core (biomero#110), like WORKFLOW_UUID_OUTPUT, instead of defining it locally. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Resolve the gated import block with #97 (the input marker cleanup, which merges first) to `from biomero_schema.zarr import CanonicalInput, TRANSFER_INPUT_MARKER`. Name the released minimum core version (2.9.0b26) in the README and in the import error messages. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…er>=1.5.0b14 The merge is clean. The scripts declared no importer minimum anywhere, so the README now names biomero-importer>=1.5.0b14 (it provides the PROCESSED_DATA_FOLDER and sidecar helpers this PR imports). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Conflicts with the PRs that merge before this one, resolved by taking their code and applying the same mechanical replacement of output-name literals with biomero.constants.results.*_OUTPUT: - SLURM_Run_Workflow.py: keep RESULTS_FOLDER_NAME and ZARR_ROOT_FILES from #97, drop the local CANONICAL_INPUTS_OUTPUT; the export, conversion and import stages of #90 use MESSAGE_OUTPUT, URL_OUTPUT and FILE_ANNOTATION_OUTPUT. - SLURM_Run_Workflow_Batched.py: use #90's register_detached_batches and MESSAGE_OUTPUT for the launcher message. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…e stores The result folder is the ANALYZED_DATA_FOLDER setting (read_analyzed_data_folder), not a local constant. The marker sits at the root of each exported store, one level below the result folder, so the sweep no longer detects Zarr stores. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Scripts and core are released together, so the README and the import error name the renamed module, not a version floor. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Independent reviewScope: only the forward-merge commit 50cfa08 ("Merge #88 into detached-ownership"), compared with both parents (c706411 and 8425807) via Verdict: the merge is correct. No blocking findings.
Nothing found in any category for the merge. 🤖 Generated with Claude Code |
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
Resolved in c7674dc: #88 (5714bb4) merged in, clean. 🤖 Generated with Claude Code |
…the folder name Core will not provide an ANALYZED_DATA_FOLDER setting for 1.8 (tracked in NL-BIOMERO#111). SLURM_Import_Results.py keeps one RESULTS_FOLDER constant, and the marker sweep accepts only a result folder of this run's workflow: <...>/<workflow id>/<timestamp>. No general constants remain in the run script. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Independent reviewReviewer: independent agent. Scope: the latest forward-merge commit of this branch (0207f8a) only (conflict resolution), plus a repository-wide search; the PR's own change was reviewed in its original review. Verdict: approve the forward merge. No blocking, no should fix.
🤖 Generated with Claude Code |
…d exact scope-check wording Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
Resolved in f4ecc98: #88 (34d9fa5) merged in, clean. 🤖 Generated with Claude Code |
Closes #86
Refs NL-BioImaging/NL-BIOMERO#92
This branch contains the merge of #88 (with #97), which merge before it; the merge was clean. After this PR: OMERO.biomero#81, then NL-BioImaging/NL-BIOMERO#94 (after the pin bump).
What
Launcher registration and resume decisions move from core (
biomero.detached) intoSLURM_Run_Workflow.py; cancellation is honoured by the pipeline.Registration (
register_detached_launcher) builds the launcher params from the script client and calls core'sWorkflowTracker.add_launcher_task(core only sets the marker). The batched script does the same inline (scripts cannot import each other). What is persisted is unchanged: all script inputs plus derived values, as before.Resume (
load_task_history,task_completed,decide_workflow_resume, ...) reads the task aggregates only (completed/failed), no longer the asynchronous execution view. Fixes:RuntimeError), it is never an empty history that restarts the run and resubmits Slurm jobs;submitted_jobreturned "not submitted" in that case, so the caller submitted it again (a double run despite its docstring);Cancellation:
execute_workflow_pipelinechecksraise_if_cancel_requestedbefore export, conversion and each submission, and on every poll. On cancel it callsSlurmClient.cancel_slurm_jobs, fails the open tasks and the workflow with the reason. The stages moved intorun_pipeline_stages;execute_workflow_pipelinekeeps its signature and wraps them. Applies to inline runs too.One reader of
BIOMERO_DETACHED_WORKFLOWS: both scripts importbiomero.detached.detached_mode_enabled; theImportErrorfallbacks are removed (scripts are version-locked to core).Cancel passes through every stage:
WorkflowCancelledraised inside export, conversion and submission is no longer turned into a generic error by those stages'except Exception;cancel_workflow_runfails every open task of the workflow (not only Slurm-job tasks, never the launcher) and cancels the jobs found in the task history.Cancel trigger: new admin script
admin/SLURM_Cancel_Workflow.py("Cancel BIOMERO Workflow"): workflow UUID and reason, allowed for the workflow's owner or an administrator, logged. It callsWorkflowTracker.request_workflow_cancel. An OMERO.biomero button can follow later.run_pipeline_stagesis split intoexport_data_stage,convert_data_stage,submit_workflow_jobsandmonitor_slurm_jobs, which return their state.Behaviour changes
Everything listed above; inline runs additionally do one tracker read per poll for the cancel check. The inline batched script has no cancel check (cancel is supported for single runs and for detached batched runs through the processor); the batched and single launcher registration are two copies of the same few lines because scripts cannot import each other, and both end in
add_launcher_task. These scripts require the matching core release (contains biomero#105 and #106; hard dependency; scripts and core are released together);VERSIONvalues are bumped in the release PR. Admin scripts are unchanged (still usebiomero.detached/biomero.maintenance, which stay in core; see biomero PR 105).Tests
On
test-suite: NL-BioImaging/biomero-scripts PR (companion, see below). 27 new tests run the real runner functions against a real tracker (resume decisions, imported results, failed/cancelled jobs, unreadable history, cancel before start and during a poll, scancel, registration); existing structural tests accept both layouts. Mutation checks (drop the failed-job filter, swallow the read error, remove the poll check, ignore imported results, skip scancel, fail completed tasks) each turn a test red. Full harness locally: 493 passed. Core 2.9.0b26 (published, contains #105 and #106) is installed by the harness once test PR #103 is merged, so these tests run in CI.Merge order (production and test PRs, interleaved)
Each step keeps
masterCI green: every (production state, test-suite state) pair on this path was run locally with core 2.9.0b26 from PyPI (482, 484, 484, 513, 524, 588, 594, 597, 606 passed; no failures). The production PRs merge cleanly in this order: each branch already contains the conflict resolutions with the PRs before it.test-suite:biomero>=2.9.0b26,<3intests/requirements.txt, so core-dependent tests run in CI)masterand for the new source)masterbefore Remove the input marker from results after import on every path #97, by design)A production PR's own check stays red until the test PRs before it in this list are merged (CI runs the shared harness from
test-suite, which is not pinned); merge the test PR, then rerun the check.🤖 Generated with Claude Code
Round 2 changes
WorkflowInterruptedwithout failing tasks; onlyWorkflowCancelledmakes the run cancel jobs and fail tasks and the workflow.execute_workflow_pipeline(..., stop_check=None)is the explicit hook for the caller (the detached worker passes its claim check); stage arguments are grouped in aPipelineRundataclass.monitor_slurm_jobssplit intoprocess_job_state,end_failed_job(one place for 'fail task, keep the log') andimport_completed_job.cancel_workflow_runstill fails tasks and the workflow whencancel_slurm_jobsraises (logged; the jobs may need cancelling by hand).WorkflowInterrupted,is_workflow_finished).Round 3 changes
PipelineRun.raise_if_claim_lost()(claim only; a late cancel still completes) is checked after the poll's progress read and before the outcome write; a stop check also runs between conversion and submission.cancel_workflow_run: the task read is inside the scancel handling; a failure puts 'Slurm jobs [...] could not be cancelled; cancel them manually' into the failure reason.run_workflowremoved (its file-transfer call is outside that try);runparameter renamedpipeline; types and docstring fortracker; the cancel script message says inline batched runs do not honour cancel.