Skip to content

Summer 26 Features: ROI padding, progress logging, scaled montage - #431

Open
animmosmith wants to merge 10 commits into
summer26-housekeepingfrom
summer26-features
Open

Summer 26 Features: ROI padding, progress logging, scaled montage#431
animmosmith wants to merge 10 commits into
summer26-housekeepingfrom
summer26-features

Conversation

@animmosmith

@animmosmith animmosmith commented Aug 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Combined feature PR for the Summer 26 Features milestone. Built on top of summer26-housekeeping so it can be reviewed/merged after that one lands.

Each bullet below links to the specific commit that makes that change, so you can review them individually rather than as one combined diff.

Test plan

  • flake8 pyopia clean
  • uv run pytest -m "not slow" - 43 passed
  • CI green on this PR (Windows/Ubuntu/macOS/License_check)

@animmosmith animmosmith added this to the Summer 26 Features milestone Aug 20, 2026
This was referenced Aug 20, 2026
Closed
Closed
Closed
@animmosmith
animmosmith force-pushed the summer26-features branch 3 times, most recently from 473a479 to 1170153 Compare August 20, 2026 22:19
This was referenced Aug 20, 2026
Closed
Closed
animmosmith and others added 10 commits August 21, 2026 08:04
@animmosmith @claude
Add scaled circular montage for fair density comparison across sample…
26fdad7 
… sizes

Fixes #407.

make_montage() always fills the same fixed rectangular canvas regardless of
how much data went into it, making visual density comparisons across
datasets with different sample sizes (e.g. depth bins with different
numbers of raw images) misleading - every montage looks equally "full."

make_montage_scaled() packs particles largest-first within a circular
boundary whose area (not radius - area scales with sqrt(rel_scale) as the
radius) is controlled by rel_scale. Setting rel_scale proportional to each
dataset's relative sample size and placing the resulting montages side by
side gives a fair visual comparison: half the raw images means half the
circle area to fill.

Also outputs grayscale rather than RGB, since particles from monochrome
instruments don't need three channels, and it makes montage_plot()'s
existing cmap='grey' argument actually take effect.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@animmosmith
Attempt every particle in make_montage_scaled instead of truncating
0a74f5c 
Prompted by Emlyn's review comment questioning the roifiles selection here.

Truncating to the max_particles largest particles (or evenly subsampling, as
gen_roifiles() does for the older make_montage) both misrepresent the true
relative abundance of particle sizes - a scaled montage's whole purpose is a
fair visual comparison, so it should reflect the real size distribution, not
an artificially selected subset. Every exported particle is now attempted,
largest first; particles that can't find a free spot are skipped as before,
but now a warning is logged summarising how many were skipped once the
montage is complete, since that means msize needs to be increased (not
rel_scale, which exists specifically to preserve relative comparisons and
would be distorted by nudging it to fit one particular montage).
@animmosmith
Add optional progress/ETA logging to Pipeline
5439189 
Closes #416: adds an opt-in Pipeline.enable_progress_tracking(total_files,
log_interval) that logs percent complete/elapsed/ETA every log_interval
calls to run(). No effect unless called. With multiple chunks
(pyopia process --num-chunks), each chunk's Pipeline instance tracks its own
progress independently - this rides on the existing per-process logger, so
it's automatically safe under the queue-based multiprocess logging already in
place.
@animmosmith
Add fixed-pixel ROI padding, complementing bbox_expansion
9e17ad0 
Closes #418.

bbox_expansion already lets a particle's bounding box be expanded before its
ROI is cropped, but that expansion is fractional - it scales with each
particle's own size. There was no way to add a fixed, consistent pixel
margin regardless of particle size, e.g. to guarantee a small non-particle
border around every ROI fed to the classifier (which is run on this same
cropped ROI as the exported file), or for visual inspection/montage context.

Adds a new pad_bbox() helper (fixed-pixel analogue of the existing
expand_bbox()) and a pad parameter threaded through the same chain
bbox_expansion already uses: extract_roi -> extract_particles -> statextract
-> CalculateStats. pad is applied on top of bbox_expansion, so either, both,
or neither can be used. Clamped to image bounds the same way.
@nepstad @animmosmith
Add ImageToDisc pipeline step for saving processed images to file
3f91d60 
Adds a new pipeline-compatible class ImageToDisc to pyopia.io that saves
processed pipeline images (raw, background, corrected, segmented, etc.)
to an output folder. Supports configurable image keys, scale factor for
downsizing, collage mode (all images in one file vs separate files),
and image format selection.

Co-authored-by: nepstad <152277+nepstad@users.noreply.github.com>
@nepstad @animmosmith
Address code review: improve memory efficiency and move imports
693e835 
- Keep original image dtypes until conversion is needed (avoid premature
  float64 conversion)
- Move matplotlib/skimage imports from __call__ to the specific methods
  that need them
- Move test-level matplotlib import to module level

Co-authored-by: nepstad <152277+nepstad@users.noreply.github.com>
@nepstad @animmosmith
Clean up redundant if/else in _prepare_image
d73bde0 
Co-authored-by: nepstad <152277+nepstad@users.noreply.github.com>
@nepstad @animmosmith
Add documentation for ImageToDisc with instructive usage examples
d099aa0 
Adds documentation in two locations:
- toml_config.ipynb: Comprehensive section with configuration options table,
  examples for separate images and collage mode, and pipeline placement guidance
- processing_raw_data.ipynb: Practical section explaining when and how to use
  ImageToDisc for quality control and visual inspection

Co-authored-by: nepstad <152277+nepstad@users.noreply.github.com>
@animmosmith
bump version
efcd52d
@animmosmith
Bound the realtime processing queue, dropping oldest images under bac…
8c324a3 
…klog

Reimplements #390 (open 3+ weeks with no response) with the configurability
suggested in review there: replaces the unbounded queue.Queue with a
deque(maxlen=queue_size), so when processing falls behind acquisition, old
images get dropped instead of an ever-growing backlog. The worker now pops
from the right (most recently queued) so a backlog is worked off newest-first,
staying close to realtime rather than grinding through stale images.

queue_size defaults to 10 (matching #390's original fixed value) and is
configurable via `pyopia process-realtime --queue-size`, the same way
--watch-folder already is - the right value depends on acquisition rate vs.
processing throughput, which is hardware/instrument-dependent.
animmosmith added a commit that referenced this pull request Aug 22, 2026
@animmosmith
Add make-montage-scaled CLI command
8b2b37a 
Wires pyopia.statistics.make_montage_scaled (#407) into the CLI, mirroring
the existing make-montage command's structure (load_stats/steps_from_xstats,
same output-filename handling). Exposes rel_scale, the parameter that makes
density comparisons across differently-sized datasets fair - see the
function's own docstring for why.

Not "Closes #407" - that issue closes via #431's own commit; this is
follow-on CLI wiring for the same feature, not the feature itself.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@nepstad nepstad Awaiting requested review from nepstad
@emlynjdavies emlynjdavies Awaiting requested review from emlynjdavies

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

Summer 26 Features

Development

Successfully merging this pull request may close these issues.

2 participants

@animmosmith