pyopia-gui-support: staging branch (not for merging as one PR) - #435
Draft
animmosmith wants to merge 10 commits into
Draft
pyopia-gui-support: staging branch (not for merging as one PR)#435animmosmith wants to merge 10 commits into
animmosmith wants to merge 10 commits into
Conversation
Closes #426: `init-project --example-data` and the exampledata helpers downloaded silently via plain urlretrieve() - on a slow connection this looked indistinguishable from a hang. Adds a reporthook logging at 10%-increments, applied to all three urlretrieve() call sites.
animmosmith
force-pushed
the
pyopia-gui-support
branch
from
August 22, 2026 17:36
92abd52 to
30fdccc
Compare
Closes #423, six related fixes from the same investigation: - init-project now exits 1 (was 0) when the target folder already exists, so scripts can detect failure without parsing printed text. - process() adds --progress-file: writes {"processed": N, "total": M} as JSON after each image (write-temp-then-rename), for external tooling to poll rather than parse logs. Per-chunk files (<path>.chunk<N>) when num_chunks > 1, avoiding same-file writes racing across processes. - process()'s stage banners and completion message now go through the same logger as everything else in the command, instead of progress.console.print (stdout only) - they now actually respect general.log_file redirection, confirmed by a manual run. The two banners before config is loaded stay as console output; no logger exists that early. Drops these banners' blue console styling as a side effect - RichHandler doesn't interpret that markup by default. - Fixed process()'s docstring (numchunks -> num_chunks, "parallell" -> "parallel", "Defult" -> "Default") and generate_config()'s (each parameter described its own name instead of what it does). - Added Windows/PowerShell Docker invocation guidance to the README - the existing instructions are bash-only. Written from general Docker/Windows knowledge, not verified on an actual Windows machine.
Closes #427: pyopia/statistics.py already has everything needed to compute particle count, d50, and size-distribution stats from a -STATS.nc file, but none of it was reachable from the CLI - only via custom Python against the library directly (which pyopia-gui was stopgapping with a vendored copy, see #427's own issue thread). `pyopia summary-stats <stats_filename> [--json-output]` follows make-montage's existing load_stats/steps_from_xstats pattern. Field names (particle_count, images_with_particles, d50_microns, dias, number_distribution) deliberately match pyopia-gui's vendored StatsSummary shape for an easy future migration. --json-output writes via sys.stdout.write, not this module's `print` (shadowed by `from rich import print`) - Rich interprets the JSON array fields' literal "[..]" as console markup, which would corrupt the output.
Wires pyopia.statistics.make_montage_scaled (#407) into the CLI, mirroring the existing make-montage command's structure (load_stats/steps_from_xstats, same output-filename handling). Exposes rel_scale, the parameter that makes density comparisons across differently-sized datasets fair - see the function's own docstring for why. Not "Closes #407" - that issue closes via #431's own commit; this is follow-on CLI wiring for the same feature, not the feature itself.
Closes #434 (holo only - see #436 opened for uvp, which has no existing example data or raw-file convention to build on): two real bugs found and fixed, both blocking holo regardless of --example-data: - raw_files was hardcoded to "images/*.silc" for every instrument, so `init-project --instrument holo` (with or without --example-data) generated a config that could never match real holo files. Now instrument-aware. - `if example_data: instrument = "silcam"` unconditionally overrode whatever --instrument was actually passed, so `init-project --example-data --instrument holo` silently got silcam data. No longer forced. `--example-data --instrument holo` now downloads a real folder of holograms via the existing get_folder_from_holo_repository() (gdown) into images/holo_test_data_01/, matching the new raw_files pattern. Known limitation, not fixed here: get_folder_from_holo_repository uses gdown/Google Drive, the same flaky download path #421 already documents - confirmed hitting that exact rate-limiting while developing this fix's own test, which now carries the same flaky-reruns mitigation already used for this root cause in test_notebooks.py.
Closes #421: get_folder_from_holo_repository downloaded via gdown/Google Drive, which has repeatedly rate-limited in CI (pipeline-holo.ipynb and single-image-stats-holo.ipynb failing with FileNotFoundError, twice on this branch alone in the last 24 hours) - the same flaky path this issue already documented from PR #419's MacOS run. Points at github.com/SINTEF/pyopia/releases/download/sample-data/ holo_test_data_01.zip instead - GitHub's own recommended place for large files associated with a repo, and keeps this in the same place as the rest of the project rather than pysilcam's separately-managed Azure blob storage (see discussion on #421). Removes the gdown dependency from the codebase entirely - it was only used in this one function. Also fixes a real pre-existing bug found while rewriting this: the old fallback branch did `foldername == "holo_test_data_01"` (a comparison, discarded) instead of `=` (an assignment) - harmless in practice since it was only reachable for an already-unsupported foldername, but dead/wrong code regardless. Verified the full download/extract/cleanup logic end-to-end against a local HTTP server serving a real copy of holo_test_data_01 (34 files) - not against the real URL, since the "sample-data" release doesn't exist yet. Needs someone with tag-creation rights (blocked by a repo ruleset from here - see #421) to create it and upload holo_test_data_01.zip before this actually works; until then, get_folder_from_holo_repository will 404.
This was referenced Aug 23, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Staging branch for pyopia-gui fixes.
Not for merging as one PR - each fix lands in
mainvia its own PR (Closes #N). SeePYOPIA_GUI_SUPPORT.mdon the branch for the branch's own purpose/scope note.Each bullet below links to the specific commit that makes that change, so you can review them individually rather than as one combined diff.
CLI improvements
8c719fd)summary-statscommand (particle count, d50, size distribution from a-STATS.ncfile, with--json-outputfor machine-readable output), field names matching pyopia-gui's vendoredStatsSummaryshape for an easy future migration. (90b639a)make-montage-scaledcommand, wiringpyopia.statistics.make_montage_scaled(Add a scaled circular montage for fair density comparison across sample sizes #407) into the CLI alongside the existingmake-montage. (8b2b37a)Example data
init-project --example-data --instrument holonow downloads real holograms and generates a config that actually matches them - two real bugs fixed along the way (raw_files was hardcoded to*.silcfor every instrument; example_data forced instrument back to silcam regardless of what was requested). (cc705d7)exampledata.py, previously silent enough to look like a hang on a slow connection. (30fdccc)gdown/Google Drive (repeatedly rate-limited in CI) onto a GitHub release, removing thegdowndependency entirely. Needs someone with tag-creation rights to actually create the release - not something I can do from here. (a9050a8)Test plan
flake8 pyopiacleanuv run pytest -m "not slow"passingsample-datarelease exists