diff --git a/docs/notebooks/processing_raw_data.ipynb b/docs/notebooks/processing_raw_data.ipynb index 489f38be..b9094211 100644 --- a/docs/notebooks/processing_raw_data.ipynb +++ b/docs/notebooks/processing_raw_data.ipynb @@ -102,6 +102,34 @@ "```" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## (Optional) Save intermediate images to disc\n", + "\n", + "During processing, it can be very helpful to save intermediate images for visual quality control or for sharing with collaborators.\n", + "For example, you may want to inspect the background-corrected images to ensure the correction is working properly.\n", + "\n", + "To do this, add a `[steps.saveimages]` step to your `config.toml` file. This uses {class}`pyopia.io.ImageToDisc`, which saves one or more pipeline images to a specified output folder.\n", + "\n", + "Here is an example that saves the raw, background, and corrected images at half resolution:\n", + "\n", + "```toml\n", + " [steps.saveimages]\n", + " pipeline_class = 'pyopia.io.ImageToDisc'\n", + " output_folder = 'processed_images'\n", + " image_keys = ['imraw', 'imbg', 'im_corrected']\n", + " scale_factor = 0.5\n", + "```\n", + "\n", + "Place this step **after** the steps that produce the images you want to save (e.g. after `correctbackground` for background-corrected images, or after `segmentation` to also include the binary segmentation mask `imbw`).\n", + "\n", + "You can also save a single **collage** image per input file, by setting `collage = true`.\n", + "\n", + "See {ref}`toml-config` for full configuration details and more examples." + ] + }, { "cell_type": "markdown", "metadata": {}, diff --git a/docs/notebooks/toml_config.ipynb b/docs/notebooks/toml_config.ipynb index 82cd49ef..373bb65b 100644 --- a/docs/notebooks/toml_config.ipynb +++ b/docs/notebooks/toml_config.ipynb @@ -199,6 +199,85 @@ "\n", "```" ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Saving processed images to disc\n", + "\n", + "It is often useful to save intermediate pipeline images to disc for visual inspection, quality control,\n", + "or sharing with collaborators. For example, you may want to visually check that the background correction\n", + "is working correctly, or provide corrected images for manual review.\n", + "\n", + "The {class}`pyopia.io.ImageToDisc` pipeline step enables this. It can be inserted at any point in the pipeline\n", + "to save the current state of one or more images from the pipeline data. Common use cases include:\n", + "\n", + "- Saving the **background-corrected image** (`im_corrected`) for visual quality control\n", + "- Saving the **raw image** (`imraw`) and **background image** (`imbg`) alongside the corrected image for comparison\n", + "- Saving the **segmented image** (`imbw`) to verify that particle detection is working as expected\n", + "- Creating a **collage** of all processing stages for a quick overview of each image\n", + "\n", + "### Configuration options\n", + "\n", + "| Option | Description | Default |\n", + "| --- | --- | --- |\n", + "| `output_folder` | Path to folder where images will be saved (created if it does not exist) | `'processed_images'` |\n", + "| `image_keys` | List of pipeline data keys to save | `['imraw', 'imbg', 'im_corrected', 'imbw']` |\n", + "| `scale_factor` | Factor to downscale images before saving (e.g. `0.5` halves the resolution) | `1.0` |\n", + "| `collage` | If `true`, combine all images into a single vertically-stacked collage per input image | `false` |\n", + "| `image_format` | Output image format | `'png'` |\n", + "\n", + "### Example: Save separate images at half resolution\n", + "\n", + "Add this step after background correction (or after segmentation, depending on which images you want to capture):\n", + "\n", + "```toml\n", + " [steps.saveimages]\n", + " pipeline_class = 'pyopia.io.ImageToDisc'\n", + " output_folder = 'processed_images'\n", + " image_keys = ['imraw', 'imbg', 'im_corrected']\n", + " scale_factor = 0.5\n", + "```\n", + "\n", + "This will create one PNG file per image key, per input image, in the `processed_images/` folder.\n", + "For example, processing an image called `image_001.silc` would produce:\n", + "```\n", + "processed_images/\n", + "├── image_001_imraw.png\n", + "├── image_001_imbg.png\n", + "└── image_001_im_corrected.png\n", + "```\n", + "\n", + "### Example: Save a collage of all processing stages\n", + "\n", + "To get a single overview image showing all stages of processing for each input image:\n", + "\n", + "```toml\n", + " [steps.saveimages]\n", + " pipeline_class = 'pyopia.io.ImageToDisc'\n", + " output_folder = 'processed_images'\n", + " image_keys = ['imraw', 'im_corrected', 'imbw']\n", + " collage = true\n", + " scale_factor = 0.5\n", + "```\n", + "\n", + "This produces a single `image_001_collage.png` per input image, with the raw, corrected and segmented\n", + "images stacked vertically.\n", + "\n", + "### Placement in the pipeline\n", + "\n", + "The `saveimages` step should be placed **after** the processing steps that produce the images you want to save.\n", + "For instance, to save background-corrected images, place it after `correctbackground`.\n", + "To also include the segmented image (`imbw`), place it after the `segmentation` step.\n", + "You can also include multiple `saveimages` steps at different points in the pipeline if needed, e.g.\n", + "one right after background correction and another after segmentation.\n", + "\n", + "```{note}\n", + "Images that are not yet available in the pipeline data at the point where `saveimages` runs will be\n", + "silently skipped. So it is safe to request keys that may not exist for all configurations.\n", + "```" + ] } ], "metadata": { diff --git a/notebooks/pyopia-classifier/pyopia-torch-dinov2-classifier-train.ipynb b/notebooks/pyopia-classifier/pyopia-torch-dinov2-classifier-train.ipynb index 3ebdcdf2..42c63d18 100644 --- a/notebooks/pyopia-classifier/pyopia-torch-dinov2-classifier-train.ipynb +++ b/notebooks/pyopia-classifier/pyopia-torch-dinov2-classifier-train.ipynb @@ -42,7 +42,7 @@ "metadata": {}, "source": [ "## Install extra dependencies\n", - "In addition to PyOPIA (for testing at the end), PyTorch, this notebook requires [TIMM](https://pypi.org/project/timm/) and scikit-learn. " + "In addition to PyOPIA (for testing at the end), PyTorch, this notebook requires [TIMM](https://pypi.org/project/timm/) and scikit-learn - already present if installed via `pyopia[classification-torch]`, otherwise installed automatically below. " ] }, { @@ -52,7 +52,22 @@ "metadata": {}, "outputs": [], "source": [ - "!uv pip install -q torch timm scikit-learn" + "import importlib.util\n", + "\n", + "# timm is pinned to match pyproject.toml's classification-torch extra: it resolves the\n", + "# DINOv2 backbone weights below (pretrained=True), and DINOv2's pretrained weights\n", + "# started under a non-commercial license before later moving to Apache 2.0 - an\n", + "# unpinned install could silently track a future timm release onto different\n", + "# weights/licensing terms. Bump only as a deliberate, reviewed change (both here and\n", + "# in pyproject.toml).\n", + "PINNED = {\"timm\": \"timm==1.0.28\"}\n", + "\n", + "missing = [pkg for pkg, mod in [(\"torch\", \"torch\"), (\"timm\", \"timm\"), (\"scikit-learn\", \"sklearn\")]\n", + " if importlib.util.find_spec(mod) is None]\n", + "if missing:\n", + " import subprocess\n", + " to_install = [PINNED.get(pkg, pkg) for pkg in missing]\n", + " subprocess.run([\"uv\", \"pip\", \"install\", \"-q\", *to_install], check=True)" ] }, { @@ -78,9 +93,15 @@ "# Option B: your own labelled data (set USE_EXAMPLE_DATABASE = False)\n", "DATA_DIR = \"\"\n", "\n", - "# Training images per class (remaining images used for validation)\n", + "# Training images per class\n", "IMAGES_PER_CLASS = 50\n", "\n", + "# Validation images per class, capped rather than using every remaining image - this\n", + "# dataset is large and unbalanced (thousands of images in some classes), so validating\n", + "# on \"everything left over\" is far more expensive than training itself for little\n", + "# extra signal.\n", + "VAL_PER_CLASS = 20\n", + "\n", "# Training epochs. 20–50 is typical; more rarely helps.\n", "EPOCHS = 30\n", "\n", @@ -178,9 +199,10 @@ " indices = np.arange(len(cls_samples))\n", " rng.shuffle(indices)\n", " n_train = min(IMAGES_PER_CLASS, len(cls_samples) - 1)\n", + " n_val = min(VAL_PER_CLASS, len(cls_samples) - n_train)\n", " for i in indices[:n_train]:\n", " train_samples.append(cls_samples[i])\n", - " for i in indices[n_train:]:\n", + " for i in indices[n_train:n_train + n_val]:\n", " val_samples.append(cls_samples[i])\n", " if n_train < IMAGES_PER_CLASS:\n", " print(f\" Warning: {classes[cls_idx]} has only {len(cls_samples)} images \"\n", diff --git a/pyopia/__init__.py b/pyopia/__init__.py index f7487666..a6b62ff3 100644 --- a/pyopia/__init__.py +++ b/pyopia/__init__.py @@ -1 +1 @@ -__version__ = "2.16.16" +__version__ = "2.17.0" diff --git a/pyopia/cli.py b/pyopia/cli.py index c6560131..6b246254 100644 --- a/pyopia/cli.py +++ b/pyopia/cli.py @@ -375,7 +375,7 @@ def process(config_filename: str, num_chunks: int = 1, strategy: str = "block"): @app.command() -def process_realtime(config_filename: str, watch_folder: str = None): +def process_realtime(config_filename: str, watch_folder: str = None, queue_size: int = 10): """Run a PyOPIA processing pipeline in realtime by watching a folder. Parameters @@ -386,6 +386,12 @@ def process_realtime(config_filename: str, watch_folder: str = None): watch_folder : str, optional Folder to monitor. If not provided, inferred from `general.raw_files` in config. + queue_size : int, optional + Maximum number of queued images retained when processing falls behind + acquisition; older images are dropped to stay close to realtime. A bigger + queue only delays which images get dropped - it doesn't fix an underlying + backlog where processing is slower than acquisition. Defaults to 10. + Notes ----- - Single-core only: files are processed sequentially by a single worker thread. @@ -412,7 +418,7 @@ def process_realtime(config_filename: str, watch_folder: str = None): output_datafile = pipeline_config["steps"]["output"]["output_datafile"] os.makedirs(os.path.split(output_datafile)[:-1][0], exist_ok=True) - pyopia.realtime.run_realtime(pipeline_config, watch_folder=watch_folder) + pyopia.realtime.run_realtime(pipeline_config, watch_folder=watch_folder, queue_size=queue_size) finally: stop_queue_logging(listener, log_queue) diff --git a/pyopia/io.py b/pyopia/io.py index 92522b0f..f1a23aa4 100644 --- a/pyopia/io.py +++ b/pyopia/io.py @@ -661,6 +661,179 @@ def __call__(self, data): return data +class ImageToDisc: + '''Pipeline-compatible class for saving processed images to disc. + + Saves specified pipeline images (e.g. raw, background corrected, segmented) + to an output folder. Can optionally downscale images and/or combine them + into a single collage per input image. + + Required keys in :class:`pyopia.pipeline.Data`: + - :attr:`pyopia.pipeline.Data.filename` + - At least one of the image keys specified in ``image_keys`` + + Parameters + ---------- + output_folder : str + Path to the output folder where images will be saved. + Created automatically if it does not exist. + image_keys : list of str, optional + List of pipeline data keys to save as images. + Defaults to ``['imraw', 'imbg', 'im_corrected', 'imbw']``. + Keys that are not present in the pipeline data for a given image + will be silently skipped. + scale_factor : float, optional + Factor to downscale images before saving. E.g. 0.5 halves the + resolution. Defaults to 1.0 (no scaling). + collage : bool, optional + If True, all specified images are combined into a single collage + image (one row per image key) rather than saved as separate files. + Defaults to False. + image_format : str, optional + Image file format extension. Defaults to ``'png'``. + + Returns + ------- + data : :class:`pyopia.pipeline.Data` + Unmodified pipeline data. + + Examples + -------- + Save background-corrected and segmented images to a folder: + + .. code-block:: toml + + [steps.saveimages] + pipeline_class = 'pyopia.io.ImageToDisc' + output_folder = 'processed_images' + image_keys = ['imraw', 'im_corrected', 'imbw'] + scale_factor = 0.5 + + Save a collage of all processing stages: + + .. code-block:: toml + + [steps.saveimages] + pipeline_class = 'pyopia.io.ImageToDisc' + output_folder = 'processed_images' + collage = true + ''' + + def __init__(self, output_folder='processed_images', + image_keys=None, + scale_factor=1.0, + collage=False, + image_format='png'): + if image_keys is None: + image_keys = ['imraw', 'imbg', 'im_corrected', 'imbw'] + self.output_folder = output_folder + self.image_keys = image_keys + self.scale_factor = scale_factor + self.collage = collage + self.image_format = image_format + + def __call__(self, data): + os.makedirs(self.output_folder, exist_ok=True) + + source_filename = data.get('filename', 'unknown') + base_name = Path(source_filename).stem + + # Collect available images (keep original dtypes for efficiency) + available_images = [] + for key in self.image_keys: + if key in data and data[key] is not None: + available_images.append((key, np.asarray(data[key]))) + + if not available_images: + logger.warning('ImageToDisc: No images found in pipeline data for the specified keys.') + return data + + if self.collage: + self._save_collage(available_images, base_name) + else: + self._save_separate(available_images, base_name) + + return data + + def _prepare_image(self, img): + '''Prepare an image for saving: handle scaling and normalisation. + + Parameters + ---------- + img : ndarray + Image array (2D or 3D, float or bool). + + Returns + ------- + img : ndarray + Prepared image array clipped to [0, 1]. + ''' + from skimage.transform import rescale + + # Convert to float64 for saving + img = img.astype(np.float64) + + if self.scale_factor != 1.0: + multichannel = img.ndim == 3 + img = rescale(img, self.scale_factor, + channel_axis=2 if multichannel else None, + anti_aliasing=True, + preserve_range=True) + # Clip to valid range for plt.imsave + img = np.clip(img, 0, 1) + return img + + def _save_separate(self, available_images, base_name): + '''Save each image key as a separate file.''' + import matplotlib.pyplot as plt + + for key, img in available_images: + img = self._prepare_image(img) + out_path = Path(self.output_folder) / f'{base_name}_{key}.{self.image_format}' + if img.ndim == 2: + plt.imsave(str(out_path), img, cmap='gray') + else: + plt.imsave(str(out_path), img) + logger.debug(f'ImageToDisc: Saved {key} to {out_path}') + + def _save_collage(self, available_images, base_name): + '''Save all images combined into a single collage image.''' + import matplotlib.pyplot as plt + from skimage.transform import resize + + # Determine target width (use first image width, after scale) + first_img = available_images[0][1] + if first_img.ndim == 2: + target_h, target_w = first_img.shape + else: + target_h, target_w = first_img.shape[:2] + + if self.scale_factor != 1.0: + target_h = int(target_h * self.scale_factor) + target_w = int(target_w * self.scale_factor) + + # Resize all images to the same dimensions and convert to 3-channel + panels = [] + for key, img in available_images: + if img.dtype == bool: + img = img.astype(np.float64) + + if img.ndim == 2: + img = resize(img, (target_h, target_w), anti_aliasing=True, preserve_range=True) + # Convert grayscale to RGB for stacking + img = np.stack([img, img, img], axis=-1) + else: + img = resize(img, (target_h, target_w, img.shape[2]), anti_aliasing=True, preserve_range=True) + + img = np.clip(img, 0, 1) + panels.append(img) + + collage = np.concatenate(panels, axis=0) + out_path = Path(self.output_folder) / f'{base_name}_collage.{self.image_format}' + plt.imsave(str(out_path), collage) + logger.debug(f'ImageToDisc: Saved collage to {out_path}') + + def load_toml(toml_file): """Load a TOML settings file from file diff --git a/pyopia/pipeline.py b/pyopia/pipeline.py index 9aeea0a4..e0a35404 100644 --- a/pyopia/pipeline.py +++ b/pyopia/pipeline.py @@ -75,6 +75,13 @@ def __init__(self, settings, # Flag used to control whether remaining pipeline steps should be skipped once it has been set to True self.data['skip_next_steps'] = False + # Progress tracking is opt-in via enable_progress_tracking(); left at these + # defaults, _log_progress() is a no-op + self._progress_total_files = None + self._progress_files_done = 0 + self._progress_start_time = None + self._progress_log_interval = 5 + self.pass_general_settings() for stepname in self.stepnames: @@ -126,10 +133,68 @@ def run(self, filename): # Reset skip flag self.data['skip_next_steps'] = False + self._log_progress() return + self._log_progress() return + def enable_progress_tracking(self, total_files, log_interval=5): + '''Opt in to periodic progress/ETA logging as `run()` is called in a loop + + Logs percent complete, elapsed time and an ETA at INFO level every + `log_interval` files. Has no effect unless called - by default `run()` does + not log any progress summary. + + Note + ---- + When running as multiple chunks (`pyopia process --num-chunks`), each chunk + runs in its own process with its own `Pipeline` instance, so progress is + reported per chunk (e.g. "3 of 20" for that chunk's own file list) rather than + as a single total across every chunk. + + Parameters + ---------- + total_files : int + Total number of files that will be passed to `run()`, used to calculate + percent complete and ETA + log_interval : int, optional + Log a progress update every this many files, by default 5 + + Examples + -------- + >>> pipeline = Pipeline(settings) + >>> pipeline.enable_progress_tracking(len(filenames)) + >>> for filename in filenames: + ... pipeline.run(filename) + ''' + self._progress_total_files = total_files + self._progress_files_done = 0 + self._progress_start_time = time.time() + self._progress_log_interval = log_interval + + def _log_progress(self): + '''Log a progress/ETA update, if progress tracking is enabled and due''' + if self._progress_total_files is None: + return + + self._progress_files_done += 1 + done = self._progress_files_done + total = self._progress_total_files + + if done % self._progress_log_interval != 0 and done != total: + return + + elapsed = time.time() - self._progress_start_time + remaining = (elapsed / done) * (total - done) + eta = (datetime.datetime.now() + datetime.timedelta(seconds=remaining)).strftime('%H:%M:%S') + + logger.info( + f'Progress: {done}/{total} ({100 * done / total:.1f}%) - ' + f'elapsed {datetime.timedelta(seconds=int(elapsed))}, ' + f'ETA {eta} (in {datetime.timedelta(seconds=int(remaining))})' + ) + def run_step(self, stepname): '''Execute a pipeline step and update the pipeline data diff --git a/pyopia/process.py b/pyopia/process.py index eeee21a7..db0695d8 100644 --- a/pyopia/process.py +++ b/pyopia/process.py @@ -183,7 +183,7 @@ def get_spine_length(imbw_roi): return spine_length -def extract_roi(input_image, bbox): +def extract_roi(input_image, bbox, pad=0): '''Given a full image and bounding box, this will return the roi image from within the bounding box Parameters @@ -192,12 +192,19 @@ def extract_roi(input_image, bbox): Full image. Can be any image, such as background-corrected image bbox : array bounding box from regionprops [r1, c1, r2, c2] + pad : int, optional + Additional fixed-pixel margin to add on every side before cropping, on top of + whatever bbox was passed in (e.g. one already expanded via :func:`expand_bbox`). + Clamped to image bounds. See :func:`pad_bbox`. Defaults to 0 (no padding). Returns ------- roi : array Image cropped to region of interest ''' + if pad: + bbox = pad_bbox(bbox, input_image.shape, pad) + # refer to skimage regionprops documentation on how bbox is structured roi = input_image[bbox[0]:bbox[2], bbox[1]:bbox[3]] @@ -257,6 +264,54 @@ def expand_bbox(bbox, image_shape, fraction): ], dtype=int) +def pad_bbox(bbox, image_shape, pad): + '''Expand a bounding box by a fixed number of pixels on every side, clamped to image bounds. + + Unlike :func:`expand_bbox`, which scales with the particle's own size, this adds the + same absolute pixel margin regardless of particle size - useful for guaranteeing a + consistent border (e.g. a small non-particle margin for the classifier, or for visual + inspection/montage context) rather than one proportional to the particle. Can be + combined with :func:`expand_bbox`: apply that first, then pass its result here. + + Parameters + ---------- + bbox : array-like of int + [min_row, min_col, max_row, max_col], following the skimage regionprops + convention where ``max_row`` and ``max_col`` are exclusive. + image_shape : tuple + Shape of the full image. Only the first two elements (H, W) are used, + so passing ``imc.shape`` works for both 2-D and 3-D images. + pad : int + Number of pixels to add on every side. ``0`` (or ``None``) returns the bbox + unchanged. Must be non-negative. + + Returns + ------- + padded : ndarray of int, shape (4,) + Padded and clamped bounding box, integer-valued. + + Raises + ------ + ValueError + If ``pad`` is negative. + ''' + bbox_int = np.asarray(bbox, dtype=int) + if pad is None or pad == 0: + return bbox_int + if pad < 0: + raise ValueError(f'pad must be non-negative, got {pad}') + + r1, c1, r2, c2 = bbox_int + H, W = image_shape[0], image_shape[1] + + return np.array([ + max(0, r1 - pad), + max(0, c1 - pad), + min(H, r2 + pad), + min(W, c2 + pad), + ], dtype=int) + + def put_roi_in_h5(export_outputpath, HDF5File, roi, filename, i): '''Adds rois to an open hdf file if export_outputpath is not None. For use within {func}`pyopia.process.export_particles` @@ -286,7 +341,7 @@ def put_roi_in_h5(export_outputpath, HDF5File, roi, filename, i): def extract_particles(imc, timestamp, Classification, region_properties, export_outputpath=None, min_length=0, propnames=['major_axis_length', 'minor_axis_length', 'equivalent_diameter'], - bbox_expansion=0.0): + bbox_expansion=0.0, pad=0): '''Extracts the particles to build stats and export particle rois to HDF5 files Parameters @@ -314,6 +369,13 @@ def extract_particles(imc, timestamp, Classification, region_properties, bounds. Only the exported ROI image is affected; the ``minr/minc/maxr/ maxc`` columns saved in stats continue to report the un-expanded regionprops bbox so that measurements are unchanged. + pad : int, optional + Fixed-pixel margin added on every side, on top of ``bbox_expansion`` (either, + both, or neither can be used). Unlike ``bbox_expansion``, this doesn't scale + with particle size, so it's useful for guaranteeing a consistent absolute + border - e.g. a small non-particle margin for the classifier, since the + classifier is run on this same cropped ROI. Clamped to image bounds. + Defaults to 0 (no padding). See :func:`pad_bbox`. Returns ------- @@ -365,10 +427,10 @@ def extract_particles(imc, timestamp, Classification, region_properties, if ((data[i, 0] > min_length) & (data[i, 1] > 2)): nb_extractable_part += 1 - # extract the region of interest from the corrected colour image, - # optionally with the bbox expanded by `bbox_expansion` to add context + # extract the region of interest from the corrected colour image, optionally + # with the bbox expanded by `bbox_expansion` and/or a fixed `pad` margin roi_bbox = expand_bbox(bboxes[i, :], imc.shape, bbox_expansion) - roi = extract_roi(imc, roi_bbox) + roi = extract_roi(imc, roi_bbox, pad=pad) if Classification is not None: # run a prediction on what type of particle this might be @@ -490,7 +552,7 @@ def statextract(imbw, timestamp, imc, export_outputpath=None, min_length=0, propnames=['major_axis_length', 'minor_axis_length', 'equivalent_diameter'], - bbox_expansion=0.0): + bbox_expansion=0.0, pad=0): '''Extracts statistics of particles in a binary images (imbw) Parameters @@ -519,6 +581,9 @@ def statextract(imbw, timestamp, imc, bbox_expansion : float, optional Fractional expansion of bounding boxes when cropping ROI images for export. See :func:`extract_particles`. Defaults to 0.0 (no expansion). + pad : int, optional + Fixed-pixel margin added on every side, on top of ``bbox_expansion``. + See :func:`extract_particles`. Defaults to 0 (no padding). Returns ------- @@ -547,7 +612,7 @@ def statextract(imbw, timestamp, imc, stats = extract_particles(imc, timestamp, Classification, region_properties, export_outputpath=export_outputpath, min_length=min_length, propnames=propnames, - bbox_expansion=bbox_expansion) + bbox_expansion=bbox_expansion, pad=pad) return stats, saturation @@ -632,6 +697,13 @@ class CalculateStats(): width and height (5% on each side, clamped to image bounds). The regionprops measurements and the ``minr/minc/maxr/maxc`` columns written into stats are unaffected. Defaults to ``0.0`` (no expansion). + pad: (int, optional) + Fixed-pixel margin added on every side of each ROI crop, on top of + ``bbox_expansion`` - either, both, or neither can be used. Unlike + ``bbox_expansion``, this doesn't scale with particle size, so it's useful + for guaranteeing a consistent absolute border, e.g. a small non-particle + margin for the classifier (which is run on this same cropped ROI). + Clamped to image bounds. Defaults to ``0`` (no padding). Configure from a TOML pipeline as:: @@ -639,6 +711,7 @@ class CalculateStats(): pipeline_class = "pyopia.process.CalculateStats" export_outputpath = "/path/to/rois" bbox_expansion = 0.1 + pad = 2 Returns ------- @@ -654,7 +727,7 @@ def __init__(self, min_length=0, propnames=['major_axis_length', 'minor_axis_length', 'equivalent_diameter'], roi_source='im_corrected', - bbox_expansion=0.0): + bbox_expansion=0.0, pad=0): self.max_coverage = max_coverage self.max_particles = max_particles @@ -663,6 +736,7 @@ def __init__(self, self.propnames = propnames self.roi_source = roi_source self.bbox_expansion = bbox_expansion + self.pad = pad self.calc_image_stats = CalculateImageStats() @@ -675,7 +749,7 @@ def __call__(self, data): export_outputpath=self.export_outputpath, min_length=self.min_length, propnames=self.propnames, - bbox_expansion=self.bbox_expansion) + bbox_expansion=self.bbox_expansion, pad=self.pad) stats['timestamp'] = data['timestamp'] stats['saturation'] = saturation diff --git a/pyopia/realtime.py b/pyopia/realtime.py index e0671f39..e9b2ba75 100644 --- a/pyopia/realtime.py +++ b/pyopia/realtime.py @@ -3,9 +3,9 @@ import fnmatch import logging import pathlib -import queue import threading import time +from collections import deque import pandas as pd from rich import print as rich_print @@ -29,7 +29,7 @@ def _resolve_watch_settings(raw_files_pattern: str, watch_folder: str | None) -> def _enqueue_file_if_new( file_path: pathlib.Path, - file_queue: queue.Queue, + file_queue: deque, file_pattern: str, seen_files: set[str], seen_lock: threading.Lock, @@ -46,14 +46,14 @@ def _enqueue_file_if_new( return False seen_files.add(file_key) - file_queue.put(file_path) + file_queue.append(file_path) return True def _enqueue_existing_files( watch_folder: str, file_pattern: str, - file_queue: queue.Queue, + file_queue: deque, seen_files: set[str], seen_lock: threading.Lock, logger: logging.Logger, @@ -64,7 +64,7 @@ def _enqueue_existing_files( def _build_event_handler( - file_queue: queue.Queue, + file_queue: deque, file_pattern: str, seen_files: set[str], seen_lock: threading.Lock, @@ -95,7 +95,7 @@ def on_moved(self, event): def _worker_loop( stop_event: threading.Event, - file_queue: queue.Queue, + file_queue: deque, processing_pipeline: pyopia.pipeline.Pipeline, logger: logging.Logger, runtime_state: dict, @@ -103,8 +103,12 @@ def _worker_loop( ): while not stop_event.is_set(): try: - filepath = file_queue.get(timeout=1) - except queue.Empty: + # Pop from the right (most recently queued) so a backlog is worked off + # newest-first, matching the point of a bounded, oldest-dropping queue: + # stay close to "now" rather than grinding through stale images. + filepath = file_queue.pop() + except IndexError: + time.sleep(0.1) continue try: @@ -123,10 +127,9 @@ def _worker_loop( finally: with state_lock: runtime_state["current_file"] = "idle" - file_queue.task_done() -def run_realtime(pipeline_config: dict, watch_folder: str | None = None): +def run_realtime(pipeline_config: dict, watch_folder: str | None = None, queue_size: int = 10): """Run a PyOPIA processing pipeline in realtime by watching a folder. Parameters @@ -135,6 +138,11 @@ def run_realtime(pipeline_config: dict, watch_folder: str | None = None): Loaded PyOPIA pipeline config. watch_folder : str, optional Folder to monitor. If not provided, inferred from ``general.raw_files``. + queue_size : int, optional + Maximum number of queued images retained when processing falls behind + acquisition; older images are dropped to stay close to realtime. A bigger + queue only delays which images get dropped - it doesn't fix an underlying + backlog where processing is slower than acquisition. """ logger = logging.getLogger("rich") logger.info(f"PyOPIA realtime process started {pd.Timestamp.now()}") @@ -145,7 +153,7 @@ def run_realtime(pipeline_config: dict, watch_folder: str | None = None): processing_pipeline = pyopia.pipeline.Pipeline(pipeline_config) - file_queue = queue.Queue() + file_queue = deque(maxlen=queue_size) stop_event = threading.Event() seen_files: set[str] = set() seen_lock = threading.Lock() @@ -212,7 +220,7 @@ def run_realtime(pipeline_config: dict, watch_folder: str | None = None): description=( "[blue]Realtime active" f" | processed: {processed_count}" - f" | queued: {file_queue.qsize()}" + f" | queued: {len(file_queue)}" f" | current: {current_file}" ), ) diff --git a/pyopia/statistics.py b/pyopia/statistics.py index 383004f5..704b12e4 100644 --- a/pyopia/statistics.py +++ b/pyopia/statistics.py @@ -5,7 +5,9 @@ import os import pandas as pd import numpy as np +from skimage.draw import disk from skimage.exposure import rescale_intensity +from skimage.morphology import binary_dilation import h5py from tqdm import tqdm from pyopia.io import write_stats, load_stats_as_dataframe @@ -478,6 +480,178 @@ def make_montage( return montage_image +def make_montage_scaled( + stats_file_or_df, + pixel_size, + roidir, + msize=1024, + rel_scale=1.0, + gap=2, + max_attempts=500, + maxlength=100000, + crop_region=None, + brightness=1, + eyecandy=True, +): + """Makes a montage of particles packed within a circular boundary, largest first + + This is an alternative to :func:`make_montage` for instruments (e.g. holographic + imaging) where particles are naturally monochrome: the montage is a single-channel + grayscale image rather than RGB, so it can be plotted directly with + :func:`pyopia.plotting.montage_plot`, including its 1mm scale reference. + + The key difference from :func:`make_montage` is `rel_scale`: rather than always + filling the same fixed canvas regardless of how much data went into it, + `rel_scale` controls the *area* of the circular region available for packing, as a + fraction of the full canvas area. This makes it possible to visually compare + particle number density across datasets with different sample sizes - e.g. several + depth bins with different numbers of raw images - fairly: set `rel_scale` + proportional to each dataset's relative sample size (e.g. number of raw images, or + total sample volume) against a shared reference, generate one montage per dataset, + and place them side by side. A bin with half the raw images of another gets half + the circle area to fill, so the resulting packed density is directly comparable + between montages, rather than every montage always looking equally "full" + regardless of how much data it actually represents. + + Every exported particle is attempted, largest first, since bigger particles are + harder to accommodate once the canvas starts filling up - there is no upfront + subsampling or truncation, since either would misrepresent the true relative + abundance of particle sizes. Each particle is given a `gap`-pixel buffer against + its neighbours (via binary dilation of its silhouette) so that packed particles + don't visually touch. A particle that can't find a free spot within `max_attempts` + random placements is skipped rather than resized or forced in; if any particles are + skipped this way, a warning is logged summarising how many, once the montage is + complete. This means `msize` is too small to fit everything - increase it (and, if + this montage is one of several being compared via `rel_scale`, increase `msize` the + same way for all of them, to keep the relative comparison valid; don't compensate + by changing `rel_scale` itself, since that would distort the comparison it exists + to preserve). + + Parameters + ---------- + stats_file_or_df : DataFrame or str + either a str specifying the location of the STATS.nc file that comes from processing, or a stats dataframe + pixel_size : float + pixel size of system in microns + roidir : str + location of roifiles + msize : int, optional + size of the (square) canvas in pixels, by default 1024 + rel_scale : float, optional + fraction (0-1) of the full canvas *area* used as the circular placement + boundary. Set this proportional to relative sample size when comparing several + montages side by side (see above); use 1.0 for a single montage that isn't + being compared against others, by default 1.0 + gap : int, optional + minimum gap in pixels enforced between packed particles, by default 2 + max_attempts : int, optional + number of random placement attempts per particle before giving up on it, by default 500 + maxlength : int, optional + maximum length in microns of particles to be included in montage, by default 100000 + crop_region : tuple, optional + None or 4-tuple of lower-left then upper-right coord of crop, passed to :func:`crop_stats`, by default None + brightness : int, optional + brightness of packaged particles used with eyecandy option, by default 1 + eyecandy : bool, optional + boolean which if True will explode the contrast of packed particles + (nice for natural particles, but not so good for oil and gas), by default True + + Returns + ------- + montage_image : array + grayscale montage image (values 0-1, dark particles on a light background, + with the region outside the circular boundary shown slightly darker than the + background so the boundary is visible) that can be plotted with + :func:`pyopia.plotting.montage_plot` + """ + if isinstance(stats_file_or_df, str): + stats = load_stats_as_dataframe(stats_file_or_df) + else: + stats = stats_file_or_df + + if crop_region is not None: + stats = crop_stats(stats, crop_region) + + # remove nans because concentrations are not important here + stats = stats[~np.isnan(stats["major_axis_length"])] + stats = stats[(stats["major_axis_length"] * pixel_size) < maxlength] + + # pack largest particles first, since they're the hardest to fit later on + stats = stats.sort_values(by=["major_axis_length"], ascending=False) + + # every exported particle is attempted - see docstring for why this isn't subsampled + roifiles = stats["export_name"][stats["export_name"] != "not_exported"].values + + # background = 1 (white); slightly darker outside the circular boundary so it's + # visible; particles are painted in as they're placed (values < 1) + montage = np.ones((msize, msize), dtype=np.float64) + # radius scales with sqrt(rel_scale) so that *area* (not radius) is proportional + # to rel_scale - see the rel_scale explanation above + radius = np.sqrt(rel_scale) * msize / 2 + rr, cc = disk((msize / 2, msize / 2), radius, shape=montage.shape) + within_boundary = np.zeros(montage.shape, dtype=bool) + within_boundary[rr, cc] = True + montage[~within_boundary] = 0.9 + + # available[i, j] is True while position (i, j) is free to place a particle in + available = within_boundary.copy() + + logger.info("making a scaled montage - this might take some time....") + n_placed = 0 + for roi_name in tqdm(roifiles): + particle_image = roi_from_export_name(roi_name, roidir) + if particle_image.ndim == 3: + particle_image = particle_image.mean(axis=2) + + if eyecandy: + particle_image = explode_contrast(particle_image) + particle_image = bright_norm(particle_image, brightness) + particle_image = np.clip(particle_image, 0, 1) + + height, width = particle_image.shape + if height >= msize or width >= msize: + continue + + # silhouette of the particle (darker-than-background pixels), padded out by + # `gap` so placed particles keep a visual buffer from their neighbours + silhouette = binary_dilation(particle_image < 0.9, footprint=np.ones((gap * 2 + 1, gap * 2 + 1))) + + placed = False + for _ in range(max_attempts): + r = np.random.randint(0, msize - height) + c = np.random.randint(0, msize - width) + + footprint = available[r:r + height, c:c + width] + if np.all(footprint[silhouette]): + canvas_region = montage[r:r + height, c:c + width] + particle_pixels = particle_image < 0.9 + canvas_region[particle_pixels] = particle_image[particle_pixels] + + available[r:r + height, c:c + width][silhouette] = False + placed = True + n_placed += 1 + break + + if not placed: + logger.debug(f"Could not find a free spot for particle: {roi_name}") + + _log_montage_placement_summary(n_placed, len(roifiles)) + + return montage + + +def _log_montage_placement_summary(n_placed, n_total): + """Log how many particles a scaled montage placed, warning if any were skipped""" + n_skipped = n_total - n_placed + if n_skipped > 0: + logger.warning( + f"{n_skipped} of {n_total} particles could not be placed and were skipped - " + "consider increasing msize to fit all particles (and, if comparing this montage " + "against others via rel_scale, increase msize consistently for all of them)." + ) + logger.info(f"scaled montage complete: placed {n_placed} of {n_total} particles") + + def gen_roifiles(stats, auto_scaler=500): """Generates a list of filenames suitable for making montages with diff --git a/pyopia/tests/test_cli.py b/pyopia/tests/test_cli.py index 2c6546be..f6a76c0f 100644 --- a/pyopia/tests/test_cli.py +++ b/pyopia/tests/test_cli.py @@ -213,8 +213,8 @@ def test_process_realtime_prepares_output_folder_and_calls_run_realtime(tmp_path recorded = {} monkeypatch.setattr( pyopia.cli.pyopia.realtime, 'run_realtime', - lambda pipeline_config, watch_folder=None: recorded.update( - pipeline_config=pipeline_config, watch_folder=watch_folder + lambda pipeline_config, watch_folder=None, queue_size=10: recorded.update( + pipeline_config=pipeline_config, watch_folder=watch_folder, queue_size=queue_size ) ) @@ -227,12 +227,14 @@ def test_process_realtime_prepares_output_folder_and_calls_run_realtime(tmp_path }, fh) result = invoke_in(tmp_path, [ - 'process-realtime', str(config_filename), '--watch-folder', str(tmp_path / 'images') + 'process-realtime', str(config_filename), '--watch-folder', str(tmp_path / 'images'), + '--queue-size', '25', ]) assert result.exit_code == 0, result.output assert (tmp_path / 'proc').is_dir() assert recorded['watch_folder'] == str(tmp_path / 'images') + assert recorded['queue_size'] == 25 assert recorded['pipeline_config']['steps']['output']['output_datafile'] == output_datafile diff --git a/pyopia/tests/test_io.py b/pyopia/tests/test_io.py index 181e2bfe..6d8f01e6 100644 --- a/pyopia/tests/test_io.py +++ b/pyopia/tests/test_io.py @@ -1,8 +1,10 @@ import os from pathlib import Path import pytest +import numpy as np +import matplotlib.pyplot as plt import pandas as pd -from pyopia.io import write_stats, load_stats, get_cf_metadata_spec +from pyopia.io import write_stats, load_stats, get_cf_metadata_spec, ImageToDisc from pyopia.instrument.silcam import generate_config @@ -64,5 +66,150 @@ def test_write_and_load_stats(tmp_path: Path): assert "PyOPIA_version" in loaded_stats.attrs +def test_image_to_disc_separate(tmp_path: Path): + """Test ImageToDisc saves separate images for each pipeline key.""" + output_folder = str(tmp_path / "output_images") + + saver = ImageToDisc( + output_folder=output_folder, + image_keys=['imraw', 'im_corrected', 'imbw'], + scale_factor=1.0, + collage=False, + ) + + # Create fake pipeline data with a mix of image types + data = { + 'filename': '/fake/path/test_image.silc', + 'imraw': np.random.rand(100, 120, 3), + 'im_corrected': np.random.rand(100, 120, 3), + 'imbw': np.random.rand(100, 120) > 0.5, # boolean segmentation mask + } + + result = saver(data) + + # Check data is returned unmodified + assert result is data + + # Check output files exist + assert os.path.isfile(os.path.join(output_folder, 'test_image_imraw.png')) + assert os.path.isfile(os.path.join(output_folder, 'test_image_im_corrected.png')) + assert os.path.isfile(os.path.join(output_folder, 'test_image_imbw.png')) + + +def test_image_to_disc_collage(tmp_path: Path): + """Test ImageToDisc saves a single collage image.""" + output_folder = str(tmp_path / "output_collage") + + saver = ImageToDisc( + output_folder=output_folder, + image_keys=['imraw', 'im_corrected'], + collage=True, + ) + + data = { + 'filename': '/fake/path/sample.silc', + 'imraw': np.random.rand(80, 100, 3), + 'im_corrected': np.random.rand(80, 100, 3), + } + + result = saver(data) + + assert result is data + assert os.path.isfile(os.path.join(output_folder, 'sample_collage.png')) + + +def test_image_to_disc_scale_factor(tmp_path: Path): + """Test ImageToDisc applies scale factor when saving separate images.""" + output_folder = str(tmp_path / "output_scaled") + + saver = ImageToDisc( + output_folder=output_folder, + image_keys=['imraw'], + scale_factor=0.5, + collage=False, + ) + + data = { + 'filename': '/fake/path/scaled_test.silc', + 'imraw': np.random.rand(100, 120, 3), + } + + saver(data) + + out_file = os.path.join(output_folder, 'scaled_test_imraw.png') + assert os.path.isfile(out_file) + + # Load the saved image and verify it was scaled down + saved_img = plt.imread(out_file) + assert saved_img.shape[0] == 50 + assert saved_img.shape[1] == 60 + + +def test_image_to_disc_missing_keys(tmp_path: Path): + """Test ImageToDisc gracefully skips missing keys.""" + output_folder = str(tmp_path / "output_missing") + + saver = ImageToDisc( + output_folder=output_folder, + image_keys=['imraw', 'imbg', 'nonexistent_key'], + ) + + data = { + 'filename': '/fake/path/missing_test.silc', + 'imraw': np.random.rand(50, 60, 3), + # 'imbg' and 'nonexistent_key' intentionally missing + } + + result = saver(data) + + assert result is data + # Only imraw should be saved + assert os.path.isfile(os.path.join(output_folder, 'missing_test_imraw.png')) + assert not os.path.isfile(os.path.join(output_folder, 'missing_test_imbg.png')) + assert not os.path.isfile(os.path.join(output_folder, 'missing_test_nonexistent_key.png')) + + +def test_image_to_disc_2d_grayscale(tmp_path: Path): + """Test ImageToDisc handles 2D grayscale images.""" + output_folder = str(tmp_path / "output_gray") + + saver = ImageToDisc( + output_folder=output_folder, + image_keys=['im_corrected'], + scale_factor=0.5, + ) + + data = { + 'filename': '/fake/path/gray_test.png', + 'im_corrected': np.random.rand(100, 120), + } + + saver(data) + + assert os.path.isfile(os.path.join(output_folder, 'gray_test_im_corrected.png')) + + +def test_image_to_disc_collage_mixed_types(tmp_path: Path): + """Test collage with a mix of 2D (binary) and 3D (RGB) images.""" + output_folder = str(tmp_path / "output_collage_mixed") + + saver = ImageToDisc( + output_folder=output_folder, + image_keys=['imraw', 'imbw'], + collage=True, + scale_factor=0.5, + ) + + data = { + 'filename': '/fake/path/mixed_test.silc', + 'imraw': np.random.rand(80, 100, 3), + 'imbw': np.random.rand(80, 100) > 0.5, + } + + saver(data) + + assert os.path.isfile(os.path.join(output_folder, 'mixed_test_collage.png')) + + if __name__ == "__main__": pytest.main() diff --git a/pyopia/tests/test_realtime.py b/pyopia/tests/test_realtime.py index 769a4018..5cc54d1a 100644 --- a/pyopia/tests/test_realtime.py +++ b/pyopia/tests/test_realtime.py @@ -1,6 +1,6 @@ import logging -import queue import threading +from collections import deque from pathlib import Path import pyopia.realtime @@ -119,7 +119,7 @@ def test_resolve_watch_settings_prefers_explicit_watch_folder(tmp_path: Path): def test_event_handler_enqueues_only_matching_moved_files(tmp_path: Path): - file_queue = queue.Queue() + file_queue = deque() logger = logging.getLogger("test") seen_files = set() seen_lock = threading.Lock() @@ -142,13 +142,13 @@ def test_event_handler_enqueues_only_matching_moved_files(tmp_path: Path): handler.on_moved(moved_event_match) handler.on_moved(moved_event_no_match) - queued = file_queue.get_nowait() + queued = file_queue.popleft() assert queued == matched - assert file_queue.empty() + assert len(file_queue) == 0 def test_event_handler_deduplicates_same_moved_file(tmp_path: Path): - file_queue = queue.Queue() + file_queue = deque() logger = logging.getLogger("test") seen_files = set() seen_lock = threading.Lock() @@ -167,13 +167,13 @@ def test_event_handler_deduplicates_same_moved_file(tmp_path: Path): handler.on_moved(moved_event) handler.on_moved(moved_event) - queued = file_queue.get_nowait() + queued = file_queue.popleft() assert queued == matched - assert file_queue.empty() + assert len(file_queue) == 0 def test_enqueue_existing_files_matches_pattern_and_deduplicates(tmp_path: Path): - file_queue = queue.Queue() + file_queue = deque() seen_files = set() seen_lock = threading.Lock() logger = logging.getLogger("test") @@ -200,13 +200,29 @@ def test_enqueue_existing_files_matches_pattern_and_deduplicates(tmp_path: Path) logger, ) - queued = file_queue.get_nowait() + queued = file_queue.popleft() assert queued == matched - assert file_queue.empty() + assert len(file_queue) == 0 + + +def test_enqueue_drops_oldest_once_queue_size_is_exceeded(tmp_path: Path): + file_queue = deque(maxlen=2) + seen_files = set() + seen_lock = threading.Lock() + + files = [] + for i in range(3): + f = tmp_path / f"image_{i}.silc" + f.write_text("ok") + files.append(f) + pyopia.realtime._enqueue_file_if_new(f, file_queue, "*.silc", seen_files, seen_lock) + + assert len(file_queue) == 2 + assert list(file_queue) == files[1:] def test_integration_existing_then_moved_files_processed_once(tmp_path: Path): - file_queue = queue.Queue() + file_queue = deque() seen_files = set() seen_lock = threading.Lock() logger = logging.getLogger("test") @@ -272,8 +288,8 @@ def run(self, filename): image_file.write_text("content") stop_event = threading.Event() - file_queue = queue.Queue() - file_queue.put(image_file) + file_queue = deque() + file_queue.append(image_file) pipeline = DummyPipeline() runtime_state = {"processed_count": 0, "current_file": "idle"} state_lock = threading.Lock() diff --git a/pyproject.toml b/pyproject.toml index 577a7c68..ea4ece9d 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -66,6 +66,13 @@ classification = [ classification-torch = [ "torch>=2.1.0", + # Pinned exactly: timm resolves the DINOv2 backbone weights used by the classifier + # training notebook (pretrained=True), and DINOv2's pretrained weights started under + # a non-commercial license before later moving to Apache 2.0 - an unpinned version + # could silently track a future timm release onto different weights/licensing terms. + # Bump this only as a deliberate, reviewed change. + "timm==1.0.28", + "scikit-learn>=1.3.0", ] [dependency-groups]