Skip to content

Preserve string-bearing NetCDF input files - #370

Open
sylvesterkaczmarek wants to merge 1 commit into
nasa:developfrom
sylvesterkaczmarek:fix/string-inputs-369
Open

sylvesterkaczmarek wants to merge 1 commit into
nasa:developfrom
sylvesterkaczmarek:fix/string-inputs-369

Conversation

@sylvesterkaczmarek

Copy link
Copy Markdown

Fixes #369.

Description

Keep nonempty NetCDF inputs that contain text variables. Empty-file detection currently applies numpy.isnan to text; the resulting TypeError is caught by validate_workable_files, which silently drops the input. A station-name variable preceding populated measurements is enough to produce an empty workable-file list and no concatenated output.

Handle variable-length strings and character arrays separately, checking their unmasked values against the fill value. Zero-size, fill-only, and fully masked string variables remain empty. Numeric handling and the concatenation algorithms are unchanged. Includes an unreleased changelog entry.

This is newly reproduced on develop 581d7d41. Issue #369 was opened on 24 September 2026. Before submission it was unassigned, with no other claimant or matching open PR.

Local test steps

  • All 19 new regression cases pass; 16 fail and three controls pass on unchanged upstream. These use real NetCDF files, covering variable-length strings, raw and decoded character arrays, scalar strings, fill values, masks, empty dimensions, and nested groups.
  • Six cases run the complete public file-to-file concatenation path and verify exact string/numeric results and unchanged input bytes. The nested fixture passes data_vars="all" because its root is empty; the ordinary root-level fixtures use the default options.
  • The complete CI unit-test selection, python -m pytest -q tests/unit --cov=stitchee --tb=short, passes 42 tests on macOS arm64/Python 3.12.11, with 91% total coverage.
  • The same 42 unit tests pass with frozen uv.lock dependencies on Python 3.13.5 (xarray 2025.12.0, NumPy 2.3.5, netCDF4 1.7.3).
  • Source distribution and wheel builds pass. Installed the wheel outside the checkout, verified the import path, and reran all 19 new cases successfully.
  • All repository pre-commit checks applicable to the changed files pass, including secret scanning, Ruff, formatting, and mypy. Full ruff check stitchee, syntax checks, and git diff --check pass. No test, dependency, or CI configuration was weakened.

Overview of integration done

The new tests exercise local NetCDF creation, validation, concatenation, and output reading without Harmony credentials. The existing unit suite's local Harmony adapter tests also pass. Live Harmony requests were not submitted.

The additional tests/local run requires datasets absent from the repository: five tests fail with missing TEMPO/CERES input directories on both this branch and unchanged upstream; its synthetic-data control passes. These are not CI unit-suite failures. Character-array operations on the newer NumPy stack emit dependency deprecation warnings, while the frozen dependency run retains the existing warning types. Other operating systems and remote-service integration were not run locally.

PR Acceptance Checklist

  • Unit tests added and passing.
  • Local file-to-file integration and installed-package validation completed.
  • CHANGELOG.md updated.
  • Maintainer-run CI and live Harmony validation.

Fixes nasa#369. Preserve existing behavior outside the corrected error path.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

String variables cause nonempty NetCDF inputs to be silently discarded

1 participant