Skip to content

Preserve single-object processing history during concatenation - #373

Closed
sylvesterkaczmarek wants to merge 1 commit into
nasa:developfrom
sylvesterkaczmarek:fix/preserve-single-record-provenance
Closed

sylvesterkaczmarek wants to merge 1 commit into
nasa:developfrom
sylvesterkaczmarek:fix/preserve-single-record-provenance

Conversation

@sylvesterkaczmarek

Copy link
Copy Markdown

GitHub Issue: #372

Description

Fixes #372.

Normalize a single-object history_json attribute to a one-element list in retrieve_history, and return an empty list for an absent attribute. Both the CLI collector and Harmony adapter extend their output history with this helper's result. Returning a dictionary caused them to append its property names and lose the corresponding provenance values.

Existing history arrays retain their order and contents. Malformed JSON still raises at the reader. This does not change concatenation, numerical values, download handling, or the new operation record. The helper return annotation and documentation now describe the record-list contract used by its callers.

Local test steps

Added eight cases covering singleton records, arrays, absent/empty histories, invalid JSON, mixed-record collection, the actual module CLI in a subprocess, and the Harmony adapter. Five cases fail against unchanged production code; three controls pass.

  • python -m pytest tests/unit -q --disable-warnings --cov=stitchee --cov-report=term-missing: 31 passed, 91% package statement coverage.
  • ruff check .: passed.
  • Ruff formatting checks on both changed Python files, Python compilation, and git diff --check: passed.

A separate baseline run including tests/local reported five failures because its untracked TEMPO/CERES data directories are not supplied in this checkout. Those local-data tests are not claimed as passing. Credentialed integration tests were not run.

Overview of integration done

The new CLI and adapter cases concatenate real synthetic NetCDF files, read the resulting history and data back, and verify byte-for-byte input preservation. The adapter uses actual local-file staging and real STAC objects; only downloads and collection-source lookup are supplied locally. The existing unit suite also exercises paged and unpaged Harmony CLI catalogs.

Tested on macOS with Python 3.12.11 against develop at 581d7d418a5f76a891fdbfebf3a9dd9687b02cea. No deployed Harmony service, live Earthdata downloads, or AWS resources were used. No dependency or workflow changes. This is separate from the string-input validation fix in #370.

PR Acceptance Checklist

  • Regression tests added and passing.
  • Deployed-service integration testing. Local CLI/adapter coverage is described above.
  • CHANGELOG.md updated.
  • Helper documentation and return annotation updated.

@ank1m ank1m closed this Oct 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Single-object processing history is replaced by its keys during concatenation

2 participants