Skip to content

Add xSV validation module#205

Merged
ialarmedalien merged 10 commits into
developfrom
develop-add-xsv-validate-module
Jul 10, 2026
Merged

Add xSV validation module#205
ialarmedalien merged 10 commits into
developfrom
develop-add-xsv-validate-module

Conversation

@mattldawson

@mattldawson mattldawson commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

Description of PR purpose/changes

This PR adds a validation option for character-separated value files. It uses the xsv-validate.sh shell script that was included in the container build scripts with PR #203

Minor Update

  • Modified the Dockerfile to install the qsv and xml_file_splitter packages for the hardware the image is built for, and to add a missing qsv dependency.

List any dependencies that are required for this change.

xml-validate.sh and qsv (already present in Docker build)

Testing Instructions

Build and run containerized tests as usual.

  • Tests pass locally and in GitHub Actions

Dev Checklist:

  • My submission follows the AI Covenant principles
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have run Ruff format to format my code
  • I have run Ruff check and fixed any errors that it uncovered

Comment thread src/cdm_data_loaders/validation/xsv.py Fixed

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an xSV (CSV/TSV/etc.) validation adapter to the cdm_data_loaders.validation package that shells out to xsv-validate.sh (qsv-backed), along with unit + e2e-style tests and the necessary marker/workflow/documentation updates to gate tool-dependent tests.

Changes:

  • Introduces cdm_data_loaders.validation.xsv.validate() plus Status/ValidationResults to run xsv-validate.sh and parse optional summary output.
  • Adds extensive tests (mocked subprocess + requires_xsv integration-style coverage) and registers the requires_xsv pytest marker.
  • Updates docs, CI test selection, and Docker image dependencies/binary installation to support running the validator.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
src/cdm_data_loaders/validation/xsv.py New validation adapter that invokes xsv-validate.sh and returns structured results (optionally from summary JSON).
tests/validation/test_xsv.py New unit + tool-gated integration-style tests for argument building, status mapping, and end-to-end behavior.
pyproject.toml Registers requires_xsv marker for pytest.
README.md Documents how to exclude requires_xsv tests when the tool isn’t installed.
Dockerfile Installs arch-appropriate qsv/xml_file_splitter and adds a missing runtime dependency.
.github/workflows/tests.yml Excludes requires_xsv tests from the “local tests” job filter.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/cdm_data_loaders/validation/xsv.py Outdated
Comment thread src/cdm_data_loaders/validation/xsv.py Outdated
Comment thread tests/validation/test_xsv.py Outdated
mattldawson and others added 7 commits July 9, 2026 12:26
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
@codecov

codecov Bot commented Jul 9, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 69.15%. Comparing base (c01480f) to head (7e9a020).
⚠️ Report is 3 commits behind head on develop.

Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff             @@
##           develop     #205      +/-   ##
===========================================
+ Coverage    68.57%   69.15%   +0.58%     
===========================================
  Files           73       75       +2     
  Lines         4462     4546      +84     
===========================================
+ Hits          3060     3144      +84     
  Misses        1402     1402              
Files with missing lines Coverage Δ
src/cdm_data_loaders/validation/xsv.py 100.00% <100.00%> (ø)

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ae0a5ce...7e9a020. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@mattldawson mattldawson requested a review from ialarmedalien July 9, 2026 20:11
Comment thread Dockerfile
Comment on lines +22 to +25
RUN ARCH=$(uname -m) && \
wget https://github.com/ialarmedalien/xml_file_splitter/releases/download/${XML_FILE_SPLITTER_VERSION}/xml_file_splitter-${ARCH}-unknown-linux-gnu.tar.gz && \
tar -xvf xml_file_splitter-${ARCH}-unknown-linux-gnu.tar.gz && \
mv xml_file_splitter-${ARCH}-unknown-linux-gnu/xml_file_splitter /usr/local/bin/ && \

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is great, thanks!

null_strings: set[str] | None = None,
skip_lines: int = 0,
summary: bool = False,
) -> ValidationResults:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All of the validation here immediately makes me think it might be easier to use a pydantic object to capture the fields so you get the validation "for free" when the object is instantiated. A pydantic object can be dumped as json or a dictionary, so depending on how the object is set up, it might be possible to have it output the args for xiv-validate in a form that will require minimal wrangling to convert to turn into args.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great suggestion! I took a stab at this and was able to remove all the validation code from the function, and I think it's a lot cleaner this way. I wasn't able to figure out an equally clean way to generate the xsv-validate args, though. Anything I considered just ended up needing a lot more code. Let me know what you think. I'm happy to make more revisions if needed.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it looks fine as-is, so not worth spending the extra time and effort on.

@ialarmedalien ialarmedalien left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great!

@ialarmedalien ialarmedalien merged commit 697c878 into develop Jul 10, 2026
14 checks passed
@ialarmedalien ialarmedalien deleted the develop-add-xsv-validate-module branch July 10, 2026 13:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants