Add xSV validation module#205
Conversation
There was a problem hiding this comment.
Pull request overview
Adds an xSV (CSV/TSV/etc.) validation adapter to the cdm_data_loaders.validation package that shells out to xsv-validate.sh (qsv-backed), along with unit + e2e-style tests and the necessary marker/workflow/documentation updates to gate tool-dependent tests.
Changes:
- Introduces
cdm_data_loaders.validation.xsv.validate()plusStatus/ValidationResultsto runxsv-validate.shand parse optional summary output. - Adds extensive tests (mocked subprocess +
requires_xsvintegration-style coverage) and registers therequires_xsvpytest marker. - Updates docs, CI test selection, and Docker image dependencies/binary installation to support running the validator.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
src/cdm_data_loaders/validation/xsv.py |
New validation adapter that invokes xsv-validate.sh and returns structured results (optionally from summary JSON). |
tests/validation/test_xsv.py |
New unit + tool-gated integration-style tests for argument building, status mapping, and end-to-end behavior. |
pyproject.toml |
Registers requires_xsv marker for pytest. |
README.md |
Documents how to exclude requires_xsv tests when the tool isn’t installed. |
Dockerfile |
Installs arch-appropriate qsv/xml_file_splitter and adds a missing runtime dependency. |
.github/workflows/tests.yml |
Excludes requires_xsv tests from the “local tests” job filter. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #205 +/- ##
===========================================
+ Coverage 68.57% 69.15% +0.58%
===========================================
Files 73 75 +2
Lines 4462 4546 +84
===========================================
+ Hits 3060 3144 +84
Misses 1402 1402
Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
| RUN ARCH=$(uname -m) && \ | ||
| wget https://github.com/ialarmedalien/xml_file_splitter/releases/download/${XML_FILE_SPLITTER_VERSION}/xml_file_splitter-${ARCH}-unknown-linux-gnu.tar.gz && \ | ||
| tar -xvf xml_file_splitter-${ARCH}-unknown-linux-gnu.tar.gz && \ | ||
| mv xml_file_splitter-${ARCH}-unknown-linux-gnu/xml_file_splitter /usr/local/bin/ && \ |
There was a problem hiding this comment.
this is great, thanks!
| null_strings: set[str] | None = None, | ||
| skip_lines: int = 0, | ||
| summary: bool = False, | ||
| ) -> ValidationResults: |
There was a problem hiding this comment.
All of the validation here immediately makes me think it might be easier to use a pydantic object to capture the fields so you get the validation "for free" when the object is instantiated. A pydantic object can be dumped as json or a dictionary, so depending on how the object is set up, it might be possible to have it output the args for xiv-validate in a form that will require minimal wrangling to convert to turn into args.
There was a problem hiding this comment.
Great suggestion! I took a stab at this and was able to remove all the validation code from the function, and I think it's a lot cleaner this way. I wasn't able to figure out an equally clean way to generate the xsv-validate args, though. Anything I considered just ended up needing a lot more code. Let me know what you think. I'm happy to make more revisions if needed.
There was a problem hiding this comment.
I think it looks fine as-is, so not worth spending the extra time and effort on.
Description of PR purpose/changes
This PR adds a validation option for character-separated value files. It uses the
xsv-validate.shshell script that was included in the container build scripts with PR #203Minor Update
qsvandxml_file_splitterpackages for the hardware the image is built for, and to add a missingqsvdependency.List any dependencies that are required for this change.
xml-validate.shandqsv(already present in Docker build)Testing Instructions
Build and run containerized tests as usual.
Dev Checklist:
formatto format my codecheckand fixed any errors that it uncovered