Skip to content

test: integration tests against fixture audio - #16

Merged
beengud merged 1 commit into
mainfrom
test/integration
May 1, 2026
Merged

beengud merged 1 commit into
mainfrom
test/integration

Conversation

@beengud

@beengud beengud commented May 1, 2026

Copy link
Copy Markdown
Contributor

Closes #7. Final issue in the v0.1 MVP.

Summary

End-to-end integration test for scribe transcribe plus targeted unit-test gap-fillers. Pipeline source from PR #15 is unchanged.

  • crates/scribe/tests/fixtures/jfk.wav (~344 KB) and tests/fixtures/README.md document license + provenance.
  • crates/scribe/tests/transcribe.rs — full pipeline run against JFK fixture into TempDir, asserts dual-output naming, frontmatter keys, ISO-8601 created, JSON sidecar shape, and JFK-specific "country" substring (proves whisper actually ran).
  • Self-skips cleanly (eprintln! + early return) when ffmpeg / ffprobe / whisper-cli / model is missing — keeps cargo test --all green on dependency-less CI runners.
  • New unit tests in output.rs (slug edge cases: long, all-punctuation, unicode + emoji, dotted version stems; frontmatter edge cases: paths with spaces, titles with colons, ISO-8601 second-precision pin, closing-blank-line convention) and config.rs (whitespace-only --model falls through to env; --keep-temp passthrough).

Fixture license attribution

jfk.wav is the canonical short-audio test sample bundled by whisper.cpp. Audio content is an excerpt of JFK's 1961 inaugural address — a U.S. federal government work, public domain in the U.S. under 17 U.S.C. § 105. Surrounding whisper.cpp project license is MIT (Copyright © 2023-2026 The ggml authors). Redistribution alongside Scribe (MIT-or-Apache-2.0) is unencumbered. See crates/scribe/tests/fixtures/README.md for the full attribution.

Test counts

Suite Before After
Unit (lib) 41 51 (+10)
tests/cli.rs 3 3
tests/transcribe.rs — 6 (1 e2e + 5 helper)

Sample output — run mode (deps available)

$ cargo test --test transcribe -- --nocapture
running 6 tests
test helper_tests::basename_well_formed_accepts_canonical ... ok
test helper_tests::basename_well_formed_rejects_non_digit_timestamp ... ok
test helper_tests::split_frontmatter_recovers_both_halves ... ok
test helper_tests::basename_well_formed_rejects_short_timestamp ... ok
test helper_tests::basename_well_formed_rejects_wrong_slug ... ok
test transcribe_jfk_end_to_end ... ok

test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.75s

Sample output — skip mode (no ffmpeg/whisper/model on PATH)

$ env -i HOME="$HOME" PATH=/nonexistent SCRIBE_WHISPER_BIN=/nonexistent SCRIBE_MODEL_PATH=/nonexistent <pre-built-test-binary> --nocapture
running 6 tests
test helper_tests::basename_well_formed_accepts_canonical ... ok
test helper_tests::basename_well_formed_rejects_non_digit_timestamp ... ok
test helper_tests::basename_well_formed_rejects_short_timestamp ... ok
test helper_tests::split_frontmatter_recovers_both_halves ... ok
SKIP transcribe_jfk_end_to_end: ffmpeg not found on PATH
test transcribe_jfk_end_to_end ... ok
test helper_tests::basename_well_formed_rejects_wrong_slug ... ok

test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

Cargo treats the early-return as a pass — exactly what CI needs.

Pipeline source unchanged

No changes to pipeline.rs, ffmpeg.rs, whisper.rs, output.rs (only new #[cfg(test)] items inside the existing mod tests block), or config.rs (same — additions only inside mod tests). The only non-test change is Cargo.toml adding serde_json and dirs as dev-dependencies for the integration test. No helpers were pub'd — integration tests live outside the binary crate and locally re-implement small lookup helpers (which_on_path, is_executable_file, env-resolvers).

Test plan

  • cargo fmt --all clean
  • cargo clippy --all-targets --all-features -- -D warnings clean
  • cargo test --all green on host with deps (51 unit + 3 cli + 6 transcribe = 60 tests pass)
  • Integration test self-skips cleanly with PATH=/nonexistent and missing env vars
  • Fixture license verified and documented

Out of scope (per spec)

  • Adding ffmpeg/whisper-cli/model to CI runners (separate self-hosted-runner follow-up).
  • Property-based tests or benchmarks.

🤖 Generated with Claude Code

…ture

Closes #7.

- Bundle whisper.cpp's `samples/jfk.wav` (~344 KB, public-domain JFK
  inaugural excerpt) under `crates/scribe/tests/fixtures/`, with a
  README documenting license + provenance.
- New `tests/transcribe.rs` runs the compiled `scribe transcribe`
  binary end-to-end into a `TempDir` and asserts: matching
  `<slug>-YYYYMMDD-HHMMSS` basenames for the `.md`/`.json` pair, full
  YAML frontmatter key set, ISO-8601 UTC `created`, sidecar field
  pointing at the JSON filename, non-empty body that doesn't bleed
  into the closing `---`, JFK-specific "country" substring (proof
  whisper actually ran), and valid whisper-cli JSON sidecar shape
  (top-level `transcription` array with `text` + `offsets.{from,to}`).
- The integration test self-skips with a clear `eprintln!` reason
  when ffmpeg / ffprobe / whisper-cli / model is missing, so
  `cargo test --all` stays green on CI runners without the AI
  toolchain. Verified locally in both modes.
- Gap-filler unit tests in `output.rs` (long stems, all-punctuation,
  unicode + emoji, dotted version stems, source paths with spaces,
  titles with colons, ISO-8601 second-precision pin, blank-line
  closing convention) and `config.rs` (whitespace-only `--model`
  flag falls through to env precedence; `--keep-temp` plumb-through).

Pipeline source (pipeline/ffmpeg/whisper/output/config) is unchanged
except for new tests inside the existing `#[cfg(test)] mod tests`
blocks; no `pub`-ing of helpers was required because integration
tests live outside the binary crate and reimplement small lookup
helpers locally.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@beengud
beengud merged commit 2b6ac3f into main May 1, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test: integration tests against fixture audio

1 participant