Skip to content

feat: add pact compatibility suite runner using Reqnroll - #571

Open
YOU54F wants to merge 2 commits into
pact-foundation:masterfrom
YOU54F:feat/compat-suite-reqnroll
Open

YOU54F wants to merge 2 commits into
pact-foundation:masterfrom
YOU54F:feat/compat-suite-reqnroll

Conversation

@YOU54F

@YOU54F YOU54F commented Nov 9, 2026

Copy link
Copy Markdown
Member

Goal

Run the shared pact-compatibility-suite against PactNet, the way pact-go, pact-php, pact-python and pact-reference already do. That way, a pact_ffi upgrade or a DSL change shows up as scenario-level differences instead of surprises in users' tests.

Changes

  • New project tests/PactNet.Compatibility.Tests (.NET 10, xUnit) using Reqnroll.
    • The suite is a git submodule pinned at dd08122.
    • Scenarios whose steps aren't implemented yet are reported as skipped, not failed.
  • Steps go through the public DSL (Pact.V2/V3/V4, WithHttpInteractions) wherever it covers them. Where it doesn't, a small Support/ layer uses the internal drivers (InternalsVisibleTo added to PactNet).
  • PACT_FFI_LIB=/path/to/libpact_ffi.{so,dylib,dll} (a file, or a directory containing one) runs the suite against a different FFI build than the bundled one.
  • baseline.yaml tracks known failures, each with a reason:
    • a new failure fails the run;
    • so does a baselined scenario that starts passing;
    • COMPAT_UPDATE_BASELINE=1 rewrites it.
  • JUnit output via JunitXml.TestLogger. compat.runsettings switches xUnit to its json reporter, because the default reporter crashes the logger on the suite's duplicate scenario titles.
  • README: a "Pact Compatibility Suite" section.

Coverage so far

Only the V1 HTTP consumer steps are implemented: 24 pass, 0 fail, 166 skipped. The same counts come out on pact_ffi 0.5.12, pact-foundation/pact-reference#568 and pact-foundation/pact-reference#584.

These DSL gaps came up while writing the steps. They decide how far the suite can get through the public API:

  • no binary body API;
  • no V1 spec version;
  • no access to mock server mismatches;
  • the interaction handle isn't exposed;
  • VerifyAsync only works inside its callback;
  • no way to apply the suite's matching-rule tables.

Not done

  • Steps for V1 provider, V2–V4, messages and sync messages.
  • CI wiring. This needs a decision on whether a suite run should gate PRs or only report.

Testing

  • dotnet test tests/PactNet.Compatibility.Tests on macOS arm64 against three pact_ffi builds: 24 pass / 166 skipped each.
  • PACT_FFI_LIB pointing at a missing file fails all 190 scenarios, so the override is what's actually loaded.

🤖 Generated with Claude Code

YOU54F and others added 2 commits October 28, 2026 23:56
Adds tests/PactNet.Compatibility.Tests, which runs the shared
pact-compatibility-suite features (git submodule pinned to dd08122)
with Reqnroll.xUnit. Feature files are linked from the submodule and
code-behind is generated under obj/.

- V1 HTTP consumer steps implemented over the public DSL, with
  NativeInterop only where the DSL cannot express the step
- baseline.yaml ratchet: unbaselined failures and baselined passes fail
  the run; undefined steps count as unsupported; COMPAT_UPDATE_BASELINE=1
  rewrites the file
- PACT_FFI_LIB overrides the native library via SetDllImportResolver
- JUnit output via JunitXml.TestLogger (xUnit json reporter set in
  compat.runsettings to avoid a logger crash on duplicate titles)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant