Skip to content

test(audit): regression harness uses CI audit entry point (run_main_audit_workflow)#6917

Merged
chubes4 merged 1 commit into
mainfrom
feat/audit-harness-workflow-entry
Jun 28, 2026
Merged

test(audit): regression harness uses CI audit entry point (run_main_audit_workflow)#6917
chubes4 merged 1 commit into
mainfrom
feat/audit-harness-workflow-entry

Conversation

@chubes4

@chubes4 chubes4 commented Jun 28, 2026

Copy link
Copy Markdown
Member

Fixes the deep fidelity gap behind #6896/#6906/#6915: the harness audited via audit_path_with_id while CI Audit uses run_main_audit_workflow + reference_paths — so reference-path/symbol-graph regressions could not reproduce locally. Switches the harness to run_main_audit_workflow (matching src/commands/audit.rs), re-captures the snapshot, keeps the skips_test_paths invariant + determinism test. Now a #6855-style structural/symbol-graph regression fails locally. Harness test + fixture only; no detector/config change.

@homeboy-ci

homeboy-ci Bot commented Jun 28, 2026

Copy link
Copy Markdown
Contributor

Homeboy Results — homeboy

Lint

lint — passed

ℹ️ Full options: homeboy docs commands/lint
ℹ️ Save lint baseline: homeboy lint homeboy --baseline
Deep dive: homeboy lint homeboy --changed-since fd23417

Artifacts and drill-down
  • CI results artifact: homeboy-ci-results-homeboy-lint-homeboy-Linux contains immediate command JSON for this action invocation.
  • Observation artifact: homeboy-observations-homeboy-lint-homeboy-Linux contains exported Homeboy run history for deeper queries.
  • Drill-down: download the observation artifact, then run homeboy runs import <dir>, homeboy runs list, and homeboy runs findings <run-id>.
  • Artifacts are attached to the workflow run: https://github.com/Extra-Chill/homeboy/actions/runs/28329868847

Test

⚠️ test — baseline red

ℹ️ No tests ran — the runner failed before producing results. See raw_output.stderr_tail / raw_output.stdout_tail for the underlying error (bootstrap failure, missing deps, DB connection, etc.).
ℹ️ To run specific tests: homeboy test homeboy -- --filter=TestName
ℹ️ Auto-fix lint issues: homeboy refactor homeboy --from lint --write
ℹ️ Collect coverage: homeboy test homeboy --coverage
ℹ️ Analyze failures: homeboy test homeboy --analyze
ℹ️ Pass args to test runner: homeboy test -- [args]
ℹ️ Full options: homeboy docs commands/test
Deep dive: homeboy test homeboy --changed-since fd23417

Artifacts and drill-down
  • CI results artifact: homeboy-ci-results-homeboy-test-homeboy-Linux contains immediate command JSON for this action invocation.
  • Observation artifact: homeboy-observations-homeboy-test-homeboy-Linux contains exported Homeboy run history for deeper queries.
  • Drill-down: download the observation artifact, then run homeboy runs import <dir>, homeboy runs list, and homeboy runs findings <run-id>.
  • Artifacts are attached to the workflow run: https://github.com/Extra-Chill/homeboy/actions/runs/28329868847

Audit

audit — passed

  • core_boundary_leak:core-agnostic-source — 70 finding(s)
  • structural — 57 finding(s)
  • intra-method-duplication — 47 finding(s)
  • field_patterns — 35 finding(s)
  • dead_code — 19 finding(s)
  • near-duplication — 17 finding(s)
  • test_quality — 13 finding(s)
  • duplication — 10 finding(s)
  • remote_execution_preflight — 9 finding(s)
  • thin_command_adapter — 8 finding(s)
  • Total: 310 finding(s)

Deep dive: homeboy audit homeboy --changed-since fd23417

Artifacts and drill-down
  • CI results artifact: homeboy-ci-results-homeboy-audit-homeboy-Linux contains immediate command JSON for this action invocation.
  • Observation artifact: homeboy-observations-homeboy-audit-homeboy-Linux contains exported Homeboy run history for deeper queries.
  • Drill-down: download the observation artifact, then run homeboy runs import <dir>, homeboy runs list, and homeboy runs findings <run-id>.
  • Artifacts are attached to the workflow run: https://github.com/Extra-Chill/homeboy/actions/runs/28329868847
Tooling versions
  • Homeboy CLI: homeboy 0.269.0+2ef12653d00f+78bc59ec5
  • Extension: rust from https://github.com/Extra-Chill/homeboy-extensions
  • Extension revision: dfe9fa31
  • Action: unknown@unknown

@chubes4

chubes4 commented Jun 28, 2026

Copy link
Copy Markdown
Member Author

Empirical finding: entry point was NOT the gap; the regression is REAL-CODEBASE-SPECIFIC

I tested this directly: cherry-picked the #6915 structural migration onto THIS branch (workflow-entry harness) and ran the harness — it still PASSES (4 tests), even via run_main_audit_workflow. So switching to the CI entry point, while strictly correct and worth keeping, does NOT reproduce the +300 regression on the synthetic fixture either.

Conclusion: the #6855 Phase 1 regression depends on the actual homeboy repository's scale/content (real cross-references, real file sizes hitting GodFile/HighItemCount thresholds, real reference_paths) — it cannot be reproduced by any synthetic fixture, however enriched.

This PR is still a real improvement (the harness should audit via the same entry point CI uses — better fidelity for the class of regressions a fixture CAN catch), so it's worth merging on its own merits.

But the actual unblocker for #6855 is a different harness shape: a test/CI step that audits the REAL homeboy repo tree and asserts zero net drift vs baseline — essentially running the differential Audit gate locally/in-test. That's the only thing that reproduces real-codebase-specific regressions. Recommend that as the next step (it overlaps with #6858 'fast audit preview/reconcile').

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant