Skip to content

fix(scoring): gate valid merged PRs on source token score#1397

Closed
Dexterity104 wants to merge 1 commit into
entrius:testfrom
Dexterity104:fix/oss-eligibility-source-token-score
Closed

fix(scoring): gate valid merged PRs on source token score#1397
Dexterity104 wants to merge 1 commit into
entrius:testfrom
Dexterity104:fix/oss-eligibility-source-token-score

Conversation

@Dexterity104
Copy link
Copy Markdown

Summary

The OSS scoring path measured one threshold constant against two different quantities. The base-score gate in calculate_base_score_for_pr_files keyed off the SOURCE-only token score, while the eligibility "valid merged PR" gate in check_eligibility keyed off pr.token_score, the all-category aggregate (SOURCE + TEST). As a result a merged PR could earn zero source base score yet still count toward min_valid_merged_prs, the gate that unlocks all scoring for a repository. A near-threshold source PR plus a small test file was enough to flip a PR from "not valid" to "valid".

This change makes validity match the base-score gate:

  • BaseScoreResult now returns source_token_score (already computed inside the helper, so no extra work).
  • ScoredPR stores source_token_score alongside the aggregate token_score, set during score_pr.
  • check_eligibility counts a merged PR as valid only when its source_token_score clears the threshold.
  • The constant comment and the check_eligibility docstring now describe the actual behavior.

The aggregate token_score is still kept for reporting rollups, the storage adapter, and the issue discovery cache, so both fields remain in use. The issue discovery path is intentionally untouched: it gates on a separate constant (min_token_score_for_valid_issue) compared consistently against the aggregate, so it has no internal mismatch.

Related Issues

Fixes #1396

Type of Change

  • Bug fix
  • New feature
  • Refactor
  • Documentation
  • Other (describe below)

Testing

  • Tests added/updated
  • Manually tested

Added test_validity_gate_keys_off_source_not_aggregate_token_score, which confirms a repo of merged PRs that pass the aggregate threshold but fail the SOURCE threshold is ineligible. Updated the _merged test helper to set the gating field, extended the base-score helper test to assert source_token_score is surfaced, and updated the BaseScoreResult stubs in test_scoring.py. The eligibility gate logic was verified directly: SOURCE-fail rows produce an ineligible result and SOURCE-pass rows produce an eligible one.

Checklist

  • Code follows project style guidelines
  • Self-review completed
  • Changes are documented (if applicable)

@xiao-xiao-mao xiao-xiao-mao Bot added the bug Something isn't working label May 29, 2026
@anderdc anderdc closed this May 29, 2026
@anderdc
Copy link
Copy Markdown
Collaborator

anderdc commented May 29, 2026

The eligibility gate this patches no longer exists on test. check_eligibility now counts merged PRs directly against min_valid_merged_prs and no longer compares token_score against min_token_score_for_base_score, so the SOURCE-vs-aggregate mismatch from #1396 is already gone and this diff conflicts against the current path. Closing as out of date.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Eligibility "valid merged PR" gate counts test content the base-score gate ignores

2 participants