Skip to content

fix: reject orphan checkpoint race - #690

Open
warren830 wants to merge 2 commits into
aws:mainfrom
warren830:fix/issue-640-checkpoint-race
Open

fix: reject orphan checkpoint race#690
warren830 wants to merge 2 commits into
aws:mainfrom
warren830:fix/issue-640-checkpoint-race

Conversation

@warren830

Copy link
Copy Markdown

Issue

Fixes #640

Description

A child checkpoint could pass the initial parent-done validation, pause while plugin hooks ran, then enqueue after its parent completed and marked it orphaned. The final enqueue path rechecked execution completion and checkpoint failure, but not the child's _parent_done state.

This change:

  • extracts the repeated orphan rejection into _reject_if_parent_done();
  • rechecks the child's orphan state in the final validation-and-enqueue region;
  • holds _completion_lock and then _parent_done_lock through queue insertion, establishing a single linearization point;
  • documents the global _completion_lock_parent_done_lock order;
  • keeps plugin callbacks outside both locks and avoids the parent-done lock for empty checkpoints.

Scope

This PR fixes only the already-known-child race in #640. It deliberately does not reject late-created descendants whose parent was already orphaned; that separate behavior remains tracked by #641.

Red → green

  • Unchanged upstream production code: the deterministic regression failed with DID NOT RAISE OrphanedChildException.
  • Patched focused test: passed.
  • Repeated fresh executions: 25/25 passed.

The regression uses two threading.Event barriers and no sleeps: the child pauses in the plugin hook after initial validation, the parent completes and marks it orphaned, and the child then resumes into the final enqueue attempt.

Testing

  • Repository tests: 3351 passed, 2 skipped, 5 subtests.
  • Core tests: 1627 passed, 5 subtests.
  • Core coverage: 98.09% (98% required).
  • OpenTelemetry tests: 149 passed, 92% coverage.
  • Examples: 87 passed, 2 skipped.
  • Type checks: 77 core source files and 23 OpenTelemetry source files, no errors.
  • Ruff format/check: core (77 files), OpenTelemetry (23 files), and examples (155 files), all passed.
  • Commit lint and git diff --check: passed.

All validation used the repository-pinned Hatch 1.16.5 and Python 3.12. No AWS call or cloud resource was used.

Fork CI note

Some repository cloud/conformance workflows require AWS secrets, SAM, and live CloudFormation resources that are unavailable to fork PRs. The authoritative local unit, integration, coverage, type-check, and formatting gates above are green; any credential-gated non-required fork check should be treated as an infrastructure limitation rather than a product regression.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Recheck parent completion while holding the queue insertion locks so an
already-known branch cannot enqueue after becoming orphaned.

Add a deterministic event-coordinated regression test.
@warren830
warren830 marked this pull request as ready for review August 31, 2026 02:09
@warren830
warren830 deployed to ai-pr-review-runtime August 31, 2026 02:09 — with GitHub Actions Active
@warren830
warren830 deployed to ai-pr-review August 31, 2026 02:09 — with GitHub Actions Active
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Orphaned map/parallel branch can checkpoint after parent completion

2 participants