safety/mads: reset heartbeat mismatch counter so a re-engage is not revoked - #493
Draft
Geomglot wants to merge 2 commits into
Draft
safety/mads: reset heartbeat mismatch counter so a re-engage is not revoked#493Geomglot wants to merge 2 commits into
Geomglot wants to merge 2 commits into
Conversation
…e-count revoke mads_exit_controls did not clear heartbeat_engaged_mads_mismatches, so a heartbeat-mismatch exit left the counter saturated at 3. The next re-engage that lands between two 1 Hz firmware ticks - before heartbeat_engaged_mads catches up - is revoked on its very first mads_heartbeat_engaged_check (counter 3 -> 4 >= 3), even though controls should have a fresh 3-tick grace window. This is a general MADS bug affecting every brand using the heartbeat engaged check. It was discovered on Rivian, where the instant revoke rejects the in-flight lateral TX frames, putting counter gaps on the bus that latch an EPAS AngleControlCntr fault (EAC fault + ToiFlt -> steerFaultPermanent). Frame-by-frame replay of the field route c17ea97dc5472650/00000006 seg 3 reproduces the field firmware bit-exactly with 0 TX mismatches; with this one-line reset the re-engage survives and the 8 previously-rejected 0x110 frames pass (fault prevented). Adds test_heartbeat_engaged_mads_reengage_after_mismatch to mads_common.py: it saturates the counter via a mismatch exit, re-engages, and asserts the next tick does not instantly revoke. Verified to fail without the fix and pass with it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The previous commit resets heartbeat_engaged_mads_mismatches in
mads_exit_controls(), which covers a re-engage that lands *after* a
heartbeat-mismatch exit. It does not cover one that lands *before* it, and that
case is worse: it leaves lateral permanently revoked rather than briefly.
When a re-engage arrives while controls_allowed_lateral is still set, the grant
block's !controls_allowed_lateral guard skips, so controls_requested_lateral is
never consumed and just sits pending. The next heartbeat tick then fires
mads_exit_controls(), which drops both controls_allowed_lateral and that pending
request. No edge remains that can re-request lateral, so the safety stays
lateral-off while openpilot MADS is enabled. openpilot's own watchdog
(lateral_mismatch_counter >= 200 in mads.py) then immediate-disables 2s later
with a red "Controls Mismatch: Lateral".
This is a general MADS bug affecting every brand using the heartbeat engaged
check. It reproduces whenever MADS is disengaged and re-engaged within the 3-tick
heartbeat window, which is an easy thing for a driver to do with a stalk. It was
observed in the field on Rivian, where the disengage gesture (stalk past detent)
disengages MADS on the openpilot side without a corresponding safety-side exit,
so the window is entered on every such disengage.
Fix: reset the counter on the rising edge of controls_requested_lateral,
mirroring the existing guard in safety.h ("reset mismatches on rising edge of
controls_allowed to avoid rare race condition"). The watchdog is not weakened -
if openpilot is genuinely not engaged, the counter re-accumulates from zero and
the safety revokes 3 ticks later.
Adds test_heartbeat_engaged_mads_reengage_before_mismatch_exit, which fails
without the fix. It drives the engage through set_mads_button_press() rather than
_lkas_button_msg(): Subaru's mads_button_press is sticky (set on HUD state 1-3,
never cleared), so it cannot produce the second rising edge the test needs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Geomglot
force-pushed
the
mads-heartbeat-counter-reset
branch
from
July 30, 2026 01:39
9a6cc50 to
aba9aee
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
heartbeat_engaged_mads_mismatchesis never cleared on the two paths that should clear it, which makes a MADS re-engage near a heartbeat-mismatch tick get revoked. There are two halves to this race, one commit each. Both are general MADS bugs affecting every brand that uses the heartbeat engaged check, not brand-specific.Half 1 — re-engage lands after the mismatch exit (commit 1)
mads_exit_controlsdoes not reset the counter, so a heartbeat-mismatch exit leaves it saturated at3. The next re-engage that lands between two 1 Hz firmware ticks — beforeheartbeat_engaged_madscatches up — is revoked on its very firstmads_heartbeat_engaged_check()(3 -> 4 >= 3), instead of getting a fresh 3-tick grace window.Fix: one line at the end of
mads_exit_controls:Half 2 — re-engage lands before the mismatch exit (commit 2)
This case is worse: it leaves lateral permanently revoked rather than briefly.
When a re-engage arrives while
controls_allowed_lateralis still set, the grant block's!controls_allowed_lateralguard skips, socontrols_requested_lateralis never consumed and just sits pending. The next heartbeat tick then firesmads_exit_controls(), which drops bothcontrols_allowed_lateraland that pending request. No edge remains that can re-request lateral, so the safety stays lateral-off while openpilot MADS is enabled. openpilot's own watchdog (lateral_mismatch_counter >= 200inmads.py) then immediate-disables 2 s later with a red "Controls Mismatch: Lateral".Fix: reset the counter on the rising edge of
controls_requested_lateral, mirroring the existing guard insafety.h:Neither fix weakens the watchdog: if openpilot is genuinely not engaged, the counter re-accumulates from zero and the safety revokes 3 ticks later.
How it was found
Both halves were observed in the field on Rivian, where the MADS disengage gesture (stalk past detent) disengages MADS on the openpilot side without a corresponding safety-side exit — so the mismatch window is entered on every such disengage, and a driver re-engaging within ~3 s hits it.
Half 1 is destructive on Rivian: the in-flight lateral TX frames sent during the one-tick revoke are safety-rejected, putting counter gaps on the bus that latch an EPAS
AngleControlCntrfault (EAC fault+ToiFlt->steerFaultPermanent, "TAKE CONTROL IMMEDIATELY").Frame-by-frame replay of the field route
c17ea97dc5472650/00000006seg 3 through the compiled safety model:0x110frames blocked → EPAS fault0x110(field_blocked=True, replay_blocked=False)Half 2 was found later in route
4440a486580ed7c6/00000059seg 0, on a build that already had the half-1 fix. Log timeline:controls_allowed_lateralmads_exit_controls()drops lateral and the pending requestlateral_mismatch_counterhits 200 →controlsMismatchLateralimmediate disableTesting
Two regression tests in
mads_common.py, both running for every MADS brand viaCarSafetyTest, each verified to fail without its fix and pass with it:test_heartbeat_engaged_mads_reengage_after_mismatch— saturates the counter with a mismatch exit, re-engages, asserts the next tick does not instantly revoke.test_heartbeat_engaged_mads_reengage_before_mismatch_exit— engages, ticks the counter to 2, re-engages while lateral is still granted, asserts the 3rd tick does not revoke. Drives the engage throughset_mads_button_press()rather than_lkas_button_msg(): Subaru'smads_button_pressis sticky (set on HUD state 1-3, never cleared), so it cannot produce the second rising edge this test needs.Full safety suite green across all brands: 7578 passed, 3126 skipped. MISRA was not run locally (cppcheck has no Darwin/x86_64 build) — relying on CI for that.
🤖 Generated with Claude Code