Skip to content

[core] Close a media bug added after hangup cleanup to avoid zombie channels - #3224

Open
renny678 wants to merge 1 commit into
signalwire:masterfrom
renny678:fix-media-bug-zombie-channel
Open

renny678 wants to merge 1 commit into
signalwire:masterfrom
renny678:fix-media-bug-zombie-channel

Conversation

@renny678

Copy link
Copy Markdown

Problem

Starting a session recording (uuid_record, switch_ivr_record_session) at the moment the channel hangs up can leave a channel that is never destroyed. It stays in show channels, hupall and uuid_kill cannot remove it, it keeps its session slot and RTP ports, and shutdown has to wait for it. Reported in #1673, #2067 and #2645.

Root cause

switch_core_media_bug_add() calls the bug's SWITCH_ABC_TYPE_INIT callback before it links the bug into session->bugs. For session recording, INIT starts recording_thread, which takes a session read lock and waits on a condition variable until SWITCH_ABC_TYPE_CLOSE signals it.

If the session thread runs switch_core_media_bug_remove_all() (right after switch_core_session_run() returns in switch_core_session_thread()) between INIT and the linking, the new bug is not on the list yet, so nothing ever closes it. The recording thread keeps its read lock, and the session thread blocks forever in switch_core_session_write_lock().

Thread dump of a stuck session on 1.11.3:

  • session thread: pthread_rwlock_wrlock <- switch_core_session_thread
  • recording thread: pthread_cond_wait <- recording_thread

Fix

After the bug is linked (and MEDIA_BUG_START is fired), if the channel is already down (switch_channel_down_nosig), remove the bug right away. Removing it runs CLOSE, which signals and joins the recording thread.

remove_all() only runs after the channel is down, and both linking and removal take bug_rwlock. So either remove_all() sees the linked bug and closes it, or this check sees the channel down and closes it; whichever comes second finds nothing to do.

The function still returns SWITCH_STATUS_SUCCESS and keeps *new_bug: callers such as switch_ivr_record_session_event() keep using the bug pointer (session pool memory), and on failure they would destroy their user data again after CLOSE already did.

Compared with #2218 (holding bug_rwlock across INIT), this does not run INIT and the execute_on_record_start / api_on_record_start hooks under the bug lock, and it does not hold up remove_all() on the hangup path while INIT waits up to 2 seconds for the recording thread.

Testing

Two FreeSWITCH 1.11.3 instances (these functions are identical on master). 200 iterations: originate to an answering peer, then bgapi uuid_record <uuid> start <file> 0 followed by bgapi uuid_kill <uuid>, with 0-9 ms between the two commands.

  • Without the patch: 162 calls answered, 51 channels never destroyed. After that the RTP port range was exhausted and further originates failed with DESTINATION_OUT_OF_ORDER.
  • With the patch: 200 calls answered, 0 channels left.

Normal recordings are unaffected (we have been running this patch since 2026-10-14).

Fixes #2067. Refs #1673, #2645.

…hannels

switch_core_media_bug_add() runs the INIT callback before it links the
bug into session->bugs. For session recording, INIT starts
recording_thread, which holds a session read lock until CLOSE signals it.
If the session thread runs switch_core_media_bug_remove_all() in between,
the bug is never closed, and the session waits forever in
switch_core_session_write_lock() (zombie channel).

After linking, close the bug right away if the channel is already down.

Fixes signalwire#2067. Refs signalwire#1673, signalwire#2645.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Hangup a call when starting recording may cause zombie channels

1 participant