Fix silent daemon hang from undrained stderr + attachment size limit#2
Open
mikesimone wants to merge 3 commits into
Open
Fix silent daemon hang from undrained stderr + attachment size limit#2mikesimone wants to merge 3 commits into
mikesimone wants to merge 3 commits into
Conversation
rpc_timeout was 30s - group sends to larger/busier destinations can legitimately take longer than that, and a timeout here doesn't cancel the request already handed to signal-cli, so it was producing spurious 504s for sends that likely succeeded anyway. Raised to 120s. /v1/search now also accepts a usernames= query param (getUserStatus supports both recipient and username independently; only recipient was wired up before). Also includes the DefaultBodyLimit fix (300MB, was axum's 2MB default) that was built and deployed 2026-07-11 but never committed.
The managed daemon's stderr was only ever read during the startup wait loop. Once the daemon signalled ready, nothing drained it again for the rest of the process's life - a Linux pipe buffer is ~64KB, and once signal-cli logs enough to fill it (e.g. a large/malformed attachment triggering warnings or a stack trace), its write() call blocks and the whole JVM silently stalls. No crash, no OOM, nothing in any log to point at; it just stops responding to JSON-RPC requests indefinitely.
signal-cli parses incoming JSON-RPC requests with Jackson, which enforces a hard 20,000,000-character limit on any single JSON string value (StreamReadConstraints.getMaxStringLength()). Attachments were embedded as base64 data URIs, so any file whose base64 form crosses that line (raw size a bit over ~15MB) trips a StreamConstraintsException that signal-cli doesn't handle cleanly - it kills the one shared JSON-RPC connection this whole process depends on, not just the offending request, taking down every other in-flight and future request too. signal-cli's `attachment` field also accepts a plain local file path, so decode data-URI attachments and write them to a temp file signal-cli can read directly instead, sidestepping the JVM's JSON parser for the bulky part entirely. Temp files are cleaned up once the send RPC call returns, success or error.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two related bugs, both found in production while running signal-cli-api with a real attachment-heavy workload (a message-forwarding service fanning out images/video to ~20 destinations):
Undrained stderr deadlocks the daemon.
daemon::spawn()pipes the managed signal-cli child's stderr (Stdio::piped()), but only reads it during the startup wait loop. Once the daemon reports ready, nothing drains that pipe for the rest of the process's life. A Linux pipe buffer is ~64KB; once signal-cli logs enough to fill it, itswrite()call blocks and the whole JVM silently stalls - no crash, no OOM, nothing in any log to explain it. From the outside it just looks like the daemon randomly stops responding to JSON-RPC requests. Fixed by spawning a task that continuously reads and logs the child's stderr for the life of the process.Large attachments exceed signal-cli's JSON string limit. Attachments are sent to
/v2/sendas base64 data URIs embedded directly in the JSON-RPC request. signal-cli parses incoming requests with Jackson, which enforces a hard 20,000,000-character limit on any single JSON string value (StreamReadConstraints.getMaxStringLength()). Any attachment whose base64 form crosses that line (raw files a bit over ~15MB) throws aStreamConstraintsExceptionthat signal-cli doesn't handle cleanly - it kills the single shared JSON-RPC connection the whole process depends on, taking down every other in-flight and future request, not just the oversized one. Fixed by decoding data-URI attachments server-side and writing them to a temp file, then passing signal-cli the file path instead (which itsattachmentfield already accepts) - sidesteps the JVM's JSON parser for the bulky part entirely. Temp files are cleaned up once the send RPC call returns.Together these two bugs compounded badly: the first bug meant the second bug's real error (
StreamConstraintsException) was completely invisible in logs until the stderr fix was in place to surface it.Test plan
cargo test --release)SUCCESSresult from Signal) after this fix, with the daemon staying healthy afterward