Skip to content

Decode server-sent events as UTF-8 and ignore a leading BOM - #1316

Closed
Kludex wants to merge 2 commits into
mainfrom
sse-utf8-sig
Closed

Kludex wants to merge 2 commits into
mainfrom
sse-utf8-sig

Conversation

@Kludex

@Kludex Kludex commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

Summary

A leading UTF-8 BOM made the first SSE event disappear, and a charset other than UTF-8 on text/event-stream garbled event data. The SSE spec says event streams are always UTF-8 and one leading BOM is ignored, which is what browsers do.

The SSE parser now reads bytes and decodes them with an incremental TextDecoder("utf-8-sig"), which handles both, including a BOM split across chunks. response.encoding is left untouched.

Checklist

  • I understand that this PR may be closed in case there was no previous discussion. (This doesn't apply to typos!)
  • I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
  • I've updated the documentation accordingly.

AI Disclaimer

This PR was developed with the assistance of either Claude or Codex. I've reviewed and verified the changes.

View guided diff

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-04T12:33:44.180084Z 1336ccf PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1336ccfa27

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/httpx2/httpx2/_sse.py Outdated
class EventSource:
def __init__(self, response: Response, max_event_size: int | None = DEFAULT_MAX_EVENT_SIZE_BYTES) -> None:
self._response = response
self._response.encoding = "utf-8-sig"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Decode SSE bytes without requiring a mutable response encoding

If a caller accesses response.text before passing a buffered response to the public EventSource constructor, this assignment now raises ValueError: the Response.encoding setter explicitly rejects changes once _text exists. This also affects client.sse() when a response hook reads and logs the response body before the source is constructed. Previously, these responses could still be parsed because iter_text() decodes the buffered bytes independently of the cached text. Apply the SSE-specific decoder to the byte iterator instead of requiring that the response encoding remain mutable.

Useful? React with 👍 / 👎.

@codspeed

codspeed Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 18 untouched benchmarks
⏩ 7 skipped benchmarks1


Comparing sse-utf8-sig (c75e90f) with main (e101f25)

Open in CodSpeed

Footnotes

  1. 7 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

Comment thread src/httpx2/httpx2/_sse.py Resolved

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found and verified against the latest diff

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="tests/httpx2/test_sse.py">

<violation number="1" location="tests/httpx2/test_sse.py:256">
P3: These decoding regressions are covered only through synchronous SSE iteration. Add async coverage for the split leading BOM and non-UTF-8 charset cases so the separate `aiter_text()` path is verified as well.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic

Comment thread tests/httpx2/test_sse.py
assert [event.data for event in events] == ["hello", "world"]


def test_leading_bom_split_across_chunks_is_ignored() -> None:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: These decoding regressions are covered only through synchronous SSE iteration. Add async coverage for the split leading BOM and non-UTF-8 charset cases so the separate aiter_text() path is verified as well.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At tests/httpx2/test_sse.py, line 256:

<comment>These decoding regressions are covered only through synchronous SSE iteration. Add async coverage for the split leading BOM and non-UTF-8 charset cases so the separate `aiter_text()` path is verified as well.</comment>

<file context>
@@ -253,6 +253,35 @@ def handler(request: httpx2.Request) -> httpx2.Response:
     assert [event.data for event in events] == ["hello", "world"]
 
 
+def test_leading_bom_split_across_chunks_is_ignored() -> None:
+    def chunks() -> Iterator[bytes]:
+        yield b"\xef"
</file context>

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 2 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="tests/httpx2/test_sse.py">

<violation number="1" location="tests/httpx2/test_sse.py:290">
P2: This test never exercises a hook consuming a live SSE stream: `sse_handler` eagerly caches `content=SSE_BODY`, making `response.read()` a no-op before the hook runs. Return a `Response(stream=...)` from this test so the hook actually buffers the response before `EventSource` iterates it.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic

Comment thread tests/httpx2/test_sse.py
response.read()
assert response.text

with httpx2.Client(transport=httpx2.MockTransport(sse_handler), event_hooks={"response": [read_text]}) as client:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This test never exercises a hook consuming a live SSE stream: sse_handler eagerly caches content=SSE_BODY, making response.read() a no-op before the hook runs. Return a Response(stream=...) from this test so the hook actually buffers the response before EventSource iterates it.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At tests/httpx2/test_sse.py, line 290:

<comment>This test never exercises a hook consuming a live SSE stream: `sse_handler` eagerly caches `content=SSE_BODY`, making `response.read()` a no-op before the hook runs. Return a `Response(stream=...)` from this test so the hook actually buffers the response before `EventSource` iterates it.</comment>

<file context>
@@ -253,6 +253,47 @@ def handler(request: httpx2.Request) -> httpx2.Response:
+        response.read()
+        assert response.text
+
+    with httpx2.Client(transport=httpx2.MockTransport(sse_handler), event_hooks={"response": [read_text]}) as client:
+        with client.sse("http://testserver/sse") as source:
+            events = list(source)
</file context>

@Kludex

Kludex commented Nov 4, 2026

Copy link
Copy Markdown
Member Author

Closing: a leading BOM or a non-UTF-8 charset on text/event-stream has not been reported by anyone (including the MCP Python SDK and httpx-sse). We will revisit if it shows up in practice.

@Kludex Kludex closed this Nov 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant