[PR] Strip C0 control characters from LLM JSON output

[BetterAndBetterII]

PR Title

[PR] Strip C0 control characters from LLM JSON output

Related Issue(s)

Closes #577

Motivation and Context

PDF text can put C0 control characters into the LLM JSON. _clean_json_output only stripped wrappers, so json.loads failed and the fallback path left residual source fragments in the translation.

Summary of Changes

Sanitize C0 controls in _clean_json_output while keeping JSON-legal whitespace (\n, \t, \r). Wrapper stripping is unchanged.

PR Type

• 🐛 Bug Fix
• 🧪 Test

Breaking Changes

• No, this PR does not introduce breaking changes.

Contributor Checklist

• I have fully read and understood the CONTRIBUTING.md guide.
• I have performed a self-review of my own code.
• My changes follow the project's code style and guidelines
• I have linked the related issue(s) in the description above (if applicable)
• I have added necessary tests that prove my fix is effective or that my feature works (if applicable)
• All new and existing tests passed locally with my changes
• My changes generate no new warnings or errors
• I understand that due to limited maintainer resources, only small PRs are accepted. Suggestions with proof-of-concept patches are appreciated, and my patch may be rewritten if necessary.

Testing Instructions

1. Check out this branch.
2. Run python -m pytest tests/test_clean_json_output.py -v.
3. Confirm both cases pass: wrappers still strip, and JSON with embedded C0 controls parses after clean.

---

Summary by cubic

Strips non-whitespace C0 control characters from LLM JSON output so json.loads succeeds and translations don’t fall back and leak source fragments. Previously, _clean_json_output only removed wrappers; JSON containing C0 controls failed to parse.

Review notes

• Change: In ILTranslatorLLMOnly._clean_json_output, after wrapper trimming and strip(), drop chars with ord(char) < 32 except \n, \t, \r. Wrapper stripping is unchanged.
• Tests: Added tests/test_clean_json_output.py to verify C0 controls are removed while JSON-legal whitespace and wrapper handling are preserved.
• Impact: Non-breaking; only removes JSON-illegal control characters. No migration required.

^{Written for commit 2ff852f. Summary will update on new commits.}

[cubic-dev-ai]

1 issue found across 2 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="babeldoc/format/pdf/document_il/midend/il_translator_llm_only.py">

<violation number="1" location="babeldoc/format/pdf/document_il/midend/il_translator_llm_only.py:1000">
P2: The filter preserves literal `\n`, `\t`, `\r` everywhere, including inside JSON string values. Those bytes are invalid inside strings, so when a translated string contains one, `json.loads` still raises exactly the failure this PR targets. Use a context-aware clean (e.g. only strip control chars outside string literals, or escape them inside strings) so in-string whitespace is handled as well as structural whitespace.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: The filter preserves literal \n, \t, \r everywhere, including inside JSON string values. Those bytes are invalid inside strings, so when a translated string contains one, json.loads still raises exactly the failure this PR targets. Use a context-aware clean (e.g. only strip control chars outside string literals, or escape them inside strings) so in-string whitespace is handled as well as structural whitespace.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At babeldoc/format/pdf/document_il/midend/il_translator_llm_only.py, line 1000:

<comment>The filter preserves literal `\n`, `\t`, `\r` everywhere, including inside JSON string values. Those bytes are invalid inside strings, so when a translated string contains one, `json.loads` still raises exactly the failure this PR targets. Use a context-aware clean (e.g. only strip control chars outside string literals, or escape them inside strings) so in-string whitespace is handled as well as structural whitespace.</comment>

<file context>
@@ -995,4 +995,8 @@ def _clean_json_output(self, llm_output: str) -> str:
-        return llm_output.strip()
+        llm_output = llm_output.strip()
+        # JSON allows \n \t \r as whitespace; other C0 controls make json.loads fail.
+        return "".join(
+            char for char in llm_output if ord(char) >= 32 or char in "\n\t\r"
+        )
</file context>