[wittjeff]

Follow-up to #337, standalone per #428.

Problem

A name object is a byte string (ISO 32000-1, 7.3.5). Producers that localise their custom structure types write them in PDFDocEncoding, for example /Absatz#fc for "Absatzü" or /Fu#dfzeile for "Fußzeile", which is not valid UTF-8. The structure reader stored such names as they are, in the role map, the element type, the attribute owner and keys, the /OBJR subtype, and the artifact type and subtype of marked content. The JSON caster raises UnicodeDecodeError at the Python boundary, so get_annotations() fails for the whole document and get_page_marked_content() for the page:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 12: unexpected end of data

Change

Every such name goes through name_to_utf8(), which #392 added for widget names: valid UTF-8 is kept, anything else is transcoded from PDFDocEncoding, the way getUTF8Value() transcodes text strings. The function gains a std::string overload for dictionary keys, which are names too (7.3.7). to_json() uses it for its keys, which were not sanitised at all, and for its name values, which used to be rewritten with U+FFFD.

Evidence

Failure on main. tests/test_unit_structure_tree.py::test_names_in_pdfdoc_encoding_are_transcoded builds the smallest document with the shape: a role map entry, an element type, an attribute owner and key, and an artifact type and subtype, all in PDFDocEncoding. On main (6dbd6a8) get_annotations() raises the error above.

Regression dataset. I scanned every PDF of the dataset at the pinned revision (456 files) plus 17 public arXiv papers with qpdf --json and qpdf --qdf: no document has a structure type, role-map entry, attribute owner or key, dictionary key, or artifact type that is not UTF-8, so no document in the dataset reproduces the failure and no ground-truth page changes. Non-UTF-8 names are not rare in those files, though: 12 documents carry font names in Shift-JIS, GBK or Big5 bytes (/ABCDEE+#82l#82r#20#82o#83S#83V#83b#83N is "ＭＳ Ｐゴシック"). Those names do not flow through to_json(), which is why they do not appear in the ground truth.

Real documents. Four files of pdfium's public test corpus (pdfium_tests, BSD-3) name a font /#cb#ce#cc#e5 in the form's /DR dictionary: the bytes are GBK for 宋体. That key goes through to_json() as part of the /AcroForm export, so on main get_annotations() fails for the whole document:

file	main (6dbd6a8)	this PR
fx/form/combo_box_format.pdf	UnicodeDecodeError: 'utf-8' codec can't decode byte 0xcb in position 1	form loads; /DR fonts /Helv, /ËÎÌå
fx/form/format_text_color.pdf	same	form loads
fx/js/submit_form.pdf	same	form loads; /DR fonts /Helv, /Helve, /ËÎÌå, /ËÎÌå4
fx/FRC_8.5_part2/FRC_8.5_E_GoTo_D.pdf	same	form loads; seven /ËÎÌå… fonts, all distinct

The transcoded key reads as Latin letters, not as 宋体: the export cannot know the code page of a name. What the transcoding guarantees is that the document loads and that distinct keys stay distinct. Replacing the bytes with U+FFFD, as the name-value branch used to do, would map /#cb#ce and /#cc#e5 to the same JSON key, and one font entry would overwrite the other.

I also scanned the 5,366 PDFs on my machine outside caches (docling, docling-core and docling-parse test data, the pdfium and qpdf test corpora, my own downloads). Beyond the four files above, only three deliberately malformed qpdf test files (bad-content.pdf, issue-1503.pdf, weird-tokens.pdf) carry such keys; their keys are not under /AcroForm, so they load on main too. No file carries a non-UTF-8 structure type, role-map entry, attribute owner or key, or artifact type, so that part of the fix rests on the synthetic test and on the fact that the same producers that write GBK font names also write custom structure types.

Why PDFDocEncoding. It is the fallback getUTF8Value() applies to text strings, and the one #392 chose for widget names. For a Latin-script producer it recovers the intended characters; for a name in a CJK code page it yields readable but wrong characters where U+FFFD stood before. Both are lossy; the transcoding keeps the bytes recoverable. If you would rather keep U+FFFD for name values in to_json(), that is one line to drop; the structure, artifact and key sites are the ones that failed.

Tests

tests/test_unit_acroform_widgets.py::test_dictionary_keys_that_are_not_utf8_keep_the_form: a form whose /DR font dictionary has two keys in GBK bytes, as in the pdfium files. On main the annotations call raises; with this change the form loads and the two keys stay distinct.

tests/test_unit_structure_tree.py::test_names_in_pdfdoc_encoding_are_transcoded: a tagged page whose role map, element type, attribute owner and key, and artifact type and subtype are PDFDocEncoding bytes. On main the annotations call raises; with this change the role map is {"/Absatzü": "/P"}, the attributes are {"/Layouté": {"/Platzierungé": "/Block"}} and the artifact is ("/Paginationé", "/Fußzeile").

Ground truth

No page of the regression sample changes: the only non-UTF-8 names in the dataset are font names, which do not pass through the changed code. tests/test_regression_parse.py::test_reference_documents_from_filenames passes against the pinned ground truth on this branch (1 passed, 42 min), and the unit lane passes (325 passed, 1 skipped).

🤖 Generated with Claude Code

[github-actions]

✅ DCO Check Passed

Thanks @wittjeff, all your commits are properly signed off. 🎉

[mergify]

Merge Protections

🟢 Merge protection satisfied — ready to merge.

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

• title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

[PeterStaar-IBM]

lgtm!

[PeterStaar-IBM]

lgtm!

[PeterStaar-IBM]

closed and reopened the PR to re-trigger the CI.