Skip to content

fix(parser): recover unencoded core font subsets - #606

Open
jmdonbaba wants to merge 4 commits into
funstory-ai:mainfrom
jmdonbaba:fix/unencoded-font-subsets
Open

fix(parser): recover unencoded core font subsets#606
jmdonbaba wants to merge 4 commits into
funstory-ai:mainfrom
jmdonbaba:fix/unencoded-font-subsets

Conversation

@jmdonbaba

@jmdonbaba jmdonbaba commented Aug 13, 2026

Copy link
Copy Markdown

Related Issue(s)

Closes #593

Motivation and Context

Some embedded Type1/CFF font subsets omit both /Encoding and /ToUnicode.
When this happens, the parser falls back to StandardEncoding and corrupts source
text in font-variant runs, such as bold, italic, or colored text, before it is
sent to the translation service.

This PR fixes the decoding issue reported in #593.

Summary of Changes

  • Recover character mappings from G<glyph-id> names in embedded subset fonts.
  • Apply this recovery only to unencoded Arial and Times New Roman subsets that
    also lack a /ToUnicode map.
  • Add regression tests for normal and custom font charmaps.

Before / After

Reproduced using LO060_EN_46C_FV_simple.pdf, a three-page minimal
reproduction extracted from the original LO060_EN_46C_FV.pdf attached in #593.

Before parsing fix After parsing fix
7KLV FRXUVH ZLOO SURYLGH \RX ZLWK: This course will provide you with:
7DUJHW *URXS Target Group
QRW intended as a VHOI-VWXG\ program not intended as a self-study program

The corrected source text is produced before any translation request is sent, so
the fix is independent of the translation model.

PR Type

  • 🐛 Bug Fix
  • 🧪 Test

Breaking Changes

No breaking changes.

Contributor Checklist

  • I have fully read and understood the CONTRIBUTING.md guide.
  • I have performed a self-review of my own code.
  • My changes follow the project's code style and guidelines.
  • I have linked the related issue(s) in the description above.
  • I have updated relevant documentation (not applicable).
  • I have added necessary tests that prove my fix is effective.
  • All new and existing tests passed locally with my changes.
  • My changes generate no new warnings or errors.
  • I understand that due to limited maintainer resources, only small PRs are accepted.

Testing Instructions

  1. Check out this branch.
  2. Run uv run pytest -q.
  3. Run uv run ruff check babeldoc/format/pdf/new_parser/active_direct_font_backend.py tests/test_active_direct_font_backend.py.
  4. Run BabelDOC with LO060_EN_46C_FV_simple.pdf, the minimal three-page reproduction extracted from the original PDF attached in PDF文件中英文字体有加粗、斜杠、颜色变成乱码或者翻译错误 #593.
  5. Verify that text such as This course will provide you with:,
    Target Group, and not intended as a self-study program is correctly
    recovered before translation.

Minimal Reproduction PDF

LO060_EN_46C_FV_simple.pdf is a three-page minimal reproduction extracted
from the original LO060_EN_46C_FV.pdf attached in #593.

LO060_EN_46C_FV_simple.pdf

Screenshots

The following screenshots were generated from LO060_EN_46C_FV_simple.pdf,
a minimal reproduction derived from the original issue attachment.

1. Course Goals

Before

ScreenShot_2026-08-13_224508_906

After

ScreenShot_2026-08-13_223913_339

2. Target Group

Before

ScreenShot_2026-08-13_224528_812

After

ScreenShot_2026-08-13_223841_807

3. Copyright

Before

ScreenShot_2026-08-13_224430_237

After

ScreenShot_2026-08-13_223757_040

Additional Notes

The recovery path is enabled only when both /Encoding and /ToUnicode are
missing, avoiding changes to normal PDF font decoding behavior.


Summary by cubic

Recovers ASCII decoding for unencoded Arial/Times New Roman Type1/CFF subsets by deriving a Unicode map from embedded G names only when /Encoding, /ToUnicode, and any embedded Type1 header encoding are all absent. Previously we fell back to StandardEncoding and produced garbled text; now those subsets decode correctly without affecting other fonts.

  • Scans the embedded font program with freetype and maps G(\d+) names (IDs 3–97 → codepoint +29) via recover_unencoded_subset_map.
  • Tries all available charmaps; selects each before scanning to find glyph names.
  • Honors authoritative encodings: parses the Type1 header in FontFile and uses its encoding if present, skipping recovery.
  • Recovery is gated to Arial and Times New Roman subset names; other fonts keep existing behavior and are not loaded for recovery.
  • Reads FontFile, FontFile2, or FontFile3; swallows read errors to keep fallback paths intact.
  • Adds tests for recovery, charmap selection, gating by /Encoding and /ToUnicode, precedence of embedded encodings, and a non-core font recovery guard.

Written for commit b7c939f. Summary will update on new commits.

Review in cubic

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 2 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread babeldoc/format/pdf/new_parser/active_direct_font_backend.py Outdated
Comment thread tests/test_active_direct_font_backend.py
Comment thread babeldoc/format/pdf/new_parser/active_direct_font_backend.py
Comment thread babeldoc/format/pdf/new_parser/active_direct_font_backend.py
jmdonbaba and others added 2 commits August 13, 2026 23:28
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 2 files (changes from recent commits).

Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic

Comment thread tests/test_active_direct_font_backend.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PDF文件中英文字体有加粗、斜杠、颜色变成乱码或者翻译错误

1 participant