Skip to content

fix: geminate sokuon across token boundaries in normal-mode romaji conversion - #122

Closed
tahodev wants to merge 1 commit into
hexenq:masterfrom
tahodev:fix/romaji-sokuon-across-tokens
Closed

tahodev wants to merge 1 commit into
hexenq:masterfrom
tahodev:fix/romaji-sokuon-across-tokens

Conversation

@tahodev

@tahodev tahodev commented Sep 17, 2026

Copy link
Copy Markdown

Fixes takuyaa/kuromoji.js#53

Problem

Reported against kuromoji.js: kuroshiro.convert('座って', { to: 'romaji', mode: 'normal' }) produces suwatsute instead of suwatte.

I verified the tokenization side is correct: kuromoji (IPADIC) returns 座っ (reading スワッ, 連用タ接続) + て (テ), which is the proper analysis. The romaji issue is on the kuroshiro side: in normal mode each token's pronunciation is converted independently and joined, so the trailing sokuon ッ of スワッ is romanized standalone (suwatsu) instead of geminating the next token's initial consonant (suwatte).

Fix

In normal mode, join the per-token pre-conversion strings (pronunciation || reading for Japanese tokens, surface_form otherwise) first, then run toRawRomaji once on the joined string. toRawRomaji already implements correct sokuon handling (verified: toRawRomaji('スワッテ') -> suwatte). Spaced mode keeps per-token conversion, since the space-separated design is intentional there.

Tests

  • Added a regression test: 座って (normal mode, hepburn) -> suwatte
  • End-to-end with KuromojiAnalyzer: 座って -> suwatte, 行って -> itte; sanity checks (つじぎり, 東京タワー) unchanged
  • Existing within-token sokuon test (勝手に買っちゃったんだ) still passes
  • Full suite: 50 passed, 0 failed; eslint clean

hexenq commented Sep 17, 2026

Copy link
Copy Markdown
Owner

Thank you for investigating this and submitting the patch. I compared it with #117 against the current master using the real Kuromoji analyzer.

For the reported 座って normal-mode case, master already returns suwatte: patchTokens merges the verb-tail sokuon before romanization. The new regression test therefore also passes without this change. The remaining reproduced problem was in furigana rendering; #117's approach, with additional boundary and contracted-kana handling, has now been merged via #125.

I am not adopting whole-string romanization here because it also changes Japanese/Latin boundaries. For example, in Hepburn normal mode, あっcat changes from atsucat to atcat, and んbase from nbase to mbase.

Closing this PR with thanks. This is not a claim that every normal-mode cross-token case is fixed: if you have another example that fails on current master, please open a focused issue or PR with the analyzer/version and failing test. A boundary-aware fix would be welcome.

@hexenq hexenq closed this Sep 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

座って -> "suwatsute", should be "suwatte"

2 participants