fix(romaji): geminate a lone sokuon instead of emitting "tsu" - #117
Conversation
Reported upstream as takuyaa/kuromoji.js#53 ("座って -> suwatsute"), filed against the tokenizer. It is not a tokenizer bug — kuromoji returns 座っ[スワッ] + て[テ], which is correct. The error is here. The furigana renderer romanises each notation independently. A lone sokuon then has no following consonant to geminate, misses in the romaji table, and falls through to the literal "tsu": 座って -> 座[suwa] っ[tsu] て[te] 真っ赤 -> 真[ma] っ[tsu] 赤[ka] カッター -> カ[ka] ッ[tsu] タ[ta] mergeSokuonForward attaches a lone sokuon to the notation that follows, so the existing gemination rule in toRawRomaji has the consonant it needs: 座って -> 座[suwa] って[tte] 真っ赤 -> 真[ma] っ赤[kka] Applied only to the romaji paths. Hiragana and katakana output already emits the sokuon as its own literal character and is unaffected — pinned by test. Note the issue as filed does not reproduce: `normal` mode romanises the whole token at once and has always produced "suwatte". It is `furigana` mode that breaks, which is the mode that matters for ruby output. A trailing sokuon (「あっ」) has nothing to geminate and no agreed romanisation, so it is deliberately left as-is and pinned by a test, to make the limitation explicit rather than forgotten. 58 tests passing, up from 49; the four new gemination assertions fail against the unfixed renderer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Thanks for the fix and the regression tests! I reproduced the furigana issue and am adopting this approach in #125. Since maintainer edits are disabled on the fork, that follow-up retains your original commit and adds the maintenance work separately. The additions prevent merging into Latin text, punctuation, spaces or vowels, and keep contracted kana together (e.g. 買っちゃった / いっしょ). Normal/spaced modes and hiragana/katakana output stay unchanged. The full local suite now passes 100 tests, including package/UMD checks. Once CI passes, I will merge #125 without squashing to preserve your contribution history. |
|
Merged via #125 after Node 22 and 24 CI passed. Your original commit is retained in master, and GitHub has marked this PR as merged. Thanks again for the contribution! This is merged source code only; no new npm version has been published yet. |
2 participants
Reported upstream as takuyaa/kuromoji.js#53 ("座って -> suwatsute"), filed against the tokenizer. It is not a tokenizer bug — kuromoji returns 座っ[スワッ] + て[テ], which is correct. The error is here.
The furigana renderer romanises each notation independently. A lone sokuon then has no following consonant to geminate, misses in the romaji table, and falls through to the literal "tsu":
座って -> 座[suwa] っ[tsu] て[te]
真っ赤 -> 真[ma] っ[tsu] 赤[ka]
カッター -> カ[ka] ッ[tsu] タ[ta]
mergeSokuonForward attaches a lone sokuon to the notation that follows, so the existing gemination rule in toRawRomaji has the consonant it needs:
座って -> 座[suwa] って[tte]
真っ赤 -> 真[ma] っ赤[kka]
Applied only to the romaji paths. Hiragana and katakana output already emits the sokuon as its own literal character and is unaffected — pinned by test.
Note the issue as filed does not reproduce:
normalmode romanises the whole token at once and has always produced "suwatte". It isfuriganamode that breaks, which is the mode that matters for ruby output.A trailing sokuon (「あっ」) has nothing to geminate and no agreed romanisation, so it is deliberately left as-is and pinned by a test, to make the limitation explicit rather than forgotten.
58 tests passing, up from 49; the four new gemination assertions fail against the unfixed renderer.