bug: Romanize company name, found repeated strings

[siripods]

Description

Found strange output from feature Romanization

engine: thai2rom_onnx

---

Sample #1
Input: เอ็มเอฟซีบัญชีเพื่อการชำระค่ารับซื้อคืน
Output (Romanized) : em-epsibanchiphueakanchiphueakanchiphueakanchiphueakanchiphueakanchiphueakanchiphueakanchiphueakanch

Notice: string "banchiphueakan" are repeated many times

---

Sample #2
Input: บัญชีเพื่อการชำระค่าขายคืนหน่วยลงทุน
Output (Romanized) : banchiphueakanchamrakhakhaikhaikhaikhaikhaikhaikhaikhaikhaikhaikhaikhaikhaikhaikhaikhaikhaikhaikhai
k

Notice: string "khai" are repeated many times

Expected results

There should not be any repeated strings in romanized output.

Current results

Found strange output from feature Romanization

engine: thai2rom_onnx

---

Sample #1
Input: เอ็มเอฟซีบัญชีเพื่อการชำระค่ารับซื้อคืน
Output (Romanized) : em-epsibanchiphueakanchiphueakanchiphueakanchiphueakanchiphueakanchiphueakanchiphueakanchiphueakanch

Notice: string "banchiphueakan" are repeated many times

---

Sample #2
Input: บัญชีเพื่อการชำระค่าขายคืนหน่วยลงทุน
Output (Romanized) : banchiphueakanchamrakhakhaikhaikhaikhaikhaikhaikhaikhaikhaikhaikhaikhaikhaikhaikhaikhaikhaikhaikhai
k

Notice: string "khai" are repeated many times

Steps to reproduce

engine: thai2rom_onnx

Sample #1
Input: เอ็มเอฟซีบัญชีเพื่อการชำระค่ารับซื้อคืน

Sample #2
Input: บัญชีเพื่อการชำระค่าขายคืนหน่วยลงทุน

PyThaiNLP version

5.3

Python version

3.12

Operating system and version

Windows

More info

No response

Possible solution

No response

Files

No response

[wannaphong]

From https://pythainlp.org/docs/5.3.4/api/transliterate.html#modules, the pythainlp.transliterate.romanize support a Thai word only. You should do preprocessing before using the function.

Example:

from pythainlp.transliterate import romanize
from pythainlp.tokenize import word_tokenize

def make_romanize(text):
    rom=""
    for word in word_tokenize(text):
        rom+=romanize(word, engine="thai2rom_onnx")
    return rom

print(make_romanize("เอ็มเอฟซีบัญชีเพื่อการชำระค่ารับซื้อคืน"))
print(make_romanize("บัญชีเพื่อการชำระค่าขายคืนหน่วยลงทุน"))

output:

em-epsibanchiphueakanchamrakharapsuekhuen
banchiphueakanchamrakhakhaikhuennuailongthun

[siripods]

Can you explain about word tokenize ?
I notice that when using word tokenize, the romanization output looks different than without word tokenize.

Without tokenize:
ศิริพจน์ สุรบถโสภณ -> siriphot suropthosophon

With tokenize
ศิริพจน์ สุรบถโสภณ -> siriphot surabotsophon

[wannaphong]

Can you explain about word tokenize ? I notice that when using word tokenize, the romanization output looks different than without word tokenize.

Without tokenize: ศิริพจน์ สุรบถโสภณ -> siriphot suropthosophon

With tokenize ศิริพจน์ สุรบถโสภณ -> siriphot surabotsophon

The romanization model was trained from word-level dataset.