Uzbek cyrillic being thrown away

[ZJaume]

Noticed that in most of HPLT documents that CLD2 says it is Uzbek and are written in cyrillic, fasttext is saying that sentences are other cyrillic langs like ru, kk, tt, ug, az. The list of possible cases is large so I this language may need a special mode where we simply check cyr and lat Uzbek dictionaries and if error is less than 30%, we keep it as uz.

There is one dictionary for both scripts here: https://github.com/u2b3k/uz-hunspell

[mbanon]

@ZJaume Hello one year after opening the issue xD Do we have any samples of cyrillic Uzbek? Current HPLT v2 is only uzbek latin (uzn_Latn)

[ZJaume]

For example

zstdcat uz_1.jsonl.zst | grep '"tt"

from v1.2 will give many of those.

Not really sure about CLD2 being correct in this, but according to Wikipedia, despite Latin being official since '92, Cyrillic is still widespread.