Lyrics language classification (Mandarin or Cantonese)

[F1shYi]

Hi there! Thanks for your wonderful work. It is indeed an efficient way to determine whether certain text is written in Mandarin or Cantonese.

Recently I'm dealing with a bunch of crawled lyrics, most of which are mandarin songs but there do exist a few cantonese songs. I would like to try some fast and efficient ways to distinguish them, and I'm wondering if this algorithm has been trained on corpus including lyrics and how well it performs on this lyrics language classification task?

Thanks for your time!

[F1shYi]

I noticed that in README, most samples categorized as Cantonese are written in '白话', yet most Cantonese lyrics are written in a much more formal fashion. Does this difference matter a lot?

[laubonghaudoi]

Hi, thanks for posting this question. For non-native Cantonese speakers this is indeed a confusing question. To put it simply: Most of the existing Canto-pop songs in Hong Kong are indeed SWC (Mandarin) lyrics. There are only very limited number of Cantonese songs whose lyrics that are truely Written Vernacular Cantonese. This is caused by complex historical and social-political factors, and it will take a thousand-word essay to explain it clearly to a non-Cantonese person. If you are interested I can explain it to you in a video chat but for now you can simply trust the output labels of this classifier. As for your question Does this difference matter a lot? yes it does matter a lot, because the classification will become Mandarin rather than Cantonese.

And to answer your question: this classifier is deterministic and rule-based. It is not ML-based so it doesn't use any training data. So your question of if this algorithm has been trained on corpus including lyrics does not apply here. This classifier detects the syntatic and lexical features of the input sentence using hard-coded rules to judge the output label, which is the reason why it achieves such a high performance and accuracy simultaneously.

[F1shYi]

Thanks for your prompt response! I kind of get it. So to take an example: for Gareth T's 《劲浪漫超温馨》and 《紧急联络人》, though both sung in Cantonese, the former will be classified as Cantonese while the latter as Mandarin, due to their respective certain linguistic features, and this outcome is exactly what is expected by performing this rule-based algorithm. Is this understanding correct? :-)

[laubonghaudoi]

Yes, the lyrics of 劲浪漫超温馨 is Cantonese and 紧急联络人 is SWC.