The challenges of AI models in non-English languages
Even major languages like Cantonese lag behind in AI model performance.
1Data scarcity for language models
AI models require massive amounts of text data for training. With approximately half of online content in English, models for other languages face data shortages. Even Cantonese, spoken by 85 million people, is considered 'low-resource'. Jacky Chan, founder of Votee, highlights how errors in digitized texts create major challenges when developing Cantonese models.
2Cultural context and translation hurdles
AI models struggle with culturally specific terms like 'dim sum' that don't have direct translations. Aliya Bhatia from the Center for Democracy Technology explains that models invent translations for unfamiliar words, producing outputs native speakers wouldn't use. This issue is particularly pronounced in non-Latin script languages.
3Local solutions and 'sovereign AI'
South Korea's Naver developed HyperCLOVA X, trained on 6,500 times more Korean data than GPT-4. In Indonesia, Indosat and Goto are working on a 70-billion parameter model for local languages. This 'sovereign AI' approach focuses on needs big tech can't address, though parameter counts still lag behind English models.
4Scale and parameter challenges
OpenAI's GPT-4 has about 1.8 trillion parameters, while SEA-LION's Southeast Asian language model has just 7 billion. This massive gap directly impacts performance. Additional challenges arise in languages like Cantonese with differences between formal and informal writing styles.
5Data quality and type issues
Low-resource languages often rely on formal sources like legal documents or Wikipedia, resulting in models that sound overly formal. Georgia Tech's study of Arabic models found Western cultural references (like alcoholic beverages in Islamic contexts) being incorrectly incorporated.
6Risks of machine translation
Translating English content is a common workaround for data scarcity, but loses linguistic nuance. Solutions like Votee's synthetic data generation attempt to bridge gaps, but machine translations often fail to capture cultural context, leading to flawed learning.
7Additional challenges in tonal languages
Tonal languages like Vietnamese present unique difficulties as written forms don't fully convey pronunciation. A model trained solely on pop music data couldn't accurately interpret historical events. Similarly, Cantonese's spoken-written divide affects model consistency.
8Pursuing solutions for linguistic diversity
Most of the world's population speaks non-English languages. For truly global AI, multilingual model development is essential. While local efforts in South Korea and Indonesia show promise, resource disparities persist. International cooperation is needed to preserve linguistic diversity and realize AI's potential across all languages.
Original article: Fortune — İngilizce dışı dillerde AI modellerinin zorlukları