How Ultimate Built the Most Accurate Language Detection Model for Chatbots

Over the past five years, Ultimate has grown from a monolingual Finnish customer-support solution into a multilingual AI provider. Its language-agnostic model can predict intent in more than 100 languages, regardless of the language of the training data. Yet understanding a language is only half the job: a chatbot must also respond in the visitor's chosen language. This requirement made accurate language detection essential, and because existing market tools did not fit the chat domain, Ultimate set out to build its own Language Detection Model that could become one of the most accurate in the field.

Language detection is the automatic identification of the language present in a document based on its content. It is typically treated as a special case of text categorization and solved with statistical methods. For well-written, long texts, detection is correct over 99% of the time, so it is not among the hardest NLP tasks. Chat is different. Conversational bots mostly handle short interactions, often just a word or a few words. Users also mix languages within a single message, and closely related pairs such as Spanish and Catalan or Serbian and Croatian are difficult to separate. Good performance on short text is therefore the decisive factor.

Four requirements guided the selection and design. First, the model had to cover all 109 languages the product supports. Second, it had to be highly accurate even when a message contained only one or two words. Third, it had to be small, keeping costs low. Fourth, it had to run in real time so that language detection would never become a bottleneck in the chat flow.

Most research on language detection was conducted on long, clean sentences and paragraphs, where results look impressive. The explosion of text messages and social media has changed the picture: slang, misspellings, and acronyms degrade performance sharply. Ultimate reviewed a range of tools, including Lingua, NLLB (a fastText model), Franc, the Google API, Langdetect, Langid, pycld2, gcld3, and fastText's lid.176 variants, comparing cost, hosting, language coverage, general and short-text accuracy, and speed. None could handle short texts quickly, accurately, and across many languages at once, so the team concluded that a ready-to-use solution did not exist and decided to build one in-house.

Before designing the model, the team identified why general tools fail on chat messages. The first problem is length: short strings often belong to several languages. The word 'api' is Indonesian for 'fire' and an English abbreviation for 'application programming interface.' Chat users also write queries such as 'Login issue' instead of full sentences, and many messages are greetings like 'Hi,' 'Ciao,' or 'Hola,' which appear across languages and can mislead a detector. The second problem is domain mismatch. Most tools were trained on Wikipedia, which features complete sentences and general vocabulary, while customer-support chat uses specialized terms such as 'reset password' or 'delivery issues.' The third problem is noise: typos, abbreviations, missing punctuation, and filler words such as 'um' and 'uh' confuse models, as in 'cn i get ma ordr statos.' The fourth problem is bias. Off-the-shelf models, including the pretrained fastText model, favor English and other European languages because they are overrepresented in training data; fastText, for instance, labels '9999999' as English with 0.357 confidence. The fifth problem is calibration. A well-calibrated model's confidence should match its accuracy, so that predictions made with 0.8 confidence are correct about 80% of the time. Many models are overconfident and wrong with high certainty, which makes them untrustworthy.

The team chose fastText as the classifier: it is open source, fast to train and run, simple in architecture, and has already performed well in language identification benchmarks. The harder question was how to feed text into the model. Splitting text into words creates an unmanageably large vocabulary across dozens of languages and fails on inflection-heavy languages such as Finnish and Hungarian, where a single noun can take many forms, and on compound words, abbreviations, and scripts without spaces such as Chinese, Japanese, and Korean. Splitting into characters avoids unknown words but produces many more tokens and makes closely related languages with similar character sets hard to tell apart. The solution was a subword-level approach using Byte-Pair Encoding (BPE). BPE starts from individual characters and repeatedly merges the most frequent adjacent pairs, learning units without any language-specific knowledge. It handles out-of-vocabulary words by breaking them into known subwords, produces shorter sequences than character tokenization, and avoids 'UNK' tokens. Because a standard fastText model treats tokens as an unordered bag and ignores word order, the team also added n-gram representations of the tokens. Since the number of unique n-grams is huge, they applied hashing into a fixed number of buckets (around two million), bounding memory requirements.

The most effective remedy for domain mismatch is a training dataset that closely fits the target domain, though in-domain data is often scarce. Ultimate combined two sources. The primary source was OpenSubtitles, a corpus of movie and TV subtitles in many languages, whose natural conversation and short sentences and greetings made it a strong starting point. To close the gap with customer-service language, the team added customer-support data. The final dataset contained around 145 million samples with balanced languages. This careful design improved overall performance, especially on short messages; reduced the domain mismatch with chat; removed languages irrelevant to the business, such as Latin and Esperanto; and eliminated the heavy English bias of the off-the-shelf model.

Noise is the easiest problem to attack. The team handled it in two steps. First, text cleaning removed line breaks, tabs, URLs, digits, punctuation, and emojis. Second, to make the model resilient to misspellings, they applied dropout during tokenization: each training sample was tokenized twice with different random subword splits, so that 'this is a sentence' might appear both as one segmentation and as another that splits 'sentence' differently. Greetings required a dedicated solution despite these measures. Ultimate built a greeting dictionary covering 3,500 greetings across all required languages. Each key is a greeting and each value is the list of possible languages ordered by probability, so that 'ciao' maps to Italian, German, and others in ranked order.

A language model must be trustworthy in production, which requires good calibration. The team measured that the fastText model was typically overconfident, meaning it predicted wrong labels with high confidence because it had overfit the data. Two ideas addressed this. The first applied dropout to highly frequent tokens during training to combat overfitting. The second was the tokenization dropout described above. Together these methods made the model far more reliable, bringing its calibration error to roughly 1%.

The model was benchmarked against Langdetect, pycld2, gcld3, fastText (large and small), Franc, NLLB, and Lingua. Two datasets were used. The first was an in-house chat dataset of 2,800 samples in English, Spanish, French, Finnish, and German, mostly single words, word pairs, and very short sentences. The second was the Lingua test set, a standardized generic-domain benchmark covering 75 languages with one-word, two-word, and short sentences. Lingua was run in its LowAccuracyMode for speed, and accuracies for 45 languages were taken from its reports for a fair comparison. On the in-house chat dataset, Ultimate's model achieved the best performance, meeting its goal of high accuracy in its own domain. On the Lingua test set, it was also the most accurate, showing that its strength extends to generic text. Speed was measured alongside accuracy because production systems cannot tolerate bottlenecks. The model predicts about 28,359 samples per second, behind only pycld2 (about 418,809), fastText base models (about 92,532 and 165,477), and ahead of Franc, gcld3, and NLLB. Langdetect and Lingua were the slowest.

Ultimate's journey shows that an older model such as fastText can achieve excellent results when paired with careful training data design, thoughtful text processing, and targeted techniques. The in-house model is the most accurate language detector in both the chat-specific and generic domains, and it is very fast, making language detection nearly solved for long messages and state-of-the-art for short ones. No model is perfect, and there will always be uncertain cases. In those situations, Ultimate's chat platform guides the user toward the best way to connect with the chatbot, preserving the continuity of the conversation.