TL;DR: The repository introduces GlotLID, an open-source language identification (LID) model with support for more than 2000 labels.
Stable: GlotLID v4 (model.bin, the same file as model_v4.bin) supports 2060 labels (1,900 ISO 639-3 language codes with script, with the zxx and und series). GlotLID v4 small (model_v4_small.bin) has the same labels and is about 4 times faster. For the supported languages, performance and the changes from v3, see languages-v4.md.
Previous: GlotLID v3 (model_v3.bin) supports 2102 labels; see languages-v3.md.
- Language Identification
- Get Sentence Vectors in respect to many languages
- Limit the language identification model to a smaller set of languages: The SET! evaluation part in the original paper.
- Code-switching: find all the languages of a mixed-language text, and the words of each, with MaskLID (try it in the Code-Switching tab of the demo).
# pip install fasttext-turbo huggingface_hub
import fasttext
from huggingface_hub import hf_hub_download
# download model and get the model path
# cache_dir is the path to the folder where the downloaded model will be stored/cached.
model_path = hf_hub_download(repo_id="cis-lmu/glotlid", filename="model.bin", cache_dir=None)
print("model path:", model_path)
# load the model
model = fasttext.load_model(model_path)We recommend fastText-turbo (pip install fasttext-turbo): it is a drop-in replacement for fastText (same import fasttext, same model files and API, prebuilt wheels) that predicts with GlotLID about 3 times faster than the original fastText in our tests (2.7x with v4, 3.4-4.3x with v3), with the same predictions. The original fasttext package (or fasttext-numpy2-wheel) works too.
If you do not want to always get the latest model (model.bin) and prefer to fix the version you are using, use model_v?.bin for example model_v4.bin (? is the version: we have 1, 2, 3, 4), or model_v4_small.bin for the faster small model. See versions for more.
Now you can use it for different applications:
"""Language Identification"""
# predict language label (call this function as many times as needed)
model.predict("Hello, world!")
# (('__label__eng_Latn',), array([0.99850202]))If you need top k prediction then, for example k = 3:
model.predict("Hello, world!", 3)
# (('__label__eng_Latn', '__label__sun_Latn', '__label__ind_Latn'), array([9.98502016e-01, 7.95848144e-04, 3.11827025e-04]))If you want to see which labels (language_scripts) are supported by the model, check the list of them by:
model.labels
# ['__label__eng_Latn', '__label__rus_Cyrl', '__label__arb_Arab', '__label__por_Latn', ...]"""Sentence Vectors"""
# get sentence vector of input sentence (call this function as many times as needed)
embedding = model.get_sentence_vector(sent)You do not need to train another model for this. Just limit the prediction to the set you want. You can do it in two ways: after the prediction over all available classes, limit to those you want, or before the prediction, first limit to those you want, and then predict. If you want your probability to also be normalized to the set you specify, choose the before mode (default). However, if you want the probability you have to be in respect to all languages and then become limited to the set you have chosen, use the after mode (a text in none of your languages then gets a low probability for all of them).
The CustomLID class is in customlid.py:
from huggingface_hub import hf_hub_download
from customlid import CustomLID
# download model and get the model path
# cache_dir is the path to the folder where the downloaded model will be stored/cached.
model_path = hf_hub_download(repo_id="cis-lmu/glotlid", filename="model.bin", cache_dir=None)
# labels with or without "__label__"; check the supported labels in model.labels
limited_languages = ['eng_Latn', 'arb_Arab', 'rus_Cyrl', 'por_Latn', 'pol_Latn', 'hin_Deva']
model = CustomLID(model_path, languages=limited_languages, mode='before')
model.predict("Hello, world!", 3)
# (('__label__eng_Latn', '__label__pol_Latn', '__label__hin_Deva'), array([9.9995399e-01, 4.2381755e-05, 3.2946982e-06], dtype=float32))
# many texts at once (faster than one by one): one result per text
model.predict(["Hello, world!", "Привет, мир!"], 3)
# [(('__label__eng_Latn', ...), array([...])), (('__label__rus_Cyrl', ...), array([...]))]model_pathcan also be a model you already loaded withfasttext.load_model, so severalCustomLIDs can share one model in memory.predict(text, k=1, threshold=0.0)works like fastText'spredict:k=-1returns all the chosen labels, andthresholddrops labels with a lower probability. Line breaks in the text count as spaces.- Labels that the model does not have are skipped with a warning. The labels keep the model's order, whatever the order of
languages.
We always maintain the previous version of GlotLID in our huggingface repository.
To access a specific version, simply append the version number to the filename.
For v1: model_v1.bin (introduced in the GlotLID paper and used in all experiments).
For v2: model_v2.bin (an edited version of v1, featuring more languages, and cleaned from noisy corpora based on the analysis of v1).
- It supports 1802 three-letter iso codes (1847 three letter iso codes with script)
- For 1626 three-letter iso codes; v2 on the test set achieved F1 of 0.996 and FPR of 0.0002.
- These 1626 languages are selected based on the 0.5 F1 threshold and 0.0005 FPR threshold for low resource languages.
For v3: model_v3.bin (an edited version of v2, featuring more languages, excluding macro languages, further cleaned from noisy corpora and incorrect metadata labels based on the analysis of v2, supporting "zxx" and "und" series labels).
- It supports 1880 three-letter ISO codes (2114 three-letter ISO codes with script).
- For more details on the supported languages and performance, significant changes from previous versions, refer to languages-v3.md.
For v4: model_v4.bin (an edited version of v3: labels kept only where they work on independent benchmarks, so some v3 labels were merged or removed, further cleaned corpora, and trained on random short spans and normalized lines, which makes it markedly better on short text and calibrated).
- It supports 2060 labels (1,900 three-letter ISO codes, with the
zxxandundseries).model_v4.ftzis its quantized version (101 MB; the same top label for 99.8% of lines). - Trained with fastText-turbo; it loads in fastText and fastText-turbo like the previous versions. It is smaller (96-dimensional, 783 MB) and uses character n-grams of 2 to 5 with word n-grams of 1, so nothing is shared between neighbouring words: the scores of a text add up from the scores of its words, which makes the per-word scores that MaskLID uses for code-switching exact.
- For more details on the supported languages and performance, and the changes from v3, refer to languages-v4.md.
For v4 small: model_v4_small.bin (the same 2060 labels in a 16-dimensional model with hierarchical softmax, 131 MB). It predicts about 4 times faster than v4 on one CPU thread, and is less accurate than v4, mostly between closely related languages. model_v4_small.ftz is its quantized version (35 MB).
model.bin always refers to the latest stable version (v4 now).
See list of data sources here. We made a version of the dataset available for research purposes, which can be requested for access: https://huggingface.co/datasets/cis-lmu/glotlid-corpus
You're welcome to open a pull request or (issue) and contribute new resources to our data list. Even for the languages we already support, we're actively seeking additional resources to mitigate domain shift issues.
- UDHR: access our clean version of udhr here.
- FLORES-200: devtest part of FLORES-200.
You can find how we compute F1, Recall, Precision, and FPR in metrics.py.
- If you see wrong predicted tags by GlotLID for a normal long text open an issue, however:
- if the script is not supported by our model then use GlotScript to verify for the predicted
lang_script, script in the sentence exists! Otherwise, you need to write a function that returns 'und_mainscript' in this situations. GlotScript can identify both the mainscript and all available scripts in the sentence. We recommend using GlotLID in conjunction with GlotScript. - The high confidence threshold for each language could be different. This is because not all languages have the same distance from each other. For one language, 0.6 is a lot because it is very close to a similar language (such as dyu and bam), while for another, 0.9 might not be.
- This model is primarily trained on longer sentences, avoid using it on very short sentences. Other language identification models are not good at short sentences as well unless you increase the ngram size (in training), which is computationally expensive.
- In GlotLID, the false positive rate (FPR) for high-resource languages is much higher than for low-resource languages. However, even with this higher FPR, it is still lower than in a situation where the language identification model only recognizes high-resource languages. We are also okay with this situation since our main concern is for the FPR of low-resource languages to be low. The high-resource base frequency is much higher than for low-resource languages, so cleanliness would not be a threat for those languages. However, for a low-resource language with a low base frequency, even a small FPR might result in most of the corpus being noisy.
- if the script is not supported by our model then use GlotScript to verify for the predicted
- Don't forget, you don't always need to run the language identification model in full. If you have a setup where you know it can only be a set of specific languages (subset of languages), then limit the prediction to those. We have provided the code for limiting it in this readme.
- If you want to add a language, provide the resource in an open issue, and we will add it. If you require the model urgently, we can expedite the process in less than a week (the training itself takes less than a day). However, if there's no immediate urgency, that language will be included in the official release according to our schedule (depends on new resources).-
- If you want to collaborate, please send us an email (to: amir@cis.lmu.de) specifying the type of collaboration you need from us.
- for the rest of requests feel free to email or open an issue.
The model is distributed under the Apache License, Version 2.0 plus notices (see LICENSE file for full terms).
-
Additional Notice: This software includes a machine learning model for language identification based on statistical representations (e.g., character n-grams and linear classification). The model was trained on a mixture of publicly available data sources, which may include materials subject to copyright or other rights. The model does not store or encode expressive text content. Instead, it represents aggregated statistical patterns (such as n-gram frequency weights) that are not intended to reproduce original source material. No training data is included or distributed with this software. Due to its design, this model performs classification only and is not capable of generating or reconstructing source texts.
-
Good-Faith Notice: If you are a rights holder and believe that your content has been included in the training data and wish to request its exclusion from future versions, please contact the authors. Reasonable efforts will be made to review and address such requests.
If you find our model, code and list of data sources useful for your research, please cite:
@inproceedings{
kargaran2023glotlid,
title={{G}lot{LID}: Language Identification for Low-Resource Languages},
author={Kargaran, Amir Hossein and Imani, Ayyoob and Yvon, Fran{\c{c}}ois and Sch{\"u}tze, Hinrich},
booktitle={The 2023 Conference on Empirical Methods in Natural Language Processing},
year={2023},
url={https://openreview.net/forum?id=dl4e3EBz5j}
}