Academic

OpenLID-v3: Improving the Precision of Closely Related Language Identification -- An Experience Report

arXiv:2602.13139v1 Announce Type: new Abstract: Language identification (LID) is an essential step in building high-quality multilingual datasets from web data. Existing LID tools (such as OpenLID or GlotLID) often struggle to identify closely related languages and to distinguish valid natural language from noise, which contaminates language-specific subsets, especially for low-resource languages. In this work we extend the OpenLID classifier by adding more training data, merging problematic language variant clusters, and introducing a special label for marking noise. We call this extended system OpenLID-v3 and evaluate it against GlotLID on multiple benchmarks. During development, we focus on three groups of closely related languages (Bosnian, Croatian, and Serbian; Romance varieties of Northern Italy and Southern France; and Scandinavian languages) and contribute new evaluation datasets where existing ones are inadequate. We find that ensemble approaches improve precision but also s

M
Mariia Fedorova, Nikolay Arefyev, Maja Buljan, Jind\v{r}ich Helcl, Stephan Oepen, Egil R{\o}nningstad, Yves Scherrer · · 1 min read · 2 views
#cs.CL

arXiv:2602.13139v1 Announce Type: new Abstract: Language identification (LID) is an essential step in building high-quality multilingual datasets from web data. Existing LID tools (such as OpenLID or GlotLID) often struggle to identify closely related languages and to distinguish valid natural language from noise, which contaminates language-specific subsets, especially for low-resource languages. In this work we extend the OpenLID classifier by adding more training data, merging problematic language variant clusters, and introducing a special label for marking noise. We call this extended system OpenLID-v3 and evaluate it against GlotLID on multiple benchmarks. During development, we focus on three groups of closely related languages (Bosnian, Croatian, and Serbian; Romance varieties of Northern Italy and Southern France; and Scandinavian languages) and contribute new evaluation datasets where existing ones are inadequate. We find that ensemble approaches improve precision but also substantially reduce coverage for low-resource languages. OpenLID-v3 is available on https://huggingface.co/HPLT/OpenLID-v3.

Executive Summary

The article 'OpenLID-v3: Improving the Precision of Closely Related Language Identification -- An Experience Report' presents an enhanced version of the OpenLID classifier, aimed at improving the identification of closely related languages and distinguishing natural language from noise. The authors, recognizing the limitations of existing tools like OpenLID and GlotLID, introduce OpenLID-v3 by incorporating more training data, merging problematic language variant clusters, and adding a special label for noise. The study focuses on three groups of closely related languages and contributes new evaluation datasets where existing ones are inadequate. The findings suggest that ensemble approaches improve precision but reduce coverage for low-resource languages. OpenLID-v3 is made available on Hugging Face.

Key Points

  • ▸ OpenLID-v3 improves the precision of language identification for closely related languages.
  • ▸ The study introduces a special label for marking noise to enhance data quality.
  • ▸ Ensemble approaches improve precision but reduce coverage for low-resource languages.
  • ▸ New evaluation datasets are contributed for groups of closely related languages.

Merits

Enhanced Precision

The article demonstrates a significant improvement in the precision of language identification, particularly for closely related languages, which is a critical advancement in the field.

Comprehensive Evaluation

The study provides a thorough evaluation against GlotLID on multiple benchmarks, contributing new datasets where existing ones are inadequate, ensuring robust and reliable results.

Accessibility

The availability of OpenLID-v3 on Hugging Face makes it accessible to researchers and practitioners, fostering further advancements in the field.

Demerits

Reduced Coverage

The use of ensemble approaches, while improving precision, substantially reduces coverage for low-resource languages, which is a notable limitation.

Focused Scope

The study focuses on specific groups of closely related languages, which may limit the generalizability of the findings to other language groups.

Expert Commentary

The article presents a significant advancement in the field of language identification, particularly in addressing the challenges posed by closely related languages. The introduction of OpenLID-v3, with its enhanced precision and special label for noise, addresses critical gaps in existing tools. The study's focus on specific language groups, while limiting in scope, provides valuable insights and contributes new evaluation datasets. The trade-off between precision and coverage, especially for low-resource languages, is a critical consideration that warrants further research. The availability of OpenLID-v3 on Hugging Face ensures that the tool is accessible to a broad audience, fostering further advancements in the field. The study's findings have practical implications for improving the quality of multilingual datasets and policy implications for supporting low-resource languages.

Recommendations

  • ✓ Further research should explore methods to balance precision and coverage, particularly for low-resource languages, to ensure equitable support across all language groups.
  • ✓ Future studies should expand the scope to include a broader range of closely related languages and low-resource languages to enhance the generalizability of the findings.

Sources

Related Articles