Machine Translation Engines Evaluation Framework

Explore a comprehensive framework for evaluating Machine Translation (MT) engines in this 32-minute conference talk from EuroPython 2022. Delve into the challenges of MT engine evaluation, including quality variations across domains and language pairs, interface differences, and the complexities of defining good translations. Learn about the universal evaluation framework developed to address these challenges, featuring a base translation class for unified file handling and result processing, a set of test datasets covering general and healthcare domains, and quality metrics such as BLEU, BERTScore, ROUGE, TER, and CHRF. Discover evaluation results for various cloud-based and open-source MT engines, including Azure Translator, Google Translate, Marian MT, NVIDIA's NeMo, and Facebook's MBart 50 and M2M100. Gain insights into fine-tuning MT engines for specific domains and understand how to extend the framework to custom MT engines and datasets.