AI, Data Science & Cloud Certificates from Google, IBM & Meta — 40% Off
One plan covers every Professional Certificate on Coursera. 40% off your first 3 months — limited time.
Unlock All Certificates
Explore an innovative approach to efficiently extend pretrained Masked Language Models (MLMs) to new languages in this 11-minute conference talk from the Center for Language & Speech Processing (CLSP) at Johns Hopkins University. Dive into the concept of mini-model adaptation, a compute-efficient alternative to traditional methods that builds a shallow mini-model from a fraction of a large model's parameters. Learn about two approaches for creating mini-models: MiniJoint and MiniPost. Discover how these techniques allow for rapid cross-lingual transfer while significantly reducing computational costs. Examine experimental results from XNLI, MLQA, and PAWS-X datasets, which demonstrate that mini-model adaptation matches the performance of standard approaches while using up to 2.4x less compute. Gain insights into this cutting-edge research based on the paper "Efficiently Extending Pretrained Models to New Languages via Aligned Shallow Training" presented at ACL Findings.
