MrT5: Dynamic Token Merging for Efficient Byte-level Language Models

This seminar presentation by Julie Kallini from Stanford University explores MrT5 (MergeT5), an innovative approach to making byte-level language models more efficient. Learn about the limitations of subword tokenization models, including their vulnerability to spelling errors and inconsistent compression across languages, and discover how MrT5 addresses these issues through a dynamic token deletion mechanism. The presentation explains how MrT5 can achieve significant improvements in inference runtime while maintaining performance, adapting to language-specific orthographic characteristics, and reducing sequence lengths by up to 75% compared to traditional byte-level models like ByT5. Kallini, a PhD student supported by multiple prestigious fellowships whose work won Best Paper Award at ACL 2024, demonstrates how MrT5 offers comparable accuracy on downstream tasks while solving practical limitations of existing byte-level models.