AI, Data Science & Cloud Certificates from Google, IBM & Meta
Live Online Classes in Design, Coding & AI — Small Classes, Free Retakes
Overview
Google, IBM & Meta Certificates – 40% Off
One Coursera Plus subscription covers most Professional Certificates on Coursera.
Unlock All Certificates
This course demonstrates how to build and train a WordPiece tokenizer for Dhivehi, a low-resource language using the right-to-left Thaana writing system. It covers tokenizer components, implementation with Hugging Face Tokenizers, saving for Transformers, and testing.
Syllabus
Intro
Dhivehi Project
Hurdles for Low Resource Domains
Dhivehi Dataset
Download Dhivehi Corpus
Tokenizer Components
Normalizer Component
Pre-tokenization Component
Post-tokenization Component
Decoder Component
Tokenizer Implementation
Tokenizer Training
Post-processing Implementation
Decoder Implementation
Saving for Transformers
Tokenizer Test and Usage
Download Dhivehi Models
First Steps
Taught by
James Briggs