Class Central is learner-supported. When you buy through links on our site, we may earn an affiliate commission.

YouTube

Building Transformer Tokenizers - Dhivehi NLP #1

James Briggs via YouTube

Overview

Google, IBM & Meta Certificates – 40% Off
One Coursera Plus subscription covers most Professional Certificates on Coursera.
Unlock All Certificates
This course demonstrates how to build and train a WordPiece tokenizer for Dhivehi, a low-resource language using the right-to-left Thaana writing system. It covers tokenizer components, implementation with Hugging Face Tokenizers, saving for Transformers, and testing.

Syllabus

Intro
Dhivehi Project
Hurdles for Low Resource Domains
Dhivehi Dataset
Download Dhivehi Corpus
Tokenizer Components
Normalizer Component
Pre-tokenization Component
Post-tokenization Component
Decoder Component
Tokenizer Implementation
Tokenizer Training
Post-processing Implementation
Decoder Implementation
Saving for Transformers
Tokenizer Test and Usage
Download Dhivehi Models
First Steps

Taught by

James Briggs

Reviews

Start your review of Building Transformer Tokenizers - Dhivehi NLP #1

Never Stop Learning.

Get personalized course recommendations, track subjects and courses with reminders, and more.

Someone learning on their laptop while sitting on the floor.