Need for High Quality Datasets for Indic-NLP

This conference talk explores the critical importance of high-quality datasets and corpus development for natural language processing (NLP) in Indian languages. Discover how better data resources can accelerate advancements in Large Language Models (LLMs) and AI applications across India. Learn why token frequency and occurrence patterns are fundamental to NLP performance, and why Indian languages require specialized corpus development that matches the domain distribution, structure, and generalization standards of Western datasets. Understand how proper datasets are essential not only for advancing research but also for accurately evaluating and refuting theories about language behavior based on statistical occurrence. The talk emphasizes the necessity for large-scale collaboration and contribution in dataset preparation to support the development of LLMs and conversational AI systems comparable to ChatGPT for Indian languages.