Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline

NVIDIA and IIT Madras' AI4Bharat lab introduced Nemotron-pretraining-Indic-v1, a corpus of approximately 2.8 trillion tokens across 22 Indian languages, alongside an open, GPU-accelerated data generation pipeline.

The corpus includes native-script data for all 22 languages and romanized text for twelve, covering 34 language-script configurations, and is built from curated English source documents via translation, quality assessment, and repair.

The pipeline and data are released openly so developers can reproduce or adapt the workflow for their own needs.