How researchers adapted Dolma for better Thai language models

Ai2 reported that Thai researchers used its open Dolma toolkit to create Mangosteen, a 47-billion-token Thai pretraining corpus.

The team modified Dolma's pipeline to handle Thai's sentence boundaries and quality filters, and found that their curation removed over 80% of Common Crawl data and nearly half of FineWeb2 while maintaining or improving model performance.