Mix, Don't Tune: Bilingual Pre-Training Outperforms Hyperparameter Search in Data-Constrained Settings

Apple researchers compared two strategies for improving language model pre-training when target-language data is scarce: aggressive hyperparameter tuning (e.g., high weight decay) versus mixing in data from a high-resource auxiliary language.

Across roughly 1000 runs at scales from 150M to 1.43B parameters, using Arabic as the low-resource target and English as the auxiliary, they found that mixing consistently outperformed hyperparameter tuning on both validation loss and downstream accuracy, with the gap widening at larger model sizes.

The paper recommends mixing in a high-resource language, prioritizing the mixing ratio over hyperparameter tuning, and transferring hyperparameters from a small proxy model via μP.