# Apple — Mix, Don't Tune: Bilingual Pre-Training Outperforms Hyperparameter Search in Data-Constrained Settings

- Company: Apple (apple.com)
- Announced: 2026-10-13
- Category: research-paper
- Coverage: not counted
- Announcement: yes
- Group: announcements
- Source: https://machinelearning.apple.com/research/mix-dont-tune
- Record: https://forck.live/items/20485-mix-don-t-tune-bilingual-pre-training-outperforms-hyperparameter-search-in
- Subject: Machine Learning Research

Apple researchers compared two strategies for improving language model pre-training when target-language data is scarce: aggressive hyperparameter tuning (e.g., high weight decay) versus mixing in data from a high-resource auxiliary language. Across roughly 1000 runs at scales from 150M to 1.43B parameters, using Arabic as the low-resource target and English as the auxiliary, they found that mixing consistently outperformed hyperparameter tuning on both validation loss and downstream accuracy, with the gap widening at larger model sizes. The paper recommends mixing in a high-resource language, prioritizing the mixing ratio over hyperparameter tuning, and transferring hyperparameters from a small proxy model via μP.

## Evidence

Verbatim from https://machinelearning.apple.com/research/mix-dont-tune:

> First, mixing yields larger improvements than hyperparameter tuning on both validation loss and downstream task accuracy, and the gap grows with model size. Second, we quantify how much mixing helps: it boosts performance by an amount equivalent to 2–3× the unique target data on validation loss and 2–13× on downstream task accuracy, with the gain scaling steeply with model size.

---

Record: https://forck.live/items/20485-mix-don-t-tune-bilingual-pre-training-outperforms-hyperparameter-search-in
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md