From the source
Apple researchers found that enhancing language discrimination during pretraining of multilingual speech models (using an auxiliary language classifier and per-language k-means targets) reduces the performance gap compared to monolingual models.
- In a controlled English/French HuBERT setting, phone discrimination error decreased from 11.6% (bilingual baseline) to 10.4%, while lexical and prosodic performance improved, in some cases matching or exceeding monolingual baselines.
- The strongest gains occurred when language discrimination was introduced in the first training iteration.