From the source
LG AI Research proposes a cluster-aware upcycling method that initializes experts and routers based on activation clustering of a pre-trained dense model, aiming to improve expert specialization in Mixture-of-Experts structures.
[CVPR 2026] An upcycling method for creating true experts within a Mixture-of-Experts structure
Research paper
From the source
LG AI Research analyzed how the final training results and performance can vary depending on the initialization method used when upcycling pre-trained dense models into MoE models, and proposed an initialization technique for more effective training.
lgresearch.ai
