Modular LLMs at scale: how FlexOlmo is helping to pool national expertise without pooling sensitive data

Researchers at the University of Southern Denmark's OdenseNLP lab built FlexMoRE, a modification of Ai2's FlexOlmo architecture that replaces most full-size experts with smaller low-rank adapters.

FlexMoRE outperforms a FlexOlmo-style baseline while using less than one-third the parameters, enabling modular language models to run on consumer hardware.

The work is part of the Danish Foundation Models (DFM) project, which aims to develop open Danish-language models for institutions that cannot share sensitive data.