From the source
The Technology Innovation Institute (TII) released the Falcon3 family of five open decoder-only large language models under 10 billion parameters: Falcon3-1B-Base, Falcon3-3B-Base, Falcon3-Mamba-7B-Base, Falcon3-7B-Base, and Falcon3-10B-Base.
The models emphasize science, math, and code capabilities, with the 7B model pretrained on 14 trillion tokens using 1024 H100 GPUs and the 10B model created via depth up-scaling.
Falcon3-10B-Base achieves state-of-the-art zero-shot and few-shot performance for models under 13B parameters, and all transformer-based models are compatible with Llama architecture.
