From the source
From the source
First general-purpose pure Mamba model at 7B scale
TII released Falcon Mamba, a 7B-parameter pure Mamba architecture model trained on ~5500GT of data.
The model can process sequences of arbitrary length without increased memory storage and fits on a single A10 24GB GPU.
It is open access under the TII Falcon Mamba 7B License 1.0 and available via Hugging Face.
From the source
Falcon Mamba is based on the original Mamba architecture, proposed in Mamba: Linear-Time Sequence Modeling with Selective State Spaces , with the addition of extra RMS normalization layers to ensure stable training at scale. This choice of architecture ensures that Falcon Mamba: can process sequences of arbitrary length without any increase in memory storage, in particular, fitting on a single A10 24GB GPU.
falcon-lm.github.io
