Falcon 2: An 11B parameter pretrained language model and VLM, trained on over 5000B tokens and 11 languages

TII released Falcon 2, a new generation of open-source models, starting with an 11B parameter language model (LLM) and an 11B vision-language model (VLM) that can answer queries about images.

The LLM was trained on over 5,000 billion tokens of RefinedWeb and curated data, supports 11 languages, and achieves performance comparable to Falcon-40B at a quarter of the size.

The VLM integrates a CLIP ViT-L/14 vision encoder with the chat-finetuned LLM.