From the source
Hugging Face published a community article introducing Alyah, a benchmark for evaluating Arabic LLMs on the Emirati dialect.
The benchmark contains 1,173 manually collected multiple-choice samples covering greetings, figurative language, poetry, and heritage. 54 models (23 base, 31 instruction-tuned) were evaluated, including Jais, Allam, Qwen, LLaMA, Fanar, and AceGPT.
Accuracy on multiple-choice questions is the primary metric.
