Improving Prompt Consistency with Structured Generations
Research paper
Hugging Face's research team, with collaborators at Dottxt, investigates how LLM evaluation results are highly sensitive to prompt format changes, showing that performance can vary by 10 points and model rankings can be inconsistent across different prompt formats.
They describe experiments on MMLU subsets with 8 prompt formats and 5 models, highlighting the need for greater prompt consistency.
