From the source
GitHub has opened a public preview of GitHub Copilot Evals, a tool that lets people test whether their own Copilot customizations still produce good results.
- An eval is a repeatable test for AI behavior: you give an agent a realistic task, describe what a good result looks like, and check both the result and how the agent got there.
- Because AI output can change from one run to the next, GitHub says a single good answer does not show whether a customization works, so users can rerun the same eval after changing a skill, agent or MCP tool to catch regressions.
- Users can ask Copilot in their own words or use the /eval command to create an eval for a skill, custom agent or MCP tool in their project, then run it and see which checks passed and failed, with each run approved before it starts.
- Results can show why a check failed and let users compare runs from before and after a change.
- Evals uses Vally, an extensible local eval tool, as its evaluation engine, and installs as a Visual Studio Code extension, a Copilot CLI plugin, or a plugin from the github/evals marketplace in the Copilot app.
- The public preview covers Copilot Individual, Copilot Business and Copilot Enterprise plans.





