# Harvey — Introducing BigLaw Bench

- Company: Harvey (harvey.ai)
- Announced: 2024-08-29T18:48:00+00:00
- Category: research-paper
- Coverage: not counted
- Announcement: yes
- Group: announcements
- Source: https://www.harvey.ai/blog/introducing-biglaw-bench
- Record: https://forck.live/items/17729-introducing-biglaw-bench
- Subject: Harvey

Harvey introduced BigLaw Bench, a public version of its internal dataset for evaluating large language models on complex legal tasks. The benchmark uses bespoke rubrics to score model responses on real-world billable tasks, measuring both answer quality and source verifiability. Harvey reported that its proprietary models significantly outperform publicly available LLMs, but all models show substantial room for improvement compared to full completion of tasks performed by lawyers.

## Evidence

Verbatim from https://www.harvey.ai/blog/introducing-biglaw-bench:

> BigLaw Bench is a framework for quantitatively evaluating the performance of LLMs on real-world legal tasks; supplementing prior work that measures LLM legal reasoning in more structured settings. Harvey’s proprietary models significantly outperform publicly available LLMs, but all models show substantial room for improvement when benchmarked against full completion of tasks performed by lawyers.

---

Record: https://forck.live/items/17729-introducing-biglaw-bench
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md