# Harvey — BigLaw Bench: Hallucinations

- Company: Harvey (harvey.ai)
- Announced: 2024-10-07T17:00:00+00:00
- Category: research-paper
- Coverage: not counted
- Announcement: yes
- Group: announcements
- Source: https://www.harvey.ai/blog/biglaw-bench-hallucinations
- Record: https://forck.live/items/17723-biglaw-bench-hallucinations
- Subject: Harvey
- Models affected: Harvey Assistant, Claude, ChatGPT, Gemini

Harvey published a benchmark evaluating hallucinations on legal tasks. Its Assistant model hallucinated at a rate of 0.2% (1 in 500 claims), lower than Claude (0.7%), ChatGPT (1.3%), and Gemini (1.9%), despite providing longer and more detailed answers. The company defines hallucinations as factual claims demonstrably disproven by a source of truth and uses a system of models to detect and measure them.

## Evidence

Verbatim from https://www.harvey.ai/blog/biglaw-bench-hallucinations:

> On BigLaw Bench tasks, Harvey’s Assistant model hallucinates around 1 in 500 claims (.2%). Foundation models hallucinate more frequently, with rates between 1 in 150 (.7%, Claude) and 1 in 110 (1.9%, Gemini).

---

Record: https://forck.live/items/17723-biglaw-bench-hallucinations
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md