# Vercel — The rise of the AI crawler

- Company: Vercel (vercel.com)
- Announced: 2024-12-17T13:00:00+00:00
- Category: research-paper
- Coverage: not counted
- Announcement: yes
- Group: announcements
- Source: https://vercel.com/blog/the-rise-of-the-ai-crawler
- Record: https://forck.live/items/20912-the-rise-of-the-ai-crawler
- Subject: v0 / AI SDK / AI Gateway

Vercel and MERJ analyzed AI crawler traffic across Vercel's network, finding that GPTBot generated 569 million requests and Claude 370 million in the past month, together about 20% of Googlebot's 4.5 billion requests. None of the major AI crawlers (OpenAI, Anthropic, Meta, ByteDance, Perplexity) currently render JavaScript, while Google's Gemini and AppleBot do. ChatGPT spends 34.82% of its fetches on 404 pages and Claude 34.16%, compared to Googlebot's 8.22%.

## Evidence

Verbatim from https://vercel.com/blog/the-rise-of-the-ai-crawler:

> Our analysis shows a clear divide in JavaScript rendering capabilities among AI crawlers. ... The results consistently show that none of the major AI crawlers currently render JavaScript.

---

Record: https://forck.live/items/20912-the-rise-of-the-ai-crawler
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md