# Hugging Face — Scaling AI-based Data Processing with Hugging Face + Dask

- Company: Hugging Face (huggingface.co)
- Announced: 2024-10-09T00:00:00+00:00
- Subject: Platform
- Source: https://huggingface.co/blog/dask-scaling
- Record: https://forck.live/items/1818-scaling-ai-based-data-processing-with-hugging-face-dask

This blog post demonstrates scaling AI-based data processing using Hugging Face and Dask, showing how to process 100 rows locally with pandas and then scale to 211 million rows with Dask across multiple GPUs on the cloud, using the FineWeb dataset and FineWeb-Edu classifier.

## Evidence

Verbatim from https://huggingface.co/blog/dask-scaling:

> In this post we show an example of data processing from the FineWeb dataset, using the FineWeb-Edu classifier to identify web pages with high educational value. We'll show: How to process 100 rows locally with pandas; Scaling to 211 million rows with Dask across multiple GPUs on the cloud.

---

Record: https://forck.live/items/1818-scaling-ai-based-data-processing-with-hugging-face-dask
Catalogue: https://forck.live/llms.txt
Feed: https://forck.live/feed.md