# Hugging Face — Why your fine-tuning job takes longer than the FLOPs say

- Company: Hugging Face (huggingface.co)
- Announced: 2026-11-18T18:53:43+00:00
- Category: research-paper
- Coverage: 1 outlet
- Announcement: yes
- Group: covered
- Source: https://huggingface.co/blog/Dharma-AI/predicting-fine-tuning-time
- Record: https://forck.live/items/38168-why-your-fine-tuning-job-takes-longer-than-the-flops-say
- Subject: Platform

A preview paper from Dharma AI compares FLOP-based estimates of fine-tuning time against learned predictors, using 272 production supervised fine-tuning runs on shared NVIDIA H200 GPUs. The authors say the standard compute-only estimate explains little of what they observed, with R² = 0.31 and a mean absolute error of 12.41 hours. A CatBoost regressor that sees only information available before dispatch reached R² = 0.98, a mean absolute error of 1.71 h and a median error of 0.51 h in 5-fold cross-validation. The gap comes from training-specific signals the closed-form estimate cannot use, such as evaluation cadence, checkpoint policy, per-GPU load and fine-tuning regime, and from nonlinear interactions between them. The runs cover several model families up to 15 billion parameters, LoRA and full fine-tuning, on 1, 2 or 4 GPUs drawn from a pool of 8 to 32 H200s, with durations from 1.6 to 172.4 hours. Sweeping MFU from 0.25 to 0.65 did not fix the FLOP baseline: the best setting, 30%, still leaves a mean error above 12 hours on jobs averaging 23 hours. The full version is to appear on arXiv.

## Evidence

Verbatim from https://huggingface.co/blog/Dharma-AI/predicting-fine-tuning-time:

> Predicting end-to-end fine-tuning time on shared GPU clusters, from 272 production runs.
> 
> A preview of our upcoming paper, Predicting End-to-End Fine-Tuning Time for Generative Language Models . The full version will be on arXiv soon.
> 
> GPUs are among the most expensive and scarcest resources in AI today. Anyone running a shared cluster has to decide, every day, which jobs go on which GPUs and in what order. That plan is only as good as the estimate of how long each job will take. When a job finishes hours early, GPUs sit idle until the next one is dispatched. When it runs hours late, everything queued behind it slips, experiments wait and delivery dates move.
> 
> Better duration estimates turn this into planning instead of guesswork: what runs, when, and on how many GPUs, with less idle capacity in between. Yet the estimate most teams use is a back-of-the-envelope FLOP calculation. It says nine hours, and the job finishes the next afternoon.
> 
> We wanted to know how far off that math really is, and whether we could do better before a job is even dispatched.

## Around this story

Outlets this record can name and link:

- Extrapolator — Fine-Tuning Time Prediction Cuts MAE to 1.7 Hours: https://extrapolator.ai/2026/11/19/fine-tuning-time-prediction

---

Record: https://forck.live/items/38168-why-your-fine-tuning-job-takes-longer-than-the-flops-say
Catalogue: https://forck.live/llms.txt
Current issue: https://forck.live/feed.md
