From the source
From the source
Together AI introduces ParallelKernelBench (PKB), a benchmark and evaluation framework for multi-GPU kernel generation, containing 87 problems from real codebases.
The benchmark tests frontier coding models on replacing PyTorch + NCCL with CUDA kernels that communicate directly over NVLink.
Evaluation shows that under a third of problems are solved correctly, and fewer than a quarter of those beat the naive baseline.
From the source
ParallelKernelBench (PKB) offers a benchmark and evaluation framework for multi-GPU kernel generation and includes 87 problems from real codebases where the task is replacing PyTorch + NCCL with a CUDA kernel that moves data directly over NVLink. We tested frontier coding models such as GPT-5.5, Gemini 3 Pro, Opus 4.7, and others. The evaluation revealed significant performance gaps across the board: under a third of problems were solved correctly, and fewer than a quarter of those beat the naive baseline.
together.ai
