From the source
From the source
Debugging GPU memory leaks caused by stuck FUSE mounts
H Company engineers investigated GPU memory leaks on Kubernetes nodes where VRAM appeared allocated without an owning process.
They traced the issue to containers that remained in RUNNING state after pod deletion, with threads stuck in D-state inside the FUSE kernel driver (`request_wait_answer`), blocking on a FUSE reply that never arrived.
The post documents the debugging methodology and identifies the root cause as a FUSE mount hang rather than a CUDA or NCCL issue.
From the source
Every single D-state thread was sleeping inside the same kernel function: request_wait_answer . That symbol is defined in `fs/fuse/dev.c` , i.e. the FUSE device driver. It is the function that parks a thread until the userspace FUSE daemon answers a request (more details about FUSE below). None of this was CUDA, NCCL, or similar: every stuck worker was blocked on a FUSE reply that never arrived.
hcompany.ai
