From the source
H Company engineers diagnosed recurring phantom GPU memory usage on SkyPilot-managed Kubernetes nodes.
On freshly booked nodes, nvidia-smi showed tens of gigabytes of allocated VRAM with no owning process.
The root cause was zombie training containers whose threads were stuck in an uninterruptible D-state waiting for a FUSE filesystem reply; the kubelet could not reap those containers, and the underlying GPU memory allocation remained pinned.
