From the source
Lead story
Ask your AI
Top stories
Models & availability
Latest
Limits of Confidence in Diffusion
Apple researchers prove that discrete diffusion models using per-position distributions (including remasking and uniform-state samplers) cannot match the training distribution when the tokens they write are dependent, and that per-position marginals alone cannot detect such dependence.
On a synthetic task, they measure the generated distribution's total variation to be 29 times the sampling-noise floor.
From the source
We show that a step matches the training distribution only when the positions it writes are conditionally independent given the tokens already fixed, that no product of per-position distributions can match a dependent group, and that per-position distributions do not determine whether a group is dependent: two joint distributions can have identical per-position marginals while differing in which combinations of values occur.
machinelearning.apple.com