From the source
From the source
A single transformer estimates density and score without retraining
Ai2 introduced DiScoFormer, a transformer that estimates both the density and score of a distribution from a set of data points in a single forward pass without retraining.
It outperforms kernel density estimation in high dimensions, cutting score error by about 6.5x and density error by more than 37x in 100 dimensions, and generalizes to out-of-distribution inputs via a label-free consistency loss.
From the source
Across the board, DiScoFormer beats KDE at both density and score estimation, and the gap widens exactly where KDE struggles. In 100 dimensions, it isn't close—against the best hand-tuned KDE, it cuts score error by about 6.5x and density error by more than 37x, and it keeps improving as you add samples, while KDE runs out of memory.
allenai.org
