If X\sim P then for any distribution Q it is unlikely that Q ascribes much greater density to X's outcome than P does. In fact if P,Q have PDFs f_P, f_Q, then:

\begin{align}
\mathbb{P …

## Point KL-Divergence is not Very Negative Very Often

If X\sim P then for any distribution Q it is unlikely that Q ascribes much greater density to X's outcome than P does. In fact if P,Q have PDFs f_P, f_Q, then:

\begin{align} \mathbb{P …