Відмінності між дистанцією Бхаттачарія та різницею KL

33

Я шукаю інтуїтивне пояснення для наступних питань:

У теорії статистики та інформації, чим відрізняється відстань Бхаттачарія від розбіжності KL як міри різниці між двома дискретними розподілами ймовірностей?

Чи не мають вони абсолютно ніяких зв’язків і вимірюють відстань між двома розподілами ймовірностей абсолютно різним чином?

— JewelSue
джерело

36

Коефіцієнт Бхаттачарія визначається як

D_{B} (p, q) = \int \sqrt{p (x) q (x)} d x

$D_B(p,q) = \int \sqrt{p(x)q(x)}\,\text{d}x$

d_{H} (p, q)

$d_H(p,q)$

d_{H} (p, q) = {1 - D_{B} (p, q)}^{1 / 2}

$d_H(p,q)=\{1-D_B(p,q)\}^{1/2}$

d_{K L} (p ‖ q) \geq 2 d_{H}^{2} (p, q) = 2 {1 - D_{B} (p, q)} .

$d_{KL}(p\|q) \geq 2 d_H^2(p,q) = 2 \{1-D_B(p,q)\}\,.$

Однак це не питання: якщо відстань Бхаттачарія визначається як тоді

г_{Б} (p, q) \overset{деф}{=} - журнал D_{Б} (p, q),

$d_B(p,q)\stackrel{\text{def}}{=}-\log D_B(p,q)\,,$

\begin{aligned} d_{B} (p, q) = - \log D_{B} (p, q) & = - \log \int \sqrt{p (x) q (x)} d x \\ \overset{def}{=} - \log \int h (x) d x \\ = - \log \int \frac{h (x)}{p (x)} p (x) d x \\ \leq \int - \log {\frac{h (x)}{p (x)}} p (x) d x \\ = \int \frac{- 1}{2} \log {\frac{h^{2} (x)}{p^{2} (x)}} p (x) d x \\ = \int \frac{- 1}{2} \log {\frac{q (x)}{p (x)}} p (x) d x = \frac{1}{2} d_{K L} (p ‖ q) \end{aligned}

$\begin{align*}d_B(p,q)=-\log D_B(p,q)&=-\log \int \sqrt{p(x)q(x)}\,\text{d}x\\ &\stackrel{\text{def}}{=}-\log \int h(x)\,\text{d}x\\ &= -\log \int \frac{h(x)}{p(x)}\,p(x)\,\text{d}x\\ &\le \int -\log \left\{\frac{h(x)}{p(x)}\right\}\,p(x)\,\text{d}x\\ &= \int \frac{-1}{2}\log \left\{\frac{h^2(x)}{p^2(x)}\right\}\,p(x)\,\text{d}x\\ &= \int \frac{-1}{2}\log \left\{\frac{q(x)}{p(x)}\right\}\,p(x)\,\text{d}x= \frac{1}{2}d_{KL}(p\|q) \end{align*}$

d_{K L} (p ‖ q) \geq 2 d_{B} (p, q) .

${d_{KL}(p\|q)\ge 2d_B(p,q)\,.}$

- l o g (x) \geq 1 - x 0 \leq x \leq 1,

$-log(x)\ge 1-x\qquad\qquad 0\le x\le 1\,,$ enter image description here

we have the complete ordering

d_{K L} (p ‖ q) \geq 2 d_{B} (p, q) \geq 2 d_{H} (p, q)^{2} .

${d_{KL}(p\|q)\ge 2d_B(p,q)\ge 2d_H(p,q)^2\,.}$

— Xi'an
джерело

2

Brilliant! This explanation should be the one I am looking for eagerly. Just one last question: in what case (or what kinds of P and Q) will the inequality becomes equality?

— JewelSue

1

Given that the

- \log (\cdot)

$-\log(\cdot)$ function is strictly convex, I would assume the only case for equality is when the ratio

p (x) / q (x)

$p(x)/q(x)$ is constant in

x

$x$ .

— Xi'an

5

And the only case when

p (x) / q (x)

$p(x)/q(x)$ is constant in

x

$x$ is when

p = q

$p=q$ .

— Xi'an

8

I don't know of any explicit relation between the two, but decided to have a quick poke at them to see what I could find. So this isn't much of an answer, but more of a point of interest.

For simplicity, let's work over discrete distributions. We can write the BC distance as

d_{BC} (p, q) = - \ln \sum_{x} (p (x) q (x))^{\frac{1}{2}}

$d_\text{BC}(p,q) = - \ln \sum_x (p(x)q(x))^\frac{1}{2}$

and the KL divergence as

d_{KL} (p, q) = \sum_{x} p (x) \ln \frac{p (x)}{q (x)}

$d_\text{KL}(p,q) = \sum_x p(x)\ln \frac{p(x)}{q(x)}$

Now we can't push the log inside the sum on the $\text{BC}$ distance, so let's try pulling the log to the outside of the $\text{KL}$ divergence:

d_{KL} (p, q) = - \ln \prod_{x} {(\frac{q (x)}{p (x)})}^{p (x)}

$d_\text{KL}(p,q) = -\ln \prod_x \left( \frac{q(x)}{p(x)} \right)^{p(x)}$

Let's consider their behaviour when $p$ is fixed to be the uniform distribution over $n$ possibilities:

d_{KL} (p, q) = - \ln n - \ln {(\prod_{x} q (x))}^{\frac{1}{n}} d_{BC} (p, q) = - \ln \frac{1}{\sqrt{n}} - \ln \sum_{x} \sqrt{q (x)}

$d_\text{KL}(p,q) = -\ln n - \ln \left(\prod_x q(x)\right)^\frac{1}{n} \qquad d_\text{BC}(p,q) = - \ln \frac{1}{\sqrt{n}} - \ln\sum_x \sqrt{q(x)}$

On the left, we have the log of something that's similar in form to the geometric mean. On the right, we have something similar to the log of the arithmetic mean. Like I said, this isn't much of an answer, but I think it gives a neat intuition of how the BC distance and the KL divergence react to deviations between $p$ and $q$ .

— Andy Jones
джерело