Sep 2026·10 min read information-theory probability

KL divergence, or why you can't just subtract two distributions

Vineet Kumar

Applied Scientist II

TL;DR

Short version: you can't just subtract two distributions and call it a day, because a plain difference treats every mistake the same whether it's a shrug or a disaster. Chase that problem far enough and you land on surprise, then entropy, then cross-entropy, and KL divergence falls out the other end — not a formula someone invented, but the only honest way to measure "how many extra bits am I paying for believing the wrong thing." And here's the twist: you've already been minimizing it. Cross-entropy loss is KL divergence, just with the name filed off.

First thing first — as you probably know, most of machine learning is about estimating a probability distribution from a finite number of samples. But to do that, we need to know when we've actually estimated it properly. But wait, what do you even mean when you say "we've estimated the distribution"?

What I mean is, we kind of know a distribution that's very similar to the true distribution, p(x). And that's the core question here — what does it even mean to say two probability distributions are "similar" or "different"?

Why not just subtract them?

One simple answer pops into everyone's mind: why not just subtract the distributions? Take p(x) − q(x) at each point and add up the sizes. Tbh, it's not crazy — it's a real distance, and it's called total variation. But it has a flaw that matters enormously in machine learning.

It treats all errors with equal importance. Being off by 0.1 when p(x) = 0.5 is treated the same as being off by 0.1 when p(x) = 0.0001, even though those are very different mistakes. I'll explain in a bit why that's an issue, but the short version is, most of the samples we actually see come from regions with higher probability, so getting those right most of the time matters a lot more than getting some rare event right. Total variation doesn't know that. It just sees a gap of 0.1 in both places and treats them the same.

Fig 1. q1 misjudges two common outcomes; q2 misjudges one common outcome and one that almost never happens. Same total variation, wildly different real-world stakes.

There's another way to see the same flaw, and it's sharper. If the truth says an event essentially never happens and your model gives it real probability, that's a serious error. And worse, if the truth says an event does happen and your model assigns it probability zero, your model is claiming something impossible is real. A plain difference barely notices either case; it just sees a small gap.

Note (still a bit unsure about this one): this might not always be true. Sometimes being correct on rare samples matters way more than being correct in general, and then you'd want to prioritize the difference on rare samples instead. But subtraction still isn't the way to do that — you'd just want to weigh those samples more.

Instead, what we'd like is a measure that autoscales itself based on how big the blunder is — so how does p/q sound? If p is very small, the penalty is small even if the blunder by q is big. And there's a second clue: independent samples multiply their probabilities, but we'd prefer our score to add, kind of like accumulating how much of a mistake we're making for every single sample. Differences don't respect products; logs turn products into sums. Two clues, one answer: we want the log of a ratio, or something similar.

Surprise, entropy, cross-entropy

Before going for KL, let's bring in the concept of surprise, entropy, and cross-entropy.

Surprise is simple: how surprised should you be when something with probability p happens? If something is really obvious, you won't really be surprised, right? Like if someone says the sun rises in the east, you're like, yeah I know — in fact, everyone knows that. But if someone says there's going to be an eclipse today, you'll definitely be more surprised, since it's rare, new information. Now if someone says the sun rose in the west this morning — you'll go nuts, right? Even though the probability of that is nearly zero, if it actually happened, the surprise would be just crazy. Infinite, basically.

Picture a function that stays tiny near probability 1 and blows up near probability 0 — that's it:

\text{Surprise}(x) = -\log p(x)

You can also, in a sense, say this is how much information x is carrying. That's why when you change the log from ln to log2, it's called bits — literally, the number of bits required to store that information.

Now suppose I want to describe a whole distribution by how much it can surprise me — then we go for something like the expectation of surprise (the average), given by:

H(p) = \sum_x p(x)\big(-\log p(x)\big)

— it's also called the entropy of the distribution (its randomness). That's also the fewest bits per symbol any code built with full knowledge of p can average — the best you could ever do.

Now suppose you only have model q. Every draw you feel surprise −log q(x), but x still comes from the real p:

H(p,q) = \sum_x p(x)\big(-\log q(x)\big)

If you analyze closely, you'll see that when x is very probable in p and less probable in q, both p(x) and −log q(x) come out large. And if x is very unlikely in p but very likely in q, they both go toward a small value — less important, like we said before, since we don't see many samples from there anyway.

H(p,q) is the average number of bits actually paid, using a code built for q, on data that really comes from p. If q = p this collapses to H(p). If q is wrong, you overpay, every time. In simpler language, what we're trying to do is store all the information that's in the distribution p into our model q — and H(p,q) is the bill for doing that with the wrong code.

Quick note: I've been writing ln above out of habit, and I'll switch to log below. Doesn't really matter which base you use as long as you're consistent — ln gives you "nats," log2 gives you "bits," and switching just rescales KL by a constant. Doesn't change where it hits zero or how asymmetric it is.

KL is just the size of the overpay

KL is just the size of this overpay. So the best we can do is H(p), right? And the cost is H(p,q). Then the price is:

\begin{aligned} KL(p\Vert q) &= H(p,q) - H(p) \\ &= \sum_x p(x)\big(-\log q(x)\big) - \sum_x p(x)\big(-\log p(x)\big) \\ &= \sum_x p(x)\big(\log p(x) - \log q(x)\big) \\ &= \sum_x p(x)\log\frac{p(x)}{q(x)} \end{aligned}

It wasn't invented — we were forced into it. It's "how many extra bits does believing q cost me, relative to the best I could ever do." Here's a toy weather example: true forecast p = (Sunny ½, Cloudy ¼, Rainy ⅛, Snowy ⅛), and a lazy model q that just guesses uniformly, ¼ each. Then H(p) = 1.75 bits, H(p,q) = 2.00 bits, and KL(p‖q) = 0.25 bits — a quarter of a bit wasted on every single forecast, forever, just for believing "all weather is equally likely." Here's where we lose those bits:

Fig 2. The gap between H(p) and H(p,q) — the amber sliver — is exactly KL(p‖q): 0.25 extra bits, spent forever, just for using the wrong code.

Always ≥ 0, sometimes infinite

There's another interesting property: KL divergence is always ≥ 0, and the proof is trivial. One line, using Jensen's inequality on the convex function −log:

KL(p\Vert q) = \mathbb{E}_p\!\left[-\log\frac{q}{p}\right] \;\ge\; -\log\,\mathbb{E}_p\!\left[\frac{q}{p}\right] = -\log\Big(\sum_x q(x)\Big) = -\log 1 = 0

Equality holds only when q = p everywhere — a model can never beat the truth's own entropy, only match it. And if p(x) > 0 somewhere q says is impossible (q(x) = 0), KL blows up to infinity: ruling out something that actually happens is infinitely costly under forward KL. That's the real reason softmax never outputs a hard zero, and why label smoothing exists.

Why it's asymmetric

Another interesting property is the asymmetry, and it comes from the exact same place. KL is an average of log(p/q), and an average needs a weighting distribution. Forward KL(p‖q) weights by p, so it only "cares" about places where p is large (more likely things). But reverse KL(q‖p) weights by q, so it only cares about where q is large (less frequent events). Same ratio, different vantage point — that's the whole source of the asymmetry, and it's also why KL fails two of the three rules for being a true distance metric: not symmetric, no triangle inequality — it's a divergence, not a metric. When you need a symmetric, bounded fix, that's Jensen–Shannon divergence, which is what the original GAN objective minimizes.

To see a clearer difference between the two, let's take an interesting example. We'll try to model a p with two modes (peaks) using a model q, which is a true Gaussian. If we go with forward KL, it tries to minimize the divergence at places where p is large, so it wants q to be higher wherever p is higher — it will not dare put q(x) = 0 where p is nonzero. Reverse KL goes the other way: it tries to get things right at the places that are less likely, so it aligns q close to zero wherever p is small. You'll get a better idea if you see the figure:

Fig 3. Fitting one Gaussian q to a two-humped truth p — forward KL hedges across both modes; reverse KL commits to one and drops the other.

Forward KL is what maximum likelihood does (classifiers, language models, most supervised learning) — it hedges, and tries to cover everything. Reverse KL is what variational inference and VAEs do (the tractable direction) — it collapses onto one mode, which is the textbook explanation for blurry VAE samples and posteriors that don't fully cover the truth.

You might have never noticed it, but KL was already running your training loop

You never actually have the true p — only samples. Define the empirical distribution p_data: equal mass on every observed example, zero everywhere else (7 heads in 10 flips → p_data = (0.7, 0.3)). This stands in for "the truth" in every practical setting. Maximum likelihood estimation is just: pick parameters that make the observed data least surprising to the model.

\begin{aligned} \text{maximize} \;\; &\prod_i q(x_i) \\ = \text{minimize} \;\; &{-\tfrac{1}{N}\sum_i \log q(x_i)} && \text{(average negative log-likelihood)}\\ = \text{minimize} \;\; &H(p_{\text{data}}, q) && \text{(cross-entropy)} \end{aligned}

Since H(p_data,q) = H(p_data) + KL(p_data‖q), and H(p_data) has no model parameters in it — it's just a constant — minimizing cross-entropy and minimizing KL(p_data‖q) are literally the same optimization. MLE is forward-KL minimization, which is exactly why it's mode-covering.

For classification specifically, the true label is one-hot, so H(p) = 0 exactly, and:

\text{cross-entropy loss} = H(p,q) = H(p) + KL(p\Vert q) = KL(p\Vert q)

The classifier or language-model loss you've been minimizing this whole time isn't "like" KL divergence — it is KL divergence, exactly, with the constant subtracted off.

The wider family

Moreover, KL is just one member of the f-divergence family — you can create your own. Just pick any convex f with f(1) = 0, and define:

D_f(p\Vert q) = \sum_x q(x)\, f\!\left(\frac{p(x)}{q(x)}\right)

Non-negativity and "zero iff p = q" come free from convexity + Jensen for every member of this family — never a KL-specific fact.

f(t)	divergence
t · log t	KL(p‖q), forward
−log t	KL(q‖p), reverse
½\|t − 1\|	total variation
(t − 1)²	chi-squared

Mutual information also turns out to be exactly a KL divergence:

I(X;Y) = KL\big(p(x,y)\,\Vert\,p(x)\,p(y)\big)

— literally, how far the true joint distribution sits from the pretend-independent world. That single identity is why mutual information is automatically ≥ 0 and automatically 0 only when X and Y are independent, with no separate proof needed.

Other ways of looking at entropy

Everything above builds entropy from one route — an axiomatic notion of "surprise." That's Shannon's cleanest route, but there are other doors from which the same number falls out, through unrelated arguments. But that's for some other blog! Here's the map, for now:

Fig 4. The same H(p) falls out of at least six unrelated arguments — surprise is just the one this post takes.

Key takeaways

Total variation is a real distance, but it treats every mistake the same no matter how rare the event is — you need a ratio, not a difference, to fix that.
KL(p‖q) isn't a formula someone made up. It falls straight out of "how many extra bits does believing q cost me," once you accept surprise → entropy → cross-entropy.
You've been minimizing KL this whole time without knowing it — MLE is forward-KL minimization, and cross-entropy loss is KL divergence, exactly.
Forward and reverse KL aren't interchangeable. They optimize for opposite failure modes, mode-covering vs. mode-seeking, and that choice quietly shapes how your model behaves.

Have a great day! Happy learning.

References

Kullback, S., & Leibler, R. A. On Information and Sufficiency. Annals of Mathematical Statistics, 22(1), 1951.
Shannon, C. E. A Mathematical Theory of Communication. Bell System Technical Journal, 27, 1948.
Cover, T. M., & Thomas, J. A. Elements of Information Theory (2nd ed.). Wiley, 2006.
Kingma, D. P., & Welling, M. Auto-Encoding Variational Bayes. arXiv:1312.6114, 2013.
Goodfellow, I., et al. Generative Adversarial Networks. arXiv:1406.2661, 2014.

← back to all posts