Entropy and Relative Entropy

By Simji (審遲)

4 August 2026

5 min read

Overview

This article is a work-in-progress. To add:

References to sources and some historical notes.
Key properties of introduced definitions.
Other interpretations. E.g. as compressability

The core idea of informational theory is that the informational value of a communicated message depends on the degree to which the content of the message is surprising. If the message communicates the occurrence of some very likely event, it has conveyed very little surprise to the recipient. On the other hand, if the message conveyed the occurrence of some very unlikely event, the message is much more informative. For instance, the knowledge that some particular number will not be the winning number of a lottery provides very little information, because any particular chosen number will almost certainly not win. However, knowledge that a particular number will win has high informational value because it communicates the outcome of a very low probability event.

The above intuition was formalised (more or less) by C. Shannon (1948) who proposed that the amount of information $I(p)$ acquired by observing and event $E$ with probability $p = \Prob(E)$ should satisfy the following properties.1 1Implicit in our mathematical description is a (Kolmogorov) probability space $(\Omega, \Field, \Prob)$ where events $E$ are sampled from the $\sigma$-field $\Field$ appropriate to the experiment at hand.

$p \mapsto I(p)$ is continuous.
$I(p) \ge 0$. Information is a non-negative quantity.
$I(p)$ is monotonically decreasing in $p$, in the sense that if $E_1$ and $E_2$ are two events such that $p_1 = \Prob(E_1) \ge \Prob(E_2) = p_2$ then $I(p_1) \le I(p_2)$. I.e., observing a less likely event has more informational content than observing a highly likely event.
$I(p) = 0$ whenever $p = \Prob(E) = 1$. Events that (almost) always occur do not convey information.
$I(p_1p_2) = I\big( \Prob(E_1 \cap E_2) \big) = I(p_1) + I(p_2)$ whenever $E_1$ and $E_2$ are independent. That is, the information learned from observing the occurrence of two independent events is the sum of the information learned from each event.

Proposition

Suppose $\Field$ has at least two independent events. Then the only twice differentiable mapping $I: [0,1] \rightarrow [0, \infty)$ satisfying the above properties is the function $I(p) = k \ \log(p)$ for $k \le 0$. Moreover, if $I(p)$ is required to be strictly monotonically decreasing in $p$ then $k < 0$.

Proof. Clearly $\log(1/p)$ satisfies all of the required properties and is twice differentiable. To check it is the only one suppose that property 5 is true for arbitrary independent events $E_1$ and $E_2$ (we have supposed that that there is at least one pair of independent events in $\Field$). Differentiating with respect to $p_1$ and then with respect to $p_2$ we get the differential equation

$$I'(p_1p_2) + p_1 p_2 I''(p_1 p_2) = 0 \quad p_1, p_2 \in [0,1]$$

which can be rewritten as the second-order differential equation with respect to $u := p_1 p_2 \in (0,1)$ so that

$$I'(u) + u I''(u) = 0 \quad u \in [0, 1].$$

Solving the resulting differential equation yields $I(u) = k \ log(u) + c$ for some constants $k, c \in \Reals$. Property 4 gives $c=0$ and Property 2 leads to $k \le 0$. Evidently, $k < 0$ is a requirement for strict monotonicity.

■

The above result leads to Shannon’s definition of entropy of a discrete random variable.

Definition (Shannon entropy)

Consider a discrete random variable $X$ with support over a denumerable set $\\{ x_1, \ldots, x_n, \ldots \\}$. Then the Shannon entropy is given by (Greek letter “eta”)

$$ \Ent(X; \Prob) := -\sum_{i=1}^\infty \Prob(X = x_i) \log_b \big[ \Prob(X = x_i) \big] $$

where $b$ is the base of the logarithm.

Definition (Differential entropy)

For a continuous random variable $Y$ whose distribution is absolutely continuous with respect to Lebesgue measure (i.e. so that its density function $f(y)$ exists by the Radon-Nikodym theorem), the differential entropy is given by

$$ \Ent(X; \Prob) := -\int_{\mathcal{Y}} f(y) \log_b \big( f(y) \big) \ dy $$

where $\mathcal{Y}$ is the (possibly unbounded) support of $Y$.

While Shannon and differential entropy might look like direct analogues, they are not. Differential entropy can be negative and has an infinitely large difference from the limit of a Shannon entropy. This fact can be verified by taking Riemann sums of the differential entropy and comparing it to the Shannon entropy in the limit. The notion of relative entropy, in contrast, does not suffer from this defect, having a seamless analogue between the continuous and discrete cases.

Definition (Kullback-Leibler divergence)

Let $X$ be a discrete random variable taking values in a denumerable set $\\{ x_1, \ldots, x_n, \ldots \\}$ and suppose that $\mathbb{Q}(E) = 0$ implies $\Prob(E) = 0$ for every $E \in \Field$ (i.e. $\Prob \ll \mathbb{Q}$; $\Prob$ is absolutely continuous with respect to $\mathbb{Q}$). The Kullback-Leibler divergence or (relative entropy) of $X$ is given by

$$ \KL(\Prob \|| \mathbb{Q}) := \sum_{i=1}^\infty \Prob (X = x_i) \ln \Big[ \frac{\Prob(X = x_i)}{\mathbb{Q}(X = x_i)}\Big]. $$

Suppose $Y$ is a continuous random variabe with support $\mathbb{Y} \subseteq \Reals$ and suppose the densities $f_\Prob$ and $f_{\mathbb{Q}}$ are well-defined. Then the relative entropy of $Y$ is given by

$$ \KL(\Prob \|| \mathbb{Q}) := \int_{y \in \mathbb{Y}} f_\Prob(y) \ln \Big[ \frac{f_\Prob(y)}{f_\mathbb{Q}(y)} \Big] dy. $$

The Kullback-Leibler divergence is a measure of how one probability distribution $\mathbb{Q}$ differs from a second reference probability distribution $\Prob$. Note that it is not a “distance” in the standard sense of metrics since it fails to satisfy symmetry.