Entropy and Relative Entropy
Overview
The core idea of informational theory is that the informational value of a communicated message depends on the degree to which the content of the message is surprising. If the message communicates the occurrence of some very likely event, it has conveyed very little surprise to the recipient. On the other hand, if the message conveyed the occurrence of some very unlikely event, the message is much more informative. For instance, the knowledge that some particular number will not be the winning number of a lottery provides very little information, because any particular chosen number will almost certainly not win. However, knowledge that a particular number will win has high informational value because it communicates the outcome of a very low probability event.
The above intuition was formalised (more or less) by C. Shannon (1948) who proposed that the amount of information $I(p)$ acquired by observing and event $E$ with probability $p = \Prob(E)$ should satisfy the following properties.
1Implicit in our mathematical description is a (Kolmogorov) probability space $(\Omega, \Field, \Prob)$ where events $E$ are sampled from the $\sigma$-field $\Field$ appropriate to the experiment at hand.
- $p \mapsto I(p)$ is continuous.
- $I(p) \ge 0$. Information is a non-negative quantity.
- $I(p)$ is monotonically decreasing in $p$, in the sense that if $E_1$ and $E_2$ are two events such that $p_1 = \Prob(E_1) \ge \Prob(E_2) = p_2$ then $I(p_1) \le I(p_2)$. I.e., observing a less likely event has more informational content than observing a highly likely event.
- $I(p) = 0$ whenever $p = \Prob(E) = 1$. Events that (almost) always occur do not convey information.
- $I(p_1p_2) = I\big( \Prob(E_1 \cap E_2) \big) = I(p_1) + I(p_2)$ whenever $E_1$ and $E_2$ are independent. That is, the information learned from observing the occurrence of two independent events is the sum of the information learned from each event.
Proposition
Suppose $\Field$ has at least two independent events. Then the only twice differentiable mapping $I: [0,1] \rightarrow [0, \infty)$ satisfying the above properties is the function $I(p) = k \log(p)$ for $k \le 0$. Moreover, if $I(p)$ is required to be strictly monotonically decreasing in $p$ then $k < 0$.
Proof. Clearly $\log(1/p)$ satisfies all of the required properties and is twice differentiable. To check it is the only one suppose that property 5 is true for arbitrary independent events $E_1$ and $E_2$ (we have supposed that that there is at least one pair of independent events in $\Field$). Differentiating with respect to $p_1$ and then with respect to $p_2$ we get the differential equation
$$I’(p_1p_2) + p_1 p_2 I’’(p_1 p_2) = 0 \quad p_1, p_2 \in [0,1]$$
which can be rewritten as the differential equation with respect to $u := p_1 p_2 \in (0,1)$ so that $$I’(u) + u I’’(u) = 0 \quad u \in [0, 1].$$ Solving the resulting differential equation yields $I(u) = k log(u) + c$ for some constants $k, c \in \Reals$. Property 4 gives $c=0$ and Property 2 leads to $k \le 0$. Evidently, $k < 0$ is a requirement for strict monotonicity.