Basic Properties of the Log-Mean-Exp Operator
Overview
Consider any $\eta \in \Reals \setminus \{ 0 \}$ Sometimes, $\eta$ is called the temperature because of analogues from thermodynamics. and vector $q = [q_1, \ldots, q_n] \in \Reals^n$. Define the log-mean-exp operator
$$ \LME_{\eta}[q] := \frac{1}{\eta}\log \Big( \sum_i p_i e^{\eta q_i} \Big) $$where the $p_i$ are positive weights summing to one.
Let $q_{\min}$ and $q_{\max}$ denote the minimal and maximal values of $q$.
Intuitive Interpretation #
This (differential) operator smoothly interpolates between key statistics as $\eta$ varies as demonstrated in the following statement.
Visualisation of $\LME_\eta[q]$ for varying temperatures.
In this example $\eta$ for $q := [e^{-1}, \phi \approx 1.618, e, \pi]$ with $p = [0.1, 0.2, 0.4, 0.3]$.
Proposition
For fixed $q$ whose values are not all identical the log-mean-exp operator increases monotonically with $\eta \in \Reals$. Moreover,
- $$ \lim_{\eta \rightarrow \infty} \LME_\eta[q] = q_{\max} \label{eq:lmelimit1} $$
- $$ \lim_{\eta \rightarrow -\infty} \LME_\eta[q] = q_{\min} \label{eq:lmelimit2} $$
- $$ \lim_{\eta \rightarrow 0} \LME_\eta[q] = \Exp_p[q] :=\sum_{i}^{n} p_i q_i. \label{eq:lmelimit3} $$
Proof. Monotonicity. Let $K(\eta) = \log(\sum_i p_i e^{\eta q_i})$, which is the cumulant-generating function of a discrete random variable $Q$ taking values $q_i$ with probability $p_i$. The operator can then be written as:
$$ f(\eta) = \frac{K(\eta)}{\eta}. $$Taking the derivative with respect to $\eta$:
$$ \frac{d}{d\eta} f(\eta) = \frac{\eta K'(\eta) - K(\eta)}{\eta^2}. $$Because $K(\eta)$ is strictly convex whenever the values $q_i$ are not all identical (it is easy and mundane to show that $K''(\eta) = \Var_\eta(Q) > 0$), any tangent line lies strictly below the function. Evaluating the first-order convexity condition at $0$:
$$ K(0) > K(\eta) + K'(\eta)(0 - \eta). $$Since $\eta^2 > 0$ for all $\eta \ne 0$, the derivative $\frac{d}{d\eta} f(\eta) > 0$ for all $\eta \ne 0$.
Asymptotic Limits. Let $I_{\max}$ and $I_{\min}$ be the set of indices for which $q$ attains its maximal or minimal value. To verify $\eqref{eq:lmelimit1}$ and $\eqref{eq:lmelimit2}$ observe that
$$ \LME_\eta[q] =q_{\max} + \frac{1}{\eta} \log \Big( \sum_{i \in I_{\max}} p_i + \sum_{j \notin I_{\max}} p_j e^{\eta (q_j - q_{\max})} \Big) \xrightarrow[\eta \rightarrow \infty]{} q_{\max} $$$$ \LME_\eta[q] =q_{\min} + \frac{1}{\eta} \log \Big( \sum_{i \in I_{\min}} p_i + \sum_{j \notin I_{\min}} p_j e^{\eta (q_j - q_{\min})} \Big) \xrightarrow[\eta \rightarrow -\infty]{} q_{\min}. $$
Finally, $\eqref{eq:lmelimit3}$ is a straightforward application of L’hôpital’s rule.
Remark