Rényi Entropy

The Rényi entropy is a spectrum of generalizations to the Shannon Entropy:

\[\RE{\alpha}{X} = \frac{1}{1-\alpha} \log_2 \left( \sum_{x \in \mathcal{X}} p(x)^\alpha \right)\]
In [1]: In [1]: from dit.other import renyi_entropy

Special Cases

For several values of \(\alpha\), the Rényi entropy takes on particular values.

\(\alpha = 0\)

When \(\alpha = 0\) the Rényi entropy becomes what is known as the Hartley entropy:

\[\RE{0}{X} = \log_2 |X|\]
In [2]: In [5]: renyi_entropy(d, 0)
Out[2]: 2.0

In [3]: Out[5]: 4.0

\(\alpha = 1\)

When \(\alpha = 1\) the Rényi entropy becomes the standard Shannon entropy:

\[\RE{1}{X} = \H{X}\]
In [4]: In [6]: renyi_entropy(d, 1)
Out[4]: 2.0

In [5]: Out[6]: 2.9688513169509623

\(\alpha = 2\)

When \(\alpha = 2\), the Rényi entropy becomes what is known as the collision entropy:

\[\RE{2}{X} = - \log_2 p(X = Y)\]

where \(Y\) is an IID copy of X. This is basically the surprisal of “rolling doubles”

In [6]: In [7]: renyi_entropy(d, 2)
Out[6]: 2.0

In [7]: Out[7]: 2.7607270851693615

\(\alpha = \infty\)

Finally, when \(\alpha = \infty\) the Rényi entropy picks out the probability of the most-probable event:

\[\RE{\infty}{X} = - \log_2 \max_{x \in \mathcal{X}} p(x)\]
In [8]: In [8]: renyi_entropy(d, np.inf)
Out[8]: 2.0

In [9]: Out[8]: 2.275104563096674

General Properies

In general, the Rényi entropy is a monotonically decreasing function in \(\alpha\):

\[\RE{\alpha}{X} \ge \RE{\beta}{X}, \quad \beta > \alpha\]

Further, the following inequality holds in the other direction:

\[\RE{2}{X} \le 2 \cdot \RE{\infty}{X}\]

API

renyi_entropy(dist, order, rvs=None, rv_mode=None)[source]

Compute the Renyi entropy of order order.

Parameters
  • dist (Distribution) – The distribution to take the Renyi entropy of.

  • order (float >= 0) – The order of the Renyi entropy.

  • rvs (list, None) – The indexes of the random variable used to calculate the Renyi entropy of. If None, then the Renyi entropy is calculated over all random variables.

  • rv_mode (str, None) – Specifies how to interpret rvs and crvs. Valid options are: {‘indices’, ‘names’}. If equal to ‘indices’, then the elements of crvs and rvs are interpreted as random variable indices. If equal to ‘names’, the the elements are interpreted as random variable names. If None, then the value of dist._rv_mode is consulted, which defaults to ‘indices’.

Returns

H_a – The Renyi entropy.

Return type

float

Raises
  • ditException – Raised if rvs or crvs contain non-existant random variables.

  • ValueError – Raised if order is not a non-negative float.