Cumulative Residual Entropy¶
The cumulative residual entropy [RCVW04] is an alternative to the differential Shannon entropy. The differential entropy has many issues, including that it can be negative even for simple distributions such as the uniform distribution; and that if one takes discrete estimates that limit to the continuous distribution, the discrete entropy does not limit to the differential (continuous) entropy. It also attempts to provide meaningful differences between numerically different random variables, such as a die labeled [1, 2, 3, 4, 5, 6] and one lebeled [1, 2, 3, 4, 5, 100].
Note
The Cumulative Residual Entropy is unrelated to Residual Entropy.
In [1]: In [1]: from dit.other import cumulative_residual_entropy
Generalized Cumulative Residual Entropy¶
The genearlized form of the cumulative residual entropy integrates over the intire set of reals rather than just the positive ones:
In [2]: In [6]: from dit.other import generalized_cumulative_residual_entropy
Conditional Cumulative Residual Entropy¶
The conditional cumulative residual entropy \(\CRE[X|Y]\) is a distribution with the same probability mass function as \(Y\), and the outcome associated with \(p(y)\) is equal to the cumulative residual entropy over probabilities conditioned on \(Y = y\). In this sense the conditional cumulative residual entropy is more akin to a distribution over \(\H[X|Y=y]\) than the single scalar quantity \(\H[X|Y]\).
Conditional Generalized Cumulative Residual Entropy¶
Conceptually the conditional generalized cumulative residual entropy is the same as the non-generalized form, but integrated over the entire real line rather than just the positive:
API¶
-
cumulative_residual_entropy
(dist, extract=False)[source]¶ The cumulative residual entropy is an alternative to the Shannon differential entropy with several desirable properties including non-negativity.
- Parameters
dist (Distribution) – The distribution to compute the cumulative residual entropy of each index for.
extract (bool) – If True and dist.outcome_length() is 1, return the single GCRE value rather than a length-1 array.
- Returns
CREs – The cumulative residual entropy for each index.
- Return type
ndarray
Examples
>>> d1 = ScalarDistribution([1, 2, 3, 4, 5, 6], [1/6]*6) >>> d2 = ScalarDistribution([1, 2, 3, 4, 5, 100], [1/6]*6) >>> cumulative_residual_entropy(d1) 2.0683182557028439 >>> cumulative_residual_entropy(d2) 22.672680046016705
-
generalized_cumulative_residual_entropy
(dist, extract=False)[source]¶ The generalized cumulative residual entropy is a generalized from of the cumulative residual entropy. Rather than integrating from 0 to infinity over the absolute value of the CDF.
- Parameters
dist (Distribution) – The distribution to compute the generalized cumulative residual entropy of each index for.
extract (bool) – If True and dist.outcome_length() is 1, return the single GCRE value rather than a length-1 array.
- Returns
GCREs – The generalized cumulative residual entropy for each index.
- Return type
ndarray
Examples
>>> generalized_cumulative_residual_entropy(uniform(-2, 3)) 1.6928786893420307 >>> generalized_cumulative_residual_entropy(uniform(0, 5)) 1.6928786893420307
Conditional Forms¶
-
conditional_cumulative_residual_entropy
(dist, rv, crvs=None, rv_mode=None)[source]¶ Returns the conditional cumulative residual entropy.
- Parameters
dist (Distribution) – The distribution to compute the conditional cumulative residual entropy of.
rv (list, None) – The possibly joint random variable to compute the conditional cumulative residual entropy of. If None, then all variables not in crvs are used.
crvs (list, None) – The random variables to condition on. If None, nothing is conditioned on.
rv_mode (str, None) – Specifies how to interpret rvs and crvs. Valid options are: {‘indices’, ‘names’}. If equal to ‘indices’, then the elements of crvs and rvs are interpreted as random variable indices. If equal to ‘names’, the the elements are interpreted as random variable names. If None, then the value of dist._rv_mode is consulted, which defaults to ‘indices’.
- Returns
CCRE – The conditional cumulative residual entropy.
- Return type
ScalarDistribution
Examples
>>> from itertools import product >>> events = [ (a, b) for a, b, in product(range(5), range(5)) if a <= b ] >>> probs = [ 1/(5-a)/5 for a, b in events ] >>> d = Distribution(events, probs) >>> print(conditional_cumulative_residual_entropy(d, 1, [0])) Class: ScalarDistribution Alphabet: (-0.0, 0.5, 0.91829583405448956, 1.3112781244591329, 1.6928786893420307) Base: linear
x p(x) -0.0 0.2 0.5 0.2 0.918295834054 0.2 1.31127812446 0.2 1.69287868934 0.2
-
conditional_generalized_cumulative_residual_entropy
(dist, rv, crvs=None, rv_mode=None)[source]¶ Returns the conditional cumulative residual entropy.
- Parameters
dist (Distribution) – The distribution to compute the conditional generalized cumulative residual entropy of.
rv (list, None) – The possibly joint random variable to compute the conditional generalized cumulative residual entropy of. If None, then all variables not in crvs are used.
crvs (list, None) – The random variables to condition on. If None, nothing is conditioned on.
rv_mode (str, None) – Specifies how to interpret rvs and crvs. Valid options are: {‘indices’, ‘names’}. If equal to ‘indices’, then the elements of crvs and rvs are interpreted as random variable indices. If equal to ‘names’, the the elements are interpreted as random variable names. If None, then the value of dist._rv_mode is consulted, which defaults to ‘indices’.
- Returns
CCRE – The conditional cumulative residual entropy.
- Return type
ScalarDistribution
Examples
>>> from itertools import product >>> events = [ (a-2, b-2) for a, b, in product(range(5), range(5)) if a <= b ] >>> probs = [ 1/(3-a)/5 for a, b in events ] >>> d = Distribution(events, probs) >>> print(conditional_generalized_cumulative_residual_entropy(d, 1, [0])) Class: ScalarDistribution Alphabet: (-0.0, 0.5, 0.91829583405448956, 1.3112781244591329, 1.6928786893420307) Base: linear
x p(x) -0.0 0.2 0.5 0.2 0.918295834054 0.2 1.31127812446 0.2 1.69287868934 0.2