Cross Mutual Information

The cross mutual information [GCS+25], denoted \(CI_{pq}\), quantifies how strongly an \(X\)-\(Y\) dependence defined by a reference distribution \(q\) is expressed in test data sampled from \(p\). In analogy to the cross entropy, the pointwise information of each outcome is evaluated using \(q\) while the expectation is taken over \(p\):

\[CI_{pq} = \sum_{x, y} p(x, y) \log_2 \frac{q(x, y)}{q(x) q(y)}\]

When \(p = q\) the cross mutual information reduces to the ordinary Mutual Information. If the reference distribution factorizes (\(X\) and \(Y\) are independent under \(q\)) then the cross mutual information is zero for any test distribution. Unlike the conventional mutual information, the cross mutual information can be negative: this occurs when the dependence in the test data is surprising relative to the reference.

In [1]: from dit import Distribution

In [2]: from dit.multivariate import cross_coinformation as CI

In [3]: p = Distribution(['00', '01', '10', '11'], [0.4, 0.1, 0.1, 0.4])

In [4]: q = Distribution(['00', '01', '10', '11'], [0.1, 0.4, 0.4, 0.1])

In [5]: CI(p, p)
Out[5]: 0.2780719051126379

In [6]: CI(p, q)
Out[6]: -0.9219280948873623

Generalizations

Each of the multivariate mutual informations is a signed sum of joint entropies, and so admits a cross generalization by replacing every entropy with the analogous cross entropy between \(p\) and \(q\). In addition to the cross co-information (cross_coinformation(), the multivariate generalization of the cross mutual information above), dit provides the cross Total Correlation, the cross Dual Total Correlation, and the cross CAEKL Mutual Information. Each reduces to its conventional counterpart when \(p = q\).

In [7]: from dit.multivariate import cross_total_correlation as CT

In [8]: from dit.multivariate import cross_dual_total_correlation as CB

In [9]: from dit.multivariate import cross_caekl_mutual_information as CJ

API

cross_coinformation(dist, ref_dist, rvs=None, crvs=None)[source]

Calculates the cross co-information, the cross-distribution generalization of the (multivariate) mutual information.

For two random variables this is the cross mutual information of [Gohil et al., arXiv:2507.15372].

Parameters:

  • dist (Distribution) – The test distribution p, over which the expectation is taken.

  • ref_dist (Distribution) – The reference distribution q, used to evaluate the pointwise information of each outcome.

  • rvs (list, None) – The indexes of the random variable used to calculate the cross co-information between. If None, then the cross co-information is calculated over all random variables.

  • crvs (list, None) – The indexes of the random variables to condition on. If None, then no variables are condition on.

Returns:

CI – The cross co-information.

Return type:

float

Raises:

ditException – Raised if dist is not a joint distribution or if rvs or crvs contain non-existant random variables.

cross_total_correlation(dist, ref_dist, rvs=None, crvs=None)[source]

Computes the cross total correlation, the cross-distribution generalization of the total correlation.

Parameters:

  • dist (Distribution) – The test distribution p, over which the expectation is taken.

  • ref_dist (Distribution) – The reference distribution q, used to evaluate the pointwise information of each outcome.

  • rvs (list, None) – A list of lists. Each inner list specifies the indexes of the random variables used to calculate the cross total correlation. If None, then the cross total correlation is calculated over all random variables, which is equivalent to passing rvs=dist.rvs.

  • crvs (list, None) – A single list of indexes specifying the random variables to condition on. If None, then no variables are conditioned on.

Returns:

CT – The cross total correlation.

Return type:

float

Raises:

ditException – Raised if dist is not a joint distribution or if rvs or crvs contain non-existant random variables.

cross_dual_total_correlation(dist, ref_dist, rvs=None, crvs=None)[source]

Calculates the cross dual total correlation, the cross-distribution generalization of the dual total correlation.

Parameters:

  • dist (Distribution) – The test distribution p, over which the expectation is taken.

  • ref_dist (Distribution) – The reference distribution q, used to evaluate the pointwise information of each outcome.

  • rvs (list, None) – The indexes of the random variable used to calculate the cross dual total correlation. If None, then the cross dual total correlation is calculated over all random variables.

  • crvs (list, None) – The indexes of the random variables to condition on. If None, then no variables are condition on.

Returns:

CB – The cross dual total correlation.

Return type:

float

Raises:

ditException – Raised if dist is not a joint distribution or if rvs or crvs contain non-existant random variables.

cross_caekl_mutual_information(dist, ref_dist, rvs=None, crvs=None)[source]

Calculates the cross CAEKL mutual information, the cross-distribution generalization of the CAEKL mutual information.

Parameters:

  • dist (Distribution) – The test distribution p, over which the expectation is taken.

  • ref_dist (Distribution) – The reference distribution q, used to evaluate the pointwise information of each outcome.

  • rvs (list, None) – A list of lists. Each inner list specifies the indexes of the random variables used to calculate the cross CAEKL mutual information. If None, then the cross CAEKL mutual information is calculated over all random variables, which is equivalent to passing rvs=dist.rvs.

  • crvs (list, None) – A single list of indexes specifying the random variables to condition on. If None, then no variables are conditioned on.

Returns:

CJ – The cross CAEKL mutual information.

Return type:

float

Raises:

ditException – Raised if dist is not a joint distribution or if rvs or crvs contain non-existant random variables.