Transmission

The transmission [Zwi04], denoted \(T\), is the central error measure of reconstructability analysis. Given a structure (a set of marginals, or “projections”, to hold fixed), it quantifies how much of a distribution’s constraint is lost when the distribution is reconstructed from those marginals alone. It is the Kullback-Leibler divergence from the data to the maximum entropy distribution consistent with the chosen marginals:

\[T(\text{structure}) = \DKL{p}{q_{\text{structure}}} = U(q_{\text{structure}}) - U(p)\]

where \(q_{\text{structure}}\) is the maximum entropy reconstruction (see Optimization) matching the specified marginals of \(p\), and \(U\) is the Shannon entropy.

A structure is specified as a list of marginals, each a list of variable indices. For example, [[0, 1], [1, 2]] is the structure AB:BC.

In [1]: from dit.multivariate import transmission

In [2]: xor = dit.example_dists.Xor()

The default structure is the independence model (each variable on its own), for which the transmission equals the Total Correlation:

In [3]: transmission(xor)
Out[3]: 1.0

In [4]: from dit.multivariate import total_correlation

In [5]: total_correlation(xor)
Out[5]: 1.0

For the saturated model (all variables held jointly), nothing is lost and the transmission is zero:

In [6]: transmission(xor, [[0, 1, 2]])
Out[6]: 0.0

Because the parity information in xor lives entirely in the three-way interaction, any proper decomposition loses all of it:

In [7]: transmission(xor, [[0, 1], [1, 2]])
Out[7]: 1.0

Model Complexity

Reconstructability analysis trades transmission (model error) against model complexity, measured by the degrees of freedom of the structure: the number of free parameters needed to specify the maximum entropy distribution. Together, transmission and degrees of freedom for every structure in the lattice form the “decomposition spectrum”; see the dependency decomposition in Information Profiles.

In [8]: from dit.algorithms import degrees_of_freedom

In [9]: degrees_of_freedom(xor, [[0, 1], [1, 2]])
Out[9]: 5

API

transmission(dist, structure=None)

Compute the transmission of dist relative to a marginal model: the Kullback-Leibler divergence from the data to the maximum entropy distribution reconstructed from the marginals in structure.

This is the decomposition error of reconstructability analysis. With Shannon entropy denoted by U, it equals U(model) - U(data): the constraint the model fails to capture. For the independence structure it reduces to the total correlation, and for the saturated structure (all variables together) it is zero.

Parameters:

• dist (Distribution) – The distribution whose decomposition error is computed.

• structure (list of lists, None) – The marginal model: a list of marginals (each a set of random variables, or “projection”) to hold fixed, e.g. [[0, 1], [1, 2]] for the structure AB:BC. If None, the independence model (each variable on its own) is used, recovering the total correlation.

Returns:

T – The transmission (decomposition error).

Return type:

float

Examples

>>> d = dit.example_dists.Xor()
>>> dit.multivariate.transmission(d)
1.0
>>> dit.multivariate.transmission(d, [[0, 1], [1, 2]])
1.0
>>> dit.multivariate.transmission(d, [[0, 1, 2]])
0.0

degrees_of_freedom(dist, structure=None)

Returns the degrees of freedom of a marginal model: the number of free parameters needed to specify the maximum entropy distribution consistent with the marginals in structure.

This is the “complexity” of a reconstructability-analysis model. It is the rank of the marginal constraint matrix (which includes the normalization constraint) minus one for that normalization constraint.

Parameters:

• dist (distribution) – The distribution defining the sample space and the marginal values.

• structure (sequence, None) – A sequence whose elements are also sequences, each specifying a marginal (a “projection”) to constrain, e.g. [[0, 1], [1, 2]] for AB:BC. If None, the independence model (each variable on its own) is used.

Returns:

df – The degrees of freedom of the model.

Return type:

int

Examples

>>> d = dit.uniform_distribution(3, 2)
>>> dit.algorithms.degrees_of_freedom(d, [[0, 1], [2]])
4
>>> dit.algorithms.degrees_of_freedom(d, [[0, 1], [1, 2]])
5