Partial Information Decomposition
The partial information decomposition (PID), put forth by Williams & Beer [WB10], is a framework for decomposing the information shared between a set of variables we will refer to as inputs, \(X_0, X_1, \ldots\), and another random variable we will refer to as the output, \(Y\). This decomposition seeks to partition the information \(\I{X_0,X_1,\ldots : Y}\) among the antichains of the inputs.
Background
It is often desirable to determine how a set of inputs influence the behavior of an output. dit.pid.distributions ships two dictionaries of standard test cases from the PID literature: bivariates (two inputs; e.g. 'and', 'synergy', 'redundant', 'unique 1', 'rdn xor', 'reduced or', 'sum', 'cat', 'boom') and trivariates (three inputs; e.g. 'synergy', 'redundant', 'dbl xor', 'giant xor', 'anddup'); sorted(bivariates) lists them all. Consider the exclusive or logic gates, for example:
In [1]: from dit.pid.distributions import bivariates, trivariates
In [2]: xor = bivariates['synergy']
In [3]: print(xor)
Class: Distribution
Alphabet: (('0', '1'), ('0', '1'), ('0', '1'))
Base: linear
x p(X0,X1,X2)
('0', '0', '0') 0.25
('0', '1', '1') 0.25
('1', '0', '1') 0.25
('1', '1', '0') 0.25
We can see from inspection that either input (the first two indexes) is independent of the output (the final index), yet the two inputs together determine the output. One could call this “synergistic” information. Next, consider the giant bit distribution:
In [4]: gb = bivariates['redundant']
In [5]: print(gb)
Class: Distribution
Alphabet: (('0', '1'), ('0', '1'), ('0', '1'))
Base: linear
x p(X0,X1,X2)
('0', '0', '0') 0.5
('1', '1', '1') 0.5
Here, we see that either input informs us of exactly what the output is. One could call this “redundant” information. Furthermore, consider the Co-Information of these distributions:
In [6]: from dit.multivariate import coinformation as I
In [7]: I(xor)
Out[7]: -1.0
In [8]: I(gb)
Out[8]: 1.0
This could lead one to intuit that negative values of the coinformation correspond to synergistic effects in a distribution, while positive values correspond to redundant effects. This intuition, however, is at best misleading: the coinformation of a 4-variable giant bit and 4-variable parity distribution are both positive:
In [9]: I(dit.example_dists.giant_bit(4, 2))
Out[9]: 1.0
In [10]: I(dit.example_dists.n_mod_m(4, 2))
Out[10]: 1.0
This, as well as other issues, lead Williams & Beer [WB10] to propose the partial information decomposition.
Framework
The goal of the partial information is to assign to each some non-negative portion of \(\I{\{X_i\} : Y}\) to each antichain over the inputs. An antichain over the inputs is a set of sets, where each of those sets is not a subset of any of the others. For example, \(\left\{ \left\{X_0, X_1\right\}, \left\{X_1, X_2\right\} \right\}\) is an antichain, but \(\left\{ \left\{X_0, X_1\right\}, \left\{X_0 X_1, X_2\right\} \right\}\) is not.
The antichains for a lattice based on this partial order:
From here, we wish to find a redundancy measure, \(\Icap{\bullet}\) which would assign a fraction of \(\I{\{X_i\} : Y}\) to each antichain intuitively quantifying what portion of the information in the output could be learned by observing any of the sets of variables within the antichain. In order to be a viable measure of redundancy, there are several axioms a redundancy measure must satisfy.
Bivariate Lattice
Let us consider the special case of two inputs. The lattice consists of four elements: \(\left\{\left\{X_0\right\}, \left\{X_1\right\}\right\}\), \(\left\{\left\{X_0\right\}\right\}\), \(\left\{\left\{X_1\right\}\right\}\), and \(\left\{\left\{X_0, X_1\right\}\right\}\). We can interpret these elements as the redundancy provided by both inputs, the information uniquely provided by \(X_0\), the information uniquely provided by \(X_1\), and the information synergistically provided only by both inputs together. Together these for elements decompose the input-output mutual information:
Furthermore, due to the self-redundancy axiom (described ahead), the single-input mutual informations decomposed in the following way:
Colloquially, from input \(X_0\) one can learn what is redundantly provided by either input, plus what is uniquely provided by \(X_0\), but not what is uniquely provided by \(X_1\) or what can only be learned synergistically from both inputs.
Axioms
The following three axioms were provided by Williams & Beer.
Symmetry
The redundancy \(\Icap{X_{0:n} : Y}\) is invariant under reorderings of \(X_i\).
Self-Redundancy
The redundancy of a single input is its mutual information with the output:
Monotonicity
The redundancy should only decrease with in inclusion of more inputs:
with equality if \(\mathcal{A}_{k-1} \subseteq \mathcal{A}_k\).
There have been other axioms proposed following from those of Williams & Beer.
Identity
The identity axiom [HSP13] states that if the output is identical to the inputs, then the redundancy is the mutual information between the inputs:
Target (output) Monotonicity
This axiom states that redundancy can not increase when replacing the output by a function of itself.
It first appeared in [BROJ13] and was expanded upon in [RBO+17].
Source Chain Rule
If a context \(C\) of sources is known, collections that each contain it share all of its information, plus what the remaining collections share once it is known [Gut26]:
where the conditional redundancy averages the redundancy under each \(p(\cdot | C = c)\). It holds exactly when conditioning on the context regroups the atoms by the further sources that unlock them, that is, when the conditional and contextual decompositions agree (see Conditional and Contextual Decompositions). With three or more sources it is incompatible with nonnegative atoms.
These and the other properties proposed in the literature can be checked on any decomposition; see Checking Properties.
Measures
We now turn our attention a variety of methods proposed to flesh out this partial information decomposition.
In [11]: from dit.pid import *
Each measure is a class, constructed from a distribution and, optionally, the sources and the target: by default every variable but the last is a source, and the last is the target. Measures differ in what they quantify directly:
a redundancy gives every node of the lattice, so the decomposition is complete for any number of sources;
a pointwise redundancy is the average of a redundancy of each outcome;
pointwise=Truealso gives the decomposition of each outcome;a bivariate redundancy is defined for two collections of sources;
a unique information is the information a source provides beyond all the others;
a union information is what any of several collections of sources provide; the redundancies follow by inclusion-exclusion.
Measures |
Quantity |
Computed by |
|---|---|---|
redundancy |
closed form |
|
pointwise |
closed form |
|
redundancy |
maximum entropy |
|
redundancy |
linear programs over extreme rays |
|
redundancy |
optimization under sampled constraints (approximate) |
|
redundancy |
search over partitions of the outcomes (small distributions) |
|
union information |
concave envelopes of ROC curves |
|
union information |
optimization |
|
bivariate |
closed form |
|
bivariate |
exact |
|
bivariate |
convex optimization |
|
bivariate |
nonconvex optimization from several starting points |
|
unique information |
convex optimization (exponential cone or admUI) |
|
unique information |
maximum entropy |
|
|
unique information |
secret key agreement rates and their bounds |
Decompositions by bivariate redundancies and unique informations are complete for two sources. For more, only some redundancies are computed. The atoms they determine are exact. Others follow from self-redundancy and the monotonicity of redundancy on the lattice, provided the atoms so obtained are nonnegative, as monotonicity assumes. The remaining atoms are nan. pid.complete tells whether every atom is known, and pid.consistent whether the known values satisfy the Möbius inversion; some unique informations, such as the secret key agreement rates and \(\Ira{\bullet}\), give inconsistent decompositions. Here the bivariate \(\Ido{\bullet}\) leaves the redundancy of the three sources, and the atoms that depend on it, unknown:
In [12]: pid = PID_Do(trivariates['anddup'])
In [13]: pid.get_red(((0,), (1,), (2,))), pid.get_pi(((1,),)), pid.complete
Out[13]: (nan, 0.22957395851362258, False)
Options of a measure are passed to its class, and an option the measure does not take is an error.
\(\Imin{\bullet}\)
\(\Imin{\bullet}\)[WB10] was Williams & Beer’s initial proposal for a redundancy measure. It is given by:
However, this measure has been criticized for acting in an unintuitive manner [GK14]:
In [14]: d = dit.Distribution(['000', '011', '102', '113'], [1/4]*4)
In [15]: PID_WB(d)
Out[15]: <dit.pid.measures.imin.PID_WB object at 0x7a17a09054f0>
We have constructed a distribution whose inputs are independent random bits, and whose output is the concatenation of those inputs. Intuitively, the output should then be informed by one bit of unique information from \(X_0\) and one bit of unique information from \(X_1\). However, \(\Imin{\bullet}\) assesses that there is one bit of redundant information, and one bit of synergistic information. This is because \(\Imin{\bullet}\) quantifies redundancy as the least amount of information one can learn about an output given any single input. Here, however, the one bit we learn from \(X_0\) is, in a sense, orthogonal from the one bit we learn from \(X_1\). This observation has lead to much of the follow-on work.
- class PID_WB(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]
The Williams & Beer partial information decomposition.
\(\Immi{\bullet}\)
One potential measure of redundancy is the minimum mutual information [BROJ13]:
This measure, though crude, is known to be correct for multivariate gaussian variables [OBR15].
- class PID_MMI(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]
The minimum mutual information partial information decomposition.
This measure is known to be accurate for gaussian random variables. It was also briefly and tangentially studied by the BROJA team.
\(\Iwedge{\bullet}\)
Redundancy seems to intuitively be related to common information Common Informations. This intuition lead to the development of \(\Iwedge{\bullet}\) [GCJ+14]:
That is, redundancy is the information the Gács-Körner Common Information of the inputs shares with the output.
Warning
This measure can result in a negative PID.
- class PID_GK(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]
The Griffith et al partial information decomposition.
This PID is known to produce negative partial information values.
\(\Iproj{\bullet}\)
Utilizing information geometry, Harder et al [HSP13] have developed a strictly bivariate measure of redundancy, \(\Iproj{\bullet}\):
where
where \(C_{cl}(\bullet)\) denotes closure. Intuitively, this measures seeks to quantify redundancy as the minimum of how much \(p(Y | X_0)\) can be expressed when \(X_0\) is projected on to \(X_1\), and vice versa.
- class PID_Proj(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]
The Harder et al partial information decomposition.
\(\Ibroja{\bullet}\)
In a very intuitive effort, Bertschinger et al (henceforth BROJA) [BRO+14, GK14] defined unique information as the minimum conditional mutual informations obtainable while holding the input-output marginals fixed:
For bivariate sources (two inputs), PID_BROJA accepts a method keyword selecting how the marginal-matching optimization is solved:
'scipy'— SLSQP on the free joint pmf parameters. It can stop well short of the minimum, overstating the unique information, so'auto'does not use it.'admui'— alternating divergence minimization [BRMontufar17]. If it has not converged withinmaxiteriterations, its last iterate, which matches the marginals, is used.'cone'— exponential cone program solved with ECOS [MTV18] (requirespip install dit[broja]).'auto'(the default) — picksconeif ECOS is installed andadmuiotherwise, falling back to the other methods if a solver fails.
Note
In the bivariate case, Griffith independently suggested the same decomposition but from the viewpoint of synergy [GK14].
The BROJA measure has recently been criticized for behaving in an unintuitive manner on some examples. Consider the reduced or distribution:
In [16]: bivariates['reduced or']
Out[16]: <Distribution p(X0,X1,X2)>
In [17]: print(PID_BROJA(bivariates['reduced or']))
+---------------------------+
| I_broja |
+---------+--------+--------+
| I_broja | I_r | pi |
+---------+--------+--------+
| {0:1} | 1.0000 | 0.6887 |
| {0} | 0.3113 | 0.0000 |
| {1} | 0.3113 | 0.0000 |
| {0}{1} | 0.3113 | 0.3113 |
+---------+--------+--------+
We see that in this instance BROJA assigns no partial information to either unique information. However, it is not difficult to argue that in the case that either input is a 1, that input then has unique information regarding the output.
\(\Iproj{\bullet}\) and \(\Ibroja{\bullet}\) are Distinct
In the BROJA paper [BRO+14] the only example given where their decomposition differs from that of Harder et al. is the dit.example_dists.summed_dice(). We can find a simpler example where they differ using hypothesis:
In [18]: from hypothesis import find
In [19]: from dit.utils.testing import distribution_structures
In [20]: find(distribution_structures(3, 2, True), lambda d: PID_Proj(d) != PID_BROJA(d))
---------------------------------------------------------------------------
DeadlineExceeded Traceback (most recent call last)
Cell In[20], line 1
----> 1 find(distribution_structures(3, 2, True), lambda d: PID_Proj(d) != PID_BROJA(d))
File ~/checkouts/readthedocs.org/user_builds/dit/envs/latest/lib/python3.12/site-packages/hypothesis/core.py:2507, in find(specifier, condition, settings, random, database_key)
2504 test._hypothesis_internal_database_key = database_key # type: ignore
2506 try:
-> 2507 test()
2508 except Found:
2509 return last[0]
File ~/checkouts/readthedocs.org/user_builds/dit/envs/latest/lib/python3.12/site-packages/hypothesis/core.py:2495, in find.<locals>.test()
2490 specifier.validate()
2492 last: list[Ex] = []
2494 @settings
-> 2495 @given(specifier)
2496 def test(v):
2497 if condition(v):
2498 last[:] = [v]
[... skipping hidden 1 frame]
File ~/checkouts/readthedocs.org/user_builds/dit/envs/latest/lib/python3.12/site-packages/hypothesis/core.py:1040, in StateForActualGivenExecution._proxied_test.<locals>.test(*args, **kwargs)
1038 current_deadline = (current_deadline // 4) * 5
1039 if runtime >= current_deadline.total_seconds():
-> 1040 raise DeadlineExceeded(
1041 datetime.timedelta(seconds=runtime), self.settings.deadline
1042 )
1043 return result
DeadlineExceeded: Test took 1171.48ms, which exceeds the deadline of 200.00ms. If you expect test cases to take this long, you can use @settings(deadline=...) to either set a higher deadline, or to disable it with deadline=None.
Failing test case: test(
# The test sometimes passed when commented parts were varied together.
v=<Distribution p(X0,X1,X2)>,
)
- class PID_BROJA(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]
The BROJA partial information decomposition.
Notes
This partial information decomposition, at least in the bivariate source case, was independently suggested by Griffith.
For two sources,
methodselects the bivariate solver:'scipy'— SLSQP on the marginal-matching polytope, which can stop short of the minimum'admui'— alternating divergence minimization ([BRMontufar17])'cone'— exponential cone program via ECOS ([MTV18])'auto'—'cone'if ECOS is installed and'admui'otherwise, with fallbacks
With more than two sources, the unique information of each source is its unique information in the bivariate decomposition of it and the union of the other sources, computed by the same solver.
\(\Iccs{\bullet}\)
Taking a pointwise point of view, Ince has proposed a measure of redundancy based on the Co-Information [Inc17a]:
While this measure behaves intuitively in many examples, it also assigns negative values to some partial information atoms in some instances.
This decomposition also displays an interesting phenomena, that of subadditive redundancy. The gband distribution is an independent mix of a giant bit (redundancy of 1 bit) and the and distribution (redundancy of 0.1038 bits), and yet gband has 0.8113 bits of redundancy:
In [21]: PID_CCS(bivariates['gband'])
Out[21]: <dit.pid.measures.iccs.PID_CCS object at 0x7a1797f611c0>
Warning
This measure can result in a negative PID.
- class PID_CCS(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]
The common change in surprisal partial information decomposition, as defined by Ince.
\(\Idep{\bullet}\)
James et al [JEC17] have developed a method of quantifying unique information based on the Dependency Decomposition. Unique information from variable \(X_i\) is evaluated as the least change in sources-target mutual information when adding the constraint \(X_i Y\).
In [22]: PID_dep(bivariates['not two'])
Out[22]: <dit.pid.measures.idep.PID_dep object at 0x7a1797f84350>
- class PID_dep(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]
The dependency partial information decomposition, as defined by James at al.
\(\Ipm{\bullet}\)
Also taking a pointwise view, Finn & Lizier’s \(\Ipm{\bullet}\) [FL18] instead splits the pointwise mutual information into two components:
They then define two partial information lattices, one quantified locally by \(h(s)\) and the other by \(h(s|t)\). By averaging these local lattices and then recombining them, we arrive at a standard Williams & Beer redundancy lattice.
In [23]: PID_PM(bivariates['pnt. unq.'])
Out[23]: <dit.pid.measures.ipm.PID_PM object at 0x7a1797f88dd0>
Warning
This measure can result in a negative PID.
- class PID_PM(dist, sources=None, target=None, reds=None, pis=None, compute=True, pointwise=False, **kwargs)[source]
The Finn & Lizier partial information decomposition (I_±).
The measure decomposes into: * h_s (specificity / plus): min over sources of the source surprisal * h_s|t (ambiguity / minus): min over sources of the conditional surprisal
When constructed with
pointwise=True, also computes the per-outcome pointwise PID and the specificity / ambiguity decomposition.
\(\Isx{\bullet}\)
\(\Irav{\bullet}\)
Taking a functional perspective as in \(\Iwedge{\bullet}\), \(\Irav{\bullet}\) defines bivariate redundancy as the maximum coinformation between the two sources \(X_0, X_1\), a target \(Y\), and a deterministic function of the inputs \(f(X_0,X_1)\).
This measure is designed to exploit the conflation of synergy and redundancy in the three variable coinformation: \(\I{X_0\!:\!X_1\!:\!Y} = R - S\).
Note
TODO: reference. No canonical publication for \(\Irav{\bullet}\) has been identified in the dit source; a citation should be added if/when the origin paper is confirmed.
In [24]: PID_RAV(bivariates['pnt. unq.'])
Out[24]: <dit.pid.measures.irav.PID_RAV object at 0x7a1797f84080>
- class PID_RAV(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]
The maximum coinformation auxiliary random variable method.
\(\Irr{\bullet}\)
In order to combine \(\Immi{\bullet}\) with the coinformation, Goodwell and Kumar [GK17] have introduced their rescaled redundancy:
In [25]: PID_RR(bivariates['pnt. unq.'])
Out[25]: <dit.pid.measures.irr.PID_RR object at 0x7a1797f62930>
- class PID_RR(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]
The rescaled redundancy of Goodwell & Kumar.
Redundancy interpolates between max(0, coinformation) and the minimum mutual information, in proportion to the normalized dependence I(X_0 : X_1) / min_i H(X_i) between the sources.
\(\Ira{\bullet}\)
Drawing from the reconstructability analysis work of Zwick [Zwi04], we can define \(Ira{\bullet}\) as a restricted form of \(\Idep{\bullet}\).
Warning
This measure can result in a negative PID.
- class PID_RA(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]
The “reproducibility analysis” partial information decomposition, derived from the work of Zwick.
\(\Irdr{\bullet}\)
The measure of Mages & Rohner [MR23] can be interpreted as a pointwise version of \(\Ibroja{\bullet}\) to provide a non-negative partial information decomposition for an arbitrary number of sources.
It obtains its operational interpretation from valuating the reachable decision regions (achievable type I/II error pairs) for each state of the target variable.
For each target outcome \(y\), each source defines a binary-input channel for the event \(Y = y\), and its reachable decision region is bounded by its ROC curve.
The union information of a collection of sources is the pointwise information of the convex hull of their regions (their join), averaged over \(y\), and redundancies follow from the union information of sub-collections by inclusion-exclusion [MAR24, MR23].
dit computes each join as the upper concave envelope of the ROC curves, and computes the union information of each collection once for the whole lattice.
- class PID_RDR(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]
The reachable decision regions partial information decomposition of Mages & Rohner.
For each target outcome, the union information of a collection of sources is the pointwise information of the join of their reachable decision regions, the convex hull of their ROC curves. Redundancies follow by inclusion-exclusion, and the union information of each collection is computed once for the whole lattice.
\(\Ict{\bullet}\)
Sigtermans [Sig20] proposes a bivariate redundancy measure based on causal tensors and path-based information flow between the two sources and the target.
- class PID_CT(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]
The bivariate PID defined by Sigtermans using causal tensors.
\(\Ideg{\bullet}\)
The degradation intersection information of Kolchinsky [Kol22] (further developed by Gomes & Figueiredo [GF23]) defines redundancy as the maximum mutual information carried by a channel that is a Blackwell degradation of every source channel:
This is the Blackwell redundancy \(\Iprec{\bullet}\) in the notation of Gomes & Figueiredo, for any number of sources.
The feasible degradation channels form a polytope and \(\I{Q : Y}\) is convex, but the maximum can be found without enumerating the vertices of that polytope.
Each outcome \(q\) of \(Q\) corresponds to the vector of its degradation probabilities \((p(q \mid x_0), \ldots, p(q \mid x_{n-1}))\), which lies in a polyhedral cone.
These vectors add up to the all-ones vector, and \(\I{Q : Y}\) is a sum over the outcomes of a convex, positively homogeneous function of them.
Splitting an outcome along the extreme rays of the cone therefore never decreases \(\I{Q : Y}\), so the redundancy is the value of a linear program over the extreme rays, whose solution is also an optimal \(Q\).
This formulation is due to André F. C. Gomes.
dit enumerates the rays with cddlib when it is installed (from the optional extra), and otherwise with an exact implementation of the double description method.
Passing method='local' instead runs a local search over the vertices of the polytope, which only gives a lower bound.
- class PID_Deg(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]
Degradation intersection information I_d^∩ from Kolchinsky (2022).
Redundancy is the maximum I(Q; T) such that Q is a Blackwell degradation of every source channel, for any number of sources. This is the Blackwell redundancy of
PID_Prec, in the notation of Gomes & Figueiredo. Satisfies the Williams-Beer axioms and the independent identity property (IIP).By default the maximum is computed exactly, by a linear program over the extreme rays of the cone of common degradations (a formulation due to A. F. C. Gomes). The rays are enumerated by cddlib (from the
optionalextra) when it is installed, and otherwise by an exact implementation of the double description method. Passingmethod='local'instead runs a local search over the vertices of the polytope of degradation channels, which only gives a lower bound.References
A. Kolchinsky, “A Novel Approach to the Partial Information Decomposition”, Entropy 24, 403, 2022.
A. F. C. Gomes and M. A. T. Figueiredo, “Orders between Channels and Implications for Partial Information Decomposition”, Entropy 25, 975, 2023.
\(\Ici{\bullet}\)
Gomes & Figueiredo [GF24] define a union information rather than a redundancy. Each source \(A_i\) contributes its channel \(p(a_i \mid y)\); by sharing channels (and marginals thereof) the sources can build any distribution \(q(a, y) = p(y) \prod_{B \in \pi} p(a_B \mid y)\), where \(\pi\) partitions the variables of \(A = \bigcup_i A_i\) into blocks that each lie within some source. The union information is the best such \(q\) can do, capped at the true mutual information:
Synergy is the information lost by assuming the sources are conditionally independent given the target, \(S^{CI} = \I{A : Y} - \Ici{A_0, \ldots, A_{m-1} \to Y} \geq 0\). Sources contained in, or a deterministic function of, other sources are removed first.
PID_CI is built on BaseSynergyPID, which turns a synergy (equivalently, union information) into redundancies by inclusion-exclusion over sub-collections of each antichain, as in the parthood formulation of Gutknecht et al. [GWM21]:
In [26]: from dit.pid import PID_CI
In [27]: PID_CI(bivariates['and'])
Out[27]: <dit.pid.measures.ici.PID_CI object at 0x7a1797fd5730>
In [28]: PID_CI(bivariates['and']).get_syn(((0,), (1,)))
Out[28]: 0.2704260414863776
- class PID_CI(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]
The conditional-independence union information I_∪^CI of Gomes & Figueiredo (2024).
Synergy is the information lost by assuming the sources are conditionally independent given the target; redundancies follow by inclusion-exclusion over union informations.
References
A. F. C. Gomes and M. A. T. Figueiredo, “A Measure of Synergy Based on Union Information”, Entropy 26, 271, 2024.
\(\Idelta{\bullet}\)
The \(\delta\)-PID of Banerjee, Olbrich, Jost & Rauh [BOJR18] quantifies unique information via the weighted output KL deficiency, the cost of approximating one source channel from the other via output randomization:
Redundancy is then min-symmetrized across the two sources.
- class PID_Delta(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]
The delta-PID based on weighted output KL deficiency.
The deficiency
delta(M : X \ Y)measures how well Y can simulate X for making decisions about M, via the expected KL divergence between the true channelP(X|M)and the best approximationP(X'|Y)composed withP(Y|M).Redundancy is symmetrized via the minimum:
RI = min{ I(M;X) - delta(M:X\Y), I(M;Y) - delta(M:Y\X) }
References
P. K. Banerjee et al., “Unique informations and deficiencies”, Allerton Conference, 2018.
\(\IdeltaLambda{\bullet}\)
Venkatesh, Gurushankar & Schamberg [VGS23] introduce a Lagrangian generalization that smoothly interpolates between the \(\delta\)-PID of [BOJR18] and the BROJA PID of [BRO+14]:
As \(\lambda \to \infty\) the measure recovers \(\Idelta{\bullet}\). As \(\lambda \to 0\), \(\delta_\lambda / \lambda\) tends to the unique information of \(\Ibroja{\bullet}\); \(\delta_\lambda\) itself vanishes, so the redundancy tends to that of \(\Immi{\bullet}\).
- class PID_DeltaLambda(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]
The delta-lambda PID from Venkatesh, Gurushankar & Schamberg (2023).
A Lagrangian generalization that interpolates between the delta-PID and the BROJA PID via a parameter lambda:
delta_lambda(M : X \ Y) = inf_{P(X'|MY)} E_M[D_KL(P(X|M) || P(X'|M))] + lambda * I(M; X' | Y)
Redundancy is symmetrized:
RI = min{ I(M;X) - delta_lambda(M:X\Y), I(M;Y) - delta_lambda(M:Y\X) }
The following parameters are passed as keyword arguments to the constructor.
- Parameters:
lam (float) – Lagrange multiplier (default 1.0). Large values approach the delta-PID. As lam tends to 0, the deficiency divided by lam tends to the BROJA unique information, while the redundancy tends to that of the MMI.
bound (int, None) – Bound on auxiliary variable cardinality.
niter (int, None) – Number of optimization restarts.
\(\Igh{\bullet}\)
Griffith & Ho [GH15] define redundancy, their \(I_\alpha\), as the most information about the target carried by a variable that is independent of the target given each source:
where \(Q\) may depend on the sources and the target jointly.
Each condition \(Q - X_i - Y\), that is \(p(q \mid x_i, y) = p(q \mid x_i)\), is linear in the channel \(p(q \mid x_{0:n}, y)\) [Kol22], so as for \(\Ideg{\bullet}\) the redundancy is the value of a linear program over the extreme rays of a polyhedral cone.
The optimal \(Q\) can need more values than the sources have; on ‘erase’ it needs four.
Passing method='optimizer' instead runs a local optimization of a bounded auxiliary variable, which only gives a lower bound.
Conditional independence implies the Blackwell order and is implied by a common function, so \(\Iwedge{\bullet} \leq \Igh{\bullet} \leq \Iprec{\bullet}\). Unlike the Blackwell order it is not transitive: \(Q - X_1 - Y\) does not imply \(Q - (X_1, X_2) - Y\), so adding a source that contains another can lower the redundancy, and with three sources the redundancies need not be monotone on the lattice.
- class PID_GH(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]
The I_GH measure defined by Griffith & Ho.
Redundancy is the maximum I(Q; Y) over variables Q which are independent of the target given each source. By default it is computed exactly, by a linear program over the extreme rays of the cone of such outcomes of Q. Passing
method='optimizer'instead runs a local optimization of a bounded auxiliary variable, which only gives a lower bound.References
V. Griffith and T. Ho, “Quantifying Redundant Information in Predicting a Target Random Variable”, Entropy 17, 4644, 2015.
\(\Iig{\bullet}\)
Niu & Quinn [NQ19] propose an information-geometric PID: the synergy is defined as the KL divergence from the true joint distribution to the nearest distribution satisfying \(X_0 - X_1 - Y\) or \(X_1 - X_0 - Y\) Markov chains, minimized along the geodesic \(q_\lambda \propto p(x_0, x_1)\, p(y \mid x_0)^\lambda p(y \mid x_1)^{1 - \lambda}\) between them; the minimizing \(\lambda\) need not lie in \([0, 1]\).
- class PID_IG(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]
The Niu & Quinn partial information decomposition.
\(\Iipid{\bullet}\)
The I-PID of Venkatesh, Gurushankar & Schamberg [VGS23] defines unique information via an information deficiency that maximizes the mutual-information gap attainable by a Markov test channel \(T - Y - (X_0, X_1)\):
Redundancy is obtained by min-symmetrization, analogously to \(\Idelta{\bullet}\).
- class PID_IPID(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]
The I-PID from Venkatesh, Gurushankar & Schamberg (2023).
- Unique information is defined via information deficiency:
delta_I(M : X Y) = sup_{P(T|M)} [ I(T;X) - I(T;Y) ]
- and redundancy is symmetrized:
RI_I = min{ I(M;X) - delta_I(M:XY), I(M;Y) - delta_I(M:YX) }
\(\Imc{\bullet}\)
The more-capable intersection information of Gomes & Figueiredo [GF23] (building on [Kol22]) replaces the Blackwell-degradation order in \(\Ideg{\bullet}\) with the weaker more-capable channel preorder:
where \(Q \preceq_{\mathrm{mc}} X_i\) iff \(\I{Q : Y} \leq \I{X_i : Y}\) for every input distribution on \(Y\).
- class PID_MC(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]
More-capable intersection information I_mc^∩ from Gomes & Figueiredo.
Redundancy is the maximum I(Q; T) such that K_Q is “less capable” than every source channel for all input distributions. Defined for any number of sources.
The more-capable constraints are imposed at sampled input distributions and the maximum is searched locally, from several starts that include an optimal common degradation of the sources. The result is therefore at least the degradation redundancy of
PID_Deg, but is not a certified bound on I_mc^cap. The search is seeded by rng (0 by default), so results are reproducible.References
A. F. C. Gomes and M. A. T. Figueiredo, “Orders between Channels and Implications for Partial Information Decomposition”, Entropy 25, 975, 2023.
\(\Iln{\bullet}\)
The less-noisy intersection information of Gomes & Figueiredo [GF23] uses the less noisy channel preorder, which sits between the Blackwell-degradation and more-capable orders, so \(\Ideg{\bullet} \leq \Iln{\bullet} \leq \Imc{\bullet}\):
where \(Q \preceq_{\mathrm{ln}} X_i\) iff \(\I{U : Q} \leq \I{U : X_i}\) for every \(U\) with \(U - Y - (Q, X_i)\). Equivalently, the one-way secret key agreement rate from \(Y\) to \(Q\) with \(X_i\) eavesdropping vanishes, so \(\Iln{\bullet}\) is the most information about the target that cannot support secret key agreement against any single source.
By [vD97], \(Q \preceq_{\mathrm{ln}} X_i\) iff \(\I{Y : X_i} - \I{Y : Q}\) is concave in the distribution of \(Y\).
dit imposes the equivalent pointwise condition on the Hessians of the output entropies at sampled distributions and directions, so the computed value is approximate.
No bound on \(|Q|\) is known; it defaults to the largest source alphabet.
Warning
This measure appears to result in a negative PID. For \(p_Y = (0.3, 0.55, 0.15)\) with \(p(x_0 | y)\) rows \((0.2, 0.5, 0.3), (0, 0, 1), (0.1, 0.9, 0)\) and \(p(x_1 | y)\) rows \((0.1, 0, 0.9), (0.05, 0.5, 0.45), (0.05, 0.5, 0.45)\), \(\Iln{\bullet} \approx 0.021\) bits, below the coinformation \(0.6925 + 0.2341 - 0.8836 = 0.043\) bits, so the synergy is about \(-0.022\) bits. More generally, since \(\Iln{\bullet}\) depends only on the source-target marginals, it gives a nonnegative synergy on every distribution with those marginals only if it is at least \(\Ibroja{\bullet}\) [BRO+14] (Lemma 3), which here is \(0.050\) bits.
- class PID_LN(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]
Less-noisy intersection information I_ln^∩ from Gomes & Figueiredo.
Redundancy is the maximum I(Q; T) over channels K_Q such that every source channel is less noisy than K_Q; equivalently, no one-way secret key can be agreed from T to Q against any single source. Defined for any number of sources.
References
A. F. C. Gomes and M. A. T. Figueiredo, “Orders between Channels and Implications for Partial Information Decomposition”, Entropy 25, 975, 2023.
\(\Imes{\bullet}\)
\(\Imes{\bullet}\) is a maximum-entropy PID inspired by BROJA’s “*” marginal-consistency assumption: the redundancy is the coinformation of the sources and the target under the maximum entropy distribution with the same source-target marginals,
The sources are independent given the target under \(q\), so for two sources the redundancy is \(I_q(X_0 : X_1)\), the same as that of \(\Ido{\bullet}\). James, Emenheiser & Crutchfield [JEC19] discuss this distribution as the maximum entropy reading of assumption *, and Liardi et al. [LDB+26] attribute the measure to them; it was not proposed as a complete decomposition, and it does not satisfy subset equality.
- class PID_MES(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]
The maximum entropy PID inspired by BROJA’s * assumption.
Redundancy is the coinformation of the sources and the target under the maximum entropy distribution with the same source-target marginals, q = p(t) prod_i p(x_i | t). The sources are independent given the target under q, so for two sources this is I_q(X_0 : X_1), which is also the redundancy of
PID_Do.
\(\Iprec{\bullet}\)
Kolchinsky [Kol22] defines a redundancy measure using the Blackwell order on channels:
This is the same quantity as \(\Ideg{\bullet}\), for any number of sources, and by default it is computed by the same linear program over the extreme rays of the cone of common degradations.
Passing n_q fixes the cardinality of \(Q\).
Every outcome of \(Q\) is a combination of extreme rays of that cone, and the objective is convex, so the maximum is attained at a vertex of the linear program’s feasible polytope with the rays in its support grouped into the outcomes of \(Q\).
Splitting an outcome never lowers the objective, so only the partitions of each support into n_q groups need be checked, and from the cardinality of Theorem A1 on, the value is that of the linear program.
The vertices are enumerated in exact arithmetic.
- class PID_Prec(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]
The \(I_{\prec}\) measure defined by Kolchinsky.
This is the same measure as
PID_Deg, the degradation intersection information of Gomes & Figueiredo. By default it is computed exactly, by a linear program over the extreme rays of the cone of common degradations. Passingn_qfixes the cardinality of the redundancy variable; the maximum is then found exactly from the vertices of that program’s feasible polytope.
\(\Ido{\bullet}\)
Lyu, Clark & Raviv [LCR24] give an explicit bivariate redundancy via Pearl-style do-calculus. An intervened source \(X'_i\) is defined by
and the redundancy is \(I(X'_1 : X_2) = I(X'_2 : X_1)\). Only defined for two sources.
In [29]: from dit.pid import PID_Do
In [30]: print(PID_Do(bivariates['synergy']))
+--------------------------+
| I_do |
+--------+--------+--------+
| I_do | I_r | pi |
+--------+--------+--------+
| {0:1} | 1.0000 | 1.0000 |
| {0} | 0.0000 | 0.0000 |
| {1} | 0.0000 | 0.0000 |
| {0}{1} | 0.0000 | 0.0000 |
+--------+--------+--------+
- class PID_Do(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]
The do-calculus partial information decomposition of Lyu, Clark & Raviv.
Notes
Bivariate only. The paper explicitly notes that I_do only fully determines the PID for n = 2 sources.
Synergistic Disclosure
Rosas, Mediano, Rassouli & Barrett [RMRB20] decompose \(I(X;Y)\) via \(\alpha\)-synergistic channels (see also Synergistic Disclosure). SynDisc inverts on the extended constraint lattice; ModifiedSynDisc is the backbone coarse-graining.
- class ModifiedSynDisc(dist, sources=None, target=None, niter=None, bound=None)[source]
Modified synergistic disclosure decomposition.
Gutknecht, Makkeh & Wibral (2023), “From Babel to Boole: The Logical Organization of Information Decompositions”, arXiv:2306.00734v2.
Identical to SynDisc except that singleton constraint nodes (
|alpha| = 1) use conditional mutual information I(T : a^C | a) instead of the optimization-based synergistic disclosure. This enforces the PID consistency condition: for every source collection a, the sum of atoms whose parthood distribution assigns 1 to a equals I(a : T).- Parameters:
dist (Distribution) – The joint distribution over sources and target.
sources (iter of iters, None) – The source variable groups. If None, all variables except the last are treated as individual sources.
target (iter, None) – The target variable. If None, the last variable is used.
niter (int, None) – Number of basin-hopping restarts per lattice node.
bound (int, None) – Cardinality bound on the auxiliary variable V.
- class SynDisc(dist, sources=None, target=None, niter=None, bound=None)[source]
Synergistic disclosure decomposition (Rosas et al. 2020).
Decomposes I(X; Y) into atoms via Mobius inversion on the extended constraint lattice. Also provides the backbone decomposition, a non-negative coarse-graining that scales linearly with system size.
- Parameters:
dist (Distribution) – The joint distribution over sources and target.
sources (iter of iters, None) – The source variable groups. If None, all variables except the last are treated as individual sources.
target (iter, None) – The target variable. If None, the last variable is used.
niter (int, None) – Number of basin-hopping restarts per lattice node.
bound (int, None) – Cardinality bound on the auxiliary variable V.
Secret Key Agreement Rates
One can associate Secret Key Agreement rates with unique informations [BG15] by considering the rate at which one source and the target can agree upon a secret key while the other source eavesdrops. The four classes are PID_SKAR_nw (no communication), PID_SKAR_owa (source to target), PID_SKAR_owb (target to source), and PID_SKAR_tw (two-way). This results in four possibilities:
neither source nor target communicate
only the source communicates
only the target communicates
both the source and the target communicate
No Communication
Warning
This measure can result in an inconsistent PID.
One-Way Communication
Camel
Elephant
Warning
This measure can result in an inconsistent PID.
Two-Way Communication
Warning
This measure can result in an inconsistent PID.
- class PID_SKAR_nw(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]
The no-communication secret key agreement rate partial information decomposition.
- class PID_SKAR_owa(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]
The one-way secret key agreement rate partial information decomposition, source to target.
- class PID_SKAR_owb(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]
The one-way secret key agreement rate partial information decomposition, target to source.
- class PID_SKAR_tw(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]
The two-way secret key agreement rate partial information decomposition.
Notes
This method progressively utilizes better bounds on the SKAR, and if even when using the tightest bounds does not result in a singular SKAR, nan is returned.
Experimental Measures
Three variants of \(\Idep{\bullet}\) take the unique information of a source from the dependency lattice of all the sources and the target, rather than of the source, the other sources together, and the target. With two sources PID_dep_a is \(\Idep{\bullet}\); PID_dep_b only considers the edges that never constrain the sources among themselves, and gives inconsistent decompositions; PID_dep_c only those that keep the joint distribution of the sources. No publication describes them, and their lattices grow quickly with the number of sources.
- class PID_dep_a(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]
An experimental variant of the dependency decomposition PID of James et al.
The unique information of a source is the least change in I(sources : target) among the edges of the dependency lattice of all the sources and the target that add the constraint of the source and the target. With two sources this is
PID_dep; with more,PID_depcoalesces the other sources into one variable.Notes
Experimental: this alternative method behaves oddly with three or more sources, and its lattice grows quickly with the number of sources.
- class PID_dep_b(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]
An experimental, reduced variant of the dependency decomposition PID of James et al.
As
PID_dep_a, but only over the edges whose constraints among several variables all involve the target, so that dependencies among the sources alone are never constrained.Notes
Experimental: this decomposition is known to be inconsistent; for instance, its unique informations for two copies of a source are inconsistent with self-redundancy.
- class PID_dep_c(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]
An experimental, reduced variant of the dependency decomposition PID of James et al.
As
PID_dep_a, but only over the edges whose constraints include the joint distribution of all the sources, so that the dependencies among the sources are always kept.Notes
Experimental: this decomposition can result in subadditive redundancy.
Conditional and Contextual Decompositions
When some variables are already known, the information the sources provide about the target beyond them can be decomposed in two ways [Gut26]. The conditional decomposition decomposes the sources under each conditional distribution given the known variables, and averages:
When the known variables are some of the sources, the context, the contextual decomposition instead regroups the atoms of the decomposition of all the sources: the atoms the context provides by itself are dropped, and the others are grouped by the remaining sources that give access to them once the context is known. Its redundancies are
Both decompose \(\I{\mathcal{A} : Y | C}\), and they agree atom by atom exactly when the measure satisfies the source chain rule. On the distribution where a third source decides which of the first two copies the target, \(\Imin{\bullet}\) gives half a bit of contextual redundancy and no conditional redundancy, while \(\Isx{\bullet}\), which satisfies the rule, gives the same decomposition both ways:
In [31]: from dit.pid import ConditionalPID, ContextualPID
In [32]: d = trivariates['max min']
In [33]: print(ContextualPID(PID_WB(d), context=[2]))
+-----------------------------+
| I_min\{2} |
+-----------+--------+--------+
| I_min\{2} | I_r | pi |
+-----------+--------+--------+
| {0:1} | 1.0000 | 0.5000 |
| {0} | 0.5000 | 0.0000 |
| {1} | 0.5000 | 0.0000 |
| {0}{1} | 0.5000 | 0.5000 |
+-----------+--------+--------+
In [34]: print(ConditionalPID(PID_WB, d, [[0], [1]], [3], crvs=[2]))
+-----------------------------+
| I_min|{2} |
+-----------+--------+--------+
| I_min|{2} | I_r | pi |
+-----------+--------+--------+
| {0:1} | 1.0000 | 0.0000 |
| {0} | 0.5000 | 0.5000 |
| {1} | 0.5000 | 0.5000 |
| {0}{1} | 0.0000 | 0.0000 |
+-----------+--------+--------+
In [35]: ContextualPID(PID_SX(d), context=[2]) == ConditionalPID(PID_SX, d, [[0], [1]], [3], crvs=[2])
Out[35]: True
crvs need not be sources: conditioning on any other variables gives the decomposition of a conditional mutual information, such as a transfer entropy.
- class ConditionalPID(measure, dist, sources=None, target=None, crvs=None, **kwargs)[source]
The conditional partial information decomposition (Definition 3.4 of [Gut26]): the decomposition of the information the sources provide about the target given the variables crvs, by decomposing the sources under each conditional distribution given crvs and averaging,
\[\Pi(\alpha; T \mid Z) = \sum_z p(z) \, \Pi^{p(\cdot \mid Z = z)}(\alpha; T).\]Redundancies average the same way, and are related to the atoms by Möbius inversion; the atoms decompose the conditional mutual informations \(I(a; T \mid Z)\).
Examples
Given one of two independent bits, their maximum and minimum each determine the other bit for one of its values, and nothing else:
>>> from dit.pid import ConditionalPID, PID_SX >>> from dit.pid.distributions import trivariates >>> d = trivariates['max min'] # max(C, T), min(C, T), C, T >>> pid = ConditionalPID(PID_SX, d, [[0], [1]], [3], crvs=[2]) >>> pid.get_red(((0,), (1,))), pid.get_pi(((0,),)), pid.get_pi(((1,),)) (0.0, 0.5, 0.5)
- class ContextualPID(pid, context)[source]
The contextual decomposition (Definition 3.1 of [Gut26]) of the information the sources of a decomposition provide about its target beyond a context, some of its sources.
The atoms that the context provides itself, those with an element made of context sources only, are dropped. Erasing the context sources from the elements of every other atom’s label, and keeping the minimal elements, gives the remaining sources that give access to the atom once the context is known: the contextual atom of each such label collects the atoms that map to it. The contextual redundancies (Proposition 3.6) are
\[I_\cap(a_1, \ldots, a_m; T \setminus c) = I_\cap(a_1 \cup c, \ldots, a_m \cup c; T) - I(c; T).\]The contextual decomposition agrees with the conditional one,
ConditionalPID(type(pid), dist, remaining sources, target, crvs=context), exactly when the measure satisfies the source chain rule.Examples
For \(I_{min}\), which does not satisfy the source chain rule, the two decompositions differ (Section 7 of [Gut26]):
>>> from dit.pid import ContextualPID, PID_WB >>> from dit.pid.distributions import trivariates >>> pid = ContextualPID(PID_WB(trivariates['max min']), context=[2]) >>> pid.get_red(((0,), (1,))) 0.5
Partial Entropy Decomposition
Ince [Inc17b] proposed applying the PID framework to decompose multivariate entropy (without considering information about a separate target variable). This partial entropy decomposition (PED), seeks to partition a mutlivariate entropy \(\H{X_0,X_1,\ldots}\) among the antichains of the variables. The PED perspective shows that bivariate mutual information is equal to the difference between redundant entropy and synergistic entropy.
\(\Hcs{\bullet}\)
Taking a pointwise point of view, following \(\Iccs{\bullet}\), Ince has proposed a measure of redundant entropy based on the Co-Information [Inc17b]:
While this measure behaves intuitively in many examples, it also assigns negative values to some partial entropy atoms in some instances. However, Ince [Inc17b] argues that concepts such as mechanistic information redundnacy (non-zero information redundancy between independent predictors, c.f. AND) necessitate negative partial entropy terms.
Like \(\Iccs{\bullet}\), \(\Hcs{\bullet}\) is also subadditive.
In [36]: PED_CS(dit.Distribution(['00','01','10','11'],[0.25]*4))
Out[36]: <dit.pid.hcs.PED_CS object at 0x7a179795cd10>
- class PED_CS(dist, sources=None, **kwargs)[source]
The change in surprisal partial entropy decomposition, as defined by Ince (2017).
\(H_{\mathrm{mos}}\)
Finn & Lizier [FL20] define a partial entropy decomposition whose redundancy functional is the expected minimum of the source-group surprisals:
In [37]: from dit.pid import PED_MOS
In [38]: print(PED_MOS(dit.Distribution(['00','01','10','11'], [0.25]*4)))
+--------------------------+
| H_mos |
+--------+--------+--------+
| H_mos | H_r | H_d |
+--------+--------+--------+
| {0:1} | 2.0000 | 1.0000 |
| {0} | 1.0000 | 0.0000 |
| {1} | 1.0000 | 0.0000 |
| {0}{1} | 1.0000 | 1.0000 |
+--------+--------+--------+
This is the decomposition that later work, such as [VPP+23], calls \(h_{\min}\).
- class PED_MOS(dist, sources=None, pointwise=False, **kwargs)[source]
The min-of-surprisals partial entropy decomposition.
Based on Finn & Lizier (2020), “Generalised Measures of Multivariate Information Content”, Entropy 22(2), 216. https://doi.org/10.3390/e22020216
This is the \(h_{\min}\) decomposition. When
pointwise=True, per-outcome redundancies and PI atoms are also computed via Mobius inversion on the lattice. Theorem 5 of the paper guarantees that every pointwise PI atom is non-negative.
\(H_{\mathrm{sx}}\)
Varley et al. [VPP+23] define the redundant entropy of an antichain \(\alpha = \{A_1, \ldots, A_k\}\) through shared probability mass exclusions: the surprisal of the event that at least one source group takes its observed value,
This is the entropy analogue of \(\Isx{\bullet}\). Like PED_MOS, it accepts pointwise=True to keep the per-outcome atoms.
In [39]: from dit.pid import PED_SX
In [40]: print(PED_SX(dit.Distribution(['00','01','10','11'], [0.25]*4)))
+--------------------------+
| H_sx |
+--------+--------+--------+
| H_sx | H_r | H_d |
+--------+--------+--------+
| {0:1} | 2.0000 | 0.4150 |
| {0} | 1.0000 | 0.5850 |
| {1} | 1.0000 | 0.5850 |
| {0}{1} | 0.4150 | 0.4150 |
+--------+--------+--------+
Unlike \(H_{\mathrm{cs}}\), which gives these independent bits no redundant entropy, \(H_{\mathrm{sx}}\) counts the \(\log_2 4/3\) bits it takes to exclude the outcome that matches neither observed value.
- class PED_SX(dist, sources=None, pointwise=False, **kwargs)[source]
The shared-exclusions partial entropy decomposition \(h_{sx}\) of Varley et al. (2023).
https://doi.org/10.1073/pnas.2300888120
When
pointwise=True, per-outcome redundancies and PI atoms are also computed via Mobius inversion on the lattice.
Generalized Information Decomposition
The local Kullback-Leibler divergence \(\log q(x)/p(x)\) of a posterior \(q\) from a prior \(p\) is the difference \(h_p(x) - h_q(x)\) of two local entropies. Varley [Var24b] therefore decomposes \(\DKL{q || p}\) over the redundancy lattice as the posterior expectation of the difference between the prior’s and the posterior’s pointwise partial entropy atoms. With the product of marginals as prior this decomposes the total correlation; with the uniform distribution, the negentropy. GID takes measure='sx' (PED_SX) or measure='mos' (PED_MOS).
In [41]: from dit.pid import GID
In [42]: from dit.distconst import product_distribution
In [43]: d = dit.example_dists.Xor()
In [44]: gid = GID(d, product_distribution(d))
In [45]: sum(gid[node] for node in gid._lattice)
Out[45]: 1.0
- class GID(posterior, prior, sources=None, measure='sx')[source]
The generalized information decomposition of \(D_{KL}[\text{posterior} \| \text{prior}]\).
- get_pi(node)[source]
The atom of node.
- get_pw_pi(node)[source]
The local atoms of node,
{outcome: float}(in bits).
- get_red(node)[source]
The redundant divergence of node: the sum of the atoms at and below it.
Integrated Information Decomposition
The integrated information decomposition (\(\Phi\mathrm{ID}\)) [MRCH+19, MRL+25] decomposes the mutual information between a set of sources (typically a system’s past) and a set of targets (its future) over the product of the two redundancy lattices. An atom \(\alpha \to \beta\) is information carried by the sources in the mode \(\alpha\) and by the targets in the mode \(\beta\). For two sources and two targets there are 16 atoms, including persistent redundancy \(\{0\}\{1\} \to \{2\}\{3\}\), information transfer \(\{0\} \to \{3\}\), and downward causation \(\{0:1\} \to \{2\}\).
The atoms are the Mobius inversion of a double redundancy \(I_\partial(\alpha \to \beta)\). PhiID provides three:
measure='mmi': \(\min_{i,j} \I{A_i : B_j}\) [MRCH+19].measure='pm': the expectation of \(\min_i h(a_i) + \min_j h(b_j) - \min_{i,j} h(a_i, b_j)\), after [FL20].measure='sx': the expectation of \(h_{\mathrm{sx}}(\alpha) + h_{\mathrm{sx}}(\beta) - h_{\mathrm{sx}}(\{A_i \cup B_j\})\) [Var23].
The atoms always sum to \(\I{X : Y}\), but individual atoms can be negative.
In [46]: from dit.pid import PhiID
In [47]: d = dit.Distribution(['0000', '1111'], [1/2, 1/2])
In [48]: PhiID(d)[(((0,), (1,)), ((2,), (3,)))]
Out[48]: 1.0
- class PhiID(dist, sources=None, targets=None, measure='mmi', pointwise=False)[source]
The Integrated Information Decomposition of \(I[X : Y]\) for sources \(X\) and targets \(Y\).
- get_red(node)[source]
The double redundancy \(I_\partial(\alpha \to \beta)\) of node.
- get_pi(node)[source]
The PhiID atom \(\alpha \to \beta\).
- get_pw_red(node)[source]
Pointwise double redundancies of node,
{outcome: float}(in bits).
- get_pw_pi(node)[source]
Pointwise atoms of node,
{outcome: float}(in bits).
- property nodes
The nodes \((\alpha, \beta)\) of the product lattice.
Representational Complexity
Ehrlich et al. [ESP+23] summarize a decomposition by the atom-weighted average degree of synergy: each atom is assigned the number of sources in its smallest group, so purely redundant (or unique) information has degree 1 and information available only from all \(n\) sources jointly has degree \(n\).
In [49]: from dit.pid.helpers import representational_complexity
In [50]: representational_complexity(PED_SX(dit.example_dists.Xor()))
Out[50]: 1.1225562489182657
- representational_complexity(decomposition, comparator=<built-in function min>)[source]
The representational complexity of a partial information (or entropy) decomposition [ESP+23].
Each atom \(\alpha\) gets a degree, the number of sources in its smallest group (
comparator=min, as in Ehrlich et al.) or its largest (comparator=max), and the complexity is the atom-weighted mean degree:\[C = \frac{\sum_\alpha \deg(\alpha) \Pi(\alpha)}{\sum_\alpha \Pi(\alpha)}\]It ranges from 1 (all redundant) to the number of sources (all synergistic).
- Parameters:
decomposition (BasePID) – A computed PID or PED, such as
PID_SXorPED_SX.comparator ({min, max}) – How the sizes of an atom’s groups are reduced to its degree.
- Returns:
complexity
- Return type:
Comparing Measures
The helpers in dit.pid.helpers print side-by-side tables.
compare_measures() runs several PID measures on one
distribution (by default, every measure in dit.pid.measures.__all_pids), and
pointwise_pid_table() lists the per-outcome atoms of a
pointwise measure such as \(\Isx{\bullet}\):
In [51]: from dit.pid.helpers import compare_measures, pointwise_pid_table
In [52]: compare_measures(bivariates['and'], pids=[PID_WB, PID_MMI, PID_CCS])
+-----------------------------------------+
| PID Comparison |
+--------+----------+----------+----------+
| | I_min | I_mmi | I_ccs |
+--------+----------+----------+----------+
| {0:1} | 0.50000 | 0.50000 | 0.29248 |
| {0} | 0.00000 | 0.00000 | 0.20752 |
| {1} | 0.00000 | 0.00000 | 0.20752 |
| {0}{1} | 0.31128 | 0.31128 | 0.10376 |
+--------+----------+----------+----------+
In [53]: pointwise_pid_table(bivariates['and'], pid_class=PID_SX)
+---------------------------------------------------------------------------------------------------------------------------------------------------------------+
| I_sx (pointwise) |
+-----------------+--------+----------+-----------+-----------+---------+---------+---------+---------+---------+---------+-----------+------------+------------+
| event | p | {0:1} pi | {0:1} pi+ | {0:1} pi- | {0} pi | {0} pi+ | {0} pi- | {1} pi | {1} pi+ | {1} pi- | {0}{1} pi | {0}{1} pi+ | {0}{1} pi- |
+-----------------+--------+----------+-----------+-----------+---------+---------+---------+---------+---------+---------+-----------+------------+------------+
| ('0', '0', '0') | 0.2500 | 0.0000 | 0.4150 | 0.4150 | -0.0000 | 0.5850 | 0.5850 | -0.0000 | 0.5850 | 0.5850 | 0.4150 | 0.4150 | -0.0000 |
| ('0', '1', '0') | 0.2500 | 0.4150 | 0.4150 | 0.0000 | 0.5850 | 0.5850 | 0.0000 | -0.4150 | 0.5850 | 1.0000 | -0.1699 | 0.4150 | 0.5850 |
| ('1', '0', '0') | 0.2500 | 0.4150 | 0.4150 | 0.0000 | -0.4150 | 0.5850 | 1.0000 | 0.5850 | 0.5850 | 0.0000 | -0.1699 | 0.4150 | 0.5850 |
| ('1', '1', '1') | 0.2500 | 0.4150 | 0.4150 | -0.0000 | 0.5850 | 0.5850 | -0.0000 | 0.5850 | 0.5850 | -0.0000 | 0.4150 | 0.4150 | -0.0000 |
+-----------------+--------+----------+-----------+-----------+---------+---------+---------+---------+---------+---------+-----------+------------+------------+
- compare_measures(dist, pids=[<class 'dit.pid.measures.immi.PID_MMI'>, <class 'dit.pid.measures.iwedge.PID_GK'>, <class 'dit.pid.measures.imin.PID_WB'>, <class 'dit.pid.measures.irr.PID_RR'>, <class 'dit.pid.measures.iccs.PID_CCS'>, <class 'dit.pid.measures.ipm.PID_PM'>, <class 'dit.pid.measures.isx.PID_SX'>, <class 'dit.pid.measures.iproj.PID_Proj'>, <class 'dit.pid.measures.igh.PID_GH'>, <class 'dit.pid.measures.ibroja.PID_BROJA'>, <class 'dit.pid.measures.imes.PID_MES'>, <class 'dit.pid.measures.idep.PID_dep'>, <class 'dit.pid.measures.idep.PID_RA'>, <class 'dit.pid.measures.irav.PID_RAV'>, <class 'dit.pid.measures.iskar.PID_SKAR_nw'>, <class 'dit.pid.measures.iskar.PID_SKAR_owa'>, <class 'dit.pid.measures.iskar.PID_SKAR_owb'>, <class 'dit.pid.measures.iskar.PID_SKAR_tw'>, <class 'dit.pid.measures.iprec.PID_Prec'>, <class 'dit.pid.measures.ict.PID_CT'>, <class 'dit.pid.measures.iig.PID_IG'>, <class 'dit.pid.measures.iipid.PID_IPID'>, <class 'dit.pid.measures.idelta.PID_Delta'>, <class 'dit.pid.measures.ideltalambda.PID_DeltaLambda'>, <class 'dit.pid.measures.irdr.PID_RDR'>, <class 'dit.pid.measures.ideg.PID_Deg'>, <class 'dit.pid.measures.imc.PID_MC'>, <class 'dit.pid.measures.iln.PID_LN'>, <class 'dit.pid.measures.ido.PID_Do'>, <class 'dit.pid.measures.ici.PID_CI'>, <class 'dit.pid.measures.idep.PID_dep_a'>, <class 'dit.pid.measures.idep.PID_dep_b'>, <class 'dit.pid.measures.idep.PID_dep_c'>], inputs=None, output=None, name='', digits=5)[source]
Print the results of several partial information decompositions.
- Parameters:
dist (dit.Distribution) – The distribution to compute PIDs of.
pids (iterable(BasePID)) – The PID classes to use.
inputs (iterable of iterables) – The variables to treat as inputs.
output (iterable) – The variables to consider as a singular output.
name (str) – The name of distribution.
digits (int) – The number of digits of precision to print.
- pointwise_pid_table(dist, sources=None, target=None, pid_class=None, digits=4)[source]
Print a table of per-outcome pointwise PID values.
Each row corresponds to one joint outcome. Columns are:
event,p, then for each lattice node (sorted by depth):{node} piand — when available —{node} pi+and{node} pi-.- Parameters:
dist (dit.Distribution) – The distribution to analyse.
sources (iter of iters, optional) – Source variable indices. Defaults to all variables except the last.
target (iter, optional) – Target variable indices. Defaults to the last variable.
pid_class (BasePointwisePID subclass, optional) – The pointwise PID measure to use. Defaults to
PID_SX.digits (int) – Number of decimal places to display.
Checking Properties
Liardi et al. [LDB+26] catalogue the properties proposed for redundancy measures, and which measures satisfy them. check() tests one property on a computed decomposition. Its result is truthy when the property holds, and lists the comparisons that fail. Here \(\Imin{\bullet}\) violates the source chain rule on the distribution of Gutknecht [Gut26], in which a third source decides which of the first two copies the target, while \(\Isx{\bullet}\) satisfies it:
In [54]: from dit.pid.distributions import trivariates
In [55]: d = trivariates['max min']
In [56]: print(PID_WB(d).check('SCR'))
SCR (source chain rule): fails, 12 tested
I_cap({0:2}{1:2}) = I(c : T) + I_cap({0}{1} | c), c = {2}: needs 0.5 = 0
In [57]: bool(PID_SX(d).check('SCR'))
Out[57]: True
check_properties() checks every property, next to what the literature claims for the measure; claims() returns just the claims, and property_table() prints those of several measures:
In [58]: print(PID_WB(bivariates['and']).check_properties())
+-----------------------------------------------------------------------------------------------------------------------+
| Properties of I_min |
+----------+-------+-------+--------+---------+-------------------------------------------------------------------------+
| property | claim | check | tested | max gap | note |
+----------+-------+-------+--------+---------+-------------------------------------------------------------------------+
| SR | ✓ | ✓ | 3 | | |
| S0 | ✓ | ✓ | 4 | | |
| M0 | ✓ | ✓ | 4 | | |
| GP | ✓ | ✓ | 4 | | |
| LP0 | ✓ | ✓ | 4 | | |
| LP1 | ✓ | ? | 0 | | concerns three or more sources |
| LP | ✓ | ✓ | 4 | | |
| IID | ✗ | ✗ | 2 | 1 | measure-level: on copy distributions |
| ID | ✗ | ✗ | 3 | 1 | measure-level: on copy distributions |
| TM | ✗ | ? | 0 | | needs a target of two or more variables |
| TC | ✗ | ? | 0 | | needs a target of two or more variables |
| SE | ✓ | ✓ | 0 | | vacuous: nothing to compare |
| LB | ✓ | ✓ | 4 | | Q ranges up to the Gacs-Korner meet |
| TE | ✓ | ✓ | 4 | | |
| M1 | ✓ | ✓ | 8 | | M0 and TE (Liardi et al., Theo. 6) |
| S1 | ✗ | ✗ | 2 | 0.3113 | |
| * | ✓ | ✓ | 1 | | same source-target marginals, sources independent given T |
| BP | ✗ | ✓ | 2 | | |
| AD | ✗ | ✓ | 4 | | an independent copy |
| CO | ✓ | ? | 0 | | cannot be checked on a single distribution |
| DI | ✗∥ | ? | 0 | | cannot be checked on a single distribution |
| EI | ✓ | ✓ | 4 | | |
| SCR | ✗ | ✓ | 0 | | has content only for three or more sources; vacuous: nothing to compare |
+----------+-------+-------+--------+---------+-------------------------------------------------------------------------+
In [59]: from dit.pid.helpers import property_table
In [60]: property_table(pids=[PID_WB, PID_MMI, PID_GK, PID_SX], props=['LP', 'TC', 'SCR'])
+---------------------------------------+
| Claimed PID properties |
+----------+-------+-------+-----+------+
| property | I_min | I_mmi | I_∧ | I_sx |
+----------+-------+-------+-----+------+
| LP | ✓ | ✓ | ✗ | ✗‡ |
| TC | ✗ | ✗ | ✗ | ✓ |
| SCR | ✗ | ✗ | ✓ | ✓ |
+----------+-------+-------+-----+------+
Properties are named by the keys of [LDB+26], or by name (e.g. 'source_chain_rule'):
Key |
Property |
Check |
|---|---|---|
SR |
self-redundancy |
The redundancy of each single collection is its mutual information with the target. |
S0 |
weak symmetry |
Recompute with the sources reordered ( |
M0 |
weak monotonicity |
Redundancy does not decrease up the lattice, and dropping a collection that determines another changes nothing. |
GP |
global positivity |
Every redundancy is nonnegative. |
LP0, LP1, LP |
local positivity |
Every atom is nonnegative; LP0 concerns two sources, LP1 three or more. |
IID, ID |
(independent) identity |
The redundancy of two sources about a target copying them, on fixed copy distributions; a property of the measure, not of this distribution. |
TM |
target monotonicity |
Recompute with each target variable dropped. |
TC |
target chain rule |
Recompute with the first target variable, and with the rest given it ( |
SE |
subset equality |
Dropping a collection that determines another changes nothing. |
LB |
lower bound |
Each redundancy is at least the information the Gács-Körner meet of its collections carries about the target. |
TE |
target equality |
Adding a copy of the target to the sources changes nothing. |
M1 |
strong monotonicity |
M0 and TE, and dropping a collection that determines the target changes nothing. |
S1 |
strong symmetry |
Exchanging the target with any source changes nothing. |
* |
assumption (*) |
Recompute under \(p(y) \prod_i p(x_i | y)\), which has the same source-target marginals. |
BP |
Blackwell property |
A unique information vanishes exactly when its source is Blackwell inferior to the other (two sources). |
AD |
additivity |
Recompute on two independent copies of the system, or on it and |
CO, DI |
continuity, differentiability |
Not checkable on a single distribution. |
EI |
equivalence-class invariance |
Recompute with every alphabet relabeled. |
SCR |
source chain rule |
Recompute the decomposition of the other sources given each context ( |
Comparisons use the absolute tolerance atol (1e-5 by default); measures computed by numerical optimization may need a looser one. Checks that recompute the decomposition run the measure again, several times; additivity squares every alphabet, which is prohibitive for measures that search exhaustively or optimize large auxiliary variables, and slow for \(\Iprec{\bullet}\) and \(\Ideg{\bullet}\) with three or more sources, whose cones of common degradations then have very many extreme rays, so leave it out of props for them. Two-source measures do not define the redundancy of three or more collections, so comparisons that need one are undetermined, and their claims concern two sources only.
The claims transcribe Table 5 of [LDB+26], whose footnote marks they keep († two sources only, § full support only, ‡ the pointwise version holds, ∥ global, * supported by simulations), and the source chain rule results of [Gut26]. \(\Ideg{\bullet}\) is the same measure as \(\Iprec{\bullet}\), so it takes that column. Entries marked ! correct Table 5: by its own appendix, \(I_{\prec}\) satisfies (*); identical sources show that \(\Iwedge{\bullet}\) violates (*); ‘rdn xor’ shows that \(\Irr{\bullet}\) violates the lower bound; the correlated two-bit copy shows that \(I_{rdr}\) satisfies only one direction of the Blackwell property; and ‘max min’ shows that \(\Igh{\bullet}\) violates weak monotonicity, and so strong monotonicity, since its order \(Q - X - Y\) is not transitive.
- class PropertyCheck(key: str, holds: bool | None, tested: int = 0, violations: list = <factory>, note: str = '')[source]
The result of checking one property on one decomposition.
A check is truthy only if the property holds. holds is None when the property does not apply, or could not be evaluated.
- key
The property’s key in PROPERTIES.
- Type:
- holds
Whether the property holds.
- Type:
bool, None
- tested
The number of (in)equalities that were compared.
- Type:
- violations
The comparisons that failed.
- Type:
list of Violation
- note
Context for the result.
- Type:
- property name
The name of the property.
- property max_violation
The largest gap among the violations (0 if there are none).
- property_table(pids=None, props=None)[source]
Print the properties claimed for PID measures in the literature.
Rows are properties and columns are measures: ✓ if the property is shown to hold, ✗ if it is shown to fail, n.a. if it does not apply, and ? if the literature is silent. Marks are the footnotes of Table 5 of [LDB+26], and ! flags a corrected entry; see BasePID.claims for the caveats.
- Parameters:
pids (iterable(BasePID), None) – The PID classes to show. If None, every measure in
dit.pid.measures.__all_pidswith claims.props (iterable of str, None) – The properties to show. If None, all of them.