Partial Information Decomposition

The partial information decomposition (PID), put forth by Williams & Beer [WB10], is a framework for decomposing the information shared between a set of variables we will refer to as inputs, \(X_0, X_1, \ldots\), and another random variable we will refer to as the output, \(Y\). This decomposition seeks to partition the information \(\I{X_0,X_1,\ldots : Y}\) among the antichains of the inputs.

Background

It is often desirable to determine how a set of inputs influence the behavior of an output. dit.pid.distributions ships two dictionaries of standard test cases from the PID literature: bivariates (two inputs; e.g. 'and', 'synergy', 'redundant', 'unique 1', 'rdn xor', 'reduced or', 'sum', 'cat', 'boom') and trivariates (three inputs; e.g. 'synergy', 'redundant', 'dbl xor', 'giant xor', 'anddup'); sorted(bivariates) lists them all. Consider the exclusive or logic gates, for example:

In [1]: from dit.pid.distributions import bivariates, trivariates

In [2]: xor = bivariates['synergy']

In [3]: print(xor)
Class:    Distribution
Alphabet: (('0', '1'), ('0', '1'), ('0', '1'))
Base:     linear

x                 p(X0,X1,X2)
('0', '0', '0')   0.25
('0', '1', '1')   0.25
('1', '0', '1')   0.25
('1', '1', '0')   0.25

We can see from inspection that either input (the first two indexes) is independent of the output (the final index), yet the two inputs together determine the output. One could call this “synergistic” information. Next, consider the giant bit distribution:

In [4]: gb = bivariates['redundant']

In [5]: print(gb)
Class:    Distribution
Alphabet: (('0', '1'), ('0', '1'), ('0', '1'))
Base:     linear

x                 p(X0,X1,X2)
('0', '0', '0')   0.5
('1', '1', '1')   0.5

Here, we see that either input informs us of exactly what the output is. One could call this “redundant” information. Furthermore, consider the Co-Information of these distributions:

In [6]: from dit.multivariate import coinformation as I

In [7]: I(xor)
Out[7]: -1.0

In [8]: I(gb)
Out[8]: 1.0

This could lead one to intuit that negative values of the coinformation correspond to synergistic effects in a distribution, while positive values correspond to redundant effects. This intuition, however, is at best misleading: the coinformation of a 4-variable giant bit and 4-variable parity distribution are both positive:

In [9]: I(dit.example_dists.giant_bit(4, 2))
Out[9]: 1.0

In [10]: I(dit.example_dists.n_mod_m(4, 2))
Out[10]: 1.0

This, as well as other issues, lead Williams & Beer [WB10] to propose the partial information decomposition.

Framework

The goal of the partial information is to assign to each some non-negative portion of \(\I{\{X_i\} : Y}\) to each antichain over the inputs. An antichain over the inputs is a set of sets, where each of those sets is not a subset of any of the others. For example, \(\left\{ \left\{X_0, X_1\right\}, \left\{X_1, X_2\right\} \right\}\) is an antichain, but \(\left\{ \left\{X_0, X_1\right\}, \left\{X_0 X_1, X_2\right\} \right\}\) is not.

The antichains for a lattice based on this partial order:

\[\alpha \leq \beta \iff \forall \mathbf{b} \in \beta, \exists \mathbf{a} \in \alpha, \mathbf{a} \subseteq \mathbf{b}\]

From here, we wish to find a redundancy measure, \(\Icap{\bullet}\) which would assign a fraction of \(\I{\{X_i\} : Y}\) to each antichain intuitively quantifying what portion of the information in the output could be learned by observing any of the sets of variables within the antichain. In order to be a viable measure of redundancy, there are several axioms a redundancy measure must satisfy.

Bivariate Lattice

Let us consider the special case of two inputs. The lattice consists of four elements: \(\left\{\left\{X_0\right\}, \left\{X_1\right\}\right\}\), \(\left\{\left\{X_0\right\}\right\}\), \(\left\{\left\{X_1\right\}\right\}\), and \(\left\{\left\{X_0, X_1\right\}\right\}\). We can interpret these elements as the redundancy provided by both inputs, the information uniquely provided by \(X_0\), the information uniquely provided by \(X_1\), and the information synergistically provided only by both inputs together. Together these for elements decompose the input-output mutual information:

\[\I{X_0, X_1 : Y} = \Ipart{\left\{X_0\right\}, \left\{X_1\right\} : Y} + \Ipart{\left\{X_0\right\} : Y} + \Ipart{\left\{X_1\right\} : Y} + \Ipart{\left\{X_0, X_1\right\} : Y}\]

Furthermore, due to the self-redundancy axiom (described ahead), the single-input mutual informations decomposed in the following way:

\[ \begin{align}\begin{aligned}\I{X_0 : Y} = \Ipart{\left\{X_0\right\}, \left\{X_1\right\} : Y} + \Ipart{\left\{X_0\right\} : Y}\\\I{X_1 : Y} = \Ipart{\left\{X_0\right\}, \left\{X_1\right\} : Y} + \Ipart{\left\{X_1\right\} : Y}\end{aligned}\end{align} \]

Colloquially, from input \(X_0\) one can learn what is redundantly provided by either input, plus what is uniquely provided by \(X_0\), but not what is uniquely provided by \(X_1\) or what can only be learned synergistically from both inputs.

Axioms

The following three axioms were provided by Williams & Beer.

Symmetry

The redundancy \(\Icap{X_{0:n} : Y}\) is invariant under reorderings of \(X_i\).

Self-Redundancy

The redundancy of a single input is its mutual information with the output:

\[\Icap{X_i : Y} = \I{X_i : Y}\]

Monotonicity

The redundancy should only decrease with in inclusion of more inputs:

\[\Icap{\mathcal{A}_1, \ldots, \mathcal{A}_{k-1}, \mathcal{A}_k : Y} \leq \Icap{\mathcal{A}_1, \ldots, \mathcal{A}_{k-1} : Y}\]

with equality if \(\mathcal{A}_{k-1} \subseteq \mathcal{A}_k\).

There have been other axioms proposed following from those of Williams & Beer.

Identity

The identity axiom [HSP13] states that if the output is identical to the inputs, then the redundancy is the mutual information between the inputs:

\[\Icap{X_0, X_1 : \left(X_0, X_1\right)} = \I{X_0 : X_1}\]

Target (output) Monotonicity

This axiom states that redundancy can not increase when replacing the output by a function of itself.

\[\Icap{X_{0:n} : Y} \ge \Icap{X_{0:n} : f(Y)}\]

It first appeared in [BROJ13] and was expanded upon in [RBO+17].

Measures

We now turn our attention a variety of methods proposed to flesh out this partial information decomposition.

In [11]: from dit.pid import *

\(\Imin{\bullet}\)

\(\Imin{\bullet}\)[WB10] was Williams & Beer’s initial proposal for a redundancy measure. It is given by:

\[\Imin{\mathcal{A}_1, \mathcal{A}_2, \ldots : Y} = \sum_{y \in Y} p(y) \min_{\mathcal{A}_i} \I{\mathcal{A}_i : Y=y}\]

However, this measure has been criticized for acting in an unintuitive manner [GK14]:

In [12]: d = dit.Distribution(['000', '011', '102', '113'], [1/4]*4)

In [13]: PID_WB(d)
Out[13]: <dit.pid.measures.imin.PID_WB object at 0x7c1fcdb99310>

We have constructed a distribution whose inputs are independent random bits, and whose output is the concatenation of those inputs. Intuitively, the output should then be informed by one bit of unique information from \(X_0\) and one bit of unique information from \(X_1\). However, \(\Imin{\bullet}\) assesses that there is one bit of redundant information, and one bit of synergistic information. This is because \(\Imin{\bullet}\) quantifies redundancy as the least amount of information one can learn about an output given any single input. Here, however, the one bit we learn from \(X_0\) is, in a sense, orthogonal from the one bit we learn from \(X_1\). This observation has lead to much of the follow-on work.

class PID_WB(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]: The Williams & Beer partial information decomposition.

\(\Immi{\bullet}\)

One potential measure of redundancy is the minimum mutual information [BROJ13]:

\[\Immi{X_{0:n} : Y} = \min_{i} \I{X_i : Y}\]

This measure, though crude, is known to be correct for multivariate gaussian variables [OBR15].

class PID_MMI(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]

The minimum mutual information partial information decomposition.

This measure is known to be accurate for gaussian random variables. It was also briefly and tangentially studied by the BROJA team.

\(\Iwedge{\bullet}\)

Redundancy seems to intuitively be related to common information Common Informations. This intuition lead to the development of \(\Iwedge{\bullet}\) [GCJ+14]:

\[\Iwedge{X_{0:n} : Y} = \I{ \meet X_i : Y}\]

That is, redundancy is the information the Gács-Körner Common Information of the inputs shares with the output.

Warning

This measure can result in a negative PID.

class PID_GK(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]

The Griffith et al partial information decomposition.

This PID is known to produce negative partial information values.

\(\Iproj{\bullet}\)

Utilizing information geometry, Harder et al [HSP13] have developed a strictly bivariate measure of redundancy, \(\Iproj{\bullet}\):

\[\Iproj{\left\{X_0\right\}\left\{X_1\right\} : Y} = \min \{ I^\pi_Y[X_0 \mss X_1], I^\pi_Y[X_1 \mss X_0] \}\]

where

\[ \begin{align}\begin{aligned}I^\pi_Y[X_0 \mss X_1] = \sum_{x_0, y} p(x_0, y) \log \frac{p_{(x_0 \mss X_1)}(y)}{p(y)}\\p_{(x_0 \mss X_1)}(Y) = \pi_{C_{cl}(\langle X_1 \rangle_Y)}(p(Y | x_0)\\\pi_B(p) = \arg \min_{r \in B} \DKL{p || r}\\C_{cl}(\langle X_1 \rangle_Y) = C_{cl}(\left\{p(Y | x_1) : x_1 \in X_1 \right\})\end{aligned}\end{align} \]

where \(C_{cl}(\bullet)\) denotes closure. Intuitively, this measures seeks to quantify redundancy as the minimum of how much \(p(Y | X_0)\) can be expressed when \(X_0\) is projected on to \(X_1\), and vice versa.

class PID_Proj(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]: The Harder et al partial information decomposition.

\(\Ibroja{\bullet}\)

In a very intuitive effort, Bertschinger et al (henceforth BROJA) [BRO+14, GK14] defined unique information as the minimum conditional mutual informations obtainable while holding the input-output marginals fixed:

\[ \begin{align}\begin{aligned}\Delta = \{ Q : \forall i : p(x_i, y) = q(x_i, y) \}\\\Ibroja{X_{0:n} : Y} = \min_{Q \in \Delta} \I{X_i : Y | X_{\overline{\{i\}}}}\end{aligned}\end{align} \]

For bivariate sources (two inputs), PID_BROJA accepts a method keyword selecting how the marginal-matching optimization is solved:

'scipy' (default for small alphabets under 'auto') — SLSQP on the free joint pmf parameters.
'admui' — alternating divergence minimization [BRMontufar17].
'cone' — exponential cone program solved with ECOS [MTV18] (requires pip install dit[broja]).
'auto' — picks scipy or admui by joint alphabet size, with fallback to the other methods on failure.

Note

In the bivariate case, Griffith independently suggested the same decomposition but from the viewpoint of synergy [GK14].

The BROJA measure has recently been criticized for behaving in an unintuitive manner on some examples. Consider the reduced or distribution:

In [14]: bivariates['reduced or']
Out[14]: <Distribution p(X0,X1,X2)>

In [15]: print(PID_BROJA(bivariates['reduced or']))
+---------------------------+
|          I_broja          |
+---------+--------+--------+
| I_broja |  I_r   |   pi   |
+---------+--------+--------+
|  {0:1}  | 1.0000 | 0.6887 |
|   {0}   | 0.3113 | 0.0000 |
|   {1}   | 0.3113 | 0.0000 |
|  {0}{1} | 0.3113 | 0.3113 |
+---------+--------+--------+

We see that in this instance BROJA assigns no partial information to either unique information. However, it is not difficult to argue that in the case that either input is a 1, that input then has unique information regarding the output.

\(\Iproj{\bullet}\) and \(\Ibroja{\bullet}\) are Distinct

In the BROJA paper [BRO+14] the only example given where their decomposition differs from that of Harder et al. is the dit.example_dists.summed_dice(). We can find a simpler example where they differ using hypothesis:

In [16]: from hypothesis import find

In [17]: from dit.utils.testing import distribution_structures

In [18]: find(distribution_structures(3, 2, True), lambda d: PID_Proj(d) != PID_BROJA(d))
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
File ~/checkouts/readthedocs.org/user_builds/dit/envs/latest/lib/python3.12/site-packages/pandas/core/indexes/base.py:3816, in Index.get_loc(self, key)
   3815 try:
-> 3816     return self._engine.get_loc(casted_key)
   3817 except KeyError as err:

File pandas/_libs/index.pyx:172, in pandas._libs.index.IndexEngine.get_loc()
--> 172 'Could not get source, probably due dynamically evaluated source code.'
File pandas/_libs/index.pyx:180, in pandas._libs.index.IndexEngine.get_loc()
--> 180 'Could not get source, probably due dynamically evaluated source code.'
File pandas/_libs/index.pyx:671, in pandas._libs.index.StringObjectEngine._check_type()
--> 671 'Could not get source, probably due dynamically evaluated source code.'
KeyError: 0

The above exception was the direct cause of the following exception:

KeyError                                  Traceback (most recent call last)
File ~/checkouts/readthedocs.org/user_builds/dit/envs/latest/lib/python3.12/site-packages/xarray/core/indexes.py:883, in PandasIndex.sel(self, labels, method, tolerance)
    882 try:
--> 883     indexer = self.index.get_loc(label_value)
    884 except KeyError as e:

File ~/checkouts/readthedocs.org/user_builds/dit/envs/latest/lib/python3.12/site-packages/pandas/core/indexes/base.py:3823, in Index.get_loc(self, key)
   3822         raise InvalidIndexError(key) from err
-> 3823     raise KeyError(key) from err
   3824 except TypeError:
   3825     # If we have a listlike key, _check_indexing_error will raise
   3826     #  InvalidIndexError. Otherwise we fall through and re-raise
   3827     #  the TypeError.

KeyError: 0

The above exception was the direct cause of the following exception:

KeyError                                  Traceback (most recent call last)
File ~/checkouts/readthedocs.org/user_builds/dit/envs/latest/lib/python3.12/site-packages/dit/distribution.py:2453, in Distribution.__getitem__(self, key)
   2452 try:
-> 2453     return self._coerce_prob(self.data.sel(sel2))
   2454 except KeyError as exc:

File ~/checkouts/readthedocs.org/user_builds/dit/envs/latest/lib/python3.12/site-packages/xarray/core/dataarray.py:1849, in DataArray.sel(self, indexers, method, tolerance, drop, **indexers_kwargs)
   1741 """Return a new DataArray whose data is given by selecting index
   1742 labels along the specified dimension(s).
   1743 
   (...)   1847 Dimensions without coordinates: points
   1848 """
-> 1849 ds = self._to_temp_dataset().sel(
   1850     indexers=indexers,
   1851     drop=drop,
   1852     method=method,
   1853     tolerance=tolerance,
   1854     **indexers_kwargs,
   1855 )
   1856 return self._from_temp_dataset(ds)

File ~/checkouts/readthedocs.org/user_builds/dit/envs/latest/lib/python3.12/site-packages/xarray/core/dataset.py:3076, in Dataset.sel(self, indexers, method, tolerance, drop, **indexers_kwargs)
   3074         """
   3075         indexers = either_dict_or_kwargs(indexers, indexers_kwargs, "sel")
-> 3076         query_results = map_index_queries(
   3077             self, indexers=indexers, method=method, tolerance=tolerance

File ~/checkouts/readthedocs.org/user_builds/dit/envs/latest/lib/python3.12/site-packages/xarray/core/indexing.py:219, in map_index_queries(obj, indexers, method, tolerance, **indexers_kwargs)
    218     else:
--> 219         results.append(index.sel(labels, **options))
    221 merged = merge_sel_results(results)

File ~/checkouts/readthedocs.org/user_builds/dit/envs/latest/lib/python3.12/site-packages/xarray/core/indexes.py:885, in PandasIndex.sel(self, labels, method, tolerance)
    884         except KeyError as e:
--> 885             raise KeyError(
    886                 f"not all values found in index {coord_name!r}. "
    887                 "Try setting the `method` keyword argument (example: method='nearest')."
    888             ) from e
    890 elif label_array.dtype.kind == "b":

KeyError: "not all values found in index 'X0'. Try setting the `method` keyword argument (example: method='nearest')."

The above exception was the direct cause of the following exception:

InvalidOutcome                            Traceback (most recent call last)
Cell In[18], line 1
----> 1 find(distribution_structures(3, 2, True), lambda d: PID_Proj(d) != PID_BROJA(d))

File ~/checkouts/readthedocs.org/user_builds/dit/envs/latest/lib/python3.12/site-packages/hypothesis/core.py:2507, in find(specifier, condition, settings, random, database_key)
   2504 test._hypothesis_internal_database_key = database_key  # type: ignore
   2506 try:
-> 2507     test()
   2508 except Found:
   2509     return last[0]

File ~/checkouts/readthedocs.org/user_builds/dit/envs/latest/lib/python3.12/site-packages/hypothesis/core.py:2495, in find.<locals>.test()
   2490 specifier.validate()
   2492 last: list[Ex] = []
   2494 @settings
-> 2495 @given(specifier)
   2496 def test(v):
   2497     if condition(v):
   2498         last[:] = [v]

    [... skipping hidden 1 frame]

Cell In[18], line 1, in <lambda>(d)
----> 1 find(distribution_structures(3, 2, True), lambda d: PID_Proj(d) != PID_BROJA(d))

File ~/checkouts/readthedocs.org/user_builds/dit/envs/latest/lib/python3.12/site-packages/dit/pid/pid.py:158, in BasePID.__init__(self, dist, sources, target, reds, pis, compute, **kwargs)
    155 self._pis = {} if pis is None else pis
    157 if compute:
--> 158     self._compute()

File ~/checkouts/readthedocs.org/user_builds/dit/envs/latest/lib/python3.12/site-packages/dit/pid/pid.py:979, in BaseBivariatePID._compute(self)
    977 for node in self._lattice:
    978     if len(node) == 2 and node not in self._reds:
--> 979         self._reds[node] = self._scaled(self._measure(self._dist, node, self._target, **self._kwargs))
    981 super()._compute()

File ~/checkouts/readthedocs.org/user_builds/dit/envs/latest/lib/python3.12/site-packages/dit/pid/measures/iproj.py:223, in PID_Proj._measure(d, sources, target)
    220     msg = f"This method needs exact two sources, {len(sources)} given."
    221     raise ditException(msg)
--> 223 pi_0 = projected_information(d, sources[0], sources[1], target)
    224 pi_1 = projected_information(d, sources[1], sources[0], target)
    225 return min(pi_0, pi_1)

File ~/checkouts/readthedocs.org/user_builds/dit/envs/latest/lib/python3.12/site-packages/dit/pid/measures/iproj.py:184, in projected_information(dist, X, Y, Z)
    182 for x, p_proj_z in zip(p_x.outcomes, projections, strict=True):
    183     for z in p_z.outcomes:
--> 184         vals.append(p_xz[(x, z)] * np.log2(p_proj_z[z] / p_z[z]))
    185 val = np.nansum(vals)
    187 return val

File ~/checkouts/readthedocs.org/user_builds/dit/envs/latest/lib/python3.12/site-packages/dit/distribution.py:2455, in Distribution.__getitem__(self, key)
   2453         return self._coerce_prob(self.data.sel(sel2))
   2454     except KeyError as exc:
-> 2455         raise InvalidOutcome(msg=f"Outcome {key!r} is not in the sample space.") from exc
   2456 raise InvalidOutcome(msg=f"Invalid outcome: {key!r}")

InvalidOutcome: Outcome (0, 0) is not in the sample space.
Failing test case: test(
    v=<Distribution p(X0,X1,X2)>,  # or any other generated value
)

class PID_BROJA(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]

The BROJA partial information decomposition.

Notes

This partial information decomposition, at least in the bivariate source case, was independently suggested by Griffith.

For two sources, method selects the bivariate solver:

'scipy' — SLSQP on the marginal-matching polytope (default for small alphabets)
'admui' — alternating divergence minimization ([BRMontufar17])
'cone' — exponential cone program via ECOS ([MTV18])
'auto' — size-based selection with scipy/cone fallbacks

\(\Iccs{\bullet}\)

Taking a pointwise point of view, Ince has proposed a measure of redundancy based on the Co-Information [Inc17a]:

\[\Iccs{X_{0:n} : Y} = \sum p(x_0, \ldots, x_n, y) \I{x_0 : \ldots : x_n : y}~~\textrm{if}~~\operatorname{sign}(\I{x_i : y}) = \operatorname{sign}(\I{x_0 : \ldots : x_n : y})\]

While this measure behaves intuitively in many examples, it also assigns negative values to some partial information atoms in some instances.

This decomposition also displays an interesting phenomena, that of subadditive redundancy. The gband distribution is an independent mix of a giant bit (redundancy of 1 bit) and the and distribution (redundancy of 0.1038 bits), and yet gband has 0.8113 bits of redundancy:

In [19]: PID_CCS(bivariates['gband'])
Out[19]: <dit.pid.measures.iccs.PID_CCS object at 0x7c1fc60de930>

Warning

This measure can result in a negative PID.

class PID_CCS(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]: The common change in surprisal partial information decomposition, as defined by Ince.

\(\Idep{\bullet}\)

James et al [JEC17] have developed a method of quantifying unique information based on the Dependency Decomposition. Unique information from variable \(X_i\) is evaluated as the least change in sources-target mutual information when adding the constraint \(X_i Y\).

In [20]: PID_dep(bivariates['not two'])
Out[20]: <dit.pid.measures.idep.PID_dep object at 0x7c1fc4f45a30>

class PID_dep(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]: The dependency partial information decomposition, as defined by James at al.

\(\Ipm{\bullet}\)

Also taking a pointwise view, Finn & Lizier’s \(\Ipm{\bullet}\) [FL18] instead splits the pointwise mutual information into two components:

\[i(s, t) = h(s) - h(s|t)\]

They then define two partial information lattices, one quantified locally by \(h(s)\) and the other by \(h(s|t)\). By averaging these local lattices and then recombining them, we arrive at a standard Williams & Beer redundancy lattice.

In [21]: PID_PM(bivariates['pnt. unq.'])
Out[21]: <dit.pid.measures.ipm.PID_PM object at 0x7c1fc827fda0>

Warning

This measure can result in a negative PID.

class PID_PM(dist, sources=None, target=None, reds=None, pis=None, compute=True, pointwise=False, **kwargs)[source]

The Finn & Lizier partial information decomposition (I_±).

The measure decomposes into: * h_s (specificity / plus): min over sources of the source surprisal * h_s|t (ambiguity / minus): min over sources of the conditional surprisal

When constructed with pointwise=True, also computes the per-outcome pointwise PID and the specificity / ambiguity decomposition.

\(\Isx{\bullet}\)

Shared Exclusions PID (\(I^{\mathrm{sx}}\)) by Makkeh et al.

Makkeh et al. propose a pointwise redundancy measure built on the notion of shared exclusions in probability space ([MCTV21]). Intuitively, the measure quantifies how much probability mass for the target Y is commonly excluded by observing sets of sources simultaneously — that common excluded mass is taken as the redundant (shared) contribution.

\[\Isx{X_{0:1} : Y} \;=\; \sum_{x_0,x_1,y} p(x_0,x_1,y)\; \log_2 \frac{p(x_0 \cup x_1 | y)}{p(x_0 \cup x_1)}\]

For the general multivariate definition (which is implemented in dit), refer to [MCTV21]. For large datasets with up to five source variables, refer to the reference implementation at https://github.com/Abzinger/SxPID

Warning

This measure can result in a negative PID.

class PID_SX(dist, sources=None, target=None, reds=None, pis=None, compute=True, pointwise=False, **kwargs)[source]

Shared Exclusion Partial Information Decomposition (I^sx) as described in:

Makkeh, A. & Gutknecht, A. & Wibral, M. (2021). Introducing a differentiable measure of pointwise shared information. Phys. Rev. E 103, 032149

When constructed with pointwise=True, also computes the per-outcome pointwise PID (PPID) and the informative / misinformative decomposition (i^sx+ and i^sx-).

\(\Irav{\bullet}\)

Taking a functional perspective as in \(\Iwedge{\bullet}\), \(\Irav{\bullet}\) defines bivariate redundancy as the maximum coinformation between the two sources \(X_0, X_1\), a target \(Y\), and a deterministic function of the inputs \(f(X_0,X_1)\).

\[\Irav{X_{0:2} : Y} = \max_f\left(\I{X_0\!:\!X_1\!:\!Y\!:\!f(X_0,X_1)}\right)\]

This measure is designed to exploit the conflation of synergy and redundancy in the three variable coinformation: \(\I{X_0\!:\!X_1\!:\!Y} = R - S\).

Note

TODO: reference. No canonical publication for \(\Irav{\bullet}\) has been identified in the dit source; a citation should be added if/when the origin paper is confirmed.

In [22]: PID_RAV(bivariates['pnt. unq.'])
Out[22]: <dit.pid.measures.irav.PID_RAV object at 0x7c1fc4930320>

class PID_RAV(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]: The maximum coinformation auxiliary random variable method.

\(\Irr{\bullet}\)

In order to combine \(\Immi{\bullet}\) with the coinformation, Goodwell and Kumar [GK17] have introduced their rescaled redundancy:

\[ \begin{align}\begin{aligned}\Irr{X_0 : X_1} = R_{\text{min}} + I_{S} (\Immi{X_{0:2} : Y} - R_{\text{min}}\\R_{\text{min}} = \max\{ 0, \I{X_0 : X_1 : Y} \}\\I_{S} = \frac{\I{X_0 : X_1}}{\min\{ \H{X_0}, \H{X_1} \}}\end{aligned}\end{align} \]

In [23]: PID_RR(bivariates['pnt. unq.'])
Out[23]: <dit.pid.measures.irr.PID_RR object at 0x7c1fd5e192b0>

class PID_RR(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]

The minimum mutual information partial information decomposition.

This measure is known to be accurate for gaussian random variables. It was also briefly and tangentially studied by the BROJA team.

\(\Ira{\bullet}\)

Drawing from the reconstructability analysis work of Zwick [Zwi04], we can define \(Ira{\bullet}\) as a restricted form of \(\Idep{\bullet}\).

Warning

This measure can result in a negative PID.

class PID_RA(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]: The “reproducibility analysis” partial information decomposition, derived from the work of Zwick.

\(\Irdr{\bullet}\)

The measure of Mages & Rohner [MR23] can be interpreted as a pointwise version of \(\Ibroja{\bullet}\) to provide a non-negative partial information decomposition for an arbitrary number of sources. It obtains its operational interpretation from valuating the reachable decision regions (achievable type I/II error pairs) for each state of the target variable.

class PID_RDR(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]: The Mages & Rohner partial information decomposition.

\(\Ict{\bullet}\)

Sigtermans [Sig20] proposes a bivariate redundancy measure based on causal tensors and path-based information flow between the two sources and the target.

class PID_CT(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]: The bivariate PID defined by Sigtermans using causal tensors.

\(\Ideg{\bullet}\)

The degradation intersection information of Kolchinsky [Kol22] (further developed by Gomes & Figueiredo [GF23]) defines redundancy as the maximum mutual information carried by a channel that is a Blackwell degradation of every source channel:

\[\Ideg{X_{0:n} \to Y} = \max_{Q :\; Q \preceq_d X_i \;\forall i} \I{Q : Y}\]

class PID_Deg(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]

Degradation intersection information I_d^∩ from Kolchinsky (2022).

Redundancy is the maximum I(Q; T) such that Q is a Blackwell degradation of every source channel. Satisfies the Williams-Beer axioms and the independent identity property (IIP).

References

A. Kolchinsky, “A Novel Approach to the Partial Information Decomposition”, Entropy 24, 403, 2022.

\(\Ici{\bullet}\)

Gomes & Figueiredo [GF24] define a union information rather than a redundancy. Each source \(A_i\) contributes its channel \(p(a_i \mid y)\); by sharing channels (and marginals thereof) the sources can build any distribution \(q(a, y) = p(y) \prod_{B \in \pi} p(a_B \mid y)\), where \(\pi\) partitions the variables of \(A = \bigcup_i A_i\) into blocks that each lie within some source. The union information is the best such \(q\) can do, capped at the true mutual information:

\[\Ici{A_0, \ldots, A_{m-1} \to Y} = \min\left\{ \I{A : Y},\; \max_{q} \I{A : Y}_q \right\}\]

Synergy is the information lost by assuming the sources are conditionally independent given the target, \(S^{CI} = \I{A : Y} - \Ici{A_0, \ldots, A_{m-1} \to Y} \geq 0\). Sources contained in, or a deterministic function of, other sources are removed first.

PID_CI is built on BaseSynergyPID, which turns a synergy (equivalently, union information) into redundancies by inclusion-exclusion over sub-collections of each antichain, as in the parthood formulation of Gutknecht et al. [GWM21]:

\[I_\cap(\alpha) = \sum_{\emptyset \neq \beta \subseteq \alpha} (-1)^{|\beta| + 1} I_\cup(\beta)\]

In [24]: from dit.pid import PID_CI

In [25]: PID_CI(bivariates['and'])
Out[25]: <dit.pid.measures.ici.PID_CI object at 0x7c1fc4f496a0>

In [26]: PID_CI(bivariates['and']).get_syn(((0,), (1,)))
Out[26]: 0.2704260414863777

class PID_CI(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]

The conditional-independence union information I_∪^CI of Gomes & Figueiredo (2024).

Synergy is the information lost by assuming the sources are conditionally independent given the target; redundancies follow by inclusion-exclusion over union informations.

References

A. F. C. Gomes and M. A. T. Figueiredo, “A Measure of Synergy Based on Union Information”, Entropy 26, 271, 2024.

\(\Idelta{\bullet}\)

The \(\delta\)-PID of Banerjee, Olbrich, Jost & Rauh [BOJR18] quantifies unique information via the weighted output KL deficiency, the cost of approximating one source channel from the other via output randomization:

\[\delta(Y : X_0 \setminus X_1) = \inf_{P(X_0'|X_1)} \mathbb{E}_{Y}\!\left[ \DKL{P(X_0|Y) \;\|\; P(X_0'|X_1) \circ P(X_1|Y)} \right]\]

Redundancy is then min-symmetrized across the two sources.

class PID_Delta(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]

The delta-PID based on weighted output KL deficiency.

The deficiency delta(M : X \ Y) measures how well Y can simulate X for making decisions about M, via the expected KL divergence between the true channel P(X|M) and the best approximation P(X'|Y) composed with P(Y|M).

Redundancy is symmetrized via the minimum:

RI = min{ I(M;X) - delta(M:X\Y), I(M;Y) - delta(M:Y\X) }

References

P. K. Banerjee et al., “Unique informations and deficiencies”, Allerton Conference, 2018.

\(\IdeltaLambda{\bullet}\)

Venkatesh, Gurushankar & Schamberg [VGS23] introduce a Lagrangian generalization that smoothly interpolates between the \(\delta\)-PID of [BOJR18] and the BROJA PID of [BRO+14]:

\[\delta_{\lambda}(Y : X_0 \setminus X_1) = \inf_{P(X_0'|X_1 Y)} \mathbb{E}_{Y}\!\left[\DKL{P(X_0|Y) \;\|\; P(X_0'|Y)}\right] + \lambda \, \I{Y : X_0' | X_1}\]

As \(\lambda \to \infty\) the measure recovers \(\Idelta{\bullet}\); as \(\lambda \to 0\) it recovers \(\Ibroja{\bullet}\).

class PID_DeltaLambda(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]

The delta-lambda PID from Venkatesh, Gurushankar & Schamberg (2023).

A Lagrangian generalization that interpolates between the delta-PID and the BROJA PID via a parameter lambda:

delta_lambda(M : X \ Y) = inf_{P(X'|MY)} E_M[D_KL(P(X|M) || P(X'|M))]
                          + lambda * I(M; X' | Y)

Redundancy is symmetrized:

RI = min{ I(M;X) - delta_lambda(M:X\Y), I(M;Y) - delta_lambda(M:Y\X) }

The following parameters are passed as keyword arguments to the constructor.

Parameters:

lam (float) – Lagrange multiplier (default 1.0). Large values approach the delta-PID; small values approach the BROJA PID.
bound (int, None) – Bound on auxiliary variable cardinality.
niter (int, None) – Number of optimization restarts.

\(\Igh{\bullet}\)

The redundancy measure of Griffith & Ho [GCJ+14] constructs a shared-information quantity via an auxiliary-variable optimization over channels from the joint sources to an intersection variable.

Note

TODO: confirm the canonical Griffith & Ho reference. The implementation in dit only identifies the authors; [GCJ+14] is used here as the best match among published Griffith papers in the PID literature.

class PID_GH(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]: The I_GH measure defined by Griffith & Ho.

\(\Iig{\bullet}\)

Niu & Quinn [NQ19] propose an information-geometric PID: the synergy is defined as the KL divergence from the true joint distribution to the nearest distribution satisfying \(X_0 - X_1 - Y\) or \(X_1 - X_0 - Y\) Markov chains, minimized over a convex mixture of the two.

class PID_IG(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]: The Niu & Quinn partial information decomposition.

\(\Iipid{\bullet}\)

The I-PID of Venkatesh, Gurushankar & Schamberg [VGS23] defines unique information via an information deficiency that maximizes the mutual-information gap attainable by a Markov test channel \(T - Y - (X_0, X_1)\):

\[\delta_I(Y : X_0 \setminus X_1) = \sup_{P(T|Y)} \left[ \I{T : X_0} - \I{T : X_1} \right]\]

Redundancy is obtained by min-symmetrization, analogously to \(\Idelta{\bullet}\).

class PID_IPID(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]

The I-PID from Venkatesh, Gurushankar & Schamberg (2023).

Unique information is defined via information deficiency:: delta_I(M : X Y) = sup_{P(T|M)} [ I(T;X) - I(T;Y) ]
and redundancy is symmetrized:: RI_I = min{ I(M;X) - delta_I(M:XY), I(M;Y) - delta_I(M:YX) }

\(\Imc{\bullet}\)

The more-capable intersection information of Gomes & Figueiredo [GF23] (building on [Kol22]) replaces the Blackwell-degradation order in \(\Ideg{\bullet}\) with the weaker more-capable channel preorder:

\[\Imc{X_{0:n} \to Y} = \max_{Q :\; Q \preceq_{\mathrm{mc}} X_i \;\forall i} \I{Q : Y}\]

where \(Q \preceq_{\mathrm{mc}} X_i\) iff \(\I{Q : Y} \leq \I{X_i : Y}\) for every input distribution on \(Y\).

class PID_MC(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]

More-capable intersection information I_mc^∩ from Gomes & Figueiredo.

Redundancy is the maximum I(Q; T) such that K_Q is “less capable” than every source channel for all input distributions.

Because the constraint set is discretized, the computed value is an upper bound that tightens with more samples.

References

A. F. C. Gomes and M. A. T. Figueiredo, “Orders between Channels and Implications for Partial Information Decomposition”, Entropy 25, 975, 2023.

\(\Iln{\bullet}\)

The less-noisy intersection information of Gomes & Figueiredo [GF23] uses the less noisy channel preorder, which sits between the Blackwell-degradation and more-capable orders, so \(\Ideg{\bullet} \leq \Iln{\bullet} \leq \Imc{\bullet}\):

\[\Iln{X_{0:n} \to Y} = \max_{Q :\; Q \preceq_{\mathrm{ln}} X_i \;\forall i} \I{Q : Y}\]

where \(Q \preceq_{\mathrm{ln}} X_i\) iff \(\I{U : Q} \leq \I{U : X_i}\) for every \(U\) with \(U - Y - (Q, X_i)\). Equivalently, the one-way secret key agreement rate from \(Y\) to \(Q\) with \(X_i\) eavesdropping vanishes, so \(\Iln{\bullet}\) is the most information about the target that cannot support secret key agreement against any single source.

By [vD97], \(Q \preceq_{\mathrm{ln}} X_i\) iff \(\I{Y : X_i} - \I{Y : Q}\) is concave in the distribution of \(Y\). dit imposes the equivalent pointwise condition on the Hessians of the output entropies at sampled distributions and directions, so the computed value is approximate. No bound on \(|Q|\) is known; it defaults to the largest source alphabet.

Warning

This measure appears to result in a negative PID. For \(p_Y = (0.3, 0.55, 0.15)\) with \(p(x_0 | y)\) rows \((0.2, 0.5, 0.3), (0, 0, 1), (0.1, 0.9, 0)\) and \(p(x_1 | y)\) rows \((0.1, 0, 0.9), (0.05, 0.5, 0.45), (0.05, 0.5, 0.45)\), \(\Iln{\bullet} \approx 0.021\) bits, below the coinformation \(0.6925 + 0.2341 - 0.8836 = 0.043\) bits, so the synergy is about \(-0.022\) bits. More generally, since \(\Iln{\bullet}\) depends only on the source-target marginals, it gives a nonnegative synergy on every distribution with those marginals only if it is at least \(\Ibroja{\bullet}\) [BRO+14] (Lemma 3), which here is \(0.050\) bits.

class PID_LN(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]

Less-noisy intersection information I_ln^∩ from Gomes & Figueiredo.

Redundancy is the maximum I(Q; T) over channels K_Q such that every source channel is less noisy than K_Q; equivalently, no one-way secret key can be agreed from T to Q against any single source.

References

A. F. C. Gomes and M. A. T. Figueiredo, “Orders between Channels and Implications for Partial Information Decomposition”, Entropy 25, 975, 2023.

\(\Imes{\bullet}\)

\(\Imes{\bullet}\) is a maximum-entropy PID inspired by BROJA’s “*” marginal-consistency assumption.

Note

TODO: reference. No canonical publication has been identified for \(\Imes{\bullet}\) in the dit source; the measure is described only as “inspired by BROJA’s * assumption”.

class PID_MES(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]

\(\Iprec{\bullet}\)

Kolchinsky [Kol22] defines a redundancy measure using the Blackwell precedence order on channels:

\[\Iprec{X_{0:n} \to Y} = \min_{s_{Q|Y}} \I{Y : Q} \quad \text{s.t.} \quad s_{Q|Y} \preceq p_{X_i|Y} \;\;\forall i\]

The constraint set is a convex polytope and the objective is convex, so the optimum lies at a vertex.

class PID_Prec(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]: The \(I_{\prec}\) measure defined by Kolchinsky.

\(\Ido{\bullet}\)

Lyu, Clark & Raviv [LCR24] give an explicit bivariate redundancy via Pearl-style do-calculus. An intervened source \(X'_i\) is defined by

\[p(X'_i, X_j, Y) = p(X_j)\, p(Y \mid X_j)\, p(X_i \mid Y)\]

and the redundancy is \(I(X'_1 : X_2) = I(X'_2 : X_1)\). Only defined for two sources.

In [27]: from dit.pid import PID_Do

In [28]: print(PID_Do(bivariates['synergy']))
+--------------------------+
|           I_do           |
+--------+--------+--------+
|  I_do  |  I_r   |   pi   |
+--------+--------+--------+
| {0:1}  | 1.0000 | 1.0000 |
|  {0}   | 0.0000 | 0.0000 |
|  {1}   | 0.0000 | 0.0000 |
| {0}{1} | 0.0000 | 0.0000 |
+--------+--------+--------+

class PID_Do(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]

The do-calculus partial information decomposition of Lyu, Clark & Raviv.

Notes

Bivariate only. The paper explicitly notes that I_do only fully determines the PID for n = 2 sources.

Synergistic Disclosure

Rosas, Mediano, Rassouli & Barrett [RMRB20] decompose \(I(X;Y)\) via \(\alpha\)-synergistic channels (see also Synergistic Disclosure). SynDisc inverts on the extended constraint lattice; ModifiedSynDisc is the backbone coarse-graining.

class ModifiedSynDisc(dist, sources=None, target=None, niter=None, bound=None)[source]

Modified synergistic disclosure decomposition.

Gutknecht, Makkeh & Wibral (2023), “From Babel to Boole: The Logical Organization of Information Decompositions”, arXiv:2306.00734v2.

Identical to SynDisc except that singleton constraint nodes (|alpha| = 1) use conditional mutual information I(T : a^C | a) instead of the optimization-based synergistic disclosure. This enforces the PID consistency condition: for every source collection a, the sum of atoms whose parthood distribution assigns 1 to a equals I(a : T).

Parameters:

dist (Distribution) – The joint distribution over sources and target.
sources (iter of iters, None) – The source variable groups. If None, all variables except the last are treated as individual sources.
target (iter, None) – The target variable. If None, the last variable is used.
niter (int, None) – Number of basin-hopping restarts per lattice node.
bound (int, None) – Cardinality bound on the auxiliary variable V.

class SynDisc(dist, sources=None, target=None, niter=None, bound=None)[source]

Synergistic disclosure decomposition (Rosas et al. 2020).

Decomposes I(X; Y) into atoms via Mobius inversion on the extended constraint lattice. Also provides the backbone decomposition, a non-negative coarse-graining that scales linearly with system size.

Parameters:

dist (Distribution) – The joint distribution over sources and target.
sources (iter of iters, None) – The source variable groups. If None, all variables except the last are treated as individual sources.
target (iter, None) – The target variable. If None, the last variable is used.
niter (int, None) – Number of basin-hopping restarts per lattice node.
bound (int, None) – Cardinality bound on the auxiliary variable V.

Secret Key Agreement Rates

One can associate Secret Key Agreement rates with unique informations [BG15] by considering the rate at which one source and the target can agree upon a secret key while the other source eavesdrops. The four classes are PID_SKAR_nw (no communication), PID_SKAR_owa (source to target), PID_SKAR_owb (target to source), and PID_SKAR_tw (two-way). This results in four possibilities:

neither source nor target communicate
only the source communicates
only the target communicates
both the source and the target communicate

No Communication

\[\Ipart{X_i \rightarrow Y \setminus X_j} = \operatorname{S}[X_i : Y || X_j]\]

Warning

This measure can result in an inconsistent PID.

One-Way Communication

Camel

\[\Ipart{X_i \rightarrow Y \setminus X_j} = \operatorname{S}[X_i \rightarrow Y || X_j]\]

Elephant

\[\Ipart{X_i \rightarrow Y \setminus X_j} = \operatorname{S}[X_i \leftarrow Y || X_j]\]

Warning

This measure can result in an inconsistent PID.

Two-Way Communication

\[\Ipart{X_i \rightarrow Y \setminus X_j} = \operatorname{S}[X_i \leftrightarrow Y || X_j]\]

Warning

This measure can result in an inconsistent PID.

class PID_SKAR_nw(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]

The two-way secret key agreement rate partial information decomposition.

Notes

This method progressively utilizes better bounds on the SKAR, and if even when using the tightest bounds does not result in a singular SKAR, nan is returned.

class PID_SKAR_owa(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]: The one-way secret key agreement rate partial information decomposition, source to target.

class PID_SKAR_owb(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]: The one-way secret key agreement rate partial information decomposition, target to source.

class PID_SKAR_tw(dist, sources=None, target=None, reds=None, pis=None, compute=True, **kwargs)[source]

The two-way secret key agreement rate partial information decomposition.

Notes

This method progressively utilizes better bounds on the SKAR, and if even when using the tightest bounds does not result in a singular SKAR, nan is returned.

Partial Entropy Decomposition

Ince [Inc17b] proposed applying the PID framework to decompose multivariate entropy (without considering information about a separate target variable). This partial entropy decomposition (PED), seeks to partition a mutlivariate entropy \(\H{X_0,X_1,\ldots}\) among the antichains of the variables. The PED perspective shows that bivariate mutual information is equal to the difference between redundant entropy and synergistic entropy.

\[\I{X_0 : X_1} = \Hpart{\left\{X_0\right\}, \left\{X_1\right\}} - \Hpart{\left\{X_0,X_1\right\}}\]

\(\Hcs{\bullet}\)

Taking a pointwise point of view, following \(\Iccs{\bullet}\), Ince has proposed a measure of redundant entropy based on the Co-Information [Inc17b]:

\[\Hcs{X_{0:n}} = \sum p(x_0, \ldots, x_n) \I{x_0 : \ldots : x_n}~~\textrm{if}~~(\I{x_0 : \ldots : x_n} > 0)\]

While this measure behaves intuitively in many examples, it also assigns negative values to some partial entropy atoms in some instances. However, Ince [Inc17b] argues that concepts such as mechanistic information redundnacy (non-zero information redundancy between independent predictors, c.f. AND) necessitate negative partial entropy terms.

Like \(\Iccs{\bullet}\), \(\Hcs{\bullet}\) is also subadditive.

In [29]: PED_CS(dit.Distribution(['00','01','10','11'],[0.25]*4))
Out[29]: <dit.pid.hcs.PED_CS object at 0x7c1fc494fdd0>

class PED_CS(dist, sources=None, **kwargs)[source]

The change in surprisal partial entropy decomposition, as defined by Ince (2017).

https://arxiv.org/abs/1702.01591

\(H_{\mathrm{mos}}\)

Finn & Lizier [FL20] define a partial entropy decomposition whose redundancy functional is the expected minimum of the source-group surprisals:

\[H_{\mathrm{mos}}(\alpha) = \mathbb{E}\bigl[\min_i h(A_i)\bigr]\]

In [30]: from dit.pid import PED_MOS

In [31]: print(PED_MOS(dit.Distribution(['00','01','10','11'], [0.25]*4)))
+--------------------------+
|          H_mos           |
+--------+--------+--------+
| H_mos  |  H_r   |  H_d   |
+--------+--------+--------+
| {0:1}  | 2.0000 | 1.0000 |
|  {0}   | 1.0000 | 0.0000 |
|  {1}   | 1.0000 | 0.0000 |
| {0}{1} | 1.0000 | 1.0000 |
+--------+--------+--------+

This is the decomposition that later work, such as [VPP+23], calls \(h_{\min}\).

class PED_MOS(dist, sources=None, pointwise=False, **kwargs)[source]

The min-of-surprisals partial entropy decomposition.

Based on Finn & Lizier (2020), “Generalised Measures of Multivariate Information Content”, Entropy 22(2), 216. https://doi.org/10.3390/e22020216

This is the \(h_{\min}\) decomposition. When pointwise=True, per-outcome redundancies and PI atoms are also computed via Mobius inversion on the lattice. Theorem 5 of the paper guarantees that every pointwise PI atom is non-negative.

\(H_{\mathrm{sx}}\)

Varley et al. [VPP+23] define the redundant entropy of an antichain \(\alpha = \{A_1, \ldots, A_k\}\) through shared probability mass exclusions: the surprisal of the event that at least one source group takes its observed value,

\[H_{\mathrm{sx}}(\alpha) = \mathbb{E}\left[-\log P\left(\bigcup_i \{A_i = a_i\}\right)\right]\]

This is the entropy analogue of \(\Isx{\bullet}\). Like PED_MOS, it accepts pointwise=True to keep the per-outcome atoms.

In [32]: from dit.pid import PED_SX

In [33]: print(PED_SX(dit.Distribution(['00','01','10','11'], [0.25]*4)))
+--------------------------+
|           H_sx           |
+--------+--------+--------+
|  H_sx  |  H_r   |  H_d   |
+--------+--------+--------+
| {0:1}  | 2.0000 | 0.4150 |
|  {0}   | 1.0000 | 0.5850 |
|  {1}   | 1.0000 | 0.5850 |
| {0}{1} | 0.4150 | 0.4150 |
+--------+--------+--------+

Unlike \(H_{\mathrm{cs}}\), which gives these independent bits no redundant entropy, \(H_{\mathrm{sx}}\) counts the \(\log_2 4/3\) bits it takes to exclude the outcome that matches neither observed value.

class PED_SX(dist, sources=None, pointwise=False, **kwargs)[source]

The shared-exclusions partial entropy decomposition \(h_{sx}\) of Varley et al. (2023).

https://doi.org/10.1073/pnas.2300888120

When pointwise=True, per-outcome redundancies and PI atoms are also computed via Mobius inversion on the lattice.

Generalized Information Decomposition

The local Kullback-Leibler divergence \(\log q(x)/p(x)\) of a posterior \(q\) from a prior \(p\) is the difference \(h_p(x) - h_q(x)\) of two local entropies. Varley [Var24b] therefore decomposes \(\DKL{q || p}\) over the redundancy lattice as the posterior expectation of the difference between the prior’s and the posterior’s pointwise partial entropy atoms. With the product of marginals as prior this decomposes the total correlation; with the uniform distribution, the negentropy. GID takes measure='sx' (PED_SX) or measure='mos' (PED_MOS).

In [34]: from dit.pid import GID

In [35]: from dit.distconst import product_distribution

In [36]: d = dit.example_dists.Xor()

In [37]: gid = GID(d, product_distribution(d))

In [38]: sum(gid[node] for node in gid._lattice)
Out[38]: 1.0

class GID(posterior, prior, sources=None, measure='sx')[source]

The generalized information decomposition of \(D_{KL}[\text{posterior} \| \text{prior}]\).

get_pi(node)[source]

The atom of node.

Parameters:: node (tuple of tuples) – A node of the redundancy lattice.
Returns:: pi
Return type:: float

get_pw_pi(node)[source]: The local atoms of node, {outcome: float} (in bits).

get_red(node)[source]: The redundant divergence of node: the sum of the atoms at and below it.

to_string(digits=4)[source]

A table of the redundant divergences and atoms.

Parameters:: digits (int) – Precision for display.
Returns:: table
Return type:: str

Integrated Information Decomposition

The integrated information decomposition (\(\Phi\mathrm{ID}\)) [MRCH+19, MRL+25] decomposes the mutual information between a set of sources (typically a system’s past) and a set of targets (its future) over the product of the two redundancy lattices. An atom \(\alpha \to \beta\) is information carried by the sources in the mode \(\alpha\) and by the targets in the mode \(\beta\). For two sources and two targets there are 16 atoms, including persistent redundancy \(\{0\}\{1\} \to \{2\}\{3\}\), information transfer \(\{0\} \to \{3\}\), and downward causation \(\{0:1\} \to \{2\}\).

The atoms are the Mobius inversion of a double redundancy \(I_\partial(\alpha \to \beta)\). PhiID provides three:

measure='mmi': \(\min_{i,j} \I{A_i : B_j}\) [MRCH+19].
measure='pm': the expectation of \(\min_i h(a_i) + \min_j h(b_j) - \min_{i,j} h(a_i, b_j)\), after [FL20].
measure='sx': the expectation of \(h_{\mathrm{sx}}(\alpha) + h_{\mathrm{sx}}(\beta) - h_{\mathrm{sx}}(\{A_i \cup B_j\})\) [Var23].

The atoms always sum to \(\I{X : Y}\), but individual atoms can be negative.

In [39]: from dit.pid import PhiID

In [40]: d = dit.Distribution(['0000', '1111'], [1/2, 1/2])

In [41]: PhiID(d)[(((0,), (1,)), ((2,), (3,)))]
Out[41]: 1.0

class PhiID(dist, sources=None, targets=None, measure='mmi', pointwise=False)[source]

The Integrated Information Decomposition of \(I[X : Y]\) for sources \(X\) and targets \(Y\).

get_red(node)[source]

The double redundancy \(I_\partial(\alpha \to \beta)\) of node.

Parameters:: node ((tuple of tuples, tuple of tuples)) – The pair \((\alpha, \beta)\) of source and target antichains.
Returns:: red
Return type:: float

get_pi(node)[source]

The PhiID atom \(\alpha \to \beta\).

Parameters:: node ((tuple of tuples, tuple of tuples)) – The pair \((\alpha, \beta)\) of source and target antichains.
Returns:: pi
Return type:: float

get_pw_red(node)[source]: Pointwise double redundancies of node, {outcome: float} (in bits).

get_pw_pi(node)[source]: Pointwise atoms of node, {outcome: float} (in bits).

property nodes: The nodes \((\alpha, \beta)\) of the product lattice.

to_string(digits=4)[source]

A table of the double redundancies and atoms.

Parameters:: digits (int) – Precision for display.
Returns:: table
Return type:: str

Representational Complexity

Ehrlich et al. [ESP+23] summarize a decomposition by the atom-weighted average degree of synergy: each atom is assigned the number of sources in its smallest group, so purely redundant (or unique) information has degree 1 and information available only from all \(n\) sources jointly has degree \(n\).

In [42]: from dit.pid.helpers import representational_complexity

In [43]: representational_complexity(PED_SX(dit.example_dists.Xor()))
Out[43]: 1.1225562489182657

representational_complexity(decomposition, comparator=<built-in function min>)[source]

The representational complexity of a partial information (or entropy) decomposition [ESP+23].

Each atom \(\alpha\) gets a degree, the number of sources in its smallest group (comparator=min, as in Ehrlich et al.) or its largest (comparator=max), and the complexity is the atom-weighted mean degree:

\[C = \frac{\sum_\alpha \deg(\alpha) \Pi(\alpha)}{\sum_\alpha \Pi(\alpha)}\]

It ranges from 1 (all redundant) to the number of sources (all synergistic).

Parameters:

decomposition (BasePID) – A computed PID or PED, such as PID_SX or PED_SX.
comparator ({min, max}) – How the sizes of an atom’s groups are reduced to its degree.

Returns:

complexity

Return type:

float

Comparing Measures

The helpers in dit.pid.helpers print side-by-side tables. compare_measures() runs several PID measures on one distribution (by default, every measure in dit.pid.measures.__all_pids), and pointwise_pid_table() lists the per-outcome atoms of a pointwise measure such as \(\Isx{\bullet}\):

In [44]: from dit.pid.helpers import compare_measures, pointwise_pid_table

In [45]: compare_measures(bivariates['and'], pids=[PID_WB, PID_MMI, PID_CCS])
+-----------------------------------------+
|              PID Comparison             |
+--------+----------+----------+----------+
|        |  I_min   |  I_mmi   |  I_ccs   |
+--------+----------+----------+----------+
| {0:1}  |  0.50000 |  0.50000 |  0.29248 |
|  {0}   |  0.00000 |  0.00000 |  0.20752 |
|  {1}   |  0.00000 |  0.00000 |  0.20752 |
| {0}{1} |  0.31128 |  0.31128 |  0.10376 |
+--------+----------+----------+----------+

In [46]: pointwise_pid_table(bivariates['and'], pid_class=PID_SX)
+---------------------------------------------------------------------------------------------------------------------------------------------------------------+
|                                                                        I_sx (pointwise)                                                                       |
+-----------------+--------+----------+-----------+-----------+---------+---------+---------+---------+---------+---------+-----------+------------+------------+
|      event      |   p    | {0:1} pi | {0:1} pi+ | {0:1} pi- |  {0} pi | {0} pi+ | {0} pi- |  {1} pi | {1} pi+ | {1} pi- | {0}{1} pi | {0}{1} pi+ | {0}{1} pi- |
+-----------------+--------+----------+-----------+-----------+---------+---------+---------+---------+---------+---------+-----------+------------+------------+
| ('0', '0', '0') | 0.2500 |  0.0000  |   0.4150  |   0.4150  | -0.0000 |  0.5850 |  0.5850 | -0.0000 |  0.5850 |  0.5850 |   0.4150  |   0.4150   |  -0.0000   |
| ('0', '1', '0') | 0.2500 |  0.4150  |   0.4150  |   0.0000  |  0.5850 |  0.5850 |  0.0000 | -0.4150 |  0.5850 |  1.0000 |  -0.1699  |   0.4150   |   0.5850   |
| ('1', '0', '0') | 0.2500 |  0.4150  |   0.4150  |   0.0000  | -0.4150 |  0.5850 |  1.0000 |  0.5850 |  0.5850 |  0.0000 |  -0.1699  |   0.4150   |   0.5850   |
| ('1', '1', '1') | 0.2500 |  0.4150  |   0.4150  |  -0.0000  |  0.5850 |  0.5850 | -0.0000 |  0.5850 |  0.5850 | -0.0000 |   0.4150  |   0.4150   |  -0.0000   |
+-----------------+--------+----------+-----------+-----------+---------+---------+---------+---------+---------+---------+-----------+------------+------------+

compare_measures(dist, pids=[<class 'dit.pid.measures.immi.PID_MMI'>, <class 'dit.pid.measures.iwedge.PID_GK'>, <class 'dit.pid.measures.imin.PID_WB'>, <class 'dit.pid.measures.irr.PID_RR'>, <class 'dit.pid.measures.iccs.PID_CCS'>, <class 'dit.pid.measures.ipm.PID_PM'>, <class 'dit.pid.measures.isx.PID_SX'>, <class 'dit.pid.measures.iproj.PID_Proj'>, <class 'dit.pid.measures.igh.PID_GH'>, <class 'dit.pid.measures.ibroja.PID_BROJA'>, <class 'dit.pid.measures.imes.PID_MES'>, <class 'dit.pid.measures.idep.PID_dep'>, <class 'dit.pid.measures.idep.PID_RA'>, <class 'dit.pid.measures.irav.PID_RAV'>, <class 'dit.pid.measures.iskar.PID_SKAR_nw'>, <class 'dit.pid.measures.iskar.PID_SKAR_owa'>, <class 'dit.pid.measures.iskar.PID_SKAR_owb'>, <class 'dit.pid.measures.iskar.PID_SKAR_tw'>, <class 'dit.pid.measures.iprec.PID_Prec'>, <class 'dit.pid.measures.ict.PID_CT'>, <class 'dit.pid.measures.iig.PID_IG'>, <class 'dit.pid.measures.iipid.PID_IPID'>, <class 'dit.pid.measures.idelta.PID_Delta'>, <class 'dit.pid.measures.ideltalambda.PID_DeltaLambda'>, <class 'dit.pid.measures.irdr.PID_RDR'>, <class 'dit.pid.measures.ideg.PID_Deg'>, <class 'dit.pid.measures.imc.PID_MC'>, <class 'dit.pid.measures.iln.PID_LN'>, <class 'dit.pid.measures.ido.PID_Do'>, <class 'dit.pid.measures.ici.PID_CI'>], inputs=None, output=None, name='', digits=5)[source]

Print the results of several partial information decompositions.

Parameters:

dist (dit.Distribution) – The distribution to compute PIDs of.
pids (iterable(BasePID)) – The PID classes to use.
inputs (iterable of iterables) – The variables to treat as inputs.
output (iterable) – The variables to consider as a singular output.
name (str) – The name of distribution.
digits (int) – The number of digits of precision to print.

pointwise_pid_table(dist, sources=None, target=None, pid_class=None, digits=4)[source]

Print a table of per-outcome pointwise PID values.

Each row corresponds to one joint outcome. Columns are: event, p, then for each lattice node (sorted by depth): {node} pi and — when available — {node} pi+ and {node} pi-.

Parameters:

dist (dit.Distribution) – The distribution to analyse.
sources (iter of iters, optional) – Source variable indices. Defaults to all variables except the last.
target (iter, optional) – Target variable indices. Defaults to the last variable.
pid_class (BasePointwisePID subclass, optional) – The pointwise PID measure to use. Defaults to PID_SX.
digits (int) – Number of decimal places to display.