Overview
Information theory is a powerful extension to probability and statistics, quantifying dependencies
among arbitrary random variables in a way that is consistent and comparable across systems and
scales. Information theory was originally developed to quantify how quickly and reliably information
could be transmitted across an arbitrary channel. The demands of modern, data-driven science have
been coopting and extending these quantities and methods into unknown, multivariate settings where
the interpretation and best practices are not known. For example, there are at least four reasonable
multivariate generalizations of the mutual information, none of which inherit all the
interpretations of the standard bivariate case. Which is best to use is context-dependent. dit
implements a vast range of multivariate information measures in an effort to allow information
practitioners to study how these various measures behave and interact in a variety of contexts. We
hope that having all these measures and techniques implemented in one place will allow the
development of robust techniques for the automated quantification of dependencies within a system
and concrete interpretation of what those dependencies mean.
Implemented Measures
dit implements the following information measures and tools. Most measures are implemented in
multivariate and conditional generality, where such generalizations either exist in the literature
or are relatively obvious — for example, though it is not in the literature, the multivariate
conditional exact common information is implemented here. Each heading links to the relevant
section of the documentation.
- Entropies
Shannon entropy (joint and conditional) · Rényi entropy · Tsallis entropy · residual entropy / independent information / variation of information · necessary conditional entropy · cumulative residual entropy (generalized and conditional) · extropy · perplexity · negentropy · disequilibrium and LMPR complexity · α-synergy entropy spectra (synergistic, unique, union, intersection, and self-synergy entropies)
- Multivariate mutual informations
mutual information · co-information · interaction information · total correlation / multi-information · dual total correlation / binding information (and its generalization) · CAEKL mutual information · O-information · S-information · cohesion · Kirkwood mutual information · frustration (order-k and pairwise) · transmission · DeWeese mutual informations · cross mutual informations · union information · Quax synergy · synergistic disclosure (and its modified and backbone variants) · logarithmic decomposition · \(\Delta^k\) / \(\Gamma^k\) · TSE complexity · α-synergy information spectra
- Common informations
Gács–Körner · stochastic Gács–Körner · Wyner · exact · functional · MSS · Kamath–Anantharam (and its directed variant) · β common information · tension common information · channel synthesis rate · Salamatian–Cohen–Médard maximum entropy function
- Other measures
lautum information · Sibson mutual information (and its conditional variants) · maximal leakage
- Divergences
Kullback–Leibler divergence / relative entropy · cross entropy · Jensen–Shannon divergence · variational distance · Hellinger distance and divergence · Bhattacharyya coefficient · Chernoff information · Rényi, Tsallis, α-, and f-divergences · earth mover’s distance · coupling metrics · copy mutual information · maximum correlation · hypercontractivity coefficient
- Secret key agreement
secret key capacity (multiterminal, with omniscience rate) · no-communication, one-way, and two-way secret key agreement rates · secrecy capacity · intrinsic, reduced intrinsic, minimal intrinsic, two-part intrinsic, relaxed two-part, upper, lower, necessary, interactive, and less-noisy intrinsic mutual informations (with total correlation, dual total correlation, and CAEKL variants) · iterated discarding · secret key cost / information of formation
- Partial information decomposition
\(I_{min}\) · \(I_{MMI}\) · \(I_{\wedge}\) · \(I_{proj}\) · \(I_{BROJA}\) · \(I_{ccs}\) · \(I_{\pm}\) · \(I_{sx}\) · \(I_{dep}\) · \(I_{RA}\) · \(I_{RAV}\) · \(I_{RR}\) · \(I_{RDR}\) · \(I_{CT}\) · \(I_{deg}\) · \(I_{MC}\) · \(I_{LN}\) · \(I_{GH}\) · \(I_{IG}\) · \(I_{IPID}\) · \(I_{\delta}\) · \(I_{\delta\Lambda}\) · \(I_{ME}\) · \(I_{\prec}\) · \(I_{do}\) · \(I_{SKAR}\) (four variants) · \(I_{\cup}^{CI}\) (union-information synergy) · partial entropy decompositions (\(H_{cs}\), \(H_{MOS}\), \(H_{sx}\)) · synergistic disclosure decomposition · integrated information decomposition (ΦID) · generalized information decomposition (GID) · representational complexity
- Profiles and visualization
Shannon and extropy partitions (information diagrams) · complexity profile · marginal utility of information · connected informations (Schneidman, dual, and m-flat / Amari) · marginal lift profile · binding mixture / shared randomness decomposition · dependency decompositions (and dual) · Shapley decompositions · entropy triangles · information UpSet plots
- Channels and coding
channel capacity (certified, accelerated Blahut–Arimoto) · capacity–cost function · Sibson capacity · a catalog of example channels · channel orderings (degraded, less noisy, more capable, Blackwell, Shannon inclusion) and Le Cam / KL deficiencies · source codes (Shannon, Fano, Shannon–Fano–Elias, Huffman, length-limited Huffman, Golomb/Rice, Tunstall, Elias, Fibonacci, unary, polar) · channel codes (repetition, parity-check, Hamming, Reed–Muller, Golay, LDPC, polar, convolutional)
- Rate-distortion and Gray–Wyner
rate-distortion curves · information bottleneck (generalized, deterministic, and divergence variants) · privacy funnel (and perfect privacy) · Gray–Wyner network, region, and extension profile · lossy Wyner common information · Körner graph entropy · maximal and private interaction informations · excess functional information
- Estimation
distributions from data and time series · 17 plug-in and bias-corrected entropy estimators (Miller–Madow, Grassberger, Chao–Shen, NSB, Zhang, shrinkage, Bayesian, jackknife, …) · alphabet-aware conditional mutual information · k-NN differential entropy and KSG total correlation · permutation entropy · entropy rates (block, Lempel–Ziv, Bayesian posterior) · Markov order tests · surrogate data
- Optimization and constructions
maximum entropy distributions · iterative proportional fitting · m-projections · mixtures of products · marginal lifts · convex maximization over polytopes · minimal sufficient statistics and information trimming · lattice joins and meets · Kirkwood, Lancaster, and Fejér–Riesz approximations · symbolic (sympy) distributions · JAX, PyTorch, and PyTensor backends
Basic Information
- Documentation
- Downloads
https://pypi.org/project/dit/ and https://anaconda.org/conda-forge/dit
- Code and bug tracker
- License
BSD 3-Clause, see
LICENSE.txtfor details.- Dependencies
Citing
If you use dit in your research, please cite it as:
@article{dit,
Author = {James, R. G. and Ellison, C. J. and Crutchfield, J. P.},
Title = {{dit}: a {P}ython package for discrete information theory},
Journal = {The Journal of Open Source Software},
Volume = {3},
Number = {25},
Pages = {738},
Year = {2018},
Doi = {https://doi.org/10.21105/joss.00738}
}