dit
: discrete information theory¶
dit
is a Python package for discrete information theory.
Introduction¶
Information theory is a powerful extension to probability and statistics, quantifying dependencies
among arbitrary random variables in a way tha tis consistent and comparable across systems and
scales. Information theory was originally developed to quantify how quickly and reliably information
could be transmitted across an arbitrary channel. The demands of modern, data-driven science have
been coopting and extending these quantities and methods into unknown, multivariate settings where
the interpretation and best practices are not known. For example, there are at least four reasonable
multivariate generalizations of the mutual information, none of which inherit all the
interpretations of the standard bivariate case. Which is best to use is context-dependent. dit
implements a vast range of multivariate information measures in an effort to allow information
practitioners to study how these various measures behave and interact in a variety of contexts. We
hope that having all these measures and techniques implemented in one place will allow the
development of robust techniques for the automated quantification of dependencies within a system
and concrete interpretation of what those dependencies mean.
For a quick tour, see the Quickstart. Otherwise, work your way through the various sections. Note that all code snippets in this documentation assume that the following lines of code have already been run:
In [1]: In [1]: from __future__ import division # true division for Python 2.7
Contents:
- General Information
- Notation
- Distributions
- Operations
- Finding Examples
- Optimization
- Information Measures
- Information Profiles
- Rate Distortion Theory
- Information Bottleneck
- APIs
- Partial Information Decomposition
- (Stumbling Blocks) On the Road to Understanding Multivariate Information Theory
- References