dit: discrete information theory

dit is a Python package for discrete information theory.

Introduction

Information theory is a powerful extension to probability and statistics, quantifying dependencies among arbitrary random variables in a way that is consistent and comparable across systems and scales. Information theory was originally developed to quantify how quickly and reliably information could be transmitted across an arbitrary channel. The demands of modern, data-driven science have been coopting and extending these quantities and methods into unknown, multivariate settings where the interpretation and best practices are not known. For example, there are at least four reasonable multivariate generalizations of the mutual information, none of which inherit all the interpretations of the standard bivariate case. Which is best to use is context-dependent. dit implements a vast range of multivariate information measures in an effort to allow information practitioners to study how these various measures behave and interact in a variety of contexts. We hope that having all these measures and techniques implemented in one place will allow the development of robust techniques for the automated quantification of dependencies within a system and concrete interpretation of what those dependencies mean.

For a quick tour, see the Quickstart. The documentation is organized as follows:

  • Getting Started — overview, installation, and a quickstart tour.

  • Distributions — building, manipulating, and conditioning joint distributions.

  • Information Measures — Shannon, multivariate, common-information, divergence, and secret-key measures.

  • Decompositions and Profiles — partial information decomposition, information profiles, and visualization.

  • Channels, Coding, and Rate-Distortion — channel capacity and ordering, source/channel codes, rate-distortion, the information bottleneck, the privacy funnel, and the Gray–Wyner network.

  • Estimation and Optimization — plug-in and bias-corrected estimators from data, and distribution optimizers such as maximum entropy.

  • API Reference — an autogenerated listing of every public function and class, by subpackage.

  • Background — notation, common stumbling blocks, and references.

Note that all code snippets in this documentation assume that the following lines of code have already been run:

In [1]: import dit

In [2]: import numpy as np

Getting Started

Distributions

Information Measures

Decompositions and Profiles

Channels, Coding, and Rate-Distortion

Estimation and Optimization

API Reference

Background

Indices and tables