dit

dit is a Python package for information theory on discrete random variables.

d = discrete i = information t = theory

However, the more precise statement (at this point) is that dit is a Python package for sigma-algebras defined on finite sets.

Classes

`CartesianProduct`	An abstract representation of a Cartesian product sample space.
`Distribution`	A distribution backed by an xarray DataArray.
`DistributionEnumerator`	This object enumerates all the uniform probability distributions over n variables each with cardinality k
`RVFunctions`	Helper class for building new random variables.
`SampleSpace`	An abstract representation of a sample space.
`ScalarSampleSpace`

Functions

`all_dist_structures`	Return an iterator of distributions over the 2(alphabet_size``outcome_length) possible combinations of joint events.
`copypmf`	Returns a NumPy array of the distribution's pmf.
`distribution_from_bayesnet`	Returns a distribution built from a Bayesian network.
`erasure`	Construct a version of dist where each variable has been passed through an erasure channel.
`expanded_samplespace`	Returns a new distribution with an expanded sample space.
`fejer_riesz_approximation`	Construct the order-k Fejér-Riesz approximation of a distribution, a nonnegative analogue of the Lancaster approximation.
`insert_rvf`	Returns a new distribution with an added random variable at index index.
`joint_from_factors`	Returns a joint distribution P(X,Y) built from P(X) and P(Y\|X).
`kirkwood_approximation`	Construct the unnormalized generalized Kirkwood superposition approximation of a distribution.
`lancaster_approximation`	Construct the order-k Lancaster approximation of a distribution, the additive analogue of the Kirkwood approximation.
`mixture_distribution`	Create a mixture distribution: \(\sum p_i d_i\)
`mixture_distribution2`	Create a mixture distribution: \(\sum p_i d_i\)
`modify_outcomes`	Returns dist but with modified outcomes, after passing them to ctor.
`noisy`	Construct a noisy version of dist.
`product_distribution`	Returns a new distribution which is the product of marginals.
`pruned_samplespace`	Returns a new distribution with pruned sample space.
`random_dist_structure`	Return a uniform distribution over a random subset of the alphabet_size`**`outcome_length possible joint events.
`random_distribution`	Returns a random distribution drawn uniformly from the simplex.
`random_scalar_distribution`	Returns a random scalar distribution over n outcomes.
`simplex_grid`	Returns a generator over distributions, determined by a grid.
`uniform`	Produces a uniform distribution over outcomes.
`uniform_distribution`	Returns a uniform distribution.
`uniform_like`	Returns a uniform distribution with the same outcome length, alphabet size, and base as dist.
`uniform_scalar_distribution`	Returns a uniform scalar distribution over n outcomes.

Reference

class CartesianProduct(alphabets, product=None)[source]

Bases: SampleSpace

An abstract representation of a Cartesian product sample space.

index(item)[source]: Returns a key for sorting items in the sample space.

coalesce(rvs, extract=False)[source]

Returns a new sample space after coalescing the specified indexes.

Given n lists, each consisting of indexes from the original sample space, the coalesced sample space has n indexes, where the alphabet for each index is a tuple of the indexes that defined the ith list.

Parameters:

rvs (sequence) – A sequence whose elements are also sequences. Each inner sequence defines an index in the new distribution as a combination of indexes in the original distribution. The length of rvs must be at least one. The inner sequences need not be pairwise mutually exclusive with one another, and each can contain repeated indexes.
extract (bool) – If the length of rvs is 1 and extract is True, then instead of the new outcomes being 1-tuples, we extract the sole element to create a sample space over the indexes in rvs[0].

Returns:

ss – The coalesced sample space.

Return type:

SampleSpace

Examples

If we have a joint distribution over 3 random variables such as:: Z = (X,Y,Z)
and would like a new sample space over 6 random variables:: Z = (X,Y,Z,X,Y,Z)
then this is achieved as:: coalesce([[0,1,2,0,1,2]], extract=True)
If you want:: Z = ((X,Y), (Y,Z))
Then you do:: coalesce([[0,1],[1,2]])

Notes

Generally, the outcomes of the new sample space will be tuples instead of matching the class of the original sample space. This is because some classes are not recursive containers. For example, one cannot have a string of strings where each string consists of more than one character. Note however, that it is perfectly valid to have a tuple of tuples. Either way, the elements within each tuple of the sample space will still match the class of the original sample space.

See also

marginal, marginalize

class Distribution(data, pmf=None, rv_names=None, free_vars=None, given_vars=None, base='linear', sample_space=None, sparse=True, trim=True, sort=True, validate=True, prng=None)[source]

Bases: object

A distribution backed by an xarray DataArray.

The distribution tracks which dimensions are “free” (joint) variables and which are “given” (conditioned on). This allows natural algebraic operations:

Multiplying p(X,Y) * p(Z|X,Y) yields p(X,Y,Z)
Summing over a free variable marginalizes it out
Dividing by a marginal conditions on it

data

The underlying probability array.

Type:: xr.DataArray

free_vars

The names of the free (joint) variables. For p(X,Y|Z) this is {'X','Y'}.

Type:: frozenset of str

given_vars

The names of the conditioned variables. For p(X,Y|Z) this is {'Z'}.

Type:: frozenset of str

ops

The operations instance for the current probability base.

Type:: Operations

Notes

Normalization convention: for a distribution p(X,Y|Z), summing over all free variables (X and Y) for each fixed value of Z should yield 1.

classmethod from_distribution(dist, rv_names=None)[source]

Create an Distribution from an existing distribution, optionally renaming its random variables.

Parameters:

dist (Distribution) – The source distribution.
rv_names (list of str, optional) – Names for each random variable. If None, uses the distribution’s existing rv_names, or defaults to 'X0', 'X1', etc.

Returns:

xrd

Return type:

Distribution

classmethod from_array(arr, dim_names, alphabets, free_vars=None, given_vars=None, base='linear')[source]

Create an Distribution from a numpy array.

Parameters:

arr (np.ndarray) – The probability array.
dim_names (list of str) – Names for each dimension.
alphabets (list of list) – The alphabet (coordinate values) for each dimension.
free_vars (set-like of str, optional) – Names of the free variables.
given_vars (set-like of str, optional) – Names of the conditioned variables.
base (str or float) – Probability base ('linear', 2, 'e', …).

Returns:

xrd

Return type:

Distribution

classmethod from_factors(marginal, conditional)[source]

Build a joint distribution from a marginal and a conditional.

p(X,Y) = p(X) * p(Y|X)

Parameters:

marginal (Distribution) – The marginal distribution, e.g. p(X).
conditional (Distribution) – The conditional distribution, e.g. p(Y|X).

Returns:

joint – The resulting joint distribution.

Return type:

Distribution

property dims: All dimension (variable) names as a tuple, in array order.

property shape: Shape of the underlying array.

property all_vars: All variable names as a frozenset.

property alphabet

Tuple of alphabets, one per dimension (in array-dimension order).

This mirrors dit.Distribution.alphabet.

property outcomes

Tuple of outcomes in lexicographic order.

When sparse (the default), only non-zero probability outcomes are included. After make_dense(), all outcomes are included.

For 1-D numerical distributions, outcomes are the coordinate values directly (e.g. (0, 1, 2)). For multi-variable distributions or distributions with non-numeric coordinates, each outcome is a tuple whose elements correspond to the dimensions in dims order.

property pmf

1-D numpy array of probabilities corresponding to outcomes.

Returns values in the current base (log if the distribution is in log space, linear otherwise), matching dit.Distribution.pmf.

to_dict()[source]

Return a dictionary mapping outcomes to probabilities.

Returns:: d – {outcome_tuple: float}
Return type:: dict

outcome_length()[source]

Number of random variables (dimensions).

Returns:: n
Return type:: int

get_rv_names()[source]

Return the variable names as a tuple, or None if not explicitly set.

Returns:: names
Return type:: tuple of str or None

set_rv_names(rv_names)[source]

Rename the dimensions (random variables).

Parameters:: rv_names (list of str) – New names, one per dimension.

is_joint()[source]

True if this distribution describes more than one random variable.

Returns False for 1-D distributions (single RV) and conditional distributions.

is_conditional()[source]: True if this is a conditional distribution.

is_log()[source]: True if the distribution stores log probabilities.

is_dense()[source]: True when the distribution reports all outcomes (including zero-probability).

is_sparse()[source]: True when the distribution reports only non-zero outcomes.

is_numerical()[source]

True if all coordinate values across all dimensions are numeric.

When True, operations like mean(), std(), and variance() are well-defined.

is_symbolic()[source]

True if the probabilities are symbolic (sympy) expressions.

Symbolic distributions store their pmf as an object-dtype array of sympy expressions rather than floats, enabling exact/algebraic computation of information measures.

is_homogeneous()[source]: True if the alphabet for each random variable is the same.

has_outcome(outcome, null=True)[source]

Check if outcome exists in the sample space.

Parameters:

outcome (tuple or str) – The outcome to check.
null (bool) – If True, accept zero-probability outcomes in the sample space. If False, only accept outcomes with positive probability.

atoms(patoms=False)[source]

Yield atoms of the probability space.

Parameters:: patoms (bool) – If True, yield only positive-probability atoms.

event_space()[source]: Return a generator over the event space (powerset of sample space).

rand(size=None, rand=None, prng=None)[source]

Return a random sample from the distribution.

Parameters:

size (int or None) – Number of samples. None for a single sample.
rand (float, array, or None) – Pre-generated random numbers. None to generate internally.
prng (random state, optional) – Random number generator. Defaults to self.prng.

property rvs

List of RV groupings, one per free variable.

Each element is a single-element list containing a sequential integer index (0-based among the free dims), matching the dit.Distribution.rvs convention. Integer indices are used (rather than names) so that helpers like flatten do not recursively split multi-character strings.

make_dense()[source]

Switch to dense mode so that outcomes and pmf include all outcomes (including zero-probability ones).

Returns:: Always returns 0 (the DataArray is inherently dense).
Return type:: int

make_sparse(trim=True)[source]

Switch to sparse mode so that outcomes and pmf include only non-zero probability outcomes (the default).

Parameters:: trim (bool, optional) – Ignored. Kept for API compatibility.
Returns:: Always returns 0.
Return type:: int

zipped(mode='pmf')[source]

Iterator over (outcome, probability) tuples.

Parameters:

mode (str) – 'pmf' to iterate over non-zero outcomes (default), 'atoms' to iterate over the full sample space, 'patoms' is treated identically to 'pmf' (provided for dit.Distribution compatibility).

Yields:

outcome (scalar or tuple)
probability (float)

get_base(numerical=False)[source]

Return the current probability base.

Parameters:: numerical (bool) – If True and the base is 'e', return its float value.
Returns:: base
Return type:: str or float

set_base(base)[source]

Change the probability base in-place.

Parameters:: base (str or float) – 'linear', 2, 'e', or any positive float.

to_string(digits=None, exact=None, tol=1e-09, show_mask=False, str_outcomes=False)[source]

Return a string representation compatible with dit.Distribution format.

Parameters:

digits (int or None) – Round probabilities. None for no rounding.
exact (bool or None) – If True, display as fractions. None uses ditParams.
tol (float) – Fraction tolerance when exact=True.
show_mask (bool) – Ignored (kept for API compatibility).
str_outcomes (bool) – If True, attempt to join tuple outcomes into strings.

to_html(digits=None, exact=None, tol=1e-09)[source]: Return an HTML representation compatible with dit.Distribution format.

validate(atol=1e-09)[source]

Validate normalisation.

For a joint distribution p(X,Y), the total sum should be 1. For a conditional p(X|Y), the sum over X for each Y should be 1.

Parameters:: atol (float) – Absolute tolerance.
Returns:: valid
Return type:: bool
Raises:: ValueError – If the distribution is not properly normalised.

marginal(*args)[source]

Marginalise to keep only the specified free variables.

Given (conditioned) variables are always kept.

Supports two call signatures:

marginal('X', 'Y') – positional variable names
marginal(['X', 'Y']) – list of names (or integer indices)

Parameters:: *args (str, or a single list/tuple) – The free variable names to keep. Integer indices are auto-resolved to dimension names.
Returns:: result
Return type:: Distribution

marginalize(*args)[source]

Marginalise out (remove) the specified free variables.

Supports two call signatures:

marginalize('X') – positional variable names
marginalize(['X']) – list of names (or integer indices)

Parameters:: *args (str, or a single list/tuple) – The free variable names to remove.
Returns:: result
Return type:: Distribution

coalesce(rvs, extract=False)[source]

Return a new distribution after coalescing random variables.

Each inner sequence in rvs defines one new random variable as a combination of original variables. The result is a joint Distribution over len(rvs) new random variables whose outcomes are tuples (or the inner values when extract=True with a single group).

Parameters:

rvs (sequence of sequences) – Each inner sequence contains variable names (or integer indices).
extract (bool) – If True and len(rvs) == 1, the single group’s values are used directly as outcomes instead of being wrapped in 1-tuples.

Returns:

d

Return type:

Distribution

condition_on(*cond_vars, rvs=None, crvs=None)[source]

Condition on the specified free variables.

Supports two call signatures:

Native: condition_on('X', 'Y') – positional var names. Returns a single conditional Distribution.
dit-compat: any of these forms triggers the dit-compatible return format (marginal, list_of_conditionals):
- condition_on(crvs=['X'], rvs=['Y'])
- condition_on(['X'], rvs=['Y']) (positional crvs)
- condition_on(crvs=['X'])
The returned list contains one Distribution per outcome of the conditioning variable.

Parameters:

*cond_vars (str, or a single list/tuple) – Variable names to condition on. If a single list/tuple is passed and rvs is provided, it is interpreted as crvs (dit-compat positional form).
rvs (list, optional) – Variables to keep in the conditional (dit-compat API).
crvs (list, optional) – Variables to condition on (dit-compat API).

Returns:

result – A single conditional distribution (native), or a (marginal, list_of_Distributions) tuple (dit-compat).

Return type:

Distribution or tuple

Examples

>>> p_xyz.condition_on('Z')   # native: returns p(X,Y|Z)
>>> p_xyz.condition_on('X', 'Y')  # native: returns p(Z|X,Y)
>>> marg, cdists = p_xyz.condition_on(crvs=['Z'])  # dit-compat
>>> marg, cdists = p_xyz.condition_on(['Z'], rvs=['X'])  # dit-compat

classmethod from_ndarray(ndarray, base=None, prng=None)[source]

Construct from a multi-dimensional numpy ndarray interpreted as a pmf.

Each axis represents a random variable, and the index along that axis is the variable’s value. For example, a (2, 3) array has two variables with alphabet sizes 2 and 3 respectively.

Parameters:

ndarray (np.ndarray)
base (str or float, optional)
prng (random state, optional)

classmethod from_rv_discrete(ssrv, base=None, prng=None)[source]

Construct from a scipy.stats.rv_discrete instance.

Parameters:

ssrv (scipy.stats.rv_discrete) – A frozen discrete random variable with .xk and .pk attributes (as produced by rv_discrete(values=...)).
base (str or float, optional) – Probability base. Defaults to 'linear'.
prng (random state, optional)

entropy(base=2)[source]

Compute the (conditional) entropy.

Note: the true conditional entropy H(X|Y) = sum_y p(y) H(X|Y=y) requires knowledge of the marginal p(Y), which is not stored. Use a joint distribution and H(X,Y) - H(Y) for the exact value.

Parameters:: base (float, optional) – Logarithm base for the result (default: 2).
Returns:: h – The (conditional) entropy.
Return type:: float

mutual_information(var_x, var_y, base=2)[source]

Compute the mutual information I(X;Y).

Only valid for joint distributions.

Parameters:

var_x (str or set of str) – Variable(s) for the first argument of I(X;Y).
var_y (str or set of str) – Variable(s) for the second argument of I(X;Y).
base (float, optional) – Logarithm base for the result (default: 2).

Returns:

mi – The mutual information I(X;Y).

Return type:

float

sel(**kwargs)[source]

Fix variables to specific values (label-based selection).

Parameters:: **kwargs – Variable-name to value mappings.
Returns:: result – If all dimensions are selected, returns a float (probability or log probability, depending on the distribution’s base). Otherwise returns a reduced Distribution.
Return type:: Distribution or float

Examples

>>> p_xyz.sel(Y='0')        # p(X,Z) at Y=0 (un-normalised slice)
>>> p_xyz.sel(X='0', Y='1') # p(Z) at X=0,Y=1

copy(base=None)[source]

Return a deep copy of this distribution.

Parameters:: base (str or float, optional) – If given, the copy will be converted to this base.
Returns:: c
Return type:: Distribution

to_distribution()[source]: Deprecated. Returns self since Distribution is now the sole distribution class.

to_numpy()[source]

Return the underlying data as a numpy array.

Returns:: Copy of the probability values (linear or log, depending on base).
Return type:: np.ndarray

property DataArray

Return the underlying xr.DataArray (read-only view).

Returns:: The probability array. Modifying it may affect this distribution.
Return type:: xr.DataArray

is_approx_equal(other, atol=1e-09, rtol=None)[source]

Check approximate equality of two distributions.

Compares by sample space and per-outcome probabilities, ignoring dimension names. This matches the old dit.Distribution behavior.

Parameters:

other (Distribution) – Distribution to compare against.
atol (float, optional) – Absolute tolerance for value comparison (default: 1e-9).
rtol (float, optional) – Ignored (kept for signature compatibility).

Returns:

eq

Return type:

bool

normalize()[source]

Normalise the distribution in-place.

For a joint distribution, divides by the total sum. For a conditional, normalises each conditional slice.

Return type:: None

sample_space()[source]

Iterator over all outcomes in the sample space.

Yields:: outcome (tuple)

event_probability(event)[source]

Compute the probability of an event (subset of outcomes).

Parameters:: event (iterable of tuples) – Outcomes in the event.
Returns:: p
Return type:: float

class DistributionEnumerator(n, k)[source]

Bases: object

This object enumerates all the uniform probability distributions over n variables each with cardinality k

class RVFunctions(d)[source]

Bases: object

Helper class for building new random variables.

Each new random variable is a function of the existing random variables. So for each outcome in the original distribution, there can be only one possible value for the new random variable.

Some methods may make assumptions about the sample space. For example, the xor() method assumes the sample space consists of 0-like and 1-like outcomes.

xor(indices)[source]

Returns a callable which returns the logical XOR of the given indices.

Outcomes are assumed to be strings of ‘0’ and ‘1’, or tuples of 0 and 1. The returned function handles both cases appropriately.

Parameters:: indices (list) – A list of two indices used to take the XOR.
Returns:: func – A callable implementing the XOR function. It receives a single argument, the outcome and returns an outcome for the calculation.
Return type:: function

Examples

>>> d = dit.Distribution(['00', '01', '10', '11'], [1/4]*4)
>>> bf = dit.RVFunctions(d)
>>> d = dit.insert_rvf(d, bf.xor([0,1]))
>>> d.outcomes
('000', '011', '101', '110')

from_mapping(mapping, force=True)[source]

Returns a callable implementing a random variable via a mapping.

Parameters:

mapping (dict) – A mapping from outcomes to values of the new random variable.
force (bool) – Ideally, the values of mapping should be satisfy the requirements of all outcomes (hashable, ordered sequences), but if force is True, we will attempt to use the distribution’s outcome constructor and make sure that they are. If they are not, then the outcomes will be placed into a 1-tuple. This is strictly a convenience for users. As an example, suppose the outcomes are strings, the values of mapping can also be strings without issue. However, if the outcomes are tuples of integers, then the values should also be tuples. When force is True, then the values can be integers and then they will be transformed into 1-tuples.

Returns:

func – A callable implementing the desired function. It receives a single argument, the outcome, and returns an outcome for the calculation.

Return type:

function

Examples

>>> d = dit.Distribution(['00', '01', '10', '11'], [1/4]*4)
>>> bf = dit.RVFunctions(d)
>>> mapping = {'00': '0', '01': '1', '10': '1', '11': '0'}
>>> d = dit.insert_rvf(d, bf.from_mapping(mapping))
>>> d.outcomes
('000', '011', '101', '110')

Same example as above but now with tuples.

>>> d = dit.Distribution([(0,0), (0,1), (1,0), (1,1)], [1/4]*4)
>>> bf = dit.RVFunctions(d)
>>> mapping = {(0,0): 0, (0,1): 1, (1,0): 1, (1,1): 0}
>>> d = dit.insert_rvf(d, bf.from_mapping(mapping, force=True))
>>> d.outcomes
((0, 0, 0), (0, 1, 1), (1, 0, 1), (1, 1, 0))

See also

marginal, marginalize

marginal(rvs)[source]

Returns a marginal distribution.

Parameters:: rvs (list of int) – The indexes to keep. All others are marginalized.
Returns:: ss – A new sample space with only indexes in rvs.
Return type:: SampleSpace

marginalize(rvs)[source]

Returns a new sample space after marginalizing the specified indexes.

Parameters:: rvs (list of int) – The indexes to marginalize. All others are kept.
Returns:: ss – A new sample space with indexes in rvs marginalized away.
Return type:: SampleSpace

class ScalarSampleSpace(samplespace)[source]: Bases: BaseSampleSpace

all_dist_structures(outcome_length, alphabet_size)[source]

Return an iterator of distributions over the 2**(alphabet_size`**`outcome_length) possible combinations of joint events.

Parameters:

outcome_length (int) – The length of outcomes to consider.
alphabet_length (int) – The size of the alphabet for each random variable.

Yields:

d (Distribution) – A uniform distribution over a subset of the possible joint events.

copypmf(d, base=None, mode='asis')[source]

Returns a NumPy array of the distribution’s pmf.

Parameters:

d (distribution) – The distribution from which the pmf is copied.
base (float, 'linear', 'e', None) – The desired base of the probabilities. If None, then the probabilities maintain their current base.
mode (['dense', 'sparse', 'asis']) – A specification of how the pmf should be construted. ‘dense’ means that the pmf should contain the entire sample space. ‘sparse’ means the pmf should only contain nonnull probabilities. ‘asis’ means to make a copy of the pmf, as it exists in the distribution.

Returns:

pmf – The pmf of the distribution.

Return type:

NumPy array

distribution_from_bayesnet(digraph, nodes=None, sample_space=None, attr='dist')[source]

Returns a distribution built from a Bayesian network.

Each node represents a random variable X_i. Every node must store its conditional probability distribution P(X_i | Y_i) where Y_i represents the parents of X_i. If a node has no in-degree, then it must store the probability distribution P(X_i).

Parameters:

digraph (NetworkX digraph) – A directed graph, representing the Bayesian graphical model.
nodes (list, None) – The order of the nodes that will determine the random variable order. If None, then we use sorted(digraph.nodes()), which assumes the nodes are sortable. The reason we assume they are sortable is because the parent values must correspond to the node order and thus, we need an unambiguous ordering that the user could have known ahead of time.
sample_space (SampleSpace) – If provided, this specifies the outcomes of the distribution to be constructed. The distributions stored on the nodes are assumed to be compatible with this space. If functions are stored on the nodes, then this parameter must be provided.
attr (str) –
The attribute for each node that holds the conditional distributions. The attribute value can take a variety of forms.

It can be a function. The function must take two arguments. The first is the value of random variable for the current node. The second is a dictionary of keyed by parents whose values are the values of the random variables corresponding to the parents. The function should return the probability P(node_val|parent_vals).

It can be a list, such as [parents, dists], that holds the parents and the conditional distributions: dists[i] = P(X | Y_i = parents[i]). It can also be a dict-like structure so that dists[y] is a distribution representing P(X | Y = y)`. If the node has no in-degree, then the attribute value should store the distribution only. When using distributions, each should have random variable names assigned that match the nodes in the graph, or alternatively, all nodes in the graph should be integers and then random variable names are not necessary. The order of elements within these lists (or the dict) does not matter, but for nodes that have parents, the order of the random variables that specify the parents, must match the order of nodes. So for example, if the node order is [2, 1, 0] and node 1 has parents 0 and 2. Then the parents for node 1 will be such that the first element corresponds to node 2 and the second to node 0, since 2 precedes 0 in the node order. All distributions will be assumed to have the same base, and this base will determine the base of the constructed distribution.

Returns:

dist – The joint distribution.

Return type:

Distribution

Examples

The Xor logic gate has the following structure:

>>> g = nx.DiGraph()
>>> g.add_edge(0, 2)
>>> g.add_edge(1, 2)

Let’s add distributions to it using functions.

>>> uniform = lambda node_val, parents: 0.5
>>> def xor(node_val, parents):
...     if '1' == parents[0] == parents[1]:
...         desired_output = '1'
...     else:
...         desired_output = '0'
...     return int(node_val == desired_output)
...
>>> g.nodes[0]['dist'] = uniform
>>> g.nodes[1]['dist'] = uniform
>>> g.nodes[2]['dist'] = xor
>>> ss = ['000', '001', '010', '011', '100', '101', '110', '111']
>>> d = dit.distribution_from_bayesnet(g, sample_space=ss)

Alternatively, we could add distributions using Distribution objects.

>>> uniform = dit.uniform_distribution(1, 2)
>>> sample_space1 = [(0,), (1,)]
>>> one = dit.Distribution(sample_space1, [0, 1])
>>> zero = dit.Distribution(sample_space1, [1, 0])
>>> sample_space2 = [(0, 0), (0, 1), (1, 0), (1, 1)]
>>> xor = [ sample_space2, [zero, one, one, zero]]
>>> g.nodes[0]['dist'] = uniform
>>> g.nodes[1]['dist'] = uniform
>>> g.nodes[2]['dist'] = xor
>>> d = dit.distribution_from_bayesnet(g)

We can add noise whenever the output would normally be 1.

>>> noisy = dit.Distribution(sample_space, [.1, .9])
>>> dists = [zero, noisy, noisy, zero]
>>> g.nodes[2]['dist'][1] = dists
>>> d = dit.distribution_from_bayesnet(g)

erasure(dist, epsilon=0.5)[source]

Construct a version of dist where each variable has been passed through an erasure channel.

Parameters:

dist (Distribution) – The distribution to fuzz.
epsilon (float, 0 <= epsilon <= 1) – The erasure probability.

Returns:

erased – The erased distribution.

Return type:

Distribution

expanded_samplespace(d, alphabets=None, union=True)[source]

Returns a new distribution with an expanded sample space.

Expand the sample space so that it is the Cartesian product of the alphabets for each random variable.

Parameters:

d (distribution) – The distribution used to create the pruned distribution.
alphabets (list) – A list of alphabets, with length equal to the outcome length in d. Each alphabet specifies the alphabet to be used for a single index random variable. The sample space of the new distribution will be the Cartesian product of these alphabets.
union (bool) – If True, then the alphabet for each random variable is unioned. The unioned alphabet is then used for each random variable.

Returns:

ed – The distribution with an expanded sample space.

Return type:

distribution

fejer_riesz_approximation(dist, rvs=None, order=None)[source]

Construct the order-k Fejér-Riesz approximation of a distribution, a nonnegative analogue of the Lancaster approximation.

The Lancaster approximation truncates the interaction expansion of the density ratio \(p / \pi\), where \(\pi(x) = \prod_i p(x_i)\), and so may be negative. In the spirit of the Fejér-Riesz theorem, which writes a nonnegative trigonometric polynomial as \(|h|^2\) [Fejer16], this approximation instead truncates the amplitude \(s = \sqrt{p / \pi}\) and squares the result:

\[s_k(x) = \sum_{|S| \leq k} c_S \, \mathbb{E}_\pi[s \mid x_S], \qquad q_k(x) = \frac{\pi(x) \, s_k(x)^2}{Z_k},\]

with the coefficients \(c_S\) of lancaster_approximation(). \(s_k\) is the \(L^2(\pi)\) projection of \(s\) onto functions of at most \(k\) variables [Lan69], so \(Z_k = \|s_k\|^2_\pi \leq \|s\|^2_\pi = 1\). Order 0 is the product of marginals, and \(q_k = p\) at every order when \(p\) is itself a product of marginals. Unlike the Lancaster approximation, \(q_k\) is a probability distribution, but it need not match the marginals of \(p\). No canonical literature source is known for this construction.

Parameters:

dist (Distribution) – The distribution to approximate.
rvs (list, None) – A list of lists. Each inner list specifies the indexes of the random variables forming one variable of the approximation. If None, then each random variable is its own variable.
order (int, None) – The largest interaction size k kept in the amplitude. Must satisfy 0 <= k <= n - 1. If None, defaults to n - 1.

Returns:

approx – The dense, normalized approximation over the outcomes of dist coalesced to rvs.

Return type:

Distribution

Raises:

ditException – Raised if order is out of range.

Examples

>>> d = dit.Distribution(['000', '010', '100', '111'], [1/4] * 4)
>>> dit.fejer_riesz_approximation(d).pmf.round(4)
array([0.3879, 0.0248, 0.1946, 0.0248, 0.1946, 0.0248, 0.0083, 0.1404])

insert_rvf(d, func, index=-1)[source]

Returns a new distribution with an added random variable at index index.

The new random variable must be a function of the other random variables. By this, we mean that the entropy of the new random variable conditioned on the original random variables should be zero.

Parameters:

dist (Distribution) – The distribution used to construct the new distribution.
func (callable | list of callable) – A function which takes a single argument—the value of the previous random variables—and returns a new random variable. Note, the return value will be added to the outcome using __add__, and so it should be a hashable, orderable sequence (as every outcome must be). If a list of callables is provided, then multiple random variables are added simultaneously and will appear in the same order as the list.
index (int) – The index at which to insert the random variable. A value of -1 is will append the random variable to the end.

Returns:

d – The new distribution.

Return type:

Distribution

Examples

>>> d = dit.Distribution(['00', '01', '10', '11'], [1/4]*4)
>>> def xor(outcome):
...    return str(int(outcome[0] != outcome[1]))
...
>>> d2 = dit.insert_rvf(d, xor)
>>> d.outcomes
('000', '011', '101', '110')

joint_from_factors(mdist, cdists, strict=True)[source]

Returns a joint distribution P(X,Y) built from P(X) and P(Y|X).

Parameters:

mdist (Distribution) – The marginal distribution P(X).
cdists (list of Distribution) – The list of conditional distributions P(Y|X=x).
strict (bool) – If True, require that the marginal and conditional masks are complementary. If False, concatenate X before Y.

Returns:

d – The joint distribution P(X,Y).

Return type:

Distribution

kirkwood_approximation(dist, rvs=None, order=None)[source]

Construct the unnormalized generalized Kirkwood superposition approximation of a distribution.

For variables \(X_0, \ldots, X_{n-1}\) the order-\(k\) approximation truncates the Möbius (interaction) expansion of \(\log p\) at subsets of size at most \(k\) [KYKG07, Wat60]:

\[\tilde{p}_k(x) = \prod_{1 \leq |S| \leq k} p(x_S)^{c_S}, \qquad c_S = (-1)^{k-|S|} \binom{n-|S|-1}{k-|S|}.\]

The default order \(k = n-1\) is the standard Kirkwood approximation, e.g. \(p(x, y) p(x, z) p(y, z) / \left(p(x) p(y) p(z)\right)\) for three variables. Order 1 is the product of marginals and order 0 is constant one. The approximation is not normalized: with \(Z\) its total mass, kirkwood_mutual_information() is \(D_{KL}(p \| \tilde{p}_k / Z)\). Wherever a marginal used by the product vanishes, the approximation is zero. See lancaster_approximation() for the additive analogue.

Parameters:

dist (Distribution) – The distribution to approximate.
rvs (list, None) – A list of lists. Each inner list specifies the indexes of the random variables forming one variable of the approximation. If None, then each random variable is its own variable.
order (int, None) – The largest marginal size k used by the approximation. Must satisfy 0 <= k <= n - 1. If None, defaults to n - 1.

Returns:

approx – The dense, unnormalized approximation over the outcomes of dist coalesced to rvs.

Return type:

Distribution

Raises:

ditException – Raised if order is out of range.

Examples

>>> d = dit.example_dists.giant_bit(3, 2)
>>> dit.kirkwood_approximation(d).pmf
array([1., 0., 0., 0., 0., 0., 0., 1.])

lancaster_approximation(dist, rvs=None, order=None)[source]

Construct the order-k Lancaster approximation of a distribution, the additive analogue of the Kirkwood approximation.

Lancaster’s expansion of \(X_0, \ldots, X_{n-1}\) writes the joint distribution relative to the product of its marginals [Lan69]:

\[p(x) = \prod_i p(x_i) \sum_{T} \delta_T(x_T), \qquad \delta_T(x_T) = \sum_{S \subseteq T} (-1)^{|T|-|S|} \frac{p(x_S)}{\prod_{i \in S} p(x_i)},\]

where \(\delta_\emptyset = 1\) and \(\delta_{\{i\}} = 0\). \(\prod_i p(x_i) \, \delta_{\{0, \ldots, n-1\}}\) is the Lancaster interaction measure \(\Delta_L p = \prod_i (p^*_{X_i} - p_{X_i})\) [SGB13]. Truncating the expansion at \(|T| \leq k\) gives

\[p_k(x) = \sum_{|S| \leq k} c_S \, p(x_S) \prod_{i \notin S} p(x_i), \qquad c_S = (-1)^{k-|S|} \binom{n-|S|-1}{k-|S|},\]

with the same coefficients as kirkwood_approximation(), but combined arithmetically rather than geometrically. The default order \(k = n-1\) is \(p - \Delta_L p\), e.g. for three variables

\[p_2(x, y, z) = p(x, y) p(z) + p(x, z) p(y) + p(y, z) p(x) - 2 p(x) p(y) p(z).\]

Orders 0 and 1 are both the product of marginals. Every order sums to one, and its marginals of size at most k agree with those of \(p\), but it may be negative. For \(n \leq 3\), the default order reproduces \(p\) whenever \(p\) factors nontrivially, although the converse fails. For \(n > 3\) even that implication fails, which motivates Streitberg’s correction [Str90]. No canonical literature source is known for orders below \(n-1\).

Parameters:

dist (Distribution) – The distribution to approximate.
rvs (list, None) – A list of lists. Each inner list specifies the indexes of the random variables forming one variable of the approximation. If None, then each random variable is its own variable.
order (int, None) – The largest interaction size k kept by the approximation. Must satisfy 0 <= k <= n - 1. If None, defaults to n - 1.

Returns:

approx – The dense, possibly signed, approximation over the outcomes of dist coalesced to rvs.

Return type:

Distribution

Raises:

ditException – Raised if order is out of range.

Examples

>>> d = dit.Distribution(['000', '010', '100', '111'], [1/4] * 4)
>>> dit.lancaster_approximation(d).pmf
array([ 0.3125, -0.0625,  0.1875,  0.0625,  0.1875,  0.0625,  0.0625,  0.1875])

mixture_distribution(dists, weights, merge=False)[source]

Create a mixture distribution: \(\sum p_i d_i\)

Parameters:

dists ([Distribution]) – List of distributions to mix. Each distribution is assumed to have the same base and sample space.
weights ([float]) – List of weights to use while mixing dists. The weights are assumed to be probability represented in the base of the distributions.
merge (bool) – If True then distributions will be mixed even if they do not share the same sample space. The idea is that each of the input distributions is reinterpreted on a common, merged sample space. If False, then an exception will be raised if incompatible distributions are mixed.

Returns:

mix – The mixture distribution.

Return type:

Distribution

Raises:

DitException – Raised if there dists and weights have unequal lengths.
InvalidNormalization – Raised if the weights do not sum to unity.
InvalidProbability – Raised if the weights are not valid probabilities.
IncompatibleOutcome – Raised if the sample spaces for each distribution are not compatible.

mixture_distribution2(dists, weights)[source]

Create a mixture distribution: \(\sum p_i d_i\)

This version assumes that the pmf for each distribution is of the same form, and as a result, will be faster than mixture_distribution. Explicitly, it assumes that the sample space is ordered exactly the same for each distribution and that the outcomes currently represented in the pmf are the same as well. Using it in any other case will result in incorrect output or an exception.

Parameters:

dists ([Distribution]) – List of distributions to mix. Each distribution is assumed to have the same base and sample space.
weights ([float]) – List of weights to use while mixing dists. The weights are assumed to be probability represented in the base of the distributions.

Returns:

mix – The mixture distribution.

Return type:

Distribution

Raises:

DitException – Raised if there dists and weights have unequal lengths.
InvalidNormalization – Raised if the weights do not sum to unity.
InvalidProbability – Raised if the weights are not valid probabilities.
IncompatibleDistribution – Raised if the sample spaces for each distribution are not compatible.

modify_outcomes(dist, ctor)[source]

Returns dist but with modified outcomes, after passing them to ctor.

Parameters:

dist (Distribution, Distribution) – The distribution to be modified.
ctor (callable) – The constructor that receives an existing outcome and returns a new modified outcome.

Returns:

d – The modified distribution.

Return type:

Distribution, Distribution

Examples

Convert joint tuple outcomes to strings. >>> d = dit.uniform_distribution(3, [‘0’, ‘1’]) >>> d2 = dit.modify_outcomes(d, lambda x: ‘’.join(x))

Increment scalar outcomes by 1. >>> d = dit.uniform_scalar_distribution(5) >>> d2 = dit.modify_outcomes(d, lambda x: x + 1)

noisy(dist, noise=0.5)[source]

Construct a noisy version of dist.

Parameters:

dist (Distribution) – The distribution to fuzz.
noise (float, 0 <= noise <= 1) – The noise level.

Returns:

fuzzy – The noisy distribution.

Return type:

Distribution

product_distribution(dist, rvs=None, base=None)[source]

Returns a new distribution which is the product of marginals.

Parameters:

dist (distribution) – The original distribution.
rvs (sequence) – A sequence whose elements are also sequences. Each inner sequence defines the marginal distribution used to create the new distribution.
base (float, 'linear', 'e') – The desired base for the distribution probabilities.

Returns:

d – The product distribution.

Return type:

Distribution

pruned_samplespace(d, sample_space=None)[source]

Returns a new distribution with pruned sample space.

The pruning is such that zero probability outcomes are removed.

Parameters:

d (distribution) – The distribution used to create the pruned distribution.
sample_space (set) – A list of outcomes with zero probability that should be kept in the sample space. If None, then all outcomes with zero probability will be removed.

Returns:

pd – The distribution with a pruned sample space.

Return type:

distribution

random_dist_structure(outcome_length, alphabet_size)[source]

Return a uniform distribution over a random subset of the alphabet_size`**`outcome_length possible joint events.

Parameters:

outcome_length (int) – The number of random variables in each joint event.
alphabet_size (int) – The size of the alphabet for each random variable.

Returns:

d – A uniform distribution over a random subset of joint events.

Return type:

Distribution

random_distribution(outcome_length, alphabet_size, base=None, alpha=None, prng=None)[source]

Returns a random distribution drawn uniformly from the simplex.

The distribution is sampled uniformly over the space of distributions on the n-simplex, where n is equal to alphabet_size**outcome_length. If alpha is not None, then the distribution is sampled from the Dirichlet distribution with parameter alpha.

Parameters:

outcome_length (int) – The length of the outcomes.
alphabet_size (int, list) – The alphabet used to construct the outcomes of the distribution. If an integer, then the alphabet will consist of integers from 0 to k-1 where k is the alphabet size. If a list, then the elements are used as the alphabet.
base (float, 'linear', 'e') – The desired base for the distribution probabilities.
alpha (list | None) – The concentration parameters defining that the Dirichlet distribution used to draw the random distribution. If None, then each of the concentration parameters are set equal to 1.

Returns:

d – A uniform sampled distribution.

Return type:

Distribution.

random_scalar_distribution(n, base=None, alpha=None, prng=None)[source]

Returns a random scalar distribution over n outcomes.

The distribution is sampled uniformly over the space of distributions on the n-simplex. If alpha is not None, then the distribution is sampled from the Dirichlet distribution with parameter alpha.

Parameters:

n (int | list) – The number of outcomes, or a list containing the outcomes.
base (float, 'linear', 'e') – The desired base for the distribution probabilities.
alpha (list | None) – The concentration parameters defining that the Dirichlet distribution used to draw the random distribution. If None, then each of the concentration parameters are set equal to 1.

simplex_grid(length, subdivisions, using=None, inplace=False)[source]

Returns a generator over distributions, determined by a grid.

The grid is “triangular” in Euclidean space.

The total number of points on the grid is:

(subdivisions + length - 1)! / (subdivisions)! / (length-1)!

and is equivalent to the total number of ways n indistinguishable items can be placed into k distinguishable slots, where n=`subdivisions` and k=`length`.

Parameters:

length (int) – The number of elements in each distribution. The dimensionality of the simplex is length-1.
subdivisions (int) – The number of subdivisions for the interval [0, 1]. Each component will take on values at the boundaries of the subdivisions. For example, one subdivision means each component can take the values 0 or 1 only. Two subdivisions corresponds to \([[0, 1/2], [1/2, 1]]\) and thus, each component can take the values 0, 1/2, or 1. A common use case is to exponentially increase the number of subdivisions at each level. That is, subdivisions would be: 2**0, 2**1, 2**2, 2**3, …
using (None, callable, or distribution) – If None, then scalar distributions on integers are yielded. If using is a distribution, then each yielded distribution is a copy of using with its pmf set appropriately. For other callables, a tuple of the pmf is passed to the callable and then yielded.
inplace (bool) – If True, then each yielded distribution is the same Python object, but with a new probability mass function. If False, then each yielded distribution is a unique Python object and can be safely stored for other calculations after the generator has finished. This keyword has an effect only when using is None or some distribution.

Examples

>>> list(dit.simplex_grid(2, 2, using=tuple))
[(0.0, 1.0), (0.25, 0.75), (0.5, 0.5), (0.75, 0.25), (1.0, 0.0)]

uniform(outcomes, base=None)[source]

Produces a uniform distribution over outcomes.

Parameters:

outcomes (iterable) – The set of outcomes with which to construct the distribution.
base (float, 'linear', 'e') – The desired base for the distribution probabilities.

Returns:

d – A uniform distribution over outcomes.

Return type:

Distribution

Raises:

ditException – Raised if the elements of outcomes do not all have the same length.
TypeError – Raised if outcomes is not iterable.

uniform_distribution(outcome_length, alphabet_size, base=None)[source]

Returns a uniform distribution.

Parameters:

outcome_length (int) – The length of the outcomes.
alphabet_size (int, list of lists) – The alphabets used to construct the outcomes of the distribution. If an integer, then the alphabet for each random variable will be the same, consisting of integers from 0 to k-1 where k is the alphabet size. If a list, then the elements are used as the alphabet for each random variable. If the list has a single element, then it will be used as the alphabet for each random variable.
base (float, 'linear', 'e') – The desired base for the distribution probabilities.

Returns:

d – A uniform distribution.

Return type:

Distribution.

Examples

Each random variable has the same standardized alphabet: [0,1] >>> d = dit.uniform_distribution(2, 2)

Each random variable has its own alphabet. >>> d = dit.uniform_distribution(2, [[0,1],[1,2]])

Both random variables have [‘H’,’T’] as an alphabet. >>> d = dit.uniform_distribution(2, [[‘H’,’T’]])

uniform_like(dist)[source]

Returns a uniform distribution with the same outcome length, alphabet size, and base as dist.

Parameters:: dist (Distribution) – The distribution to mimic.

uniform_scalar_distribution(n, base=None)[source]

Returns a uniform scalar distribution over n outcomes.

Parameters:: n (int, list) – If an integer, then the outcomes are integers from 0 to n-1. If a list then the elements are treated as the outcomes.
Returns:: d – A uniform scalar distribution.
Return type:: Distribution