tfp.vi.mutual_information.lower_bound_barber_agakov
Stay organized with collections
Save and categorize content based on your preferences.
Lower bound on mutual information from [Barber and Agakov (2003)][1].
tfp.vi.mutual_information.lower_bound_barber_agakov(
logu, entropy, name=None
)
This method gives a lower bound on the mutual information I(X; Y),
by replacing the unknown conditional p(x|y) with a variational
decoder q(x|y), but it requires knowing the entropy of X, h(X).
The lower bound was introduced in [Barber and Agakov (2003)][1].
I(X; Y) = E_p(x, y)[log( p(x|y) / p(x) )]
= E_p(x, y)[log( q(x|y) / p(x) )] + E_p(y)[KL[ p(x|y) || q(x|y) ]]
>= E_p(x, y)[log( q(x|y) )] + h(X) = I_[lower_bound_barbar_agakov]
Example:
x, y are samples from a joint Gaussian distribution, with correlation
0.8 and both of dimension 1.
batch_size, rho, dim = 10000, 0.8, 1
y, eps = tf.split(
value=tf.random.normal(shape=(2 * batch_size, dim), seed=7),
num_or_size_splits=2, axis=0)
mean, conditional_stddev = rho * y, tf.sqrt(1. - tf.square(rho))
x = mean + conditional_stddev * eps
# Conditional distribution of p(x|y)
conditional_dist = tfd.MultivariateNormalDiag(
mean, scale_diag=conditional_stddev * tf.ones((batch_size, dim)))
# Scores/unnormalized likelihood of pairs of joint samples `x[i], y[i]`
joint_scores = conditional_dist.log_prob(x)
# Differential entropy of `X` that is `1-D` Normal distributed.
entropy_x = 0.5 * np.log(2 * np.pi * np.e)
# Barber and Agakov lower bound on mutual information
lower_bound_barber_agakov(logu=joint_scores, entropy=entropy_x)
Args |
|---|
logu
|
float-like Tensor of size [batch_size] representing
log(q(x_i | y_i)) for each (x_i, y_i) pair.
|
entropy
|
float-like scalar representing the entropy of X.
|
name
|
Python str name prefixed to Ops created by this function.
Default value: None (i.e., 'lower_bound_barber_agakov').
|
Returns |
|---|
lower_bound
|
float-like scalar for lower bound on mutual information.
|
References
[1]: David Barber, Felix V. Agakov. The IM algorithm: a variational
approach to Information Maximization. In Conference on Neural
Information Processing Systems, 2003.
Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.
Last updated 2023-11-21 UTC.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Missing the information I need","missingTheInformationINeed","thumb-down"],["Too complicated / too many steps","tooComplicatedTooManySteps","thumb-down"],["Out of date","outOfDate","thumb-down"],["Samples / code issue","samplesCodeIssue","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2023-11-21 UTC."],[],[],null,["# tfp.vi.mutual_information.lower_bound_barber_agakov\n\n\u003cbr /\u003e\n\n|--------------------------------------------------------------------------------------------------------------------------------------------------|\n| [View source on GitHub](https://github.com/tensorflow/probability/blob/v0.23.0/tensorflow_probability/python/vi/mutual_information.py#L172-L237) |\n\nLower bound on mutual information from \\[Barber and Agakov (2003)\\]\\[1\\]. \n\n tfp.vi.mutual_information.lower_bound_barber_agakov(\n logu, entropy, name=None\n )\n\nThis method gives a lower bound on the mutual information I(X; Y),\nby replacing the unknown conditional p(x\\|y) with a variational\ndecoder q(x\\|y), but it requires knowing the entropy of X, h(X).\nThe lower bound was introduced in \\[Barber and Agakov (2003)\\]\\[1\\]. \n\n I(X; Y) = E_p(x, y)[log( p(x|y) / p(x) )]\n = E_p(x, y)[log( q(x|y) / p(x) )] + E_p(y)[KL[ p(x|y) || q(x|y) ]]\n \u003e= E_p(x, y)[log( q(x|y) )] + h(X) = I_[lower_bound_barbar_agakov]\n\n#### Example:\n\n`x`, `y` are samples from a joint Gaussian distribution, with correlation\n`0.8` and both of dimension `1`. \n\n batch_size, rho, dim = 10000, 0.8, 1\n y, eps = tf.split(\n value=tf.random.normal(shape=(2 * batch_size, dim), seed=7),\n num_or_size_splits=2, axis=0)\n mean, conditional_stddev = rho * y, tf.sqrt(1. - tf.square(rho))\n x = mean + conditional_stddev * eps\n\n # Conditional distribution of p(x|y)\n conditional_dist = tfd.MultivariateNormalDiag(\n mean, scale_diag=conditional_stddev * tf.ones((batch_size, dim)))\n\n # Scores/unnormalized likelihood of pairs of joint samples `x[i], y[i]`\n joint_scores = conditional_dist.log_prob(x)\n\n # Differential entropy of `X` that is `1-D` Normal distributed.\n entropy_x = 0.5 * np.log(2 * np.pi * np.e)\n\n\n # Barber and Agakov lower bound on mutual information\n lower_bound_barber_agakov(logu=joint_scores, entropy=entropy_x)\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n| Args ---- ||\n|-----------|------------------------------------------------------------------------------------------------------------------------|\n| `logu` | `float`-like `Tensor` of size \\[batch_size\\] representing log(q(x_i \\| y_i)) for each (x_i, y_i) pair. |\n| `entropy` | `float`-like `scalar` representing the entropy of X. |\n| `name` | Python `str` name prefixed to Ops created by this function. Default value: `None` (i.e., 'lower_bound_barber_agakov'). |\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n| Returns ------- ||\n|---------------|--------------------------------------------------------------|\n| `lower_bound` | `float`-like `scalar` for lower bound on mutual information. |\n\n\u003cbr /\u003e\n\n#### References\n\n\\[1\\]: David Barber, Felix V. Agakov. The IM algorithm: a variational\napproach to Information Maximization. In *Conference on Neural\nInformation Processing Systems*, 2003."]]
View source on GitHub