tf.compat.v1.uniform_unit_scaling_initializer

View source on GitHub

Initializer that generates tensors without scaling variance.

Inherits From: Initializer

tf.compat.v1.uniform_unit_scaling_initializer(
    factor=1.0, seed=None, dtype=tf.dtypes.float32
)

When initializing a deep network, it is in principle advantageous to keep the scale of the input variance constant, so it does not explode or diminish by reaching the final layer. If the input is x and the operation x * W, and we want to initialize W uniformly at random, we need to pick W from

[-sqrt(3) / sqrt(dim), sqrt(3) / sqrt(dim)]

to keep the scale intact, where dim = W.shape[0] (the size of the input). A similar calculation for convolutional networks gives an analogous result with dim equal to the product of the first 3 dimensions. When nonlinearities are present, we need to multiply this by a constant factor. See (Sussillo et al., 2014) for deeper motivation, experiments and the calculation of constants. In section 2.3 there, the constants were numerically computed: for a linear layer it's 1.0, relu: ~1.43, tanh: ~1.15.

References:

Sussillo et al., 2014 (pdf)

Methods

from_config

View source

@classmethod
from_config(
    config
)

Instantiates an initializer from a configuration dictionary.

Example:

initializer = RandomUniform(-1, 1)
config = initializer.get_config()
initializer = RandomUniform.from_config(config)

get_config

View source

get_config()

Returns the configuration of the initializer as a JSON-serializable dict.

__call__

View source

__call__(
    shape, dtype=None, partition_info=None
)

Returns a tensor object initialized as specified by the initializer.