tfm.nlp.layers.FeedForwardExperts
Stay organized with collections
Save and categorize content based on your preferences.
Feed-forward layer with multiple experts.
tfm.nlp.layers.FeedForwardExperts(
num_experts: int,
d_ff: int,
*,
inner_dropout: float = 0.0,
output_dropout: float = 0.0,
activation: Callable[[tf.Tensor], tf.Tensor] = tf_keras.activations.gelu,
kernel_initializer: _InitializerType = _DEFAULT_KERNEL_INITIALIZER,
bias_initializer: _InitializerType = _DEFAULT_BIAS_INITIALIZER,
name: str = 'experts',
**kwargs
)
Note that call() takes inputs with shape
[num_groups, num_experts, expert_capacity, hidden_dim]
which is different from the usual [batch_size, seq_len, hidden_dim] used by
the FeedForward layer.
The experts are independent FeedForward layers of the
same shape, i.e. the kernel doesn't have shape [hidden_dim, out_dim], but
[num_experts, hidden_dim, out_dim].
Args |
num_experts
|
Number of experts (i.e. number of independent feed-forward
blocks).
|
d_ff
|
Dimension of feed-forward layer of each expert.
|
inner_dropout
|
The dropout probability to be applied after intermediate
activations.
|
output_dropout
|
The dropout probability to be applied after output layer.
|
activation
|
(Nonlinear) transform applied in layer.
|
kernel_initializer
|
Initialization scheme for kernel.
|
bias_initializer
|
Initialization scheme for bias.
|
name
|
Layer name.
|
**kwargs
|
Forwarded to super.
|
Methods
call
View source
call(
inputs: tf.Tensor, *, training: Optional[bool] = None
) -> tf.Tensor
Applies layer to inputs.
| Args |
inputs
|
Inputs of shape
[num_groups, num_experts, expert_capacity, hidden_dim].
|
training
|
Only apply dropout during training.
|
| Returns |
|
Transformed inputs with the same shape as inputs
[num_groups, num_experts, expert_capacity, hidden_dim].
|
Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates. Some content is licensed under the numpy license.
Last updated 2026-11-06 UTC.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Missing the information I need","missingTheInformationINeed","thumb-down"],["Too complicated / too many steps","tooComplicatedTooManySteps","thumb-down"],["Out of date","outOfDate","thumb-down"],["Samples / code issue","samplesCodeIssue","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2026-11-06 UTC."],[],[]]