tfm.nlp.layers.GatedFeedforward
Stay organized with collections
Save and categorize content based on your preferences.
Gated linear feedforward layer.
tfm.nlp.layers.GatedFeedforward(
inner_dim=768,
inner_activation=tfm.utils.activations.gelu,
dropout=0.0,
use_gate=True,
apply_output_layer_norm=True,
num_blocks=1,
dropout_position='before_residual',
kernel_initializer='glorot_uniform',
bias_initializer='zeros',
kernel_regularizer=None,
bias_regularizer=None,
activity_regularizer=None,
kernel_constraint=None,
bias_constraint=None,
**kwargs
)
This layer follows the paper "GLU Variants Improve Transformer"
(https://arxiv.org/abs/2002.05202). In additional, it allows to stack
multiple feedforward blocks and specify the position of dropout layer.
Args |
intermediate_size
|
Size of the intermediate layer.
|
intermediate_activation
|
Activation for the intermediate layer.
|
dropout
|
Dropout probability for the output dropout.
|
use_gate
|
Whether to use gated linear units. If True, assuming GELU as the
activation and omitting bias, will apply
GEGLU(x, W, V, W_2) = (GELU(xW) * xV)W2; if False, will follow
"Attention Is All You Need" (https://arxiv.org/abs/1706.03762) paper and
apply FFN(x, W, W_2) = GELU(xW_1)W_2.
|
num_blocks
|
The number of feedforward blocks to stack. Each block contains a
(gated) linear layer and a fully connected layer followed by dropout,
layer norm and residual.
|
dropout_position
|
Where to apply the dropout, the value can be either
before_residual or after_residual. If before_residual, will apply
layer_output = layer_norm(dropout(layer_output) + layer_input); if
after residual, will apply
layer_output = dropout(layer_norm(layer_output + layer_input)).
|
kernel_initializer
|
Initializer for dense layer kernels.
|
bias_initializer
|
Initializer for dense layer biases.
|
kernel_regularizer
|
Regularizer for dense layer kernels.
|
bias_regularizer
|
Regularizer for dense layer biases.
|
activity_regularizer
|
Regularizer for dense layer activity.
|
kernel_constraint
|
Constraint for dense layer kernels.
|
bias_constraint
|
Constraint for dense layer kernels.
|
Methods
call
View source
call(
inputs
)
This is where the layer's logic lives.
The call() method may not create state (except in its first
invocation, wrapping the creation of variables or other resources in
tf.init_scope()). It is recommended to create state, including
tf.Variable instances and nested Layer instances,
in __init__(), or in the build() method that is
called automatically before call() executes for the first time.
| Args |
inputs
|
Input tensor, or dict/list/tuple of input tensors.
The first positional inputs argument is subject to special rules:
inputs must be explicitly passed. A layer cannot have zero
arguments, and inputs cannot be provided via the default value
of a keyword argument.
- NumPy array or Python scalar values in
inputs get cast as
tensors.
- TF-Keras mask metadata is only collected from
inputs.
- Layers are built (
build(input_shape) method)
using shape info from inputs only.
input_spec compatibility is only checked against inputs.
- Mixed precision input casting is only applied to
inputs.
If a layer has tensor arguments in *args or **kwargs, their
casting behavior in mixed precision should be handled manually.
- The SavedModel input specification is generated using
inputs
only.
- Integration with various ecosystem packages like TFMOT, TFLite,
TF.js, etc is only supported for
inputs and not for tensors in
positional and keyword arguments.
|
*args
|
Additional positional arguments. May contain tensors, although
this is not recommended, for the reasons above.
|
**kwargs
|
Additional keyword arguments. May contain tensors, although
this is not recommended, for the reasons above.
The following optional keyword arguments are reserved:
training: Boolean scalar tensor of Python boolean indicating
whether the call is meant for training or inference.
mask: Boolean input mask. If the layer's call() method takes a
mask argument, its default value will be set to the mask
generated for inputs by the previous layer (if input did come
from a layer that generated a corresponding mask, i.e. if it came
from a TF-Keras layer with masking support).
|
| Returns |
|
A tensor or list/tuple of tensors.
|
Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates. Some content is licensed under the numpy license.
Last updated 2026-11-06 UTC.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Missing the information I need","missingTheInformationINeed","thumb-down"],["Too complicated / too many steps","tooComplicatedTooManySteps","thumb-down"],["Out of date","outOfDate","thumb-down"],["Samples / code issue","samplesCodeIssue","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2026-11-06 UTC."],[],[]]