tf.clip_by_global_norm
Stay organized with collections
Save and categorize content based on your preferences.
Clips values of multiple tensors by the ratio of the sum of their norms.
tf.clip_by_global_norm(
t_list, clip_norm, use_norm=None, name=None
)
Given a tuple or list of tensors t_list, and a clipping ratio clip_norm,
this operation returns a list of clipped tensors list_clipped
and the global norm (global_norm) of all tensors in t_list. Optionally,
if you've already computed the global norm for t_list, you can specify
the global norm with use_norm.
To perform the clipping, the values t_list[i] are set to:
t_list[i] * clip_norm / max(global_norm, clip_norm)
where:
global_norm = sqrt(sum([l2norm(t)**2 for t in t_list]))
If clip_norm > global_norm then the entries in t_list remain as they are,
otherwise they're all shrunk by the global ratio.
If global_norm == infinity then the entries in t_list are all set to NaN
to signal that an error occurred.
Any of the entries of t_list that are of type None are ignored.
This is the correct way to perform gradient clipping (Pascanu et al., 2012).
However, it is slower than clip_by_norm() because all the parameters must be
ready before the clipping operation can be performed.
Args |
|---|
t_list
|
A tuple or list of mixed Tensors, IndexedSlices, or None.
|
clip_norm
|
A 0-D (scalar) Tensor > 0. The clipping ratio.
|
use_norm
|
A 0-D (scalar) Tensor of type float (optional). The global
norm to use. If not provided, global_norm() is used to compute the norm.
|
name
|
A name for the operation (optional).
|
Returns |
|---|
list_clipped
|
A list of Tensors of the same type as list_t.
|
global_norm
|
A 0-D (scalar) Tensor representing the global norm.
|
Raises |
|---|
TypeError
|
If t_list is not a sequence.
|
Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates. Some content is licensed under the numpy license.
Last updated 2024-04-26 UTC.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Missing the information I need","missingTheInformationINeed","thumb-down"],["Too complicated / too many steps","tooComplicatedTooManySteps","thumb-down"],["Out of date","outOfDate","thumb-down"],["Samples / code issue","samplesCodeIssue","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2024-04-26 UTC."],[],[],null,["# tf.clip_by_global_norm\n\n\u003cbr /\u003e\n\n|----------------------------------------------------------------------------------------------------------------------------|\n| [View source on GitHub](https://github.com/tensorflow/tensorflow/blob/v2.16.1/tensorflow/python/ops/clip_ops.py#L298-L392) |\n\nClips values of multiple tensors by the ratio of the sum of their norms.\n\n#### View aliases\n\n\n**Compat aliases for migration**\n\nSee\n[Migration guide](https://www.tensorflow.org/guide/migrate) for\nmore details.\n\n[`tf.compat.v1.clip_by_global_norm`](https://www.tensorflow.org/api_docs/python/tf/clip_by_global_norm)\n\n\u003cbr /\u003e\n\n tf.clip_by_global_norm(\n t_list, clip_norm, use_norm=None, name=None\n )\n\nGiven a tuple or list of tensors `t_list`, and a clipping ratio `clip_norm`,\nthis operation returns a list of clipped tensors `list_clipped`\nand the global norm (`global_norm`) of all tensors in `t_list`. Optionally,\nif you've already computed the global norm for `t_list`, you can specify\nthe global norm with `use_norm`.\n\nTo perform the clipping, the values `t_list[i]` are set to: \n\n t_list[i] * clip_norm / max(global_norm, clip_norm)\n\nwhere: \n\n global_norm = sqrt(sum([l2norm(t)**2 for t in t_list]))\n\nIf `clip_norm \u003e global_norm` then the entries in `t_list` remain as they are,\notherwise they're all shrunk by the global ratio.\n\nIf `global_norm == infinity` then the entries in `t_list` are all set to `NaN`\nto signal that an error occurred.\n\nAny of the entries of `t_list` that are of type `None` are ignored.\n\nThis is the correct way to perform gradient clipping (Pascanu et al., 2012).\n\nHowever, it is slower than `clip_by_norm()` because all the parameters must be\nready before the clipping operation can be performed.\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n| Args ---- ||\n|-------------|-------------------------------------------------------------------------------------------------------------------------------------------|\n| `t_list` | A tuple or list of mixed `Tensors`, `IndexedSlices`, or None. |\n| `clip_norm` | A 0-D (scalar) `Tensor` \\\u003e 0. The clipping ratio. |\n| `use_norm` | A 0-D (scalar) `Tensor` of type `float` (optional). The global norm to use. If not provided, `global_norm()` is used to compute the norm. |\n| `name` | A name for the operation (optional). |\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n| Returns ------- ||\n|----------------|-------------------------------------------------------|\n| `list_clipped` | A list of `Tensors` of the same type as `list_t`. |\n| `global_norm` | A 0-D (scalar) `Tensor` representing the global norm. |\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n| Raises ------ ||\n|-------------|--------------------------------|\n| `TypeError` | If `t_list` is not a sequence. |\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n| References ---------- ||\n|---|---|\n| On the difficulty of training Recurrent Neural Networks: [Pascanu et al., 2012](http://proceedings.mlr.press/v28/pascanu13.html) ([pdf](http://proceedings.mlr.press/v28/pascanu13.pdf)) ||\n\n\u003cbr /\u003e"]]
View source on GitHub