Introduction to TensorFlow Text
Stay organized with collections
Save and categorize content based on your preferences.
TensorFlow Text provides a collection of text related classes and ops ready to
use with TensorFlow 2.0. The library can perform the preprocessing regularly
required by text-based models, and includes other features useful for sequence
modeling not provided by core TensorFlow.
The benefit of using these ops in your text preprocessing is that they are done
in the TensorFlow graph. You do not need to worry about tokenization in training
being different than the tokenization at inference, or managing preprocessing
scripts.
Install TensorFlow Text
Install using pip
When installing TF Text with pip install, note the version of TensorFlow you are
running, as you should specify the corresponding version of TF Text.
pip install -U tensorflow-text==<version>
Build from source
TensorFlow Text must be built in the same environment as TensorFlow. Thus, if
you manually build TF Text, it is highly recommended that you also build
TensorFlow.
If building on MacOS, you must have coreutils installed. It is probably easiest
to do with Homebrew. First, build TensorFlow
from source.
Clone the TF Text repo.
git clone https://github.com/tensorflow/text.git
Finally, run the build script to create a pip package.
./oss_scripts/run_build.sh
Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.
Last updated 2021-05-21 UTC.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Missing the information I need","missingTheInformationINeed","thumb-down"],["Too complicated / too many steps","tooComplicatedTooManySteps","thumb-down"],["Out of date","outOfDate","thumb-down"],["Samples / code issue","samplesCodeIssue","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2021-05-21 UTC."],[],[],null,["# Introduction to TensorFlow Text\n\n\u003cbr /\u003e\n\nTensorFlow Text provides a collection of text related classes and ops ready to\nuse with TensorFlow 2.0. The library can perform the preprocessing regularly\nrequired by text-based models, and includes other features useful for sequence\nmodeling not provided by core TensorFlow.\n\nThe benefit of using these ops in your text preprocessing is that they are done\nin the TensorFlow graph. You do not need to worry about tokenization in training\nbeing different than the tokenization at inference, or managing preprocessing\nscripts.\n\nInstall TensorFlow Text\n-----------------------\n\n### Install using pip\n\nWhen installing TF Text with pip install, note the version of TensorFlow you are\nrunning, as you should specify the corresponding version of TF Text. \n\n pip install -U tensorflow-text==\u003cversion\u003e\n\n### Build from source\n\nTensorFlow Text must be built in the same environment as TensorFlow. Thus, if\nyou manually build TF Text, it is highly recommended that you also build\nTensorFlow.\n\nIf building on MacOS, you must have coreutils installed. It is probably easiest\nto do with Homebrew. First, build TensorFlow\n[from source](https://www.tensorflow.org/install/source).\n\nClone the TF Text repo. \n\n git clone https://github.com/tensorflow/text.git\n\nFinally, run the build script to create a pip package. \n\n ./oss_scripts/run_build.sh"]]
