View source on GitHub
|
Maps strings from a vocabulary to integer indices.
Inherits From: StringLookup, PreprocessingLayer, Layer, Module
tf.compat.v1.keras.layers.experimental.preprocessing.StringLookup(
max_tokens=None, num_oov_indices=1, mask_token='',
oov_token='[UNK]', vocabulary=None, encoding=None, invert=False,
**kwargs
)
Methods
adapt
adapt(
data, reset_state=True
)
Fits the state of the preprocessing layer to the dataset.
Overrides the default adapt method to apply relevant preprocessing to the inputs before passing to the combiner.
| Arguments | |
|---|---|
data
|
The data to train on. It can be passed either as a tf.data Dataset, or as a numpy array. |
reset_state
|
Optional argument specifying whether to clear the state of
the layer at the start of the call to adapt. This must be True for
this layer, which does not support repeated calls to adapt.
|
get_vocabulary
get_vocabulary()
set_vocabulary
set_vocabulary(
vocab
)
Sets vocabulary data for this layer with inverse=False.
This method sets the vocabulary for this layer directly, instead of analyzing a dataset through 'adapt'. It should be used whenever the vocab information is already known. If vocabulary data is already present in the layer, this method will either replace it
| Arguments | |
|---|---|
vocab
|
An array of string tokens. |
| Raises | |
|---|---|
ValueError
|
If there are too many inputs, the inputs do not match, or input data is missing. |
vocab_size
vocab_size()