tf.contrib.framework.RecordInput
Stay organized with collections
Save and categorize content based on your preferences.
RecordInput asynchronously reads and randomly yields TFRecords.
tf.contrib.framework.RecordInput(
file_pattern, batch_size=1, buffer_size=1, parallelism=1, shift_ratio=0, seed=0,
name=None, batches=None, compression_type=None
)
A RecordInput Op will continuously read a batch of records asynchronously
into a buffer of some fixed capacity. It can also asynchronously yield
random records from this buffer.
It will not start yielding until at least buffer_size / 2 elements have been
placed into the buffer so that sufficient randomization can take place.
The order the files are read will be shifted each epoch by shift_amount so
that the data is presented in a different order every epoch.
Args |
|---|
file_pattern
|
File path to the dataset, possibly containing wildcards.
All matching files will be iterated over each epoch.
|
batch_size
|
How many records to return at a time.
|
buffer_size
|
The maximum number of records the buffer will contain.
|
parallelism
|
How many reader threads to use for reading from files.
|
shift_ratio
|
What percentage of the total number files to move the start
file forward by each epoch.
|
seed
|
Specify the random number seed used by generator that randomizes
records.
|
name
|
Optional name for the operation.
|
batches
|
None by default, creating a single batch op. Otherwise specifies
how many batches to create, which are returned as a list when
get_yield_op() is called. An example use case is to split processing
between devices on one computer.
|
compression_type
|
The type of compression for the file. Currently ZLIB and
GZIP are supported. Defaults to none.
|
Raises |
|---|
ValueError
|
If one of the arguments is invalid.
|
Methods
get_yield_op
View source
get_yield_op()
Adds a node that yields a group of records every time it is executed.
If RecordInput batches parameter is not None, it yields a list of
record batches with the specified batch_size.
Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.
Last updated 2020-10-01 UTC.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Missing the information I need","missingTheInformationINeed","thumb-down"],["Too complicated / too many steps","tooComplicatedTooManySteps","thumb-down"],["Out of date","outOfDate","thumb-down"],["Samples / code issue","samplesCodeIssue","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2020-10-01 UTC."],[],[],null,["# tf.contrib.framework.RecordInput\n\n\u003cbr /\u003e\n\n|-----------------------------------------------------------------------------------------------------------------------------------|\n| [View source on GitHub](https://github.com/tensorflow/tensorflow/blob/v1.15.0/tensorflow/python/ops/data_flow_ops.py#L2381-L2467) |\n\nRecordInput asynchronously reads and randomly yields TFRecords. \n\n tf.contrib.framework.RecordInput(\n file_pattern, batch_size=1, buffer_size=1, parallelism=1, shift_ratio=0, seed=0,\n name=None, batches=None, compression_type=None\n )\n\nA RecordInput Op will continuously read a batch of records asynchronously\ninto a buffer of some fixed capacity. It can also asynchronously yield\nrandom records from this buffer.\n\nIt will not start yielding until at least `buffer_size / 2` elements have been\nplaced into the buffer so that sufficient randomization can take place.\n\nThe order the files are read will be shifted each epoch by `shift_amount` so\nthat the data is presented in a different order every epoch.\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n| Args ---- ||\n|--------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| `file_pattern` | File path to the dataset, possibly containing wildcards. All matching files will be iterated over each epoch. |\n| `batch_size` | How many records to return at a time. |\n| `buffer_size` | The maximum number of records the buffer will contain. |\n| `parallelism` | How many reader threads to use for reading from files. |\n| `shift_ratio` | What percentage of the total number files to move the start file forward by each epoch. |\n| `seed` | Specify the random number seed used by generator that randomizes records. |\n| `name` | Optional name for the operation. |\n| `batches` | None by default, creating a single batch op. Otherwise specifies how many batches to create, which are returned as a list when `get_yield_op()` is called. An example use case is to split processing between devices on one computer. |\n| `compression_type` | The type of compression for the file. Currently ZLIB and GZIP are supported. Defaults to none. |\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n| Raises ------ ||\n|--------------|-------------------------------------|\n| `ValueError` | If one of the arguments is invalid. |\n\n\u003cbr /\u003e\n\nMethods\n-------\n\n### `get_yield_op`\n\n[View source](https://github.com/tensorflow/tensorflow/blob/v1.15.0/tensorflow/python/ops/data_flow_ops.py#L2442-L2467) \n\n get_yield_op()\n\nAdds a node that yields a group of records every time it is executed.\nIf RecordInput `batches` parameter is not None, it yields a list of\nrecord batches with the specified `batch_size`."]]
View source on GitHub