DatasetIterator

public class DatasetIterator
Known Direct Subclasses

Represents the state of an iteration through a tf.data Datset. DatasetIterator is not a java.util.Iterator. In eager mode, `Dataset` can be used as an Iterable, returning dataset elements each iteration.

Example: Iteration in graph mode. 

// Create input tensors
 Operand<?> features = tf.constant( ... );
 Operand<?> labels = tf.constant( ... );


 Dataset dataset = Dataset
         .fromTensorSlices(XTensor, yTensor);
         .batch(BATCH_SIZE);

 DatasetIterator iterator = dataset.makeInitializeableIterator();
 List<Operand<?>> components = iterator.getNext();
 Operand<?> featureBatch = components.get(0);
 Operand<?> labelBatch = components.get(1);

 // Build a TensorFlow graph that does something on each element.
 loss = computeModelLoss(featureBatch, labelBatch);

 optimizer = ... // create an optimizer
 trainOp = optimizer.minimize(loss);

 try (Session session = new Session(graph) {
   while (true) {
     session.run(iterator.getInitializer());
     try {
         session
           .addTarget(trainOp)
           .fetch(loss)
           .run();

         ...
      catch (TFOutOfRangeError e) {
         System.out.println("finished iterating.");
         break;
     }
   }
 }

 }

Example: Iteration in eager mode. 

// Create input tensors
 Operand<?> features = tf.constant( ... );
 Operand<?> labels = tf.constant( ... );

 int BATCH_SIZE = ...

 Dataset dataset = Dataset
         .fromTensorSlices(features, labels)
         .batch(BATCH_SIZE);
 DatasetIterator iterator = dataset.makeIterator();

 Optimizer optimizer = ... // create an optimizer

 for (List<Operand<?>> components : dataset) {
     Operand<?> featureBatch = components.get(0);
     Operand<?> labelBatch = components.get(1);

     loss = computeModelLoss(featureBatch, labelBatch);
     trainOp = optimizer.minimize(loss);
 
 }

Constants

String EMPTY_SHARED_NAME

Public Constructors

DatasetIterator(Ops tf, Operand<?> iteratorResource, Op initializer, List<Class<? extends TType>> outputTypes, List<Shape> outputShapes)
DatasetIterator(Ops tf, Operand<?> iteratorResource, List<Class<? extends TType>> outputTypes, List<Shape> outputShapes)

Public Methods

static DatasetIterator
fromStructure(Ops tf, List<Class<? extends TType>> outputTypes, List<Shape> outputShapes)
Creates a new iterator from a "structure" defined by `outputShapes` and `outputTypes`.
Op
getInitializer()
Operand<?>
getIteratorResource()
List<Operand<?>>
getNext()
Returns a list of Operand<?> representing the components of the next dataset element.
DatasetOptional
getNextAsOptional()
Returns a `DatasetOptional` representing the components of the next dataset element.
Ops
getOpsInstance()
Iterator<List<Operand<?>>>
iterator()
Op
makeInitializer(Dataset dataset)
Creates and returns a TF `Op` that can be run to initialize this iterator on a dataset.

Inherited Methods

Constants

public static final String EMPTY_SHARED_NAME

Constant Value: ""

Public Constructors

public DatasetIterator (Ops tf, Operand<?> iteratorResource, Op initializer, List<Class<? extends TType>> outputTypes, List<Shape> outputShapes)

Parameters
tf Ops accessor corresponding to the same `ExecutionEnvironment` as the `iteratorResource`.
iteratorResource An Operand representing the iterator (e.g. constructed from `tf.data.iterator` or `tf.data.anonymousIterator`)
initializer An `Op` that should be run to initialize this iterator
outputTypes A list of classes corresponding to the tensor type of each component of a dataset element.
outputShapes A list of `Shape` objects corresponding to the shapes of each component of a dataset element.

public DatasetIterator (Ops tf, Operand<?> iteratorResource, List<Class<? extends TType>> outputTypes, List<Shape> outputShapes)

Public Methods

public static DatasetIterator fromStructure (Ops tf, List<Class<? extends TType>> outputTypes, List<Shape> outputShapes)

Creates a new iterator from a "structure" defined by `outputShapes` and `outputTypes`.

Parameters
tf Ops accessor
outputTypes A list of classes repesenting the tensor type of each component of a dataset element.
outputShapes A list of Shape objects representing the shape of each component of a dataset element.
Returns
  • A new DatasetIterator

public Op getInitializer ()

public Operand<?> getIteratorResource ()

public List<Operand<?>> getNext ()

Returns a list of Operand<?> representing the components of the next dataset element.

In graph mode, call this method once, and use its result as input to another computation. Then in the training loop, on successive calls to session.run(), successive dataset elements will be retrieved through these components.

In eager mode, each time this method is called, the next dataset element will be returned. (This is done automatically by iterating through `Dataset` as a Java `Iterable`).

Returns
  • A List<Operand<?>> representing dataset element components.

public DatasetOptional getNextAsOptional ()

Returns a `DatasetOptional` representing the components of the next dataset element.

In eager mode, each time this method is called, the next dataset element will be returned as a `DatasetOptional`.

Use `DatasetOptional.hasValue` to check if this optional has a value, and `DatasetOptional.getValue` to retrieve the value.

Returns
  • A `DatasetOptional` representing dataset element components.

public Ops getOpsInstance ()

public Iterator<List<Operand<?>>> iterator ()

public Op makeInitializer (Dataset dataset)

Creates and returns a TF `Op` that can be run to initialize this iterator on a dataset. The dataset must have a structure (outputTypes, outputShapes) that match this iterator, and share the same ExecutionEnvironment as this iterator.

When this `Op` is run, this iterator will be "re-initialized" at the first element of the input dataset.

In eager mode, the op will be run automatically as part of a call to `makeIterator`.

Parameters
dataset An `org.tensorflow.data.Dataset` to initialize this iterator on.
Returns
  • A TF `Op` that can be used to initialize this iterator on the dataset.
Throws
IllegalArgumentException if the dataset's ExecutionEnvironment or structure doesn't match this iterator.