IO pipelines with queues

IO

Pipeline functions

add_queue_runner(runner::QueueRunner)

Manually adds a queue runner to the graph's collection of queue runners. All runners in the collection will be executed in parallel when start_queue_runners is invoked.

source

start_queue_runners(session::Session)

Run all queue runners in parallel that were previously added to the graph's collection of queue runners via add_queue_runner.

Args:

  • session: The TensorFlow session containing the queues

source

range_input_producer(limit; num_epochs=nothing, do_shuffle=true, seed=0, capacity=32)

Produces the integers from 1 to limit in a queue.

Args:

  • limit: Inclusive upper bound on the endpoint of the range of integers to produce

  • num_epochs: Number of times to produce the integers.

  • do_shuffle: If true, shuffle the inputs each epoch.

  • seed: Seed to use for the RNG if do_shuffle is true.

  • capacity: Sets the queue capacity. Default is 32.

source

input_producer(input; element_shape=nothing, num_epochs=nothing, do_shuffle=true, seed=0, capacity=32)

Outputs the rows of input to a queue for input pipelining.

Args:

  • input: A Tensor with the rows to produce.

  • element_shape: The shape of the input rows, in case it can't be inferred. Defaults to nothing.

  • num_epochs: Number of times to produce each row. If unspecified (default), input_producer can produce each row an unlimited number of times.

  • do_shuffle: If true, shuffle the inputs each epoch.

  • seed: Seed to use for the RNG if do_shuffle is true.

  • capacity: Sets the queue capacity. Default is 32.

source

string_input_producer(string_tensors; num_epochs=nothing, do_shuffle=true, seed=0, capacity=32)

Output strings to a queue for an input pipeline.

Args:

  • string_tensor: A one dimensional Tensor of strings to produce.

  • num_epochs: Number of times to produce the strings.

  • do_shuffle: If true, shuffle the inputs each epoch.

  • seed: Seed to use for the RNG if do_shuffle is true.

  • capacity: Sets the queue capacity. Default is 32.

source

shuffle_batch(tensors, batch_size; capacity=32, enqueue_many=false, shapes=nothing, dynamic_pad=false, allow_smaller_final_batch=false)

Create batches by randomly shuffling tensors.

Args:

  • tensors: A list of tensors to enqueue.

  • batch_size: The batch size which will be pulled from the queue.

  • capacity: Sets the queue capacity. Default is 32.

  • dynamic_pad: If true all tensors will be padded on unknown dimensions to batch_size. Otherwise tensors shapes must be fully known. Currently only false is supported.

  • enqueue_many: If false, tensors represents a single example. Otherwise tensors represents a batch of examples. Currently only false is supported.

  • shapes: The shapes for each example. Defaults to the inferred shapes from tensors.

  • allow_smaller_final_batch: If true (default false), the final batch is allowed to be smaller than the other batches if there are not enough samples remaining.

source

QueueRunner

Represents an object that continually enqueues elements to a TensorFlow queue in parallel with other operations.

source

create_threads(runner::QueueRunner, session::Session)

Creates tasks that run the enqueue operations in runner in parallel.

source

Readers

io.WholeFileReader
io.TextLineReader
read