grain.experimental.torch module

Contents

grain.experimental.torch module#

Experimental Grain APIs for PyTorch pipelines.

grain.experimental.torch provides Grain’s Dataset API with PyTorch defaults:

  • Random transformations receive PyTorch PRNG keys (stateless_random_map).

  • batch collates elements into ``torch.Tensor``s.

  • mp_prefetch limits PyTorch threads in worker processes and passes CPU tensors between processes through shared memory.

  • Iterators can be checkpointed with torch.distributed.checkpoint.

PyTorch remains an optional dependency: importing grain or grain.experimental.torch does not import torch. Checkpointing more than one rank with torch.distributed.checkpoint requires PyTorch 2.4 or later.

Import the module under an alias so that it does not shadow torch:

from grain.experimental import torch as grain_torch

Example usage:

import torch
import torch.distributed.checkpoint as dcp
import torch.func._random as torch_random
from grain.experimental import torch as grain_torch

def augment(sample, key):
  image, label = sample
  noise = torch_random.normal(key, image.shape)
  return image + 0.1 * noise, label

# The rank and the number of ranks come from `torch.distributed`; they are
# `(0, 1)` outside of a distributed job.
rank, world_size = grain_torch.get_process_index_and_count()

ds = (
    # Any indexable source, such as a map-style `torch.utils.data.Dataset`.
    grain_torch.MapDataset.source(source)
    .seed(42)
    .shuffle()
    .repeat(num_epochs)
    [rank::world_size]  # Each rank reads its own part of the data.
    .stateless_random_map(augment)  # `key` is a PyTorch PRNG key.
    .batch(256, drop_remainder=True)  # `torch.utils.data.default_collate`.
    .to_iter_dataset()
    .with_options(grain_torch.DatasetOptions(min_shm_size=64 << 10))
    # Each worker process uses one PyTorch intra-op thread by default.
    .mp_prefetch(grain_torch.MultiprocessingOptions(num_workers=16))
)

it = iter(ds)
for step, (images, labels) in enumerate(it):
  ...  # Run a training step.
  if step % checkpoint_every == 0:
    dcp.save({"model": model, "data": it}, checkpoint_id=checkpoint_path)

# To resume, rebuild `ds` and restore the position of a new iterator.
it = iter(ds)
dcp.load({"model": model, "data": it}, checkpoint_id=checkpoint_path)

The datasets support the same methods as grain.MapDataset and grain.IterDataset; see their documentation for the transformations that are not specific to PyTorch.

This module is experimental and its API may change. It relies on PyTorch’s torch.func._random module, which is experimental as well.

List of Members#

class grain.experimental.torch.MapDataset(inner)[source]#

Bases: MapDataset[T]

grain.MapDataset with PyTorch defaults.

Supports the same methods as grain.MapDataset, including the factory methods such as source and range; see its documentation for the full reference and examples. Only the differences are documented here:

  • stateless_random_map passes PyTorch PRNG keys to plain callables (TorchStatelessRandomMap).

  • random_map is not supported, because it passes a NumPy generator to the transformation. Use stateless_random_map instead.

  • batch collates with torch.utils.data.default_collate by default.

  • Transformations return grain.experimental.torch.MapDataset, and to_iter_dataset returns grain.experimental.torch.IterDataset.

Each instance wraps a plain grain.MapDataset and builds every transformation on it, so default seeds, slicing, and shared-memory batching behave exactly as in grain.MapDataset.

Example usage:

import torch
from grain.experimental import torch as grain_torch

ds = (
    grain_torch.MapDataset.source([torch.zeros(3)] * 8)
    .shuffle(seed=42)
    .batch(4)
)
ds[0].shape  # torch.Size([4, 3])
Parameters:

inner (dataset.MapDataset[T])

__getitem__(index: slice) → MapDataset[T][source]#
__getitem__(index: int) → T | None

Returns the element for the index or None if missing.

__init__(inner)[source]#
Parameters:

inner (MapDataset[T])

__iter__()[source]#
Return type:

DatasetIterator[T]

__len__()[source]#

Returns length of a single epoch of this dataset.

Return type:

int

apply(transformations)[source]#

Returns a dataset with the given transformation(s) applied.

Syntactic sugar to avoid dispatch by transformation type.

Example usage:

ds = grain.MapDataset.range(5)
ds = ds.apply([AddOne(), grain.transforms.Batch(2)])
list(ds) == [np.ndarray([1, 2]), np.ndarray([3, 4]), np.ndarray([5])]
Parameters:

transformations (Batch | Map | RandomMap | StatelessRandomMap | TfRandomMap | Filter | FlatMap | MapWithIndex | Sequence[Batch | Map | RandomMap | StatelessRandomMap | TfRandomMap | Filter | FlatMap | MapWithIndex]) – one or more transformations to apply.

Returns:

Dataset with the given transformations applied.

Return type:

MapDataset[Any]

batch(batch_size, *, drop_remainder=False, batch_fn=None)[source]#

Returns a dataset of elements batched along a new first dimension.

Same as grain.MapDataset.batch, except that batch_fn defaults to torch.utils.data.default_collate. Batches are ``torch.Tensor``s; NumPy arrays and Python numbers are converted to tensors too.

Example usage:

ds = grain_torch.MapDataset.range(5)
ds = ds.batch(batch_size=2)
list(ds)  # [tensor([0, 1]), tensor([2, 3]), tensor([4])]
Parameters:
  • batch_size (int) – The number of elements to batch together.

  • drop_remainder (bool) – Whether to drop the last batch if it is smaller than batch_size.

  • batch_fn (Callable[[Sequence[T]], S] | None) – A function that takes a list of elements and returns a batch. Defaults to torch.utils.data.default_collate.

Returns:

A dataset of batches. A batch has the PyTree structure of an element, except that torch.utils.data.default_collate returns a list for a tuple.

Return type:

MapDataset[Any]

filter(transform)[source]#

Returns a dataset containing only the elements that match the filter.

Accessing an element of the returned dataset using subscription (ds[i]) returns:

  • None if transform returned False

  • the element if transform returned True

Iterating over a filtered dataset skips None elements by default.

Example usage:

ds = MapDataset.range(5)
ds = ds.filter(lambda x: x % 2 == 0)
ds[2] == 2
ds[1] == None
list(ds) == [0, 2, 4]

NOTE: list(ds) converts the dataset to an IterDataset with to_iter_dataset() under the hood which by default skips None elements.

to_iter_dataset produces a warning when iterating through a filtered dataset if the filter removes more than 90% of the elements. You can adjust the threshold through grain.experimental.DatasetOptions.filter_warn_threshold_ratio used in WithOptionsIterDataset. In order to produce an exception in such case use filter_raise_threshold_ratio.

Parameters:

transform (Filter | Callable[[T], bool]) – Either a FilterTransform containing the filter method or a callable that takes an element and returns a boolean.

Returns:

A dataset of the same type containing only the elements for which the filter transform returns True.

Return type:

MapDataset[T]

classmethod from_dataset(ds)[source]#

Returns ds as a grain.experimental.torch.MapDataset.

Use it to get the PyTorch defaults for a pipeline that was built with grain.MapDataset, for example by a library function. ds is returned unchanged if it already is a grain.experimental.torch.MapDataset.

Example usage:

ds = grain.MapDataset.range(4)  # A plain Grain dataset.
ds = grain_torch.MapDataset.from_dataset(ds)
ds = ds.batch(2)  # Batches are `torch.Tensor`s.
Parameters:

ds (MapDataset[T]) – A grain.MapDataset.

Returns:

A grain.experimental.torch.MapDataset over the same pipeline.

Return type:

MapDataset[T]

map(transform)[source]#

Returns a dataset containing the elements transformed by transform.

Example usage:

ds = MapDataset.range(5)
ds = ds.map(lambda x: x + 10)
list(ds) == [10, 11, 12, 13, 14]
Parameters:

transform (Map | Callable[[T], S]) – Either a MapTransform containing the map method or a callable that takes an element and returns a new element.

Returns:

A dataset containing the elements of the original dataset transformed by transform.

Return type:

MapDataset[S]

map_with_index(transform)[source]#

Returns a dataset containing the elements transformed by transform.

The transform is called with the index of the element within the dataset and the element itself.

Example usage:

ds = MapDataset.source(["a", "b", "c", "d"])
ds = ds.map_with_index(lambda i, x: x + str(i))
list(ds) == ["a0", "b1", "c2", "d3"]
Parameters:

transform (MapWithIndex | Callable[[int, T], S]) – Either a MapWithIndexTransform containing the map_with_index method or a callable that takes an index and an element and returns a new element.

Returns:

A dataset containing the elements of the original dataset transformed by transform.

Return type:

MapDataset[S]

property parents: Sequence[MapDataset]#
pipe(func, /, *args, **kwargs)[source]#

Returns func(self, *args, **kwargs).

Same as grain.MapDataset.pipe, except that a grain.MapDataset or grain.IterDataset returned by func is returned as a grain.experimental.torch dataset. This way, plain Grain dataset classes can be used in a PyTorch pipeline. Such a class is applied to the grain.MapDataset that this dataset wraps.

Example usage:

ds = grain_torch.MapDataset.range(8)
ds = ds.pipe(
    grain.experimental.WindowShuffleMapDataset, window_size=4, seed=0
)
ds = ds.batch(2)  # Batches are `torch.Tensor`s.
Parameters:
  • func (Callable[[...], Any]) – A callable that takes this dataset as its first argument.

  • *args – Positional arguments to pass to func after the dataset.

  • **kwargs – Keyword arguments to pass to func.

Returns:

The result of func. A grain.MapDataset or grain.IterDataset is returned as a grain.experimental.torch dataset.

Return type:

Any

random_map(transform, *, seed=None)[source]#

Not supported; use stateless_random_map instead.

grain.MapDataset.random_map passes a NumPy generator to transform, which makes it easy to mix NumPy arrays into a PyTorch pipeline. stateless_random_map passes a PyTorch PRNG key instead.

Parameters:
  • transform (RandomMap | Callable[[...], Any]) – Unused.

  • seed (int | None) – Unused.

Raises:

NotImplementedError – Always.

Return type:

NoReturn

repeat(num_epochs=None, *, reseed_each_epoch=True)[source]#

Returns a dataset repeating the elements of this dataset multiple times.

Specifying None for num_epochs will repeat the dataset infinitely, and causes len(ds) to return sys.maxsize.

Can not be called on an infinite dataset.

By default, random upstream transformations such as shuffle and random_map will be seeded differently (but deterministically) for each epoch. This means that each epoch will be shuffled differently. If you want to repeat the first epoch exactly as is, set reseed_each_epoch to False.

Implementation note: since MapDataset allows accessing elements past len(ds) - 1 anyway (and uses the index modulo len(ds)), this transformation effectively only changes the length of the dataset.

Example usage:

list(MapDataset.range(5).repeat(2)) == [0, 1, 2, 3, 4, 0, 1, 2, 3, 4]
ds = MapDataset.range(5).repeat()
len(ds) == sys.maxsize
ds[11111] == 1
Parameters:
  • num_epochs (int | None) – Either a positive integer representing the number of times this dataset should be repeated or None to repeat infinitely.

  • reseed_each_epoch (bool) – If True (default), all random upstream transformations use a seed folded with the epoch number, so that each epoch is shuffled and processed differently. If False, the first epoch is repeated exactly as is. In both cases, the random transformations are deterministic.

Returns:

A dataset repeating the elements of this dataset multiple times.

Return type:

MapDataset[T]

seed(seed)[source]#

Returns a dataset that uses the seed for default seed generation.

When default seed generation is enabled by calling ds.seed, every downstream random transformation will be automatically seeded with a unique seed by default. This simplifies seed management, making it easier to avoid:

  • Having to provide a seed in multiple transformations.

  • Accidentally reusing the same seed across transformations.

It is recommended to call this right after the source. ds.seed has to be called before any random transformations (such as shuffle or random_map that rely on default seed generation to control their seeding). Given the same seed, the pipeline is guaranteed to always use the same seeds for each transformation.

WARNING: The seed for random downstream transformations is derived from the seed passed to ds.seed and the absolute position of the transformation in the pipeline. This means that if you add transformations before the random transformation, its seed will change. For instance, if this random transformation is shuffle, adding a transformation before shuffle will change its seed and data order, consequently. To avoid this, pass the seed to the transformation directly.

Note about custom dataset implementations: the default seed generation is available through _default_seed, but the private API is not guaranteed to be stable.

Example 1:

ds = ds.seed(seed).shuffle()

shuffle will automatically derive its own seed (different from seed).

Example 2:

ds = ds.seed(seed).shuffle().random_map(...)

shuffle and random_map will each derive their own seed and the seeds are going to be different.

Example 3:

ds = ds.seed(seed).random_map(transform, seed=seed1)

random_map will use seed1 and will not be affected by seed. This can be used to control individual transformation seeding independently from the rest of the pipeline.

Example 4:

ds = ds.seed(seed1).shuffle().seed(seed2).random_map(...)

ds.seed only affects the downstream transformations and can be overridden by a subsequent seed call. shuffle will derive its seed from seed1, random_map - from seed2 and will not be affected by seed1. This can be used to control your transformation seeding even if you don’t own the first part of the pipeline.

Example 5:

ds1 = ds.source(...).seed(seed1).shuffle()
ds2 = ds.source(...).seed(seed2).shuffle()
ds = MapDataset.mix([ds1, ds2], ...).random_map(...)

Each shuffle will derive its own seed from seed1 or seed2 respectively. random_map will derive its seed from both seed1 and seed2.

Parameters:

seed (int) – Seed to use.

Returns:

A dataset with elements unchanged.

Return type:

MapDataset[T]

shuffle(seed=None)[source]#

Returns a dataset with the same elements in a globally shuffled order.

The shuffle is deterministic and will always produce the same result given the same seed. The seed can be either provided explicitly or set via ds.seed(seed). Prefer the latter if you don’t need to control the shuffle seed individually. It allows to pass a single seed to derive seeds for all downstream random transformations in the pipeline.

In multi-epoch training each epoch will be shuffled differently (i.e. the seed is combined with epoch number). In such case it is recommended to shuffle before repeat to avoid mixing elements from different epochs.

Example usage:

ds = MapDataset.range(5).shuffle()
set(ds) == {0, 1, 2, 3, 4}
list(ds) != [0, 1, 2, 3, 4]  # With probability (1 - 1/5!).
Parameters:

seed (int | None) – An optional integer between 0 and 2**32-1 representing the seed used by the shuffling algorithm. If you don’t need to control the shuffle seed individually, prefer setting the pipeline-level seed with ds.seed(seed) instead.

Returns:

A dataset containing the same elements but in a shuffled order.

Return type:

MapDataset[T]

slice(sl, wrap_around=False)[source]#

Returns a dataset containing only the elements with indices in sl.

For most implementations of MapDataset slicing is also available through subscript operator: list(ds.slice(slice(1, 10, 2))) == ds[1:10:2].

Example usage with a slice:

ds = MapDataset.range(5)
list(ds.slice(slice(1, 3))) == [1, 2]
list(ds.slice(slice(1, None, 2))) == [1, 3]

Example usage with a sequence of indices:

ds = MapDataset.range(10)
list(ds.slice([3, 1, 4, 1, 5])) == [3, 1, 4, 1, 5]

The indices sequence can also be a MapDataset of integers:

ds = MapDataset.range(10)
idx = MapDataset.range(3)
list(ds.slice(idx)) == [0, 1, 2]

Commonly used for sharding: ds = ds.slice(slice(shard_index, None, shard_count)), or, equivalently, ds = ds[shard_index::shard_count].

Parameters:
  • sl (slice | Sequence[int]) – A slice object (https://docs.python.org/3/library/functions.html#slice) representing the slice of elements to that should constitute the returned dataset, or a Sequence[int] (including MapDataset[int]) of indices into the parent dataset.

  • wrap_around (bool) – If True, indices wrap modularly within the slice range. This is needed when transformations that do their own epoch math (like shuffle or mix) sit between the slice and a repeat.

Returns:

A dataset containing only the elements with indices in sl.

Return type:

MapDataset[T]

stateless_random_map(transform, *, seed=None)[source]#

Returns a dataset transformed with a per-element PyTorch PRNG key.

Same as grain.MapDataset.stateless_random_map, except that a plain callable fn(element, key) is wrapped in TorchStatelessRandomMap instead of JaxStatelessRandomMap: key is a key tensor for the stateless random functions in torch.func._random. A StatelessRandomMap instance is used as is.

Example usage:

import torch
import torch.func._random as torch_random
from grain.experimental import torch as grain_torch

def add_noise(x, key):
  return x + torch_random.normal(key, x.shape)

ds = grain_torch.MapDataset.source([torch.zeros(2)] * 4)
ds = ds.seed(42).stateless_random_map(add_noise)
# The key depends only on the seed and the element's index, so the
# output is the same on every run.
Parameters:
  • transform (StatelessRandomMap | Callable[[T, Any], S]) – A callable fn(element, key) (wrapped in TorchStatelessRandomMap) or a StatelessRandomMap implementation.

  • seed (int | None) – An optional non-negative 64-bit integer seed. Defaults to the seed derived from ds.seed(...).

Returns:

A MapDataset containing the transformed elements.

Return type:

MapDataset[Any]

to_iter_dataset(read_options=None, *, allow_nones=False)[source]#

Converts this dataset to an IterDataset.

Elements from this dataset may be processed in multiple threads.

Note that some of the transformations are not available on IterDataset. These are roughly transformations operating on element index such as shuffle, map_with_index, slice and repeat.

Parameters:
  • read_options (ReadOptions | None) – Controls multithreading when reading the data and applying transformations in this dataset.

  • allow_nones (bool) – Whether to allow None values in the dataset (e.g. produced by filter). If False (the default), None values will be filtered out.

Returns:

An IterDataset with the same non- None elements as this dataset.

Return type:

IterDataset[T]

class grain.experimental.torch.IterDataset(inner)[source]#

Bases: IterDataset[T]

grain.IterDataset with PyTorch defaults.

Supports the same methods as grain.IterDataset; see its documentation for the full reference and examples. Only the differences are documented here:

  • stateless_random_map passes PyTorch PRNG keys to plain callables (TorchStatelessRandomMap).

  • random_map is not supported, because it passes a NumPy generator to the transformation. Use stateless_random_map instead.

  • batch collates with torch.utils.data.default_collate by default.

  • mp_prefetch limits PyTorch threads in worker processes and passes CPU tensors between processes through Grain’s shared memory.

  • with_options applies DatasetOptions.

  • Transformations return grain.experimental.torch.IterDataset, and iterating returns grain.experimental.torch.DatasetIterator.

Each instance wraps a plain grain.IterDataset and builds every transformation on it, so the pipeline behaves exactly as in grain.IterDataset.

Parameters:

inner (dataset.IterDataset[T])

__init__(inner)[source]#
Parameters:

inner (IterDataset[T])

__iter__()[source]#

Returns an iterator for this dataset.

Return type:

DatasetIterator[T]

apply(transformations)[source]#

Returns a dataset with the given transformation(s) applied.

Syntactic sugar to avoid dispatch by transformation type.

Example usage:

ds = grain.MapDataset.range(5).to_iter_dataset()
ds = ds.apply([AddOne(), grain.transforms.Batch(2)])
list(ds) == [np.ndarray([1, 2]), np.ndarray([3, 4]), np.ndarray([5])]
Parameters:

transformations (Batch | Map | RandomMap | StatelessRandomMap | TfRandomMap | Filter | FlatMap | MapWithIndex | Sequence[Batch | Map | RandomMap | StatelessRandomMap | TfRandomMap | Filter | FlatMap | MapWithIndex]) – one or more transformations to apply.

Returns:

Dataset with the given transformations applied.

Return type:

IterDataset[Any]

batch(batch_size, *, drop_remainder=False, batch_fn=None)[source]#

Returns a dataset of elements batched along a new first dimension.

Same as grain.IterDataset.batch, except that batch_fn defaults to torch.utils.data.default_collate. Batches are ``torch.Tensor``s; NumPy arrays and Python numbers are converted to tensors too.

Example usage:

ds = grain_torch.MapDataset.range(5).to_iter_dataset()
ds = ds.batch(batch_size=2)
list(ds)  # [tensor([0, 1]), tensor([2, 3]), tensor([4])]
Parameters:
  • batch_size (int) – The number of elements to batch together.

  • drop_remainder (bool) – Whether to drop the last batch if it is smaller than batch_size.

  • batch_fn (Callable[[Sequence[T]], S] | None) – A function that takes a list of elements and returns a batch. Defaults to torch.utils.data.default_collate.

Returns:

A dataset of batches. A batch has the PyTree structure of an element, except that torch.utils.data.default_collate returns a list for a tuple.

Return type:

IterDataset[Any]

enable_shared_memory_output()[source]#

Makes this dataset write its output to shared memory, if it can.

Called by mp_prefetch on the dataset that it prefetches from. It is not meant to be called directly.

Return type:

None

filter(transform)[source]#

Returns a dataset containing only the elements that match the filter.

Example usage:

ds = MapDataset.range(5).to_iter_dataset()
ds = ds.filter(lambda x: x % 2 == 0)
list(ds) == [0, 2, 4]

Produces a warning if the filter removes more than 90% of the elements. You can adjust the threshold through grain.experimental.DatasetOptions.filter_warn_threshold_ratio used in WithOptionsIterDataset. In order to produce an exception in such case use filter_raise_threshold_ratio.

Parameters:

transform (Filter | Callable[[T], bool]) – Either a FilterTransform containing the filter method or a callable that takes an element and returns a boolean.

Returns:

A dataset of the same type containing only the elements for which the filter transform returns True .

Return type:

IterDataset[T]

classmethod from_dataset(ds)[source]#

Returns ds as a grain.experimental.torch.IterDataset.

Use it to get the PyTorch defaults for a pipeline that was built with grain.IterDataset, for example by a library function. ds is returned unchanged if it already is a grain.experimental.torch.IterDataset.

Example usage:

ds = grain.MapDataset.range(4).to_iter_dataset()  # A plain Grain dataset.
ds = grain_torch.IterDataset.from_dataset(ds)
ds = ds.batch(2)  # Batches are `torch.Tensor`s.
Parameters:

ds (IterDataset[T]) – A grain.IterDataset.

Returns:

A grain.experimental.torch.IterDataset over the same pipeline.

Return type:

IterDataset[T]

property has_child: bool#
map(transform)[source]#

Returns a dataset containing the elements transformed by transform.

Example usage:

ds = MapDataset.range(5).to_iter_dataset()
ds = ds.map(lambda x: x + 10)
list(ds) == [10, 11, 12, 13, 14]
Parameters:

transform (Map | Callable[[T], S]) – Either a MapTransform containing the map method or a callable that takes an element and returns a new element.

Returns:

A dataset containing the elements of the original dataset transformed by transform.

Return type:

IterDataset[S]

map_with_index(transform)[source]#

Returns a dataset of the elements transformed by the transform.

The transform is called with the index of the element in the dataset as the first argument and the element as the second argument.

Example usage:

ds = MapDataset.range(5).to_iter_dataset()
ds = ds.map(lambda i, x: (i, 2**x))
list(ds) == [(0, 1), (1, 2), (2, 4), (3, 8), (4, 16)]
Parameters:

transform (MapWithIndex | Callable[[int, T], S]) – Either a MapWithIndexTransform containing the map_with_index method or a callable that takes an index and an element and returns a new element.

Returns:

A dataset containing the elements of the original dataset transformed by transform.

Return type:

IterDataset[S]

mp_prefetch(options=None, worker_init_fn=None, sequential_slice=False)[source]#

Returns a dataset prefetching elements in multiple processes.

Same as grain.IterDataset.mp_prefetch; see its documentation for how workers slice the dataset and for cleanup recommendations. There are two PyTorch-specific differences:

  • Each worker process is limited to options.worker_intra_op_threads PyTorch intra-op threads (1 by default), like torch.utils.data.DataLoader workers. The limit is applied before worker_init_fn runs. To use another thread count, set worker_intra_op_threads rather than calling torch.set_num_threads in worker_init_fn, which some PyTorch builds ignore after the first call.

  • CPU tensors are passed from the workers through Grain’s shared memory, so DatasetOptions.min_shm_size applies to them as it does to NumPy arrays.

Example usage:

ds = grain_torch.MapDataset.source([torch.zeros(1024)] * 8).batch(4)
ds = ds.to_iter_dataset().mp_prefetch(
    grain_torch.MultiprocessingOptions(num_workers=2)
)
it = iter(ds)
next(it).shape  # torch.Size([4, 1024])
it.close()
Parameters:
  • options (MultiprocessingOptions | None) – Options for the prefetching processes, either grain.experimental.torch.MultiprocessingOptions or grain.MultiprocessingOptions. If options.num_workers is 0, mp_prefetch has no effect. Defaults to 10 workers with one intra-op thread each.

  • worker_init_fn (Callable[[int, int], None] | None) – A function that is called in each worker process before the data is processed. The function takes two arguments: the current worker index and the total worker count.

  • sequential_slice (bool) – Whether the slice for each worker should be sequential (consecutive at the source dataset). Experimental.

Returns:

A dataset prefetching input elements in separate processes.

Return type:

IterDataset[T]

property parents: Sequence[MapDataset | IterDataset]#
pipe(func, /, *args, **kwargs)[source]#

Returns func(self, *args, **kwargs).

Same as grain.IterDataset.pipe, except that a grain.MapDataset or grain.IterDataset returned by func is returned as a grain.experimental.torch dataset. This way, plain Grain dataset classes can be used in a PyTorch pipeline. Such a class is applied to the grain.IterDataset that this dataset wraps.

Example usage:

ds = grain_torch.MapDataset.range(8).to_iter_dataset()
ds = ds.pipe(
    grain.experimental.WindowShuffleIterDataset, window_size=4, seed=0
)
ds = ds.batch(2)  # Batches are `torch.Tensor`s.
Parameters:
  • func (Callable[[...], Any]) – A callable that takes this dataset as its first argument.

  • *args – Positional arguments to pass to func after the dataset.

  • **kwargs – Keyword arguments to pass to func.

Returns:

The result of func. A grain.MapDataset or grain.IterDataset is returned as a grain.experimental.torch dataset.

Return type:

Any

prefetch(multiprocessing_options)[source]#

Deprecated, use mp_prefetch instead.

Returns a dataset prefetching the elements in multiple processes.

Each of the processes will process a slice of the dataset after all MapDataset transformations.

WARNING: If the dataset contains many-to-one transformations (such as batch), output of prefetch may change if you change the number of workers. However, it is still going to be determisitic.

Parameters:

multiprocessing_options (MultiprocessingOptions) – options for the prefetching processes. num_workers must be greater than 0.

Returns:

A dataset prefetching input elements concurrently.

Return type:

IterDataset[T]

random_map(transform, *, seed=None)[source]#

Not supported; use stateless_random_map instead.

grain.IterDataset.random_map passes a NumPy generator to transform, which makes it easy to mix NumPy arrays into a PyTorch pipeline. stateless_random_map passes a PyTorch PRNG key instead.

Parameters:
  • transform (RandomMap | Callable[[...], Any]) – Unused.

  • seed (int | None) – Unused.

Raises:

NotImplementedError – Always.

Return type:

NoReturn

seed(seed)[source]#

Returns a dataset that uses the seed for default seed generation.

When default seed generation is enabled by calling ds.seed, every downstream random transformation will be automatically seeded with a unique seed by default. This simplifies seed management, making it easier to avoid:

  • Having to provide a seed in multiple transformations.

  • Accidentally reusing the same seed across transformations.

It is recommended to call this right after the source. ds.seed has to be called before any random transformations (such as random_map that rely on default seed generation to control their seeding). Given the same seed, the pipeline is guaranteed to always use the same seeds for each transformation.

Note about custom dataset implementations: the default seed generation is available through _default_seed, but the private API is not guaranteed to be stable.

Example 1:

ds = ds.seed(seed).random_map(...)

random_map will automatically derive its own seed (different from seed).

Example 2:

ds = ds.seed(seed).random_map().random_map(...)

The first and second random_map s will each derive their own seed and the seeds are going to be different.

Example 3:

ds = ds.seed(seed).random_map(transform, seed=seed1)

random_map will use seed1 and will not be affected by seed. This can be used to control individual transformation seeding independently from the rest of the pipeline.

Example 4:

ds = ds.seed(seed1).random_map(...).seed(seed2).random_map(...)

ds.seed only affects the downstream transformations and can be overridden by a subsequent seed call. The first random_map will derive its seed from seed1, the second - from seed2 and will not be affected by seed1. This can be used to control your transformation seeding even if you don’t own the first part of the pipeline.

Example 5:

ds1 = ds.source(...).seed(seed2).shuffle().to_iter_dataset()
ds2 = ds.source(...).seed(seed2).shuffle().to_iter_dataset()
ds = IterDataset.mix([ds1, ds2], ...).random_map(...)

Each shuffle will derive its own seed from seed1 or seed2 respectively. random_map will derive its seed from both seed1 and seed2.

Parameters:

seed (int) – Seed to use.

Returns:

A dataset with elements unchanged.

Return type:

IterDataset[T]

set_slice(sl, sequential_slice=False)[source]#

Restricts this dataset, in place, to a slice of its elements.

Called by mp_prefetch on each worker’s copy of the dataset to give the worker its share of the data. It is not meant to be called directly.

Parameters:
  • sl (slice) – The slice of elements to keep.

  • sequential_slice (bool) – Whether the slice is a contiguous range of the source rather than every n-th element.

Return type:

None

stateless_random_map(transform, *, seed=None)[source]#

Returns a dataset transformed with a per-element PyTorch PRNG key.

Same as grain.IterDataset.stateless_random_map, except that a plain callable fn(element, key) is wrapped in TorchStatelessRandomMap instead of JaxStatelessRandomMap: key is a key tensor for the stateless random functions in torch.func._random. A StatelessRandomMap instance is used as is.

Example usage:

import torch
import torch.func._random as torch_random
from grain.experimental import torch as grain_torch

def add_noise(x, key):
  return x + torch_random.normal(key, x.shape)

ds = grain_torch.MapDataset.source([torch.zeros(2)] * 4).to_iter_dataset()
ds = ds.seed(42).stateless_random_map(add_noise)
# The key depends only on the seed and the element's index, so the
# output is the same on every run.
Parameters:
  • transform (StatelessRandomMap | Callable[[T, Any], S]) – A callable fn(element, key) (wrapped in TorchStatelessRandomMap) or a StatelessRandomMap implementation.

  • seed (int | None) – An optional non-negative 64-bit integer seed. Defaults to the seed derived from ds.seed(...).

Returns:

An IterDataset containing the transformed elements.

Return type:

IterDataset[Any]

with_options(options)[source]#

Returns a dataset with options applied to the pipeline.

Method form of grain.experimental.WithOptionsIterDataset that keeps the result a grain.experimental.torch.IterDataset. The options apply to all transformations in the pipeline, before and after this call. Options set by a later call take precedence.

Example usage:

ds = ds.to_iter_dataset()
# Tensors smaller than 64 KiB are copied between processes instead of
# being placed in shared memory.
ds = ds.with_options(grain_torch.DatasetOptions(min_shm_size=64 << 10))
ds = ds.mp_prefetch(grain_torch.MultiprocessingOptions(num_workers=4))
Parameters:

options (DatasetOptions) – The options to apply.

Returns:

An IterDataset with the options applied.

Return type:

IterDataset[T]

class grain.experimental.torch.DatasetIterator(parent)[source]#

Bases: DatasetIterator[T]

grain.DatasetIterator that supports PyTorch checkpointing.

Supports the same methods as grain.DatasetIterator; see its documentation for the full reference. Only the differences are documented here:

  • state_dict and load_state_dict make the iterator a torch.distributed.checkpoint Stateful object, so it can be passed to dcp.save and dcp.load next to the model and the optimizer. The state is stored per rank, so every rank restores its own position.

Example usage:

import torch.distributed.checkpoint as dcp

it = iter(ds)  # `ds` is a `grain.experimental.torch.IterDataset`.
...
dcp.save({"model": model, "data": it}, checkpoint_id=path)

# To resume, rebuild `ds` and restore the position of a new iterator.
it = iter(ds)
dcp.load({"model": model, "data": it}, checkpoint_id=path)

state_dict and load_state_dict use the default process group. If the ranks that read different data form another group, such as the data-parallel group under tensor parallelism, wrap the iterator in IteratorCheckpointWrapper to pass that group:

data = grain_torch.IteratorCheckpointWrapper(it, group=dp_group)
dcp.save({"model": model, "data": data}, checkpoint_id=path)

Like any grain.DatasetIterator, this iterator closes the iterator that it wraps when it is closed or garbage-collected. To checkpoint an iterator of a grain.MapDataset or grain.IterDataset, wrap it in IteratorCheckpointWrapper rather than in this class.

Parameters:

parent (dataset.DatasetIterator[T])

__init__(parent)[source]#

Initializes a DatasetIterator.

Parameters:
  • parents – The parent iterator or sequence of parent iterators.

  • parent (DatasetIterator[T])

__iter__()[source]#

Returns the iterator instance itself.

Return type:

DatasetIterator[T]

close()[source]#

Closes the iterator and releases resources.

This method is idempotent and safe to call multiple times. After calling close, the iterator will raise ValueError on any subsequent calls to __next__.

Recursively closes all parent iterators in the pipeline to ensure complete resource cleanup.

NOTE: Resource cleanup occurs via __del__ when the iterator is garbage collected, but since garbage collection is not guaranteed in CPython, this method can be called explicitly to ensure explicit blocking cleanup.

Return type:

None

get_state()[source]#

Returns the current state of the iterator.

We reserve the right to evolve the state format over time. The states returned from this method are only guaranteed to be restorable by the same version of the code that produced them.

Implementation Note: It is recommended that iterator implementations always produce states with the same shapes and types throughout the lifetime of the iterator. Some frameworks rely on this property to perform checkpointing, and all standard library iterators are compliant. It is also recommended to produce state values that support shapes and types, e.g. using np.int64 instead of int. The standard library iterators are not currently compliant with this recommendation.

Users can also use grain.checkpoint.CheckpointSave (which calls this function) to save the model and the iterator in a bundle when using Orbax.

Return type:

dict[str, Any]

async load(directory)[source]#

Loads the iterator state from a directory.

The state may be loaded and set in a background thread. The main thread should not alter the state content while the load is in progress.

Parameters:

directory (Path) – The directory to load the state from.

Returns:

A coroutine that has not been awaited. This is called by Orbax in a background thread to perform I/O without blocking the main thread.

Return type:

Awaitable[None]

load_state_dict(state_dict)[source]#

Restores iterator state from a dcp.load dictionary.

Parameters:

state_dict (Mapping[str, Any])

Return type:

None

async save(directory)[source]#

Saves the iterator state to a directory.

The current state (get_state) is used for saving, so any updates to the state after returning from this method will not affect the saved checkpoint.

Parameters:

directory (PathAwaitingCreation) – A path in the process of being created. Must call await_creation before accessing the physical path.

Returns:

A coroutine that has not been awaited. This is called by Orbax in a background thread to perform I/O without blocking the main thread.

Return type:

Awaitable[None]

set_state(state)[source]#

Sets the current state of the iterator.

Restores the internal execution state of this iterator and its parent iterators in-place, allowing iteration to resume from the exact step where get_state() was called without re-creating the iterator.

Users can also use grain.checkpoint.CheckpointRestore (which calls this function) to restore the model and the iterator in a bundle when using Orbax.

Parameters:

state (dict[str, Any]) – A dictionary containing the serialized iterator state previously produced by get_state().

Return type:

None

start_prefetch()[source]#

Starts processing elements in the first asynchronous parent iterator.

Useful when the iterator can be created in advance but the elements are not needed immediately. For instance, when recovering iterator and model from a checkpoint, recover the iterator first, call start_prefech and then recover the model. This way the time to get the first batch from the iterator will be partially or fully hidden behind the time it takes to recover the model.

This method is idempotent and safe to call multiple times.

Return type:

None

state_dict()[source]#

Returns a rank-keyed checkpoint dictionary for dcp.save.

Return type:

dict[str, Any]

class grain.experimental.torch.TorchStatelessRandomMap(fn=None)[source]#

Stateless PyTorch random map implementing StatelessRandomMap.

Can be used in two ways:

  1. Wrapping a callable fn(element, key): TorchStatelessRandomMap(fn) (used automatically when passing a plain callable to grain.experimental.torch.MapDataset.stateless_random_map(fn) or IterDataset.stateless_random_map(fn)).

  2. Subclassing and overriding stateless_random_map(self, element, key).

Each element receives an immutable CPU uint64[2] Philox PRNG key tensor derived via torch.func._random.key(seed) and torch.func._random.fold_in(op_key, index). Pass key to stateless random operations in torch.func._random (normal, uniform, randint, bits) and use torch.func._random.split(key) when drawing multiple independent random values within one transform.

Each element’s key depends only on the dataset seed and the element’s index, so results are deterministic under random access, prefetch threads, worker processes, and checkpoint restores. The global PyTorch RNG is neither used nor modified.

Experimental: both this class and PyTorch’s torch.func._random stateless PRNG module are experimental, and their APIs or random streams may change as PyTorch stabilizes public stateless RNG support.

Example

Adding Gaussian noise drawn from each element’s stateless PRNG key:

import torch
import torch.func._random as torch_random
from grain.experimental import torch as grain_torch

def add_noise(x, key):
  return x + torch_random.normal(key, x.shape)

# `grain.experimental.torch` datasets wrap plain callables in
# `TorchStatelessRandomMap` automatically.
ds = (
    grain_torch.MapDataset.source([torch.zeros(2)] * 4)
    .stateless_random_map(add_noise, seed=42)
    .batch(2)
)

# The same index always produces the same random values.
print(torch.equal(ds[1], ds[1]))
# True

grain.MapDataset and grain.IterDataset wrap plain callables in JaxStatelessRandomMap, so wrap the callable explicitly there:

import grain

ds = grain.MapDataset.source([torch.zeros(2)] * 4).stateless_random_map(
    grain_torch.TorchStatelessRandomMap(add_noise), seed=42
)
Parameters:

fn (Callable[[Any, Any], Any] | None)

make_key_for_index(seed)[source]#

Returns index -> key closure derived lazily via torch.func._random.

Parameters:

seed (int)

Return type:

Callable[[int], Any]

stateless_random_map(element, key)[source]#

Applies the wrapped callable or subclass override with key.

Parameters:
  • element (Any)

  • key (Any)

Return type:

Any

class grain.experimental.torch.MultiprocessingOptions(num_workers=0, per_worker_buffer_size=1, enable_profiling=False, worker_intra_op_threads=1)[source]#

Options for Python multiprocessing in PyTorch pipelines.

Extends grain.MultiprocessingOptions with worker_intra_op_threads (default 1) so PyTorch workers do not oversubscribe host CPU cores by default, like torch.utils.data.DataLoader workers.

Parameters:
  • num_workers (int)

  • per_worker_buffer_size (int)

  • enable_profiling (bool)

  • worker_intra_op_threads (int | None)

worker_intra_op_threads#

Number of PyTorch intra-op threads that each worker process uses. Grain applies it with torch.set_num_threads (and OMP_NUM_THREADS / MKL_NUM_THREADS) before worker_init_fn runs. To use another count, set this field rather than calling torch.set_num_threads in worker_init_fn: some PyTorch builds only honor the first call. If None, Grain does not set a thread count and each worker uses PyTorch’s own default: OMP_NUM_THREADS / MKL_NUM_THREADS when they are set in the environment, otherwise typically one thread per CPU core.

Type:

int | None

grain.experimental.torch.mp_prefetch(ds: IterDataset[T], options: MultiprocessingOptions | None = None, worker_init_fn: Callable[[int, int], None] | None = None, sequential_slice: bool = False) → IterDataset[T][source]#
grain.experimental.torch.mp_prefetch(ds: IterDataset[T], options: MultiprocessingOptions | None = None, worker_init_fn: Callable[[int, int], None] | None = None, sequential_slice: bool = False) → IterDataset[T]

Returns a dataset prefetching the elements of ds in multiple processes.

The function form of grain.experimental.torch.IterDataset.mp_prefetch; see its documentation. It can also be applied to a grain.IterDataset, for example with grain.IterDataset.pipe, and returns the same kind of dataset as ds.

Parameters:
  • ds (IterDataset[T]) – The dataset to prefetch from: a grain.experimental.torch.IterDataset or a grain.IterDataset.

  • options (MultiprocessingOptions | None) – Options for the prefetching processes, either grain.experimental.torch.MultiprocessingOptions or grain.MultiprocessingOptions. Defaults to 10 workers with one intra-op thread each.

  • worker_init_fn (Callable[[int, int], None] | None) – A function that is called in each worker process before the data is processed. The function takes two arguments: the current worker index and the total worker count.

  • sequential_slice (bool) – Whether the slice for each worker should be sequential (consecutive at the source dataset). Experimental.

Returns:

a grain.experimental.torch.IterDataset if ds is one, and a grain.IterDataset otherwise.

Return type:

A dataset prefetching input elements in separate processes

Raises:

TypeError – If ds is not a grain.IterDataset, for example if it is a MapDataset.

grain.experimental.torch.get_process_index_and_count(group=None)[source]#

Returns (process_index, process_count) for PyTorch distributed jobs.

Resolution order:

  1. If torch.distributed is available and initialized, queries dist.get_rank(group) and dist.get_world_size(group). Raises ValueError if the current process is not a member of group.

  2. If group was explicitly provided when torch.distributed is not initialized, raises RuntimeError.

  3. Falls back to the RANK and WORLD_SIZE environment variables (for torchrun before init_process_group). Raises ValueError if both are set but invalid. If only one of them is set, warns and ignores it.

  4. Otherwise returns (0, 1).

Never imports JAX, avoiding accidental PJRT initialization in PyTorch jobs.

Example

Sharding a dataset across torch.distributed ranks with a slice:

rank, world_size = grain.experimental.torch.get_process_index_and_count()
ds = ds.shuffle().repeat(num_epochs)[rank::world_size]
Parameters:

group (Any) – Optional torch.distributed.ProcessGroup (e.g., DP group under 2D/3D parallelism). Defaults to the default process group.

Returns:

Tuple (process_index, process_count).

Raises:
  • ValueError – If the current process is not a member of group, or if the RANK and WORLD_SIZE environment variables are consulted and both are set but are not integers with 0 <= RANK < WORLD_SIZE.

  • RuntimeError – If group is specified when torch.distributed is not initialized.

Return type:

tuple[int, int]

class grain.experimental.torch.IteratorCheckpointWrapper(iterator, *, group=None)[source]#

Wraps a Grain iterator to implement DCP’s Stateful protocol.

Implements torch.distributed.checkpoint.stateful.Stateful (state_dict / load_state_dict) on top of Grain’s get_state / set_state.

torch.distributed.checkpoint (DCP) keeps a single copy of each non-tensor state-dict key across ranks on save, so returning get_state() directly from state_dict() would make every rank restore one rank’s progress. IteratorCheckpointWrapper stores each rank’s serialized state under rank_{i} (scoped to group when DP is a subgroup of the global world, such as under tensor parallelism).

Saving from more than one rank with DCP requires PyTorch 2.4 or later. Earlier releases write non-tensor values only from the coordinator rank, so the state of the other ranks is not saved and cannot be restored.

Iterators of grain.experimental.torch datasets (grain.experimental.torch.DatasetIterator) implement Stateful on top of this class, so they can be passed to dcp.save and dcp.load directly. Use IteratorCheckpointWrapper explicitly in two cases:

  • To pass group. The iterators above always use the default process group.

  • To checkpoint an iterator of a grain.MapDataset or grain.IterDataset, which does not implement Stateful.

IteratorCheckpointWrapper holds only a reference to the iterator: state_dict() reads the iterator’s state when it is called, and nothing is cached. Create it at each save or load call, as in the example below, so that it always wraps the iterator in use. An instance that is kept around still wraps the old iterator after the iterator is replaced.

Example

An iterator of a grain.experimental.torch dataset needs no wrapper:

import torch.distributed.checkpoint as dcp
from grain.experimental import torch as grain_torch

it = iter(grain_torch.MapDataset.range(8).to_iter_dataset())
dcp.save({"data": it}, checkpoint_id=path)
dcp.load({"data": it}, checkpoint_id=path)

A process group other than the default one (here dp_group) is passed through the wrapper:

dcp.save(
    {"data": grain_torch.IteratorCheckpointWrapper(it, group=dp_group)},
    checkpoint_id=path,
)
dcp.load(
    {"data": grain_torch.IteratorCheckpointWrapper(it, group=dp_group)},
    checkpoint_id=path,
)

An iterator of a grain.MapDataset or grain.IterDataset always needs the wrapper, with or without group:

plain_it = iter(grain.MapDataset.range(8).to_iter_dataset())
dcp.save(
    {"data": grain_torch.IteratorCheckpointWrapper(plain_it)},
    checkpoint_id=path,
)
dcp.load(
    {"data": grain_torch.IteratorCheckpointWrapper(plain_it)},
    checkpoint_id=path,
)
Parameters:
  • iterator (Any)

  • group (Any)

load_state_dict(state_dict)[source]#

Restores this rank’s iterator progress from state_dict.

Parameters:

state_dict (Mapping[str, Any])

Return type:

None

state_dict()[source]#

Returns a per-rank dictionary suitable for dcp.save.

Return type:

dict[str, Any]