grain.experimental.torch module#
Experimental Grain APIs for PyTorch pipelines.
grain.experimental.torch provides Grain’s Dataset API with PyTorch defaults:
Random transformations receive PyTorch PRNG keys (
stateless_random_map).batchcollates elements into ``torch.Tensor``s.mp_prefetchlimits PyTorch threads in worker processes and passes CPU tensors between processes through shared memory.Iterators can be checkpointed with
torch.distributed.checkpoint.
PyTorch remains an optional dependency: importing grain or
grain.experimental.torch does not import torch. Checkpointing more than
one rank with torch.distributed.checkpoint requires PyTorch 2.4 or later.
Import the module under an alias so that it does not shadow torch:
from grain.experimental import torch as grain_torch
Example usage:
import torch
import torch.distributed.checkpoint as dcp
import torch.func._random as torch_random
from grain.experimental import torch as grain_torch
def augment(sample, key):
image, label = sample
noise = torch_random.normal(key, image.shape)
return image + 0.1 * noise, label
# The rank and the number of ranks come from `torch.distributed`; they are
# `(0, 1)` outside of a distributed job.
rank, world_size = grain_torch.get_process_index_and_count()
ds = (
# Any indexable source, such as a map-style `torch.utils.data.Dataset`.
grain_torch.MapDataset.source(source)
.seed(42)
.shuffle()
.repeat(num_epochs)
[rank::world_size] # Each rank reads its own part of the data.
.stateless_random_map(augment) # `key` is a PyTorch PRNG key.
.batch(256, drop_remainder=True) # `torch.utils.data.default_collate`.
.to_iter_dataset()
.with_options(grain_torch.DatasetOptions(min_shm_size=64 << 10))
# Each worker process uses one PyTorch intra-op thread by default.
.mp_prefetch(grain_torch.MultiprocessingOptions(num_workers=16))
)
it = iter(ds)
for step, (images, labels) in enumerate(it):
... # Run a training step.
if step % checkpoint_every == 0:
dcp.save({"model": model, "data": it}, checkpoint_id=checkpoint_path)
# To resume, rebuild `ds` and restore the position of a new iterator.
it = iter(ds)
dcp.load({"model": model, "data": it}, checkpoint_id=checkpoint_path)
The datasets support the same methods as grain.MapDataset and
grain.IterDataset; see their documentation for the transformations that are
not specific to PyTorch.
This module is experimental and its API may change. It relies on PyTorch’s
torch.func._random module, which is experimental as well.
List of Members#
- class grain.experimental.torch.MapDataset(inner)[source]#
Bases:
MapDataset[T]grain.MapDatasetwith PyTorch defaults.Supports the same methods as
grain.MapDataset, including the factory methods such assourceandrange; see its documentation for the full reference and examples. Only the differences are documented here:stateless_random_mappasses PyTorch PRNG keys to plain callables (TorchStatelessRandomMap).random_mapis not supported, because it passes a NumPy generator to the transformation. Usestateless_random_mapinstead.batchcollates withtorch.utils.data.default_collateby default.Transformations return
grain.experimental.torch.MapDataset, andto_iter_datasetreturnsgrain.experimental.torch.IterDataset.
Each instance wraps a plain
grain.MapDatasetand builds every transformation on it, so default seeds, slicing, and shared-memory batching behave exactly as ingrain.MapDataset.Example usage:
import torch from grain.experimental import torch as grain_torch ds = ( grain_torch.MapDataset.source([torch.zeros(3)] * 8) .shuffle(seed=42) .batch(4) ) ds[0].shape # torch.Size([4, 3])
- Parameters:
inner (dataset.MapDataset[T])
- __getitem__(index: slice) MapDataset[T][source]#
- __getitem__(index: int) T | None
Returns the element for the index or None if missing.
- __init__(inner)[source]#
- Parameters:
inner (MapDataset[T])
- apply(transformations)[source]#
Returns a dataset with the given transformation(s) applied.
Syntactic sugar to avoid dispatch by transformation type.
Example usage:
ds = grain.MapDataset.range(5) ds = ds.apply([AddOne(), grain.transforms.Batch(2)]) list(ds) == [np.ndarray([1, 2]), np.ndarray([3, 4]), np.ndarray([5])]
- Parameters:
transformations (Batch | Map | RandomMap | StatelessRandomMap | TfRandomMap | Filter | FlatMap | MapWithIndex | Sequence[Batch | Map | RandomMap | StatelessRandomMap | TfRandomMap | Filter | FlatMap | MapWithIndex]) – one or more transformations to apply.
- Returns:
Dataset with the given transformations applied.
- Return type:
MapDataset[Any]
- batch(batch_size, *, drop_remainder=False, batch_fn=None)[source]#
Returns a dataset of elements batched along a new first dimension.
Same as
grain.MapDataset.batch, except thatbatch_fndefaults totorch.utils.data.default_collate. Batches are ``torch.Tensor``s; NumPy arrays and Python numbers are converted to tensors too.Example usage:
ds = grain_torch.MapDataset.range(5) ds = ds.batch(batch_size=2) list(ds) # [tensor([0, 1]), tensor([2, 3]), tensor([4])]
- Parameters:
batch_size (int) – The number of elements to batch together.
drop_remainder (bool) – Whether to drop the last batch if it is smaller than batch_size.
batch_fn (Callable[[Sequence[T]], S] | None) – A function that takes a list of elements and returns a batch. Defaults to
torch.utils.data.default_collate.
- Returns:
A dataset of batches. A batch has the PyTree structure of an element, except that
torch.utils.data.default_collatereturns a list for a tuple.- Return type:
MapDataset[Any]
- filter(transform)[source]#
Returns a dataset containing only the elements that match the filter.
Accessing an element of the returned dataset using subscription (
ds[i]) returns:NoneiftransformreturnedFalsethe element if
transformreturnedTrue
Iterating over a filtered dataset skips
Noneelements by default.Example usage:
ds = MapDataset.range(5) ds = ds.filter(lambda x: x % 2 == 0) ds[2] == 2 ds[1] == None list(ds) == [0, 2, 4]
NOTE:
list(ds)converts the dataset to anIterDatasetwithto_iter_dataset()under the hood which by default skipsNoneelements.to_iter_datasetproduces a warning when iterating through a filtered dataset if the filter removes more than 90% of the elements. You can adjust the threshold throughgrain.experimental.DatasetOptions.filter_warn_threshold_ratioused inWithOptionsIterDataset. In order to produce an exception in such case usefilter_raise_threshold_ratio.- Parameters:
transform (Filter | Callable[[T], bool]) – Either a
FilterTransformcontaining thefiltermethod or a callable that takes an element and returns a boolean.- Returns:
A dataset of the same type containing only the elements for which the filter transform returns
True.- Return type:
MapDataset[T]
- classmethod from_dataset(ds)[source]#
Returns
dsas agrain.experimental.torch.MapDataset.Use it to get the PyTorch defaults for a pipeline that was built with
grain.MapDataset, for example by a library function.dsis returned unchanged if it already is agrain.experimental.torch.MapDataset.Example usage:
ds = grain.MapDataset.range(4) # A plain Grain dataset. ds = grain_torch.MapDataset.from_dataset(ds) ds = ds.batch(2) # Batches are `torch.Tensor`s.
- Parameters:
ds (MapDataset[T]) – A
grain.MapDataset.- Returns:
A
grain.experimental.torch.MapDatasetover the same pipeline.- Return type:
MapDataset[T]
- map(transform)[source]#
Returns a dataset containing the elements transformed by
transform.Example usage:
ds = MapDataset.range(5) ds = ds.map(lambda x: x + 10) list(ds) == [10, 11, 12, 13, 14]
- Parameters:
transform (Map | Callable[[T], S]) – Either a
MapTransformcontaining themapmethod or a callable that takes an element and returns a new element.- Returns:
A dataset containing the elements of the original dataset transformed by
transform.- Return type:
MapDataset[S]
- map_with_index(transform)[source]#
Returns a dataset containing the elements transformed by
transform.The transform is called with the index of the element within the dataset and the element itself.
Example usage:
ds = MapDataset.source(["a", "b", "c", "d"]) ds = ds.map_with_index(lambda i, x: x + str(i)) list(ds) == ["a0", "b1", "c2", "d3"]
- Parameters:
transform (MapWithIndex | Callable[[int, T], S]) – Either a
MapWithIndexTransformcontaining themap_with_indexmethod or a callable that takes an index and an element and returns a new element.- Returns:
A dataset containing the elements of the original dataset transformed by
transform.- Return type:
MapDataset[S]
- property parents: Sequence[MapDataset]#
- pipe(func, /, *args, **kwargs)[source]#
Returns
func(self, *args, **kwargs).Same as
grain.MapDataset.pipe, except that agrain.MapDatasetorgrain.IterDatasetreturned byfuncis returned as agrain.experimental.torchdataset. This way, plain Grain dataset classes can be used in a PyTorch pipeline. Such a class is applied to thegrain.MapDatasetthat this dataset wraps.Example usage:
ds = grain_torch.MapDataset.range(8) ds = ds.pipe( grain.experimental.WindowShuffleMapDataset, window_size=4, seed=0 ) ds = ds.batch(2) # Batches are `torch.Tensor`s.
- Parameters:
func (Callable[[...], Any]) – A callable that takes this dataset as its first argument.
*args – Positional arguments to pass to
funcafter the dataset.**kwargs – Keyword arguments to pass to
func.
- Returns:
The result of
func. Agrain.MapDatasetorgrain.IterDatasetis returned as agrain.experimental.torchdataset.- Return type:
Any
- random_map(transform, *, seed=None)[source]#
Not supported; use
stateless_random_mapinstead.grain.MapDataset.random_mappasses a NumPy generator totransform, which makes it easy to mix NumPy arrays into a PyTorch pipeline.stateless_random_mappasses a PyTorch PRNG key instead.- Parameters:
transform (RandomMap | Callable[[...], Any]) – Unused.
seed (int | None) – Unused.
- Raises:
NotImplementedError – Always.
- Return type:
NoReturn
- repeat(num_epochs=None, *, reseed_each_epoch=True)[source]#
Returns a dataset repeating the elements of this dataset multiple times.
Specifying
Nonefornum_epochswill repeat the dataset infinitely, and causeslen(ds)to returnsys.maxsize.Can not be called on an infinite dataset.
By default, random upstream transformations such as
shuffleandrandom_mapwill be seeded differently (but deterministically) for each epoch. This means that each epoch will be shuffled differently. If you want to repeat the first epoch exactly as is, setreseed_each_epochtoFalse.Implementation note: since
MapDatasetallows accessing elements pastlen(ds) - 1anyway (and uses the index modulolen(ds)), this transformation effectively only changes the length of the dataset.Example usage:
list(MapDataset.range(5).repeat(2)) == [0, 1, 2, 3, 4, 0, 1, 2, 3, 4] ds = MapDataset.range(5).repeat() len(ds) == sys.maxsize ds[11111] == 1
- Parameters:
num_epochs (int | None) – Either a positive integer representing the number of times this dataset should be repeated or
Noneto repeat infinitely.reseed_each_epoch (bool) – If True (default), all random upstream transformations use a seed folded with the epoch number, so that each epoch is shuffled and processed differently. If False, the first epoch is repeated exactly as is. In both cases, the random transformations are deterministic.
- Returns:
A dataset repeating the elements of this dataset multiple times.
- Return type:
MapDataset[T]
- seed(seed)[source]#
Returns a dataset that uses the seed for default seed generation.
When default seed generation is enabled by calling
ds.seed, every downstream random transformation will be automatically seeded with a unique seed by default. This simplifies seed management, making it easier to avoid:Having to provide a seed in multiple transformations.
Accidentally reusing the same seed across transformations.
It is recommended to call this right after the source.
ds.seedhas to be called before any random transformations (such asshuffleorrandom_mapthat rely on default seed generation to control their seeding). Given the same seed, the pipeline is guaranteed to always use the same seeds for each transformation.WARNING: The seed for random downstream transformations is derived from the seed passed to
ds.seedand the absolute position of the transformation in the pipeline. This means that if you add transformations before the random transformation, its seed will change. For instance, if this random transformation isshuffle, adding a transformation beforeshufflewill change its seed and data order, consequently. To avoid this, pass the seed to the transformation directly.Note about custom dataset implementations: the default seed generation is available through
_default_seed, but the private API is not guaranteed to be stable.Example 1:
ds = ds.seed(seed).shuffle()
shufflewill automatically derive its own seed (different fromseed).Example 2:
ds = ds.seed(seed).shuffle().random_map(...)
shuffleandrandom_mapwill each derive their own seed and the seeds are going to be different.Example 3:
ds = ds.seed(seed).random_map(transform, seed=seed1)
random_mapwill useseed1and will not be affected byseed. This can be used to control individual transformation seeding independently from the rest of the pipeline.Example 4:
ds = ds.seed(seed1).shuffle().seed(seed2).random_map(...)
ds.seedonly affects the downstream transformations and can be overridden by a subsequentseedcall.shufflewill derive its seed fromseed1,random_map- fromseed2and will not be affected byseed1. This can be used to control your transformation seeding even if you don’t own the first part of the pipeline.Example 5:
ds1 = ds.source(...).seed(seed1).shuffle() ds2 = ds.source(...).seed(seed2).shuffle() ds = MapDataset.mix([ds1, ds2], ...).random_map(...)
Each
shufflewill derive its own seed fromseed1orseed2respectively.random_mapwill derive its seed from bothseed1andseed2.- Parameters:
seed (int) – Seed to use.
- Returns:
A dataset with elements unchanged.
- Return type:
MapDataset[T]
- shuffle(seed=None)[source]#
Returns a dataset with the same elements in a globally shuffled order.
The shuffle is deterministic and will always produce the same result given the same seed. The seed can be either provided explicitly or set via
ds.seed(seed). Prefer the latter if you don’t need to control the shuffle seed individually. It allows to pass a single seed to derive seeds for all downstream random transformations in the pipeline.In multi-epoch training each epoch will be shuffled differently (i.e. the seed is combined with epoch number). In such case it is recommended to
shufflebeforerepeatto avoid mixing elements from different epochs.Example usage:
ds = MapDataset.range(5).shuffle() set(ds) == {0, 1, 2, 3, 4} list(ds) != [0, 1, 2, 3, 4] # With probability (1 - 1/5!).
- Parameters:
seed (int | None) – An optional integer between 0 and 2**32-1 representing the seed used by the shuffling algorithm. If you don’t need to control the shuffle seed individually, prefer setting the pipeline-level seed with
ds.seed(seed)instead.- Returns:
A dataset containing the same elements but in a shuffled order.
- Return type:
MapDataset[T]
- slice(sl, wrap_around=False)[source]#
Returns a dataset containing only the elements with indices in
sl.For most implementations of
MapDatasetslicing is also available through subscript operator:list(ds.slice(slice(1, 10, 2))) == ds[1:10:2].Example usage with a slice:
ds = MapDataset.range(5) list(ds.slice(slice(1, 3))) == [1, 2] list(ds.slice(slice(1, None, 2))) == [1, 3]
Example usage with a sequence of indices:
ds = MapDataset.range(10) list(ds.slice([3, 1, 4, 1, 5])) == [3, 1, 4, 1, 5]
The indices sequence can also be a
MapDatasetof integers:ds = MapDataset.range(10) idx = MapDataset.range(3) list(ds.slice(idx)) == [0, 1, 2]
Commonly used for sharding:
ds = ds.slice(slice(shard_index, None, shard_count)), or, equivalently,ds = ds[shard_index::shard_count].- Parameters:
sl (slice | Sequence[int]) – A
sliceobject (https://docs.python.org/3/library/functions.html#slice) representing the slice of elements to that should constitute the returned dataset, or aSequence[int](includingMapDataset[int]) of indices into the parent dataset.wrap_around (bool) – If True, indices wrap modularly within the slice range. This is needed when transformations that do their own epoch math (like
shuffleormix) sit between the slice and arepeat.
- Returns:
A dataset containing only the elements with indices in
sl.- Return type:
MapDataset[T]
- stateless_random_map(transform, *, seed=None)[source]#
Returns a dataset transformed with a per-element PyTorch PRNG key.
Same as
grain.MapDataset.stateless_random_map, except that a plain callablefn(element, key)is wrapped inTorchStatelessRandomMapinstead ofJaxStatelessRandomMap:keyis a key tensor for the stateless random functions intorch.func._random. AStatelessRandomMapinstance is used as is.Example usage:
import torch import torch.func._random as torch_random from grain.experimental import torch as grain_torch def add_noise(x, key): return x + torch_random.normal(key, x.shape) ds = grain_torch.MapDataset.source([torch.zeros(2)] * 4) ds = ds.seed(42).stateless_random_map(add_noise) # The key depends only on the seed and the element's index, so the # output is the same on every run.
- Parameters:
transform (StatelessRandomMap | Callable[[T, Any], S]) – A callable
fn(element, key)(wrapped inTorchStatelessRandomMap) or aStatelessRandomMapimplementation.seed (int | None) – An optional non-negative 64-bit integer seed. Defaults to the seed derived from
ds.seed(...).
- Returns:
A
MapDatasetcontaining the transformed elements.- Return type:
MapDataset[Any]
- to_iter_dataset(read_options=None, *, allow_nones=False)[source]#
Converts this dataset to an
IterDataset.Elements from this dataset may be processed in multiple threads.
Note that some of the transformations are not available on
IterDataset. These are roughly transformations operating on element index such asshuffle,map_with_index,sliceandrepeat.- Parameters:
read_options (ReadOptions | None) – Controls multithreading when reading the data and applying transformations in this dataset.
allow_nones (bool) – Whether to allow
Nonevalues in the dataset (e.g. produced byfilter). IfFalse(the default),Nonevalues will be filtered out.
- Returns:
An
IterDatasetwith the same non-Noneelements as this dataset.- Return type:
IterDataset[T]
- class grain.experimental.torch.IterDataset(inner)[source]#
Bases:
IterDataset[T]grain.IterDatasetwith PyTorch defaults.Supports the same methods as
grain.IterDataset; see its documentation for the full reference and examples. Only the differences are documented here:stateless_random_mappasses PyTorch PRNG keys to plain callables (TorchStatelessRandomMap).random_mapis not supported, because it passes a NumPy generator to the transformation. Usestateless_random_mapinstead.batchcollates withtorch.utils.data.default_collateby default.mp_prefetchlimits PyTorch threads in worker processes and passes CPU tensors between processes through Grain’s shared memory.with_optionsappliesDatasetOptions.Transformations return
grain.experimental.torch.IterDataset, and iterating returnsgrain.experimental.torch.DatasetIterator.
Each instance wraps a plain
grain.IterDatasetand builds every transformation on it, so the pipeline behaves exactly as ingrain.IterDataset.- Parameters:
inner (dataset.IterDataset[T])
- __init__(inner)[source]#
- Parameters:
inner (IterDataset[T])
- apply(transformations)[source]#
Returns a dataset with the given transformation(s) applied.
Syntactic sugar to avoid dispatch by transformation type.
Example usage:
ds = grain.MapDataset.range(5).to_iter_dataset() ds = ds.apply([AddOne(), grain.transforms.Batch(2)]) list(ds) == [np.ndarray([1, 2]), np.ndarray([3, 4]), np.ndarray([5])]
- Parameters:
transformations (Batch | Map | RandomMap | StatelessRandomMap | TfRandomMap | Filter | FlatMap | MapWithIndex | Sequence[Batch | Map | RandomMap | StatelessRandomMap | TfRandomMap | Filter | FlatMap | MapWithIndex]) – one or more transformations to apply.
- Returns:
Dataset with the given transformations applied.
- Return type:
IterDataset[Any]
- batch(batch_size, *, drop_remainder=False, batch_fn=None)[source]#
Returns a dataset of elements batched along a new first dimension.
Same as
grain.IterDataset.batch, except thatbatch_fndefaults totorch.utils.data.default_collate. Batches are ``torch.Tensor``s; NumPy arrays and Python numbers are converted to tensors too.Example usage:
ds = grain_torch.MapDataset.range(5).to_iter_dataset() ds = ds.batch(batch_size=2) list(ds) # [tensor([0, 1]), tensor([2, 3]), tensor([4])]
- Parameters:
batch_size (int) – The number of elements to batch together.
drop_remainder (bool) – Whether to drop the last batch if it is smaller than batch_size.
batch_fn (Callable[[Sequence[T]], S] | None) – A function that takes a list of elements and returns a batch. Defaults to
torch.utils.data.default_collate.
- Returns:
A dataset of batches. A batch has the PyTree structure of an element, except that
torch.utils.data.default_collatereturns a list for a tuple.- Return type:
IterDataset[Any]
Makes this dataset write its output to shared memory, if it can.
Called by
mp_prefetchon the dataset that it prefetches from. It is not meant to be called directly.- Return type:
None
- filter(transform)[source]#
Returns a dataset containing only the elements that match the filter.
Example usage:
ds = MapDataset.range(5).to_iter_dataset() ds = ds.filter(lambda x: x % 2 == 0) list(ds) == [0, 2, 4]
Produces a warning if the filter removes more than 90% of the elements. You can adjust the threshold through
grain.experimental.DatasetOptions.filter_warn_threshold_ratioused inWithOptionsIterDataset. In order to produce an exception in such case usefilter_raise_threshold_ratio.- Parameters:
transform (Filter | Callable[[T], bool]) – Either a
FilterTransformcontaining thefiltermethod or a callable that takes an element and returns a boolean.- Returns:
A dataset of the same type containing only the elements for which the filter transform returns
True.- Return type:
IterDataset[T]
- classmethod from_dataset(ds)[source]#
Returns
dsas agrain.experimental.torch.IterDataset.Use it to get the PyTorch defaults for a pipeline that was built with
grain.IterDataset, for example by a library function.dsis returned unchanged if it already is agrain.experimental.torch.IterDataset.Example usage:
ds = grain.MapDataset.range(4).to_iter_dataset() # A plain Grain dataset. ds = grain_torch.IterDataset.from_dataset(ds) ds = ds.batch(2) # Batches are `torch.Tensor`s.
- Parameters:
ds (IterDataset[T]) – A
grain.IterDataset.- Returns:
A
grain.experimental.torch.IterDatasetover the same pipeline.- Return type:
IterDataset[T]
- property has_child: bool#
- map(transform)[source]#
Returns a dataset containing the elements transformed by
transform.Example usage:
ds = MapDataset.range(5).to_iter_dataset() ds = ds.map(lambda x: x + 10) list(ds) == [10, 11, 12, 13, 14]
- Parameters:
transform (Map | Callable[[T], S]) – Either a
MapTransformcontaining themapmethod or a callable that takes an element and returns a new element.- Returns:
A dataset containing the elements of the original dataset transformed by
transform.- Return type:
IterDataset[S]
- map_with_index(transform)[source]#
Returns a dataset of the elements transformed by the
transform.The
transformis called with the index of the element in the dataset as the first argument and the element as the second argument.Example usage:
ds = MapDataset.range(5).to_iter_dataset() ds = ds.map(lambda i, x: (i, 2**x)) list(ds) == [(0, 1), (1, 2), (2, 4), (3, 8), (4, 16)]
- Parameters:
transform (MapWithIndex | Callable[[int, T], S]) – Either a
MapWithIndexTransformcontaining themap_with_indexmethod or a callable that takes an index and an element and returns a new element.- Returns:
A dataset containing the elements of the original dataset transformed by
transform.- Return type:
IterDataset[S]
- mp_prefetch(options=None, worker_init_fn=None, sequential_slice=False)[source]#
Returns a dataset prefetching elements in multiple processes.
Same as
grain.IterDataset.mp_prefetch; see its documentation for how workers slice the dataset and for cleanup recommendations. There are two PyTorch-specific differences:Each worker process is limited to
options.worker_intra_op_threadsPyTorch intra-op threads (1 by default), liketorch.utils.data.DataLoaderworkers. The limit is applied beforeworker_init_fnruns. To use another thread count, setworker_intra_op_threadsrather than callingtorch.set_num_threadsinworker_init_fn, which some PyTorch builds ignore after the first call.CPU tensors are passed from the workers through Grain’s shared memory, so
DatasetOptions.min_shm_sizeapplies to them as it does to NumPy arrays.
Example usage:
ds = grain_torch.MapDataset.source([torch.zeros(1024)] * 8).batch(4) ds = ds.to_iter_dataset().mp_prefetch( grain_torch.MultiprocessingOptions(num_workers=2) ) it = iter(ds) next(it).shape # torch.Size([4, 1024]) it.close()
- Parameters:
options (MultiprocessingOptions | None) – Options for the prefetching processes, either
grain.experimental.torch.MultiprocessingOptionsorgrain.MultiprocessingOptions. Ifoptions.num_workersis 0,mp_prefetchhas no effect. Defaults to 10 workers with one intra-op thread each.worker_init_fn (Callable[[int, int], None] | None) – A function that is called in each worker process before the data is processed. The function takes two arguments: the current worker index and the total worker count.
sequential_slice (bool) – Whether the slice for each worker should be sequential (consecutive at the source dataset). Experimental.
- Returns:
A dataset prefetching input elements in separate processes.
- Return type:
IterDataset[T]
- property parents: Sequence[MapDataset | IterDataset]#
- pipe(func, /, *args, **kwargs)[source]#
Returns
func(self, *args, **kwargs).Same as
grain.IterDataset.pipe, except that agrain.MapDatasetorgrain.IterDatasetreturned byfuncis returned as agrain.experimental.torchdataset. This way, plain Grain dataset classes can be used in a PyTorch pipeline. Such a class is applied to thegrain.IterDatasetthat this dataset wraps.Example usage:
ds = grain_torch.MapDataset.range(8).to_iter_dataset() ds = ds.pipe( grain.experimental.WindowShuffleIterDataset, window_size=4, seed=0 ) ds = ds.batch(2) # Batches are `torch.Tensor`s.
- Parameters:
func (Callable[[...], Any]) – A callable that takes this dataset as its first argument.
*args – Positional arguments to pass to
funcafter the dataset.**kwargs – Keyword arguments to pass to
func.
- Returns:
The result of
func. Agrain.MapDatasetorgrain.IterDatasetis returned as agrain.experimental.torchdataset.- Return type:
Any
- prefetch(multiprocessing_options)[source]#
Deprecated, use
mp_prefetchinstead.Returns a dataset prefetching the elements in multiple processes.
Each of the processes will process a slice of the dataset after all
MapDatasettransformations.WARNING: If the dataset contains many-to-one transformations (such as
batch), output of prefetch may change if you change the number of workers. However, it is still going to be determisitic.- Parameters:
multiprocessing_options (MultiprocessingOptions) – options for the prefetching processes.
num_workersmust be greater than 0.- Returns:
A dataset prefetching input elements concurrently.
- Return type:
IterDataset[T]
- random_map(transform, *, seed=None)[source]#
Not supported; use
stateless_random_mapinstead.grain.IterDataset.random_mappasses a NumPy generator totransform, which makes it easy to mix NumPy arrays into a PyTorch pipeline.stateless_random_mappasses a PyTorch PRNG key instead.- Parameters:
transform (RandomMap | Callable[[...], Any]) – Unused.
seed (int | None) – Unused.
- Raises:
NotImplementedError – Always.
- Return type:
NoReturn
- seed(seed)[source]#
Returns a dataset that uses the seed for default seed generation.
When default seed generation is enabled by calling
ds.seed, every downstream random transformation will be automatically seeded with a unique seed by default. This simplifies seed management, making it easier to avoid:Having to provide a seed in multiple transformations.
Accidentally reusing the same seed across transformations.
It is recommended to call this right after the source.
ds.seedhas to be called before any random transformations (such asrandom_mapthat rely on default seed generation to control their seeding). Given the same seed, the pipeline is guaranteed to always use the same seeds for each transformation.Note about custom dataset implementations: the default seed generation is available through
_default_seed, but the private API is not guaranteed to be stable.Example 1:
ds = ds.seed(seed).random_map(...)
random_mapwill automatically derive its own seed (different fromseed).Example 2:
ds = ds.seed(seed).random_map().random_map(...)
The first and second
random_maps will each derive their own seed and the seeds are going to be different.Example 3:
ds = ds.seed(seed).random_map(transform, seed=seed1)
random_mapwill useseed1and will not be affected byseed. This can be used to control individual transformation seeding independently from the rest of the pipeline.Example 4:
ds = ds.seed(seed1).random_map(...).seed(seed2).random_map(...)
ds.seedonly affects the downstream transformations and can be overridden by a subsequentseedcall. The firstrandom_mapwill derive its seed fromseed1, the second - fromseed2and will not be affected byseed1. This can be used to control your transformation seeding even if you don’t own the first part of the pipeline.Example 5:
ds1 = ds.source(...).seed(seed2).shuffle().to_iter_dataset() ds2 = ds.source(...).seed(seed2).shuffle().to_iter_dataset() ds = IterDataset.mix([ds1, ds2], ...).random_map(...)
Each
shufflewill derive its own seed fromseed1orseed2respectively.random_mapwill derive its seed from bothseed1andseed2.- Parameters:
seed (int) – Seed to use.
- Returns:
A dataset with elements unchanged.
- Return type:
IterDataset[T]
- set_slice(sl, sequential_slice=False)[source]#
Restricts this dataset, in place, to a slice of its elements.
Called by
mp_prefetchon each worker’s copy of the dataset to give the worker its share of the data. It is not meant to be called directly.- Parameters:
sl (slice) – The slice of elements to keep.
sequential_slice (bool) – Whether the slice is a contiguous range of the source rather than every n-th element.
- Return type:
None
- stateless_random_map(transform, *, seed=None)[source]#
Returns a dataset transformed with a per-element PyTorch PRNG key.
Same as
grain.IterDataset.stateless_random_map, except that a plain callablefn(element, key)is wrapped inTorchStatelessRandomMapinstead ofJaxStatelessRandomMap:keyis a key tensor for the stateless random functions intorch.func._random. AStatelessRandomMapinstance is used as is.Example usage:
import torch import torch.func._random as torch_random from grain.experimental import torch as grain_torch def add_noise(x, key): return x + torch_random.normal(key, x.shape) ds = grain_torch.MapDataset.source([torch.zeros(2)] * 4).to_iter_dataset() ds = ds.seed(42).stateless_random_map(add_noise) # The key depends only on the seed and the element's index, so the # output is the same on every run.
- Parameters:
transform (StatelessRandomMap | Callable[[T, Any], S]) – A callable
fn(element, key)(wrapped inTorchStatelessRandomMap) or aStatelessRandomMapimplementation.seed (int | None) – An optional non-negative 64-bit integer seed. Defaults to the seed derived from
ds.seed(...).
- Returns:
An
IterDatasetcontaining the transformed elements.- Return type:
IterDataset[Any]
- with_options(options)[source]#
Returns a dataset with
optionsapplied to the pipeline.Method form of
grain.experimental.WithOptionsIterDatasetthat keeps the result agrain.experimental.torch.IterDataset. The options apply to all transformations in the pipeline, before and after this call. Options set by a later call take precedence.Example usage:
ds = ds.to_iter_dataset() # Tensors smaller than 64 KiB are copied between processes instead of # being placed in shared memory. ds = ds.with_options(grain_torch.DatasetOptions(min_shm_size=64 << 10)) ds = ds.mp_prefetch(grain_torch.MultiprocessingOptions(num_workers=4))
- Parameters:
options (DatasetOptions) – The options to apply.
- Returns:
An
IterDatasetwith the options applied.- Return type:
IterDataset[T]
- class grain.experimental.torch.DatasetIterator(parent)[source]#
Bases:
DatasetIterator[T]grain.DatasetIteratorthat supports PyTorch checkpointing.Supports the same methods as
grain.DatasetIterator; see its documentation for the full reference. Only the differences are documented here:state_dictandload_state_dictmake the iterator atorch.distributed.checkpointStatefulobject, so it can be passed todcp.saveanddcp.loadnext to the model and the optimizer. The state is stored per rank, so every rank restores its own position.
Example usage:
import torch.distributed.checkpoint as dcp it = iter(ds) # `ds` is a `grain.experimental.torch.IterDataset`. ... dcp.save({"model": model, "data": it}, checkpoint_id=path) # To resume, rebuild `ds` and restore the position of a new iterator. it = iter(ds) dcp.load({"model": model, "data": it}, checkpoint_id=path)
state_dictandload_state_dictuse the default process group. If the ranks that read different data form another group, such as the data-parallel group under tensor parallelism, wrap the iterator inIteratorCheckpointWrapperto pass that group:data = grain_torch.IteratorCheckpointWrapper(it, group=dp_group) dcp.save({"model": model, "data": data}, checkpoint_id=path)
Like any
grain.DatasetIterator, this iterator closes the iterator that it wraps when it is closed or garbage-collected. To checkpoint an iterator of agrain.MapDatasetorgrain.IterDataset, wrap it inIteratorCheckpointWrapperrather than in this class.- Parameters:
parent (dataset.DatasetIterator[T])
- __init__(parent)[source]#
Initializes a DatasetIterator.
- Parameters:
parents – The parent iterator or sequence of parent iterators.
parent (DatasetIterator[T])
- close()[source]#
Closes the iterator and releases resources.
This method is idempotent and safe to call multiple times. After calling
close, the iterator will raiseValueErroron any subsequent calls to__next__.Recursively closes all parent iterators in the pipeline to ensure complete resource cleanup.
NOTE: Resource cleanup occurs via
__del__when the iterator is garbage collected, but since garbage collection is not guaranteed in CPython, this method can be called explicitly to ensure explicit blocking cleanup.- Return type:
None
- get_state()[source]#
Returns the current state of the iterator.
We reserve the right to evolve the state format over time. The states returned from this method are only guaranteed to be restorable by the same version of the code that produced them.
Implementation Note: It is recommended that iterator implementations always produce states with the same shapes and types throughout the lifetime of the iterator. Some frameworks rely on this property to perform checkpointing, and all standard library iterators are compliant. It is also recommended to produce state values that support shapes and types, e.g. using
np.int64instead ofint. The standard library iterators are not currently compliant with this recommendation.Users can also use
grain.checkpoint.CheckpointSave(which calls this function) to save the model and the iterator in a bundle when using Orbax.- Return type:
dict[str, Any]
- async load(directory)[source]#
Loads the iterator state from a directory.
The state may be loaded and set in a background thread. The main thread should not alter the state content while the load is in progress.
- Parameters:
directory (Path) – The directory to load the state from.
- Returns:
A coroutine that has not been awaited. This is called by Orbax in a background thread to perform I/O without blocking the main thread.
- Return type:
Awaitable[None]
- load_state_dict(state_dict)[source]#
Restores iterator state from a
dcp.loaddictionary.- Parameters:
state_dict (Mapping[str, Any])
- Return type:
None
- async save(directory)[source]#
Saves the iterator state to a directory.
The current state (get_state) is used for saving, so any updates to the state after returning from this method will not affect the saved checkpoint.
- Parameters:
directory (PathAwaitingCreation) – A path in the process of being created. Must call await_creation before accessing the physical path.
- Returns:
A coroutine that has not been awaited. This is called by Orbax in a background thread to perform I/O without blocking the main thread.
- Return type:
Awaitable[None]
- set_state(state)[source]#
Sets the current state of the iterator.
Restores the internal execution state of this iterator and its parent iterators in-place, allowing iteration to resume from the exact step where
get_state()was called without re-creating the iterator.Users can also use
grain.checkpoint.CheckpointRestore(which calls this function) to restore the model and the iterator in a bundle when using Orbax.- Parameters:
state (dict[str, Any]) – A dictionary containing the serialized iterator state previously produced by
get_state().- Return type:
None
- start_prefetch()[source]#
Starts processing elements in the first asynchronous parent iterator.
Useful when the iterator can be created in advance but the elements are not needed immediately. For instance, when recovering iterator and model from a checkpoint, recover the iterator first, call
start_prefechand then recover the model. This way the time to get the first batch from the iterator will be partially or fully hidden behind the time it takes to recover the model.This method is idempotent and safe to call multiple times.
- Return type:
None
- class grain.experimental.torch.TorchStatelessRandomMap(fn=None)[source]#
Stateless PyTorch random map implementing
StatelessRandomMap.Can be used in two ways:
Wrapping a callable
fn(element, key):TorchStatelessRandomMap(fn)(used automatically when passing a plain callable tograin.experimental.torch.MapDataset.stateless_random_map(fn)orIterDataset.stateless_random_map(fn)).Subclassing and overriding
stateless_random_map(self, element, key).
Each element receives an immutable CPU
uint64[2]Philox PRNG key tensor derived viatorch.func._random.key(seed)andtorch.func._random.fold_in(op_key, index). Passkeyto stateless random operations intorch.func._random(normal,uniform,randint,bits) and usetorch.func._random.split(key)when drawing multiple independent random values within one transform.Each element’s key depends only on the dataset seed and the element’s index, so results are deterministic under random access, prefetch threads, worker processes, and checkpoint restores. The global PyTorch RNG is neither used nor modified.
Experimental: both this class and PyTorch’s
torch.func._randomstateless PRNG module are experimental, and their APIs or random streams may change as PyTorch stabilizes public stateless RNG support.Example
Adding Gaussian noise drawn from each element’s stateless PRNG key:
import torch import torch.func._random as torch_random from grain.experimental import torch as grain_torch def add_noise(x, key): return x + torch_random.normal(key, x.shape) # `grain.experimental.torch` datasets wrap plain callables in # `TorchStatelessRandomMap` automatically. ds = ( grain_torch.MapDataset.source([torch.zeros(2)] * 4) .stateless_random_map(add_noise, seed=42) .batch(2) ) # The same index always produces the same random values. print(torch.equal(ds[1], ds[1])) # True
grain.MapDatasetandgrain.IterDatasetwrap plain callables inJaxStatelessRandomMap, so wrap the callable explicitly there:import grain ds = grain.MapDataset.source([torch.zeros(2)] * 4).stateless_random_map( grain_torch.TorchStatelessRandomMap(add_noise), seed=42 )
- Parameters:
fn (Callable[[Any, Any], Any] | None)
- class grain.experimental.torch.MultiprocessingOptions(num_workers=0, per_worker_buffer_size=1, enable_profiling=False, worker_intra_op_threads=1)[source]#
Options for Python multiprocessing in PyTorch pipelines.
Extends
grain.MultiprocessingOptionswithworker_intra_op_threads(default1) so PyTorch workers do not oversubscribe host CPU cores by default, liketorch.utils.data.DataLoaderworkers.- Parameters:
num_workers (int)
per_worker_buffer_size (int)
enable_profiling (bool)
worker_intra_op_threads (int | None)
- worker_intra_op_threads#
Number of PyTorch intra-op threads that each worker process uses. Grain applies it with
torch.set_num_threads(andOMP_NUM_THREADS/MKL_NUM_THREADS) beforeworker_init_fnruns. To use another count, set this field rather than callingtorch.set_num_threadsinworker_init_fn: some PyTorch builds only honor the first call. IfNone, Grain does not set a thread count and each worker uses PyTorch’s own default:OMP_NUM_THREADS/MKL_NUM_THREADSwhen they are set in the environment, otherwise typically one thread per CPU core.- Type:
int | None
- grain.experimental.torch.mp_prefetch(ds: IterDataset[T], options: MultiprocessingOptions | None = None, worker_init_fn: Callable[[int, int], None] | None = None, sequential_slice: bool = False) IterDataset[T][source]#
- grain.experimental.torch.mp_prefetch(ds: IterDataset[T], options: MultiprocessingOptions | None = None, worker_init_fn: Callable[[int, int], None] | None = None, sequential_slice: bool = False) IterDataset[T]
Returns a dataset prefetching the elements of
dsin multiple processes.The function form of
grain.experimental.torch.IterDataset.mp_prefetch; see its documentation. It can also be applied to agrain.IterDataset, for example withgrain.IterDataset.pipe, and returns the same kind of dataset asds.- Parameters:
ds (IterDataset[T]) – The dataset to prefetch from: a
grain.experimental.torch.IterDatasetor agrain.IterDataset.options (MultiprocessingOptions | None) – Options for the prefetching processes, either
grain.experimental.torch.MultiprocessingOptionsorgrain.MultiprocessingOptions. Defaults to 10 workers with one intra-op thread each.worker_init_fn (Callable[[int, int], None] | None) – A function that is called in each worker process before the data is processed. The function takes two arguments: the current worker index and the total worker count.
sequential_slice (bool) – Whether the slice for each worker should be sequential (consecutive at the source dataset). Experimental.
- Returns:
a
grain.experimental.torch.IterDatasetifdsis one, and agrain.IterDatasetotherwise.- Return type:
A dataset prefetching input elements in separate processes
- Raises:
TypeError – If
dsis not agrain.IterDataset, for example if it is aMapDataset.
- grain.experimental.torch.get_process_index_and_count(group=None)[source]#
Returns
(process_index, process_count)for PyTorch distributed jobs.Resolution order:
If
torch.distributedis available and initialized, queriesdist.get_rank(group)anddist.get_world_size(group). RaisesValueErrorif the current process is not a member ofgroup.If
groupwas explicitly provided whentorch.distributedis not initialized, raisesRuntimeError.Falls back to the
RANKandWORLD_SIZEenvironment variables (fortorchrunbeforeinit_process_group). RaisesValueErrorif both are set but invalid. If only one of them is set, warns and ignores it.Otherwise returns
(0, 1).
Never imports JAX, avoiding accidental PJRT initialization in PyTorch jobs.
Example
Sharding a dataset across
torch.distributedranks with a slice:rank, world_size = grain.experimental.torch.get_process_index_and_count() ds = ds.shuffle().repeat(num_epochs)[rank::world_size]
- Parameters:
group (Any) – Optional
torch.distributed.ProcessGroup(e.g., DP group under 2D/3D parallelism). Defaults to the default process group.- Returns:
Tuple
(process_index, process_count).- Raises:
ValueError – If the current process is not a member of
group, or if theRANKandWORLD_SIZEenvironment variables are consulted and both are set but are not integers with0 <= RANK < WORLD_SIZE.RuntimeError – If
groupis specified whentorch.distributedis not initialized.
- Return type:
tuple[int, int]
- class grain.experimental.torch.IteratorCheckpointWrapper(iterator, *, group=None)[source]#
Wraps a Grain iterator to implement DCP’s
Statefulprotocol.Implements
torch.distributed.checkpoint.stateful.Stateful(state_dict/load_state_dict) on top of Grain’sget_state/set_state.torch.distributed.checkpoint(DCP) keeps a single copy of each non-tensor state-dict key across ranks on save, so returningget_state()directly fromstate_dict()would make every rank restore one rank’s progress.IteratorCheckpointWrapperstores each rank’s serialized state underrank_{i}(scoped togroupwhen DP is a subgroup of the global world, such as under tensor parallelism).Saving from more than one rank with DCP requires PyTorch 2.4 or later. Earlier releases write non-tensor values only from the coordinator rank, so the state of the other ranks is not saved and cannot be restored.
Iterators of
grain.experimental.torchdatasets (grain.experimental.torch.DatasetIterator) implementStatefulon top of this class, so they can be passed todcp.saveanddcp.loaddirectly. UseIteratorCheckpointWrapperexplicitly in two cases:To pass
group. The iterators above always use the default process group.To checkpoint an iterator of a
grain.MapDatasetorgrain.IterDataset, which does not implementStateful.
IteratorCheckpointWrapperholds only a reference to the iterator:state_dict()reads the iterator’s state when it is called, and nothing is cached. Create it at each save or load call, as in the example below, so that it always wraps the iterator in use. An instance that is kept around still wraps the old iterator after the iterator is replaced.Example
An iterator of a
grain.experimental.torchdataset needs no wrapper:import torch.distributed.checkpoint as dcp from grain.experimental import torch as grain_torch it = iter(grain_torch.MapDataset.range(8).to_iter_dataset()) dcp.save({"data": it}, checkpoint_id=path) dcp.load({"data": it}, checkpoint_id=path)
A process group other than the default one (here
dp_group) is passed through the wrapper:dcp.save( {"data": grain_torch.IteratorCheckpointWrapper(it, group=dp_group)}, checkpoint_id=path, ) dcp.load( {"data": grain_torch.IteratorCheckpointWrapper(it, group=dp_group)}, checkpoint_id=path, )
An iterator of a
grain.MapDatasetorgrain.IterDatasetalways needs the wrapper, with or withoutgroup:plain_it = iter(grain.MapDataset.range(8).to_iter_dataset()) dcp.save( {"data": grain_torch.IteratorCheckpointWrapper(plain_it)}, checkpoint_id=path, ) dcp.load( {"data": grain_torch.IteratorCheckpointWrapper(plain_it)}, checkpoint_id=path, )
- Parameters:
iterator (Any)
group (Any)