scaffold.data.iterstream.source

Classes

IterableSamplerSource	A class that samples from iterables into an iterstream.
IterableSource	A class that turns an iterable to a source of a stream and provides stream manipulation functionalities on top,

Module Contents

class scaffold.data.iterstream.source.IterableSamplerSource(iterables: List[Iterable], probs: List[float] | None = None, rng: random.Random | None = None, seed: int | None = None)

Bases: scaffold.data.iterstream.base.Composable

A class that samples from iterables into an iterstream.

Initialize IterableSamplerSource.

Parameters:

• iterables (List[Iterable]) – List of iterables to sample from.

• probs (Optional[List[float]], optional) – [description]. Defaults to None.

• rng (random.Random, optional) – Random number generator to use.

• seed (Optional[int]) – An int or other acceptable types that works for random.seed(). Will be used to seed rng. If None, a unique identifier will be used to seed.

__iter__() → Iterator

Samples items from the iterables, returns all samples until all iterables are exhausted.

iterables

probs = None

rng = None

class scaffold.data.iterstream.source.IterableSource(source: Iterable | Callable | None = ())

Bases: scaffold.data.iterstream.base.Composable

A class that turns an iterable to a source of a stream and provides stream manipulation functionalities on top, for instance: - map - map_async - filter - batched - shuffle - and more

For the detailed description of each, please refer to the corresponding docstring in Composable.

Initialize IterableSource.

Parameters:

source (Union[Iterable, Callable], Optional) – An Iterable that the IterableSource is built based on, or a callable that generates items when called.

__iter__() → Iterator

Iterates over the items in the iterable