scaffold.data.iterstream.iterators

Functions

batched_(→ Iterator[List])

Yield batches of the given size.

filter_(→ Iterator)

Filter items in the iterable by the predicate callable.

flatten_(→ Iterator)

Iterate over iterables in the stream and yield their items.

get_random_range(→ random.Random)

Returns a random number as range, calculated based on the input rng and seed.

getsize(→ int)

Return estimated size (in terms of bytes) of a python object. Currently use numpy method to calculate size.

map_(→ Iterator)

Apply the callback to each item in the iterable and yield the item.

shuffle_(→ Iterator)

Shuffle the data in the stream.

take_(→ Iterator)

Yield the first n elements from the iterable.

tqdm_(→ Iterator)

Iterate while using tqdm.

Module Contents

scaffold.data.iterstream.iterators.batched_(iterable: Iterable, batchsize: int = 20, collation_fn: Callable | None = None, drop_last_if_not_full: bool = True) → Iterator[List]

Yield batches of the given size.

Parameters:

  • iterable (Iterable) – Iterable to be batched.

  • batchsize (int, optional) – Target batch size. Defaults to 20.

  • collation_fn (Callable, optional) – Collation function. Defaults to None.

  • drop_last_if_not_full (bool, optional) – If the length of the last batch is less than batchsize, drop it. Defaults to True.

Yields:

Batches (i.e. lists) of samples.

scaffold.data.iterstream.iterators.filter_(iterable: Iterable, predicate: Callable) → Iterator

Filter items in the iterable by the predicate callable.

scaffold.data.iterstream.iterators.flatten_(iterables: Iterable[Iterable]) → Iterator

Iterate over iterables in the stream and yield their items.

scaffold.data.iterstream.iterators.get_random_range(rng: random.Random | None = None, seed: scaffold.data.constants.SeedType = None) → random.Random

Returns a random number as range, calculated based on the input rng and seed.

Parameters:

  • rng (random.Random, optional) – Either random module or a random.Random instance. If None, a random.Random() is used.

  • seed (Union[int, float, str, bytes, bytearray, None]) – seed (Optional[int]): An int or other acceptable types that works for random.seed(). Will be used to seed rng. If None, a unique identifier will be used to seed.

scaffold.data.iterstream.iterators.getsize(item: Any) → int

Return estimated size (in terms of bytes) of a python object. Currently use numpy method to calculate size. Otherwise, the size is estimated through its pickled size. This is considered a better performant option than e.g. zarr.storage.getsize().

scaffold.data.iterstream.iterators.map_(iterable: Iterable, callback: Callable) → Iterator

Apply the callback to each item in the iterable and yield the item.

scaffold.data.iterstream.iterators.shuffle_(iterable: Iterable, bufsize: int = 1000, initial: int = 100, rng: random.Random | None = None, seed: scaffold.data.constants.SeedType = None) → Iterator

Shuffle the data in the stream.

Uses a buffer of size bufsize. Shuffling at startup is less random; this is traded off against yielding samples quickly.

Parameters:

  • iterable (Iterable) – Iterable to shuffle.

  • bufsize (int, optional) – Buffer size for shuffling. Defaults to 1000.

  • initial (int, optional) – Minimum number of elements in the buffer before yielding the first element. Must be less than or equal to bufsize, otherwise will be set to bufsize. Defaults to 100.

  • rng (random.Random, optional) – Either random module or a random.Random instance. If None, a random.Random() is used.

  • seed (Union[int, float, str, bytes, bytearray, None]) – A data input that can be used for random.seed().

Yields:

Any – Shuffled items of iterable.

scaffold.data.iterstream.iterators.take_(iterable: Iterable, n: int) → Iterator

Yield the first n elements from the iterable.

Parameters:

  • iterable (Iterable) – Iterable to take from.

  • n (int) – Number of samples to take.

Yields:

Any –

First n elements of iterable. Less elements can be yielded if the iterable does not have enough

elements.

scaffold.data.iterstream.iterators.tqdm_(iterable: Iterable, **kwargs) → Iterator

Iterate while using tqdm.