scaffold.data.iterstream.iterators
==================================

.. py:module:: scaffold.data.iterstream.iterators


Functions
---------

.. autoapisummary::

   scaffold.data.iterstream.iterators.batched_
   scaffold.data.iterstream.iterators.filter_
   scaffold.data.iterstream.iterators.flatten_
   scaffold.data.iterstream.iterators.get_random_range
   scaffold.data.iterstream.iterators.getsize
   scaffold.data.iterstream.iterators.map_
   scaffold.data.iterstream.iterators.shuffle_
   scaffold.data.iterstream.iterators.take_
   scaffold.data.iterstream.iterators.tqdm_


Module Contents
---------------

.. py:function:: batched_(iterable: Iterable, batchsize: int = 20, collation_fn: Optional[Callable] = None, drop_last_if_not_full: bool = True) -> Iterator[List]

   Yield batches of the given size.

   :param iterable: Iterable to be batched.
   :type iterable: Iterable
   :param batchsize: Target batch size. Defaults to 20.
   :type batchsize: int, optional
   :param collation_fn: Collation function. Defaults to None.
   :type collation_fn: Callable, optional
   :param drop_last_if_not_full: If the length of the last batch is less than `batchsize`, drop it.
                                 Defaults to True.
   :type drop_last_if_not_full: bool, optional

   :Yields: Batches (i.e. lists) of samples.


.. py:function:: filter_(iterable: Iterable, predicate: Callable) -> Iterator

   Filter items in the `iterable` by the `predicate` callable.


.. py:function:: flatten_(iterables: Iterable[Iterable]) -> Iterator

   Iterate over iterables in the stream and yield their items.


.. py:function:: get_random_range(rng: Optional[random.Random] = None, seed: scaffold.data.constants.SeedType = None) -> random.Random

   Returns a random number as range, calculated based on the input `rng` and `seed`.

   :param rng: Either `random` module or a :py:class:`random.Random` instance. If None,
               a `random.Random()` is used.
   :type rng: random.Random, optional
   :param seed: seed (Optional[int]): An int or other acceptable types
                that works for random.seed(). Will be used to seed `rng`. If None, a unique identifier will be used to seed.
   :type seed: Union[int, float, str, bytes, bytearray, None]


.. py:function:: getsize(item: Any) -> int

   Return estimated size (in terms of bytes) of a python object. Currently use numpy method to calculate size.
   Otherwise, the size is estimated through its pickled size. This is considered a better performant option than e.g.
   `zarr.storage.getsize()`.


.. py:function:: map_(iterable: Iterable, callback: Callable) -> Iterator

   Apply the `callback` to each item in the `iterable` and yield the item.


.. py:function:: shuffle_(iterable: Iterable, bufsize: int = 1000, initial: int = 100, rng: Optional[random.Random] = None, seed: scaffold.data.constants.SeedType = None) -> Iterator

   Shuffle the data in the stream.

   Uses a buffer of size `bufsize`. Shuffling at startup is less random; this is traded off against yielding samples
   quickly.

   :param iterable: Iterable to shuffle.
   :type iterable: Iterable
   :param bufsize: Buffer size for shuffling. Defaults to 1000.
   :type bufsize: int, optional
   :param initial: Minimum number of elements in the buffer before yielding the first element. Must be
                   less than or equal to `bufsize`, otherwise will be set to `bufsize`. Defaults to 100.
   :type initial: int, optional
   :param rng: Either `random` module or a :py:class:`random.Random` instance. If None,
               a `random.Random()` is used.
   :type rng: random.Random, optional
   :param seed: A data input that can be used for `random.seed()`.
   :type seed: Union[int, float, str, bytes, bytearray, None]

   :Yields: *Any* -- Shuffled items of `iterable`.


.. py:function:: take_(iterable: Iterable, n: int) -> Iterator

   Yield the first n elements from the iterable.

   :param iterable: Iterable to take from.
   :type iterable: Iterable
   :param n: Number of samples to take.
   :type n: int

   :Yields: *Any* --

            First `n` elements of `iterable`. Less elements can be yielded if the iterable does not have enough
                elements.


.. py:function:: tqdm_(iterable: Iterable, **kwargs) -> Iterator

   Iterate while using tqdm.