scaffold.torch.distributed.ddp
==============================

.. py:module:: scaffold.torch.distributed.ddp


Functions
---------

.. autoapisummary::

   scaffold.torch.distributed.ddp.check_distributed_setup
   scaffold.torch.distributed.ddp.is_distributed
   scaffold.torch.distributed.ddp.is_same_accross_workers
   scaffold.torch.distributed.ddp.model_to_ddp
   scaffold.torch.distributed.ddp.should_distribute


Module Contents
---------------

.. py:function:: check_distributed_setup() -> bool

   Tests distributed setup with a manual calculation.


.. py:function:: is_distributed() -> bool

   Returns true if job is distributed.


.. py:function:: is_same_accross_workers(local_tensor: torch.Tensor) -> bool

   Checks for a tensor if it is the same across all workers.
   Needs to be able to execute on all nodes, since we call all_gather. Otherwise this leads to deadlocks.


.. py:function:: model_to_ddp(model: torch.nn.Module) -> torch.nn.parallel.DistributedDataParallel

   Converts a torch model into a distributed data parallel model.


.. py:function:: should_distribute() -> bool

   Returns true if job should be distributed.