scaffold.torch.distributed.ddp

Functions

check_distributed_setup(→ bool)	Tests distributed setup with a manual calculation.
is_distributed(→ bool)	Returns true if job is distributed.
is_same_accross_workers(→ bool)	Checks for a tensor if it is the same across all workers.
model_to_ddp(→ torch.nn.parallel.DistributedDataParallel)	Converts a torch model into a distributed data parallel model.
should_distribute(→ bool)	Returns true if job should be distributed.

Module Contents

scaffold.torch.distributed.ddp.check_distributed_setup() → bool

Tests distributed setup with a manual calculation.

scaffold.torch.distributed.ddp.is_distributed() → bool

Returns true if job is distributed.

scaffold.torch.distributed.ddp.is_same_accross_workers(local_tensor: torch.Tensor) → bool

Checks for a tensor if it is the same across all workers. Needs to be able to execute on all nodes, since we call all_gather. Otherwise this leads to deadlocks.

scaffold.torch.distributed.ddp.model_to_ddp(model: torch.nn.Module) → torch.nn.parallel.DistributedDataParallel

Converts a torch model into a distributed data parallel model.

scaffold.torch.distributed.ddp.should_distribute() → bool

Returns true if job should be distributed.