scaffold.system_monitor

This file is strongly informed by the design of WandB’s client published under MIT license

A good overview and explanation of the different logged quantities can be found at: https://lambdalabs.com/blog/weights-and-bias-gpu-cpu-utilization/

Attributes

`GPUHandle`
`SamplerDict`
`StatsDict`

Classes

`AbstractBackendSender`	Abstract interface that works in conjunction with the system monitor.
`SystemMonitor`	Initializes the SystemMonitor using the parent threads PID and a backend_sender that implements the

Module Contents

class scaffold.system_monitor.AbstractBackendSender

Abstract interface that works in conjunction with the system monitor. Should be used to configure where the system metrics are being sent to.

abstractmethod publish(event: StatsDict, **kwargs) → None: Implements sending event to the corresponding backend.

class scaffold.system_monitor.SystemMonitor(pid: int, backend_sender: AbstractBackendSender, sample_rate_seconds: int = 2, rank: int | None = None)

Bases: object

Initializes the SystemMonitor using the parent threads PID and a backend_sender that implements the logic to send the collectesd stats to the corresponding backend, e.g. CSV, Tensorboard, WandB.

Parameters:

pid (int) – Process id of the parent process.
backend_sender (AbstractBackendSender) – Class that publishes the stats, e.g. to WandB.
sample_rate_seconds (int) – The rate of sampling system stats in seconds.
rank (int) – Rank of the current process. Only needed in a distributed setting.

flush() → None: Used the backend senders publish method to sent the latest metrics to the configured backend

shutdown() → None: Closes the SystemMonitor’s thread.

start() → None: Starts the the monitor by launching a separate thread.

gpu_count: int

network_init: Dict | None

property proc: psutil.Process: Returns the process corresponding to the parents PID

sampler: StatsDict

scaffold.system_monitor.GPUHandle

scaffold.system_monitor.SamplerDict

scaffold.system_monitor.StatsDict