scaffold.aim_utils.fsspec_artifact_storage

Classes

`FsspecArtifactStorage`	Fsspec storage backend for Aim.
`FsspecArtifactStorageAutoClean`	Makes sure all upload threads finish before the FsspecArtifactStorage is destroyed.

class scaffold.aim_utils.fsspec_artifact_storage.FsspecArtifactStorage(storage_root: str, max_upload_workers: int = 4)

Bases: aim.storage.artifacts.artifact_storage.AbstractArtifactStorage

Fsspec storage backend for Aim.

Parameters:

storage_root – URL with an optional prefix (e.g., ‘gs://my-bucket/path/prefix’).
fs – Optional filesystem object to be used. It will be created based on the storage root if not specified.

Note: Make sure to have the correct fsspec dependencies installed for the protocol you want to use (e.g., ‘gcsfs’ for ‘gs://’).

download_artifact(artifact_path: str, dest_dir: str | None = None) → str

Using fsspec, download an artifact to a local directory.

Parameters:

artifact_path – path to the artifact in storage, relative to the storage root specified in the constructor.
dest_dir – Local directory to download the artifact to. If None, a temporary directory is created.

Returns:

The local path to the downloaded artifact.

upload_artifact(file_path: str, artifact_path: str, block: bool = False)

Using fsspec, upload a local file to the artifact storage in a non-blocking manner by default.

Parameters:

file_path – Local path to the file to upload.
artifact_path – Path to the artifact in storage, relative to the storage root specified in the constructor.
block – If True, the method will block until the upload is complete.

class scaffold.aim_utils.fsspec_artifact_storage.FsspecArtifactStorageAutoClean(instance: FsspecArtifactStorage)

Bases: aim.ext.cleanup.AutoClean[FsspecArtifactStorage]

Makes sure all upload threads finish before the FsspecArtifactStorage is destroyed.

Initialize object resources that need to be cleaned up automatically.