scaffold.data.artifact_manager.base

Attributes

logger

Classes

`Artifact`	Immutable dataclass representing an artifact with its metadata.
`ArtifactManager`	Helper class that provides a standard way to create an ABC using
`DirectoryLogger`	Context manager for logging a directory as an artifact.
`TmpArtifact`	Context manager for temporary artifact download.

Module Contents

class scaffold.data.artifact_manager.base.Artifact

Immutable dataclass representing an artifact with its metadata.

collection: str

name: str

version: str

class scaffold.data.artifact_manager.base.ArtifactManager(collection: str = 'default')

Bases: abc.ABC

Helper class that provides a standard way to create an ABC using inheritance.

Artifact manager interface for various backends.

abstractmethod download_artifact(artifact_name: str, collection: str | None = None, version: str | None = None, to: str | None = None) → Artifact | TmpArtifact

Download artifact contents (from current collection) to specific location and return a source listing them.

If no target location is specified, a context manager for a temporary directory is created and the path to it is returned. Retrieve latest version unless specified.

Parameters:

artifact_name – The artifact name.
collection – The collection name. Defaults to the active collection.
version – The artifact version. If None, the latest version is used.
to – The destination path. If not provided, a TmpArtifact context manager is returned.

Returns:

If to is provided, returns an Artifact with metadata.: Otherwise, returns a TmpArtifact context manager that also has an artifact property.

Return type:

Union[Artifact, TmpArtifact]

exists(artifact_name: str) → bool: Check if artifact exists in specified collection.

abstractmethod exists_in_collection(artifact_name: str, collection: str | None = None) → bool: Check if artifact exists in specified collection.

abstractmethod list_collection_names() → Iterable: Return list of all collections in the artifact store

abstractmethod log_files(artifact_name: str, local_path: pathlib.Path, description: str, collection: str | None = None, artifact_path: pathlib.Path | None = None) → Artifact

Upload a file or folder into (current) collection, increment version automatically

Parameters:

artifact_name – Name of artifact to log
local_path – Local path to the file or folder to log
description – Description of the artifact, will be logged to improve documentation.
collection – Name of collection to log to, defaults to the active collection
artifact_path – path under which to log the files within the artifact, defaults to “./”

Returns:

The logged artifact with its metadata (name, collection, version).

Return type:

Artifact

log_folder(artifact_name: str, artifact_description: str, collection: str | None = None) → DirectoryLogger: Create a context manager for logging a directory of files as a single artifact.

property active_collection: str

Collections act as folders of artifacts.

It is ultimately up to the user how to structure their artifact store and the collections therein. All operations accessing artifacts allow explicit specification of the collection to use. Therefore, users could use collections to separate different artifact types or different experiments / runs. The manager maintains an ‘active’ collection which it logs to by default. This can be set at the run start when the manager is initialized and then remain unchanged for the duration of the run.

To avoid incompatibility between different backends, collections cannot be nested (e.g. as subfolders on a filesystem) as in particular the WandB backend has no real notion of nested folder structures.

class scaffold.data.artifact_manager.base.DirectoryLogger(artifact_manager: ArtifactManager, artifact_name: str, artifact_description: str, collection: str | None = None)

Context manager for logging a directory as an artifact.

This class creates a temporary directory where files can be written. When exiting the context, if the directory contains files, it is logged as an artifact.

Initialize a DirectoryLogger.

Parameters:

artifact_manager (ArtifactManager) – The artifact manager to use.
artifact_name (str) – The artifact name.
artifact_description – Description of the artifact. Will be logged at <artifact_root>/ARTIFACT_META_DIR/ARTIFACT_DESCRIPTION_FILE and serves to reduce undocumented artifact clutter.
collection (Optional[str]) – The collection name. Defaults to the artifact manager’s active collection.

__enter__() → str

Create and return a directory for logging files.

Returns:: The path to the logging directory.
Return type:: str

__exit__(exc_type: Any, exc_val: Any, exc_tb: Any) → None: Log the folder if non-empty and clean up the temporary directory.

artifact_manager

tempdir

class scaffold.data.artifact_manager.base.TmpArtifact(artifact_manager: ArtifactManager, collection: str, artifact_name: str, version: str)

Context manager for temporary artifact download.

This class downloads an artifact to a temporary directory when entering a context, and cleans up the directory upon exit.

Initialize a temporary artifact context.

Parameters:

artifact_manager (ArtifactManager) – The artifact manager to use.
collection (str) – The collection name.
artifact_name (str) – The artifact name.
version (str) – The artifact version.

__enter__() → str

Download the artifact into a temporary directory.

Returns:: The path to the temporary directory containing the artifact.
Return type:: str

__exit__(exc_type: Any, exc_val: Any, exc_tb: Any) → None: Clean up the temporary directory.

artifact

artifact_manager

tempdir

scaffold.data.artifact_manager.base.logger