scaffold.data.artifact_manager.filesystem

Classes

FileSystemArtifactManager

Artifact manager backed by a file system using fsspec.

Module Contents

class scaffold.data.artifact_manager.filesystem.FileSystemArtifactManager(url: str, collection: str = 'default', **fs_kwargs: Any)

Bases: scaffold.data.artifact_manager.base.ArtifactManager

Artifact manager backed by a file system using fsspec.

This implementation logs and retrieves artifacts from a specified URL.

Initialize a FileSystemArtifactManager.

Parameters:

  • url (str) – The base URL of the artifact store.

  • collection (str) – The default collection name. Defaults to “default”.

  • **fs_kwargs (Any) – Additional keyword arguments for the file system.

download_artifact(artifact_name: str, collection: str | None = None, version: str | None = None, to: str | None = None) → scaffold.data.artifact_manager.base.Artifact | scaffold.data.artifact_manager.base.TmpArtifact

Download an artifact from the artifact store.

If a destination path to is provided, the contents of the artifact version are copied there. Otherwise, a TmpArtifact context manager is returned.

Parameters:

  • artifact_name (str) – The artifact name.

  • collection (Optional[str]) – The collection name. Defaults to the active collection.

  • version (Optional[str]) – The version of the artifact. If None or “latest”, the latest version is used.

  • to (Optional[str]) – The destination path. If not provided, a temporary directory is used.

Returns:

If to is provided, returns an Artifact with metadata.

Otherwise, returns a TmpArtifact context manager that also has an artifact property.

Return type:

Union[Artifact, TmpArtifact]

exists_in_collection(artifact_name: str, collection: str | None = None) → bool

Check if an artifact exists in a specific collection.

Parameters:

  • artifact_name (str) – The artifact name.

  • collection (Optional[str]) – The collection name. Defaults to the active collection.

Returns:

True if the artifact exists, False otherwise.

Return type:

bool

list_artifacts(collection: str = None) → List[str]

Get sorted versions for an artifact.

Parameters:

collection (str) – Collection name.

Returns:

List of artifact strings.

Return type:

List[str]

list_collection_names() → Iterable[str]

List all collections in the artifact store.

Returns:

A list of collection names.

Return type:

Iterable[str]

list_versions(artifact_name: str, collection: str = None) → List[str]

Get sorted versions for an artifact.

Parameters:

  • artifact_name (str) – Artifact name.

  • collection (str) – Collection name.

Returns:

List of version strings sorted by version number (e.g., [“v0”, “v1”, “v2”]).

Return type:

List[str]

log_files(artifact_name: str, local_path: str, description: str, collection: str | None = None, artifact_path: str | None = None) → scaffold.data.artifact_manager.base.Artifact

Log a file or folder as an artifact.

This method uploads the file (or folder) located at local_path to the artifact store. If artifact_path is provided, the file is uploaded to a subpath within the artifact; otherwise, the entire folder is uploaded.

Parameters:

  • artifact_name (str) – The artifact name.

  • local_path (str) – The local path to the file or folder.

  • description – Description of the artifact. Will be logged at <artifact_root>/ARTIFACT_META_DIR/ARTIFACT_DESCRIPTION_FILE and serves to reduce undocumented artifact clutter.

  • collection (Optional[str]) – The collection name. Defaults to the active collection.

  • artifact_path (Optional[str]) – The subpath within the artifact for single file uploads.

Returns:

The logged artifact with its metadata (name, collection, version).

Return type:

Artifact

fs
url