scaffold.data.artifact_manager.base =================================== .. py:module:: scaffold.data.artifact_manager.base Attributes ---------- .. autoapisummary:: scaffold.data.artifact_manager.base.logger Classes ------- .. autoapisummary:: scaffold.data.artifact_manager.base.Artifact scaffold.data.artifact_manager.base.ArtifactManager scaffold.data.artifact_manager.base.DirectoryLogger scaffold.data.artifact_manager.base.TmpArtifact Module Contents --------------- .. py:class:: Artifact Immutable dataclass representing an artifact with its metadata. .. py:attribute:: collection :type: str .. py:attribute:: name :type: str .. py:attribute:: version :type: str .. py:class:: ArtifactManager(collection: str = 'default') Bases: :py:obj:`abc.ABC` Helper class that provides a standard way to create an ABC using inheritance. Artifact manager interface for various backends. .. py:method:: download_artifact(artifact_name: str, collection: Optional[str] = None, version: Optional[str] = None, to: Optional[str] = None) -> Union[Artifact, TmpArtifact] :abstractmethod: Download artifact contents (from current collection) to specific location and return a source listing them. If no target location is specified, a context manager for a temporary directory is created and the path to it is returned. Retrieve latest version unless specified. :param artifact_name: The artifact name. :param collection: The collection name. Defaults to the active collection. :param version: The artifact version. If None, the latest version is used. :param to: The destination path. If not provided, a TmpArtifact context manager is returned. :returns: If `to` is provided, returns an Artifact with metadata. Otherwise, returns a TmpArtifact context manager that also has an `artifact` property. :rtype: Union[Artifact, TmpArtifact] .. py:method:: exists(artifact_name: str) -> bool Check if artifact exists in specified collection. .. py:method:: exists_in_collection(artifact_name: str, collection: Optional[str] = None) -> bool :abstractmethod: Check if artifact exists in specified collection. .. py:method:: list_collection_names() -> Iterable :abstractmethod: Return list of all collections in the artifact store .. py:method:: log_files(artifact_name: str, local_path: pathlib.Path, description: str, collection: Optional[str] = None, artifact_path: Optional[pathlib.Path] = None) -> Artifact :abstractmethod: Upload a file or folder into (current) collection, increment version automatically :param artifact_name: Name of artifact to log :param local_path: Local path to the file or folder to log :param description: Description of the artifact, will be logged to improve documentation. :param collection: Name of collection to log to, defaults to the active collection :param artifact_path: path under which to log the files within the artifact, defaults to "./" :returns: The logged artifact with its metadata (name, collection, version). :rtype: Artifact .. py:method:: log_folder(artifact_name: str, artifact_description: str, collection: Optional[str] = None) -> DirectoryLogger Create a context manager for logging a directory of files as a single artifact. .. py:property:: active_collection :type: str Collections act as folders of artifacts. It is ultimately up to the user how to structure their artifact store and the collections therein. All operations accessing artifacts allow explicit specification of the collection to use. Therefore, users could use collections to separate different artifact types or different experiments / runs. The manager maintains an 'active' collection which it logs to by default. This can be set at the run start when the manager is initialized and then remain unchanged for the duration of the run. To avoid incompatibility between different backends, collections cannot be nested (e.g. as subfolders on a filesystem) as in particular the WandB backend has no real notion of nested folder structures. .. py:class:: DirectoryLogger(artifact_manager: ArtifactManager, artifact_name: str, artifact_description: str, collection: Optional[str] = None) Context manager for logging a directory as an artifact. This class creates a temporary directory where files can be written. When exiting the context, if the directory contains files, it is logged as an artifact. Initialize a DirectoryLogger. :param artifact_manager: The artifact manager to use. :type artifact_manager: ArtifactManager :param artifact_name: The artifact name. :type artifact_name: str :param artifact_description: Description of the artifact. Will be logged at /ARTIFACT_META_DIR/ARTIFACT_DESCRIPTION_FILE and serves to reduce undocumented artifact clutter. :param collection: The collection name. Defaults to the artifact manager's active collection. :type collection: Optional[str] .. py:method:: __enter__() -> str Create and return a directory for logging files. :returns: The path to the logging directory. :rtype: str .. py:method:: __exit__(exc_type: Any, exc_val: Any, exc_tb: Any) -> None Log the folder if non-empty and clean up the temporary directory. .. py:attribute:: artifact_manager .. py:attribute:: tempdir .. py:class:: TmpArtifact(artifact_manager: ArtifactManager, collection: str, artifact_name: str, version: str) Context manager for temporary artifact download. This class downloads an artifact to a temporary directory when entering a context, and cleans up the directory upon exit. Initialize a temporary artifact context. :param artifact_manager: The artifact manager to use. :type artifact_manager: ArtifactManager :param collection: The collection name. :type collection: str :param artifact_name: The artifact name. :type artifact_name: str :param version: The artifact version. :type version: str .. py:method:: __enter__() -> str Download the artifact into a temporary directory. :returns: The path to the temporary directory containing the artifact. :rtype: str .. py:method:: __exit__(exc_type: Any, exc_val: Any, exc_tb: Any) -> None Clean up the temporary directory. .. py:attribute:: artifact .. py:attribute:: artifact_manager .. py:attribute:: tempdir .. py:data:: logger