mlmd.MetadataStore

A store for the metadata.

mlmd.MetadataStore(
    config, enable_upgrade_migration: bool = False
)

Args
`config`	`proto.ConnectionConfig` or `proto.MetadataStoreClientConfig`. Configuration to connect to the database or the metadata store server.
`enable_upgrade_migration`	if set to True, the library upgrades the db schema and migrates all data if it connects to an old version backend. It is ignored when using gRPC `proto.MetadataStoreClientConfig`.

Attributes
`pipeline_asset`	Returns None as the pipeline asset.

Methods

`get_artifact_by_type_and_name`

View source

get_artifact_by_type_and_name(
    type_name: str,
    artifact_name: str,
    type_version: Optional[str] = None,
    extra_options: Optional[ExtraOptions] = None
) -> Optional[mlmd.proto.Artifact]

Get the artifact of the given type and name.

The API fails if more than one artifact is found.

Args
`type_name`	The artifact type name to look for.
`artifact_name`	The artifact name to look for.
`type_version`	An optional artifact type version. If not given, then only the type_name and artifact_name are used to look for the artifact with default version.
`extra_options`	ExtraOptions instance.

Returns
The Artifact matching the type and name. None if no matched Artifact was found.

`get_artifact_type`

View source

get_artifact_type(
    type_name: str,
    type_version: Optional[str] = None,
    extra_options: Optional[ExtraOptions] = None
) -> mlmd.proto.ArtifactType

Gets an artifact type by name and version.

Args
`type_name`	the type with that name.
`type_version`	an optional version of the type, if not given, then only the type_name is used to look for types with no versions.
`extra_options`	ExtraOptions instance.

Returns
The type with name type_name and version type version.

Raises
`errors.NotFoundError`	if no type exists.
`errors.InternalError`	if query execution fails.

`get_artifact_types`

View source

get_artifact_types(
    extra_options: Optional[ExtraOptions] = None
) -> List[mlmd.proto.ArtifactType]

Gets all artifact types.

Args
`extra_options`	ExtraOptions instance.

Returns
A list of all known ArtifactTypes.

Raises
`errors.InternalError`	if query execution fails.

`get_artifact_types_by_external_ids`

View source

get_artifact_types_by_external_ids(
    external_ids: Iterable[str], extra_options: Optional[ExtraOptions] = None
) -> List[mlmd.proto.ArtifactType]

Gets all artifact types with matching external ids.

Args
`external_ids`	A list of external_ids for retrieving the ArtifactTypes.
`extra_options`	ExtraOptions instance.

Returns
ArtifactTypes with matching external_ids.

`get_artifact_types_by_id`

View source

get_artifact_types_by_id(
    type_ids: Iterable[int], extra_options: Optional[ExtraOptions] = None
) -> List[mlmd.proto.ArtifactType]

Gets artifact types by ID.

Args
`type_ids`	a sequence of artifact type IDs.
`extra_options`	ExtraOptions instance.

Returns
A list of artifact types.

Raises
`errors.InternalError`	if query execution fails.

`get_artifacts`

View source

get_artifacts(
    list_options: Optional[mlmd.ListOptions] = None,
    extra_options: Optional[ExtraOptions] = None
) -> List[mlmd.proto.Artifact]

Gets artifacts.

Args
`list_options`	A set of options to specify the conditions, limit the size and adjust order of the returned artifacts.
`extra_options`	ExtraOptions instance.

Returns
A list of artifacts.

Raises
`errors.InternalError`	if query execution fails.
`errors.InvalidArgument`	if list_options is invalid.

`get_artifacts_and_types_by_artifact_ids`

View source

get_artifacts_and_types_by_artifact_ids(
    artifact_ids: Iterable[int], extra_options: Optional[ExtraOptions] = None
) -> Tuple[List[proto.Artifact], List[proto.ArtifactType]]

Gets all artifacts with matching ids and populates types.

The result is not index-aligned: if an id is not found, it is not returned.

Args
`artifact_ids`	A list of artifact ids to retrieve.
`extra_options`	ExtraOptions instance.

Returns
Artifacts with matching ids and ArtifactTypes which can be matched by type_ids from Artifacts. Each ArtifactType contains id, name, properties and custom_properties fields.

`get_artifacts_by_context`

View source

get_artifacts_by_context(
    context_id: int,
    list_options: Optional[mlmd.ListOptions] = None,
    extra_options: Optional[ExtraOptions] = None
) -> List[mlmd.proto.Artifact]

Gets all direct artifacts that are attributed to the given context.

Args
`context_id`	The id of the querying context
`list_options`	A set of options to specify the conditions, limit the size and adjust order of the returned executions.
`extra_options`	ExtraOptions instance.

Returns
Artifacts attributing to the context.

`get_artifacts_by_external_ids`

View source

get_artifacts_by_external_ids(
    external_ids: Iterable[str], extra_options: Optional[ExtraOptions] = None
) -> List[mlmd.proto.Artifact]

Gets all artifacts with matching external ids.

Args
`external_ids`	A list of external_ids for retrieving the Artifacts.
`extra_options`	ExtraOptions instance.

Returns
Artifacts with matching external_ids.

`get_artifacts_by_id`

View source

get_artifacts_by_id(
    artifact_ids: Iterable[int], extra_options: Optional[ExtraOptions] = None
) -> List[mlmd.proto.Artifact]

Gets all artifacts with matching ids.

The result is not index-aligned: if an id is not found, it is not returned.

Args
`artifact_ids`	A list of artifact ids to retrieve.
`extra_options`	ExtraOptions instance.

Returns
Artifacts with matching ids.

`get_artifacts_by_type`

View source

get_artifacts_by_type(
    type_name: str,
    type_version: Optional[str] = None,
    extra_options: Optional[ExtraOptions] = None
) -> List[mlmd.proto.Artifact]

Gets all the artifacts of a given type.

Args
`type_name`	The artifact type name to look for.
`type_version`	An optional artifact type version. If not given, then only the type_name are used to look for the artifacts with default version.
`extra_options`	ExtraOptions instance.

Returns
Artifacts that matches the type.

`get_artifacts_by_uri`

View source

get_artifacts_by_uri(
    uri: str, extra_options: Optional[ExtraOptions] = None
) -> List[mlmd.proto.Artifact]

Gets all the artifacts of a given uri.

Args
`uri`	The artifact uri to look for.
`extra_options`	ExtraOptions instance.

Returns
The Artifacts matching the uri.

`get_children_contexts_by_context`

View source

get_children_contexts_by_context(
    context_id: int, extra_options: Optional[ExtraOptions] = None
) -> List[mlmd.proto.Context]

Gets all children contexts of a context.

Args
`context_id`	The id of the querying context.
`extra_options`	ExtraOptions instance.

Returns
Children contexts of the querying context.

Raises
`errors.InternalError`	if query execution fails.

`get_context_by_type_and_name`

View source

get_context_by_type_and_name(
    type_name: str,
    context_name: str,
    type_version: Optional[str] = None,
    extra_options: Optional[ExtraOptions] = None
) -> Optional[mlmd.proto.Context]

Get the context of the given type and context name.

The API fails if more than one contexts are found.

Args
`type_name`	The context type name to look for.
`context_name`	The context name to look for.
`type_version`	An optional context type version. If not given, then only the type_name and context_name are used to look for the context with default version.
`extra_options`	ExtraOptions instance.

Returns
The Context matching the type and context name. None if no matched Context found.

`get_context_type`

View source

get_context_type(
    type_name: str,
    type_version: Optional[str] = None,
    extra_options: Optional[ExtraOptions] = None
) -> mlmd.proto.ContextType

Gets a context type by name and version.

Args
`type_name`	the type with that name.
`type_version`	an optional version of the type, if not given, then only the type_name is used to look for types with no versions.
`extra_options`	ExtraOptions instance.

Returns
The type with name type_name and version type_version.

Raises
`errors.NotFoundError`	if no type exists.
`errors.InternalError`	if query execution fails.

`get_context_types`

View source

get_context_types(
    extra_options: Optional[ExtraOptions] = None
) -> List[mlmd.proto.ContextType]

Gets all context types.

Args
`extra_options`	ExtraOptions instance.

Returns
A list of all known ContextTypes.

Raises
`errors.InternalError`	if query execution fails.

`get_context_types_by_external_ids`

View source

get_context_types_by_external_ids(
    external_ids: Iterable[str], extra_options: Optional[ExtraOptions] = None
) -> List[mlmd.proto.ContextType]

Gets all context types with matching external ids.

Args
`external_ids`	A list of external_ids for retrieving the ContextTypes.
`extra_options`	ExtraOptions instance.

Returns
ContextTypes with matching external_ids.

`get_context_types_by_id`

View source

get_context_types_by_id(
    type_ids: Iterable[int], extra_options: Optional[ExtraOptions] = None
) -> List[mlmd.proto.ContextType]

Gets context types by ID.

Args
`type_ids`	a sequence of context type IDs.
`extra_options`	ExtraOptions instance.

Returns
A list of context types.

Raises
`errors.InternalError`	if query execution fails.

`get_contexts`

View source

get_contexts(
    list_options: Optional[mlmd.ListOptions] = None,
    extra_options: Optional[ExtraOptions] = None
) -> List[mlmd.proto.Context]

Gets contexts.

Args
`list_options`	A set of options to specify the conditions, limit the size and adjust order of the returned contexts.
`extra_options`	ExtraOptions instance.

Returns
A list of contexts.

Raises
`errors.InternalError`	if query execution fails.
`errors.InvalidArgument`	if list_options is invalid.

`get_contexts_by_artifact`

View source

get_contexts_by_artifact(
    artifact_id: int, extra_options: Optional[ExtraOptions] = None
) -> List[mlmd.proto.Context]

Gets all context that an artifact is attributed to.

Args
`artifact_id`	The id of the querying artifact
`extra_options`	ExtraOptions instance.

Returns
Contexts that the artifact is attributed to.

`get_contexts_by_execution`

View source

get_contexts_by_execution(
    execution_id: int, extra_options: Optional[ExtraOptions] = None
) -> List[mlmd.proto.Context]

Gets all context that an execution is associated with.

Args
`execution_id`	The id of the querying execution
`extra_options`	ExtraOptions instance.

Returns
Contexts that the execution is associated with.

`get_contexts_by_external_ids`

View source

get_contexts_by_external_ids(
    external_ids: Iterable[str], extra_options: Optional[ExtraOptions] = None
) -> List[mlmd.proto.Context]

Gets all contexts with matching external ids.

Args
`external_ids`	A list of external_ids for retrieving the Contexts.
`extra_options`	ExtraOptions instance.

Returns
Contexts with matching external_ids.

`get_contexts_by_id`

View source

get_contexts_by_id(
    context_ids: Iterable[int], extra_options: Optional[ExtraOptions] = None
) -> List[mlmd.proto.Context]

Gets all contexts with matching ids.

The result is not index-aligned: if an id is not found, it is not returned.

Args
`context_ids`	A list of context ids to retrieve.
`extra_options`	ExtraOptions instance.

Returns
Contexts with matching ids.

`get_contexts_by_type`

View source

get_contexts_by_type(
    type_name: str,
    type_version: Optional[str] = None,
    extra_options: Optional[ExtraOptions] = None
) -> List[mlmd.proto.Context]

Gets all the contexts of a given type.

Args
`type_name`	The context type name to look for.
`type_version`	An optional context type version. If not given, then only the type_name are used to look for the contexts with default version.
`extra_options`	ExtraOptions instance.

Returns
Contexts that matches the type.

`get_events_by_artifact_ids`

View source

get_events_by_artifact_ids(
    artifact_ids: Iterable[int], extra_options: Optional[ExtraOptions] = None
) -> List[mlmd.proto.Event]

Gets all events with matching artifact ids.

Args
`artifact_ids`	a list of artifact ids.
`extra_options`	ExtraOptions instance.

Returns
Events with the execution IDs given.

Raises
`errors.InternalError`	if query execution fails.

`get_events_by_execution_ids`

View source

get_events_by_execution_ids(
    execution_ids: Iterable[int], extra_options: Optional[ExtraOptions] = None
) -> List[mlmd.proto.Event]

Gets all events with matching execution ids.

Args
`execution_ids`	a list of execution ids.
`extra_options`	ExtraOptions instance.

Returns
Events with the execution IDs given.

Raises
`errors.InternalError`	if query execution fails.

`get_execution_by_type_and_name`

View source

get_execution_by_type_and_name(
    type_name: str,
    execution_name: str,
    type_version: Optional[str] = None,
    extra_options: Optional[ExtraOptions] = None
) -> Optional[mlmd.proto.Execution]

Get the execution of the given type and name.

The API fails if more than one execution is found.

Args
`type_name`	The execution type name to look for.
`execution_name`	The execution name to look for.
`type_version`	An optional execution type version. If not given, then only the type_name and execution_name are used to look for the execution with default version.
`extra_options`	ExtraOptions instance.

Returns
The Execution matching the type and name. None if no matched Execution found.

`get_execution_type`

View source

get_execution_type(
    type_name: str,
    type_version: Optional[str] = None,
    extra_options: Optional[ExtraOptions] = None
) -> mlmd.proto.ExecutionType

Gets an execution type by name and version.

Args
`type_name`	the type with that name.
`type_version`	an optional version of the type, if not given, then only the type_name is used to look for types with no versions.
`extra_options`	ExtraOptions instance.

Returns
The type with name type_name and version type_version.

Raises
`errors.NotFoundError`	if no type exists.
`errors.InternalError`	if query execution fails.

`get_execution_types`

View source

get_execution_types(
    extra_options: Optional[ExtraOptions] = None
) -> List[mlmd.proto.ExecutionType]

Gets all execution types.

Args
`extra_options`	ExtraOptions instance.

Returns
A list of all known ExecutionTypes.

Raises
`errors.InternalError`	if query execution fails.

`get_execution_types_by_external_ids`

View source

get_execution_types_by_external_ids(
    external_ids: Iterable[str], extra_options: Optional[ExtraOptions] = None
) -> List[mlmd.proto.ExecutionType]

Gets all execution types with matching external ids.

Args
`external_ids`	A list of external_ids for retrieving the ExecutionTypes.
`extra_options`	ExtraOptions instance.

Returns
ExecutionTypes with matching external_ids.

`get_execution_types_by_id`

View source

get_execution_types_by_id(
    type_ids: Iterable[int], extra_options: Optional[ExtraOptions] = None
) -> List[mlmd.proto.ExecutionType]

Gets execution types by ID.

Args
`type_ids`	a sequence of execution type IDs.
`extra_options`	ExtraOptions instance.

Returns
A list of execution types.

Raises
`errors.InternalError`	if query execution fails.

`get_executions`

View source

get_executions(
    list_options: Optional[mlmd.ListOptions] = None,
    extra_options: Optional[ExtraOptions] = None
) -> List[mlmd.proto.Execution]

Gets executions.

Args
`list_options`	A set of options to specify the conditions, limit the size and adjust order of the returned executions.
`extra_options`	ExtraOptions instance.

Returns
A list of executions.

Raises
`errors.InternalError`	if query execution fails.
`errors.InvalidArgument`	if list_options is invalid.

`get_executions_by_context`

View source

get_executions_by_context(
    context_id: int,
    list_options: Optional[mlmd.ListOptions] = None,
    extra_options: Optional[ExtraOptions] = None
) -> List[mlmd.proto.Execution]

Gets all direct executions that a context associates with.

Args
`context_id`	The id of the querying context
`list_options`	A set of options to specify the conditions, limit the size and adjust order of the returned executions.
`extra_options`	ExtraOptions instance.

Returns
Executions associating with the context.

`get_executions_by_external_ids`

View source

get_executions_by_external_ids(
    external_ids: Iterable[str], extra_options: Optional[ExtraOptions] = None
) -> List[mlmd.proto.Execution]

Gets all executions with matching external ids.

Args
`external_ids`	A list of external_ids for retrieving the Executions.
`extra_options`	ExtraOptions instance.

Returns
Executions with matching external_ids.

`get_executions_by_id`

View source

get_executions_by_id(
    execution_ids: Iterable[int], extra_options: Optional[ExtraOptions] = None
) -> List[mlmd.proto.Execution]

Gets all executions with matching ids.

The result is not index-aligned: if an id is not found, it is not returned.

Args
`execution_ids`	A list of execution ids to retrieve.
`extra_options`	ExtraOptions instance.

Returns
Executions with matching ids.

`get_executions_by_type`

View source

get_executions_by_type(
    type_name: str,
    type_version: Optional[str] = None,
    extra_options: Optional[ExtraOptions] = None
) -> List[mlmd.proto.Execution]

Gets all the executions of a given type.

Args
`type_name`	The execution type name to look for.
`type_version`	An optional execution type version. If not given, then only the type_name are used to look for the executions with default version.
`extra_options`	ExtraOptions instance.

Returns
Executions that matches the type.

`get_lineage_subgraph`

View source

get_lineage_subgraph(
    query_options: metadata_store_pb2.LineageSubgraphQueryOptions,
    field_mask_paths: Optional[Sequence[str]] = None,
    extra_options: Optional[ExtraOptions] = None
) -> metadata_store_pb2.LineageGraph

Gets lineage graph including fields specified in a field mask.

Args

query_options metadata_store_pb2.LineageSubgraphQueryOptions object. It allows users to specify query options for lineage graph tracing from a list of interested nodes (limited to 100). Please refer to LineageSubgraphQueryOptions for more details.

field_mask_paths

a list of user specified paths of fields that should be included in the returned lineage graph. If field_mask_paths is specified and non-empty:

If 'artifacts', 'executions', or 'contexts' is specified in read_mask, the nodes with details will be included.
If 'artifact_types', 'execution_types', or 'context_types' is specified in read_mask, all the node types with matched type_id in nodes in the returned graph will be included.
If 'events', 'associations', or 'attributions' is specified in read_mask, the corresponding edges will be included. If field_mask_paths is unspecified or is empty, it will return all the fields in the returned graph.

extra_options ExtraOptions instance.

Returns
metadata_store_pb2.LineageGraph object that contains the lineage graph.

`get_parent_contexts_by_context`

View source

get_parent_contexts_by_context(
    context_id: int, extra_options: Optional[ExtraOptions] = None
) -> List[mlmd.proto.Context]

Gets all parent contexts of a context.

Args
`context_id`	The id of the querying context.
`extra_options`	ExtraOptions instance.

Returns
Parent contexts of the querying context.

Raises
`errors.InternalError`	if query execution fails.

`put_artifact_type`

View source

put_artifact_type(
    artifact_type: mlmd.proto.ArtifactType,
    can_add_fields: bool = False,
    can_omit_fields: bool = False,
    extra_options: Optional[ExtraOptions] = None
) -> int

Inserts or updates an artifact type.

A type has a set of strong typed properties describing the schema of any stored instance associated with that type. A type is identified by a name and an optional version.

Type Creation:

If no type exists in the database with the given identifier (name, version), it creates a new type and returns the type_id.

Type Evolution:

If the request type with the same (name, version) already exists (let's call it stored_type), the method enforces the stored_type can be updated only when the request type is backward compatible for the already stored instances.

Backwards compatibility is violated iff:

there is a property where the request type and stored_type have different value type (e.g., int vs. string)
can_add_fields = false and the request type has a new property that is not stored.
can_omit_fields = false and stored_type has an existing property that is not provided in the request type.

If non-backward type change is required in the application, e.g., deprecate properties, re-purpose property name, change value types, a new type can be created with a different (name, version) identifier. Note the type version is optional, and a version value with empty string is treated as unset.

Args
`artifact_type`	the request type to be inserted or updated.
`can_add_fields`	when true, new properties can be added; when false, returns ALREADY_EXISTS if the request type has properties that are not in stored_type.
`can_omit_fields`	when true, stored properties can be omitted in the request type; when false, returns ALREADY_EXISTS if the stored_type has properties not in the request type.
`extra_options`	ExtraOptions instance.

Returns
the type_id of the response.

Raises
`errors.AlreadyExistsError`	If the type is not backward compatible.
`errors.InvalidArgumentError`	If the request type has no name, or any property value type is unknown.

`put_artifacts`

View source

put_artifacts(
    artifacts: Sequence[mlmd.proto.Artifact],
    field_mask_paths: Optional[Sequence[str]] = None,
    extra_options: Optional[ExtraOptions] = None
) -> List[int]

Inserts or updates artifacts in the database.

If an artifact id is specified for an artifact, it is an update. If an artifact id is unspecified, it will insert a new artifact. For new artifacts, type must be specified. For old artifacts, type must be unchanged or unspecified. When the name of an artifact is given, it should be unique among artifacts of the same ArtifactType.

It is not guaranteed that the created or updated artifacts will share the same create_time_since_epoch or last_update_time_since_epoch timestamps.

If field_mask_paths is specified and non-empty:

while updating an existing artifact, it only updates fields specified in field_mask_paths.
while inserting a new artifact, field_mask_paths will be ignored.
otherwise, field_mask_paths will be applied to all artifacts. If field_mask_paths is unspecified or is empty, it updates the artifact as a whole.

Args
`artifacts`	A list of artifacts to insert or update.
`field_mask_paths`	A list of field mask paths for masked update.
`extra_options`	ExtraOptions instance.

Returns
A list of artifact ids index-aligned with the input.

Raises
`errors.AlreadyExistsError`	If artifact's name is specified and it is already used by stored artifacts of that ArtifactType.

`put_attributions_and_associations`

View source

put_attributions_and_associations(
    attributions: Sequence[mlmd.proto.Attribution],
    associations: Sequence[mlmd.proto.Association],
    extra_options: Optional[ExtraOptions] = None
) -> None

Inserts attribution and association relationships in the database.

The context_id, artifact_id, and execution_id must already exist. If the relationship exists, this call does nothing. Once added, the relationships cannot be modified.

Args
`attributions`	A list of attributions to insert.
`associations`	A list of associations to insert.
`extra_options`	ExtraOptions instance.

`put_context_type`

View source

put_context_type(
    context_type: mlmd.proto.ContextType,
    can_add_fields: bool = False,
    can_omit_fields: bool = False,
    extra_options: Optional[ExtraOptions] = None
) -> int

Inserts or updates a context type.

A type has a set of strong typed properties describing the schema of any stored instance associated with that type. A type is identified by a name and an optional version.

Type Creation:

If no type exists in the database with the given identifier (name, version), it creates a new type and returns the type_id.

Type Evolution:

Backwards compatibility is violated iff:

there is a property where the request type and stored_type have different value type (e.g., int vs. string)
can_add_fields = false and the request type has a new property that is not stored.
can_omit_fields = false and stored_type has an existing property that is not provided in the request type.

Args
`context_type`	the request type to be inserted or updated.
`can_add_fields`	when true, new properties can be added; when false, returns ALREADY_EXISTS if the request type has properties that are not in stored_type.
`can_omit_fields`	when true, stored properties can be omitted in the request type; when false, returns ALREADY_EXISTS if the stored_type has properties not in the request type.
`extra_options`	ExtraOptions instance.

Returns
the type_id of the response.

Raises
`errors.AlreadyExistsError`	If the type is not backward compatible.
`errors.InvalidArgumentError`	If the request type has no name, or any property value type is unknown.

`put_contexts`

View source

put_contexts(
    contexts: Sequence[mlmd.proto.Context],
    field_mask_paths: Optional[Sequence[str]] = None,
    extra_options: Optional[ExtraOptions] = None
) -> List[int]

Inserts or updates contexts in the database.

If an context id is specified for an context, it is an update. If an context id is unspecified, it will insert a new context. For new contexts, type must be specified. For old contexts, type must be unchanged or unspecified. The name of a context cannot be empty, and it should be unique among contexts of the same ContextType.

It is not guaranteed that the created or updated contexts will share the same create_time_since_epoch or last_update_time_since_epoch timestamps.

If field_mask_paths is specified and non-empty:

while updating an existing context, it only updates fields specified in field_mask_paths.
while inserting a new context, field_mask_paths will be ignored.
otherwise, field_mask_paths will be applied to all contexts. If field_mask_paths is unspecified or is empty, it updates the context as a whole.

Args
`contexts`	A list of contexts to insert or update.
`field_mask_paths`	A list of field mask paths for masked update.
`extra_options`	ExtraOptions instance.

Returns
A list of context ids index-aligned with the input.

Raises
`errors.InvalidArgumentError`	If name of the new contexts are empty.
`errors.AlreadyExistsError`	If name of the new contexts already used by stored contexts of that ContextType.

`put_events`

View source

put_events(
    events: Sequence[mlmd.proto.Event],
    extra_options: Optional[ExtraOptions] = None
) -> None

Inserts events in the database.

The execution_id and artifact_id must already exist. Once created, events cannot be modified.

It is not guaranteed that the created or updated events will share the same milliseconds_since_epoch timestamps.

Args
`events`	A list of events to insert.
`extra_options`	ExtraOptions instance.

`put_execution`

View source

put_execution(
    execution: mlmd.proto.Execution,
    artifact_and_events: Sequence[Tuple[proto.Artifact, Optional[proto.Event]]],
    contexts: Optional[Sequence[proto.Context]],
    reuse_context_if_already_exist: bool = False,
    reuse_artifact_if_already_exist_by_external_id: bool = False,
    force_reuse_context: bool = False,
    force_update_time: bool = False,
    extra_options: Optional[ExtraOptions] = None
) -> Tuple[int, List[int], List[int]]

Inserts or updates an Execution with artifacts, events and contexts.

In contrast with other put methods, the method update an execution atomically with its input/output artifacts and events and adds attributions and associations to related contexts.

If an execution_id, artifact_id or context_id is specified, it is an update, otherwise it does an insertion.

It is not guaranteed that the created or updated executions, artifacts, contexts and events will share the same create_time_since_epoch, last_update_time_since_epoch, or milliseconds_since_epoch timestamps.

Args
`execution`	The execution to be created or updated.
`artifact_and_events`	a pair of Artifact and Event that the execution uses or generates. The event's execution id or artifact id can be empty, as the artifact or execution may not be stored beforehand. If given, the ids must match with the paired Artifact and the input execution.
`contexts`	The Contexts that the execution should be associated with and the artifacts should be attributed to.
`reuse_context_if_already_exist`	When there's a race to publish executions with a new context (no id) with the same context.name, by default there will be one writer succeeds and the rest of the writers fail with AlreadyExists errors. If set is to True, failed writers will reuse the stored context.
`reuse_artifact_if_already_exist_by_external_id`	When there's a race to publish executions with a new artifact with the same artifact.external_id, by default there'll be one writer succeeds and the rest of the writers returning AlreadyExists errors. If set to true and an Artifact has non-empty external_id, the API will reuse the stored artifact in the transaction and perform an update. Otherwise, it will fall back to relying on `id` field to decide if it's update (if `id` exists) or insert (if `id` is empty).
`force_reuse_context`	If True, for contexts with a context.id, the stored context will NOT be updated. For such contexts, we will only look at the context.id to associate the context with the execution.
`force_update_time`	If it is true, `last_update_time_since_epoch` is updated even if input execution is the same as stored execution.
`extra_options`	ExtraOptions instance.

Returns
the execution id, the list of artifact's id, and the list of context's id.

Raises
`errors.InvalidArgumentError`	If the id of the input nodes do not align with the store. Please refer to InvalidArgument errors in other put methods.
`errors.AlreadyExistsError`	If the new nodes to be created is already exists. Please refer to AlreadyExists errors in other put methods.

`put_execution_type`

View source

put_execution_type(
    execution_type: mlmd.proto.ExecutionType,
    can_add_fields: bool = False,
    can_omit_fields: bool = False,
    extra_options: Optional[ExtraOptions] = None
) -> int

Inserts or updates an execution type.

A type has a set of strong typed properties describing the schema of any stored instance associated with that type. A type is identified by a name and an optional version.

Type Creation:

If no type exists in the database with the given identifier (name, version), it creates a new type and returns the type_id.

Type Evolution:

Backwards compatibility is violated iff:

there is a property where the request type and stored_type have different value type (e.g., int vs. string)
can_add_fields = false and the request type has a new property that is not stored.
can_omit_fields = false and stored_type has an existing property that is not provided in the request type.

Args
`execution_type`	the request type to be inserted or updated.
`can_add_fields`	when true, new properties can be added; when false, returns ALREADY_EXISTS if the request type has properties that are not in stored_type.
`can_omit_fields`	when true, stored properties can be omitted in the request type; when false, returns ALREADY_EXISTS if the stored_type has properties not in the request type.
`extra_options`	ExtraOptions instance.

Returns
the type_id of the response.

Raises
`errors.AlreadyExistsError`	If the type is not backward compatible.
`errors.InvalidArgumentError`	If the request type has no name, or any property value type is unknown.

`put_executions`

View source

put_executions(
    executions: Sequence[mlmd.proto.Execution],
    field_mask_paths: Optional[Sequence[str]] = None,
    extra_options: Optional[ExtraOptions] = None
) -> List[int]

Inserts or updates executions in the database.

If an execution id is specified for an execution, it is an update. If an execution id is unspecified, it will insert a new execution. For new executions, type must be specified. For old executions, type must be unchanged or unspecified. When the name of an execution is given, it should be unique among executions of the same ExecutionType.

It is not guaranteed that the created or updated executions will share the same create_time_since_epoch or last_update_time_since_epoch timestamps.

If field_mask_paths is specified and non-empty:

while updating an existing execution, it only updates fields specified in field_mask_paths.
while inserting a new execution, field_mask_paths will be ignored.
otherwise, field_mask_paths will be applied to all executions. If field_mask_paths is unspecified or is empty, it updates the execution as a whole.

Args
`executions`	A list of executions to insert or update.
`field_mask_paths`	A list of field mask paths for masked update.
`extra_options`	ExtraOptions instance.

Returns
A list of execution ids index-aligned with the input.

Raises
`errors.AlreadyExistsError`	If execution's name is specified and it is already used by stored executions of that ExecutionType.

`put_lineage_subgraph`

View source

put_lineage_subgraph(
    executions: Sequence[mlmd.proto.Execution],
    artifacts: Sequence[mlmd.proto.Artifact],
    contexts: Sequence[mlmd.proto.Context],
    event_edges: Sequence[Tuple[Optional[int], Optional[int], proto.Event]],
    reuse_context_if_already_exist: bool = False,
    reuse_artifact_if_already_exist_by_external_id: bool = False,
    extra_options: Optional[ExtraOptions] = None
) -> Tuple[List[int], List[int], List[int]]

Inserts a collection of executions, artifacts, contexts, and events.

This method atomically inserts or updates all specified executions, artifacts, and events and adds attributions and associations to related contexts.

Args
`executions`	List of executions to be created or updated.
`artifacts`	List of artifacts to be created or updated.
`contexts`	List of contexts to be created or reused. Contexts will be associated with the inserted executions and attributed to the inserted artifacts.
`event_edges`	List of event edges in the subgraph to be inserted. Event edges are defined as an optional execution_index, an optional artifact_index, and a required event. Event edges must have an execution_index and/or an event.execution_id. Execution_index corresponds to an execution in the executions list at the specified index. If both execution_index and event.execution_id are provided, the execution ids of the execution and the event must match. The same rules apply to artifact_index and event.artifact_id.
`reuse_context_if_already_exist`	When there's a race to publish executions with a new context (no id) with the same context.name, by default there will be one writer that succeeds and the rest of the writers will fail with AlreadyExists errors. If set to True, failed writers will reuse the stored context.
`reuse_artifact_if_already_exist_by_external_id`	When there's a race to publish executions with a new artifact with the same artifact.external_id, by default there'll be one writer succeeds and the rest of the writers returning AlreadyExists errors. If set to true and an Artifact has non-empty external_id, the API will reuse the stored artifact in the transaction and perform an update. Otherwise, it will fall back to relying on `id` field to decide if it's update (if `id` exists) or insert (if `id` is empty).
`extra_options`	ExtraOptions instance.

Returns
The lists of execution ids, artifact ids, and context ids index aligned to the input executions, artifacts, and contexts.

Raises
`errors.InvalidArgumentError`	If the id of the input nodes do not align with the store. Please refer to InvalidArgument errors in other put methods.
`errors.AlreadyExistsError`	If the new nodes to be created already exist. Please refer to AlreadyExists errors in other put methods.
`errors.OutOfRangeError`	If event_edge indices do not correspond to existing indices in the input lists of executions and artifacts.

`put_parent_contexts`

View source

put_parent_contexts(
    parent_contexts: Sequence[mlmd.proto.ParentContext],
    extra_options: Optional[ExtraOptions] = None
) -> None

Inserts parent contexts in the database.

The child_id and parent_id in every parent context must already exist.

Args
`parent_contexts`	A list of parent contexts to insert.
`extra_options`	ExtraOptions instance.

Raises
`errors.InvalidArgumentError`	if no context matches the `child_id` or no context matches the `parent_id` in any parent context.
`errors.AlreadyExistsError`	if the same parent context already exists.

mlmd.MetadataStore

Args

Attributes

Methods

get_artifact_by_type_and_name

get_artifact_type

get_artifact_types

get_artifact_types_by_external_ids

get_artifact_types_by_id

get_artifacts

get_artifacts_and_types_by_artifact_ids

get_artifacts_by_context

get_artifacts_by_external_ids

get_artifacts_by_id

get_artifacts_by_type

get_artifacts_by_uri

get_children_contexts_by_context

get_context_by_type_and_name

get_context_type

get_context_types

get_context_types_by_external_ids

get_context_types_by_id

get_contexts

get_contexts_by_artifact

get_contexts_by_execution

get_contexts_by_external_ids

get_contexts_by_id

get_contexts_by_type

get_events_by_artifact_ids

get_events_by_execution_ids

get_execution_by_type_and_name

get_execution_type

get_execution_types

get_execution_types_by_external_ids

get_execution_types_by_id

get_executions

get_executions_by_context

get_executions_by_external_ids

get_executions_by_id

get_executions_by_type

get_lineage_subgraph

get_parent_contexts_by_context

put_artifact_type

Type Creation:

Type Evolution:

put_artifacts

put_attributions_and_associations

put_context_type

Type Creation:

Type Evolution:

put_contexts

put_events

put_execution

put_execution_type

Type Creation:

Type Evolution:

put_executions

put_lineage_subgraph

put_parent_contexts

`get_artifact_by_type_and_name`

`get_artifact_type`

`get_artifact_types`

`get_artifact_types_by_external_ids`

`get_artifact_types_by_id`

`get_artifacts`

`get_artifacts_and_types_by_artifact_ids`

`get_artifacts_by_context`

`get_artifacts_by_external_ids`

`get_artifacts_by_id`

`get_artifacts_by_type`

`get_artifacts_by_uri`

`get_children_contexts_by_context`

`get_context_by_type_and_name`

`get_context_type`

`get_context_types`

`get_context_types_by_external_ids`

`get_context_types_by_id`

`get_contexts`

`get_contexts_by_artifact`

`get_contexts_by_execution`

`get_contexts_by_external_ids`

`get_contexts_by_id`

`get_contexts_by_type`

`get_events_by_artifact_ids`

`get_events_by_execution_ids`

`get_execution_by_type_and_name`

`get_execution_type`

`get_execution_types`

`get_execution_types_by_external_ids`

`get_execution_types_by_id`

`get_executions`

`get_executions_by_context`

`get_executions_by_external_ids`

`get_executions_by_id`

`get_executions_by_type`

`get_lineage_subgraph`

`get_parent_contexts_by_context`

`put_artifact_type`

`put_artifacts`

`put_attributions_and_associations`

`put_context_type`

`put_contexts`

`put_events`

`put_execution`

`put_execution_type`

`put_executions`

`put_lineage_subgraph`

`put_parent_contexts`