tensorflow::serving::BasicManager

This is an abstract class.

#include <basic_manager.h>

Helps manage the lifecycle of servables including loading, serving and unloading them.

Summary

The manager accepts servables in the form of Loaders.

We start managing a servable through one of the ManageServable* methods. You can go on to load the servable after this by calling LoadServable(). Loading will also make the servable available to serve. Once you decide to unload it, you can call UnloadServable() on it, which will make it unavailable to serve, then unload the servable.

Servables are retained until StopManagingServable() is called. This allows a higher level manager with more information to decide when it's safe to forget about a servable.

BasicManager tracks the resources (e.g. RAM) used by loaded servables, and only allows loading new servables that fit within the overall resource pool.

BasicManager can be configured to use a thread-pool to do it's load and unloads. This makes the LoadServable() and UnloadServable() methods schedule the load/unloads rather than executing them synchronously. If there are more pending load/unloads than threads in the thread pool, they are processed in FIFO order.

In the presence of loaders that over-estimate their servables' resource needs and/or only bind their servables' resources to device instances, load/unload concurrency can be reduced below the thread-pool size. That is because we may have to wait for one servable's load/unload to finish to pin down the resource availability for loading another servable.

REQUIRES:

  1. Order of method calls - ManageServable() (and variants) -> LoadServable() -> UnloadServable() -> StopManagingServable().
  2. Do not schedule concurrent load and unloads of the same servable.
  3. Do not call load or unload multiple times on the same servable.

This class is thread-safe.

Example usage: 

const ServableId id = {kServableName, 0};
std::unique_ptr<Loader> loader = ...;
...
BasicManager manager;
TF_CHECK_OK(manager.ManageServable(
  CreateServableData(id, std::move(loader))));
TF_CHECK_OK(manager.LoadServable(id));

...
TF_CHECK_OK(manager.GetServableHandle(
    ServableRequest::Latest(kServableName), &handle));
...

TF_CHECK_OK(manager.UnloadServable(id));
TF_CHECK_OK(manager.StopManagingServable(id));

Inheritance

Inherits from: tensorflow::serving::Manager

Constructors and Destructors
~BasicManager()
If configured to use a load/unload thread-pool, waits until all scheduled loads and unloads have finished and then destroys the set of threads.

Public types
DoneCallback using
std::function< void(const Status &status)>
Callback called at the end of {Load,Unload}Servable().
PreLoadHook using
std::function< void(const ServableId &)>

Friend classes
test_util::BasicManagerTestAccess
friend class

Public static functions
Create(Options options, std::unique_ptr< BasicManager > *manager)
Status

Public functions
CancelLoadServableRetry(const ServableId & id)
void
Cancels retrying the servable load during LoadServable().
GetAdditionalServableState(const ServableId & id)
T *
GetAvailableUntypedServableHandles() const override
virtual std::map< ServableId, std::unique_ptr< UntypedServableHandle > >
GetManagedServableNames() const
std::vector< string >
GetManagedServableStateSnapshot(const ServableId & id)
absl::optional< ServableStateSnapshot< T > >
GetManagedServableStateSnapshots(const string & servable_name) const
std::vector< ServableStateSnapshot< T > >
GetUntypedServableHandle(const ServableRequest & request, std::unique_ptr< UntypedServableHandle > *untyped_handle) override
virtual Status
ListAvailableServableIds() const override
virtual std::vector< ServableId >
Gets a list of all available servable ids, i.e.
LoadServable(const ServableId & id, DoneCallback done_callback)
void
Loads the servable with this id, and updates the serving map too.
ManageServable(ServableData< std::unique_ptr< Loader >> servable)
Status
Starts managing the servable.
ManageServableWithAdditionalState(ServableData< std::unique_ptr< Loader >> servable, std::unique_ptr< T > additional_state)
Status
Similar to the above method, but callers, usually other managers built on top of this one, can associate additional state with the servable.
StopManagingServable(const ServableId & id)
Status
Tells the manager to stop managing this servable.
UnloadServable(const ServableId & id, DoneCallback done_callback)
void
Unloads the servable with this id, and updates the serving map too.

Structs
tensorflow::serving::BasicManager::Options

Config options and pluggable objects that will be used by the BasicManager.

Public types

DoneCallback

std::function< void(const Status &status)> DoneCallback

Callback called at the end of {Load,Unload}Servable().

We pass in the status of the operation to the callback.

PreLoadHook

std::function< void(const ServableId &)> PreLoadHook

Friend classes

test_util::BasicManagerTestAccess

friend class test_util::BasicManagerTestAccess

Public static functions

Create

Status Create(
  Options options,
  std::unique_ptr< BasicManager > *manager
)

Public functions

CancelLoadServableRetry

void CancelLoadServableRetry(
  const ServableId & id
)

Cancels retrying the servable load during LoadServable().

Does nothing if the servable isn't managed.

If the retries are cancelled, the servable goes into a state dependent on the last Load() called on it. If the last Load() was successful, it will be in state kReady, else in kError.

GetAdditionalServableState

T * GetAdditionalServableState(
  const ServableId & id
)

REQUIRES: This manager should have been managing this servable already, else we return nullptr.

Details
Returns
the additional state for the servable. Returns nullptr if there is no additional state setup or if there is a type mismatch between what was setup and what is being asked for.

GetAvailableUntypedServableHandles

virtual std::map< ServableId, std::unique_ptr< UntypedServableHandle > > GetAvailableUntypedServableHandles() const override

GetManagedServableNames

std::vector< string > GetManagedServableNames() const

Details
Returns
the names of all the servables managed by this manager. The names will be duplicate-free and not in any particular order.

GetManagedServableStateSnapshot

absl::optional< ServableStateSnapshot< T > > GetManagedServableStateSnapshot(
  const ServableId & id
)

REQUIRES: This manager should have been managing this servable already, else we return nullopt.

Details
Returns
the state snapshot of a particular servable-id managed by this manager if available.

GetManagedServableStateSnapshots

std::vector< ServableStateSnapshot< T > > GetManagedServableStateSnapshots(
  const string & servable_name
) const

T is the additional-state type, if any.

Details
Returns
the state snapshots of all the servables of a particular stream, managed by this manager.

GetUntypedServableHandle

virtual Status GetUntypedServableHandle(
  const ServableRequest & request,
  std::unique_ptr< UntypedServableHandle > *untyped_handle
) override

ListAvailableServableIds

virtual std::vector< ServableId > ListAvailableServableIds() const override

Gets a list of all available servable ids, i.e.

each of these can be retrieved using GetServableHandle.

LoadServable

void LoadServable(
  const ServableId & id,
  DoneCallback done_callback
)

Loads the servable with this id, and updates the serving map too.

Calls done_callback with ok iff the servable was loaded successfully, else returns an error status.

If using a thread-pool, this method transitions the servable harness to kLoading state, schedules the load and returns, otherwise it completes the load before returning.

REQUIRES: This manager should have been managing this servable already, for it to be loaded, else we call done_callback with an error status. Do not call this multiple times on the same servable. Only one of those will succeed and the rest will fail with an error status.

ManageServable

Status ManageServable(
  ServableData< std::unique_ptr< Loader >> servable
)

Starts managing the servable.

Returns an error if given a servable that is already being managed.

If servable is in an error state, this method does not return an error. Instead, the manager accepts the servable, puts it in state kError (with a notification sent to the event bus), and then immediately stops managing it. This behavior facilitates uniform handling of errors that occur in sources (e.g. invalid file path to servable data) and ones that occur in the manager (e.g. insufficient resources to load servable).

ManageServableWithAdditionalState

Status ManageServableWithAdditionalState(
  ServableData< std::unique_ptr< Loader >> servable,
  std::unique_ptr< T > additional_state
)

Similar to the above method, but callers, usually other managers built on top of this one, can associate additional state with the servable.

Additional state may be ACL or lifetime metadata for that servable. The ownership of the state is transferred to this class.

StopManagingServable

Status StopManagingServable(
  const ServableId & id
)

Tells the manager to stop managing this servable.

Requires that the servable is currently being managed and that its state is one of {kNew, kError, kDisabled}.

UnloadServable

void UnloadServable(
  const ServableId & id,
  DoneCallback done_callback
)

Unloads the servable with this id, and updates the serving map too.

Calls done_callback with ok iff the servable was unloaded successfully, else returns an error status.

If using a thread-pool, this method transitions the servable harness to kQuiescing state, schedules the unload and returns, otherwise it completes the unload before returning.

REQUIRES: This manager should have loaded and made this servable available, for it to be unloaded, else calls done_callback with an error status. Do not call this multiple times on the same servable. Only one of those will succeed and the rest will fail with an error status.

~BasicManager

 ~BasicManager() override

If configured to use a load/unload thread-pool, waits until all scheduled loads and unloads have finished and then destroys the set of threads.