tensorflow::serving::ServableStateMonitor

#include <servable_state_monitor.h>

A utility that listens to an EventBus<ServableState>, and keeps track of the state of each servable mentioned on the bus.

Summary

The intended use case is to track the states of servables in a Manager.

Offers an interface for querying the servable states. It may be useful as the basis for dashboards, as well as for testing a manager.

IMPORTANT: You must create this monitor before arranging for events to be published on the event bus, e.g. giving the event bus to a Manager.

Constructors and Destructors
ServableStateMonitor(EventBus< ServableState > *bus, const Options & options)
~ServableStateMonitor()

Public types
BoundedLog using
std::deque< ServableStateAndTime >
NotifyFn using
std::function< void(const ServableState &)>
ServableStateNotifierFn using
std::function< void(bool reached_goal_state, const std::map< ServableId, ServableState::ManagerState > &states_reached)>
Notifies when all of the servables have reached the 'goal_state'.

Public functions
ForgetUnloadedServableStates()
void
Removes all servable versions from the ServableMap whose states have transitioned to kEnd.
GetAllServableStates()
ServableMap
Returns the current states of all tracked versions of all servables.
GetAvailableServableStates()
ServableSet
GetBoundedLog()
BoundedLog
Returns the current bounded log of handled servable state events.
GetLiveServableStates()
ServableMap
Returns the current states of all versions of all servables which have not transitioned to state ServableState::ManagerState::kEnd.
GetState(const ServableId & servable_id)
absl::optional< ServableState >
Returns the current state of one servable, or nullopt if that servable is not being tracked.
GetStateAndTime(const ServableId & servable_id)
absl::optional< ServableStateAndTime >
Returns the current state and time of one servable, or nullopt if that servable is not being tracked.
GetVersionStates(const string & servable_name)
VersionMap
Returns the current states of all tracked versions of the given servable, if any.
Notify(const NotifyFn & notify_fn)
void
NotifyWhenServablesReachState(const std::vector< ServableRequest > & servables, ServableState::ManagerState goal_state, const ServableStateNotifierFn & notifier_fn)
void
WaitUntilServablesReachState(const std::vector< ServableRequest > & servables, ServableState::ManagerState goal_state, std::map< ServableId, ServableState::ManagerState > *states_reached) TF_MUST_USE_RESULT
bool
WaitUntilServablesReachStateWithTimeout(const std::vector< ServableRequest > & servables, ServableState::ManagerState goal_state, absl::Duration timeout, std::map< ServableId, ServableState::ManagerState > *states_reached) TF_MUST_USE_RESULT
bool
Similar to NotifyWhenServablesReachState(...), but instead of notifying, we wait until the 'goal_state' or kEnd is reached.

Public types

BoundedLog

std::deque< ServableStateAndTime > BoundedLog

NotifyFn

std::function< void(const ServableState &)> NotifyFn

ServableStateNotifierFn

std::function< void(bool reached_goal_state, const std::map< ServableId, ServableState::ManagerState > &states_reached)> ServableStateNotifierFn

Notifies when all of the servables have reached the 'goal_state'.

Servables can be specified in two ways:

  1. As specific versions of a servable stream name. In this case, we check whether the specific version has reached the 'goal_state' or kEnd.
  2. As latest versions, in which case any version for a servable stream name will be matched against the 'goal_state' or kEnd.

We call the 'notifier_fn' when both conditions are true -

  1. All of the specific servable requests have either reached the 'goal_state' or kEnd.
  2. All of the latest servable requests have reached 'goal_state' or kEnd. The 'notifier_fn' will be called only once, and not repeatedly.

The 'reached_goal_state' argument is set as true iff all of the specific servables have reached 'goal_state'. So callers should verify that 'reached_goal_state' is true in the 'notifier_fn'.

The 'states_reached' argument is populated with the servable's id and the state it reached. The state would be 'goal_state' if 'reached_goal_state' is true, else it will contain one or more servables in kEnd state. For latest servable requests, the servable id will be the id of the servable in the stream which reached the state.

Public functions

ForgetUnloadedServableStates

void ForgetUnloadedServableStates()

Removes all servable versions from the ServableMap whose states have transitioned to kEnd.

GetAllServableStates

ServableMap GetAllServableStates()

Returns the current states of all tracked versions of all servables.

GetAvailableServableStates

ServableSet GetAvailableServableStates()

GetBoundedLog

BoundedLog GetBoundedLog()

Returns the current bounded log of handled servable state events.

GetLiveServableStates

ServableMap GetLiveServableStates()

Returns the current states of all versions of all servables which have not transitioned to state ServableState::ManagerState::kEnd.

GetState

absl::optional< ServableState > GetState(
  const ServableId & servable_id
)

Returns the current state of one servable, or nullopt if that servable is not being tracked.

GetStateAndTime

absl::optional< ServableStateAndTime > GetStateAndTime(
  const ServableId & servable_id
)

Returns the current state and time of one servable, or nullopt if that servable is not being tracked.

GetVersionStates

VersionMap GetVersionStates(
  const string & servable_name
)

Returns the current states of all tracked versions of the given servable, if any.

Notify

void Notify(
  const NotifyFn & notify_fn
)

NotifyWhenServablesReachState

void NotifyWhenServablesReachState(
  const std::vector< ServableRequest > & servables,
  ServableState::ManagerState goal_state,
  const ServableStateNotifierFn & notifier_fn
)

ServableStateMonitor

 ServableStateMonitor(
  EventBus< ServableState > *bus,
  const Options & options
)

WaitUntilServablesReachState

bool WaitUntilServablesReachState(
  const std::vector< ServableRequest > & servables,
  ServableState::ManagerState goal_state,
  std::map< ServableId, ServableState::ManagerState > *states_reached
) TF_MUST_USE_RESULT

WaitUntilServablesReachStateWithTimeout

bool WaitUntilServablesReachStateWithTimeout(
  const std::vector< ServableRequest > & servables,
  ServableState::ManagerState goal_state,
  absl::Duration timeout,
  std::map< ServableId, ServableState::ManagerState > *states_reached
) TF_MUST_USE_RESULT

Similar to NotifyWhenServablesReachState(...), but instead of notifying, we wait until the 'goal_state' or kEnd is reached.

To understand the return value and the return parameter 'states_reached', please read the documentation on NotifyWhenServablesReachState(...). WaitUntilServablesReachStateWithTimeout and WaitUntilServablesReachState perform the same function, but the former has a timeout while the latter waits indefinitely.

~ServableStateMonitor

virtual  ~ServableStateMonitor()