BentoService API

BentoService

class bentoml.BentoService

BentoService is the base component for building prediction services using BentoML.

BentoService provide an abstraction for describing model artifacts and environment dependencies required for a prediction service. And allows users to create inference APIs that defines the inferencing logic and how the underlying model can be served. Each BentoService can contain multiple models and serve multiple inference APIs.

Usage example:

>>>  from bentoml import BentoService, env, api, artifacts
>>>  from bentoml.adapters import DataframeInput
>>>  from bentoml.frameworks.sklearn import SklearnModelArtifact
>>>
>>>  @artifacts([SklearnModelArtifact('clf')])
>>>  @env(pip_packages=["scikit-learn"])
>>>  class MyMLService(BentoService):
>>>
>>>     @api(input=DataframeInput(), batch=True)
>>>     def predict(self, df):
>>>         return self.artifacts.clf.predict(df)
>>>
>>>  if __name__ == "__main__":
>>>     bento_service = MyMLService()
>>>     bento_service.pack('clf', trained_classifier_model)
>>>     bento_service.save_to_dir('/bentoml_bundles')
property artifacts

Returns the ArtifactCollection instance specified with this BentoService class

Returns

A dictionary of packed artifacts from the artifact name to the BentoServiceArtifact instance

Return type

artifacts(ArtifactCollection)

get_inference_api(api_name)

Find the inference API in this BentoService with a specific name.

When the api_name is None, this returns the first Inference API found in the self.inference_apis list.

Parameters

api_name – the target Inference API’s name

Returns

property inference_apis

Return a list of user defined API functions

Returns

List of Inference API objects

Return type

list(InferenceAPI)

pack(*args, **kwargs)

Deprecated: Legacy BentoService#pack class method, no longer supported

save(yatai_url=None, version=None, labels=None)

Save and register this BentoService via BentoML’s built-in model management system. BentoML by default keeps track of all the SavedBundle’s files and metadata in local file system under the $BENTOML_HOME(~/bentoml) directory. Users can also configure BentoML to save their BentoService to a shared Database and cloud object storage such as AWS S3.

Parameters

  • yatai_url – optional - URL path to Yatai server

  • version – optional - save with version override

  • labels – optional - labels dictionary

Returns

saved_path: file path to where the BentoService is saved

save_to_dir(path, version=None)

Save this BentoService along with all its artifacts, source code and dependencies to target file path, assuming path exist and empty. If target path is not empty, this call may override existing files in the given path.

Parameters

  • (str) (path) – Destination of where the bento service will be saved

  • version – optional - save with version override

set_version(version_str=None)

Set the version of this BentoService instance. Once the version is set explicitly via set_version, the self.versioneer method will no longer be invoked when saving this BentoService.

property tag

Bento tag is simply putting its name and version together, separated by a colon tag is mostly used in Yatai model management related APIs and operations

property version

Return the version of this BentoService. If the version of this BentoService has not been set explicitly via self.set_version, a new version will be generated with the self.versioneer method. User can customize this version str either by setting the version with self.set_version before a save call, or override the self.versioneer method to customize the version str generator logic.

For BentoService loaded from a saved bundle, this will simply return the version information found in the saved bundle.

Returns

BentoService version str

versioneer()

Function used to generate a new version string when saving a new BentoService bundle. User can also override this function to get a customized version format

api

bentoml.api(*args, input: Optional[bentoml.adapters.base_input.BaseInputAdapter] = None, output: Optional[bentoml.adapters.base_output.BaseOutputAdapter] = None, api_name: Optional[str] = None, route: Optional[str] = None, api_doc: Optional[str] = None, mb_max_batch_size: int = 4000, mb_max_latency: int = 20000, batch=False, **kwargs)

A decorator exposed as bentoml.api for defining Inference API in a BentoService class.

Parameters

  • input – InputAdapter instance of the inference API

  • output – OutputAdapter instance of the inference API

  • api_name – API name, default to the user-defined callback function’s function name

  • route – Specify HTTP URL route of this inference API. By default, api.name is used as the route. This parameter can be used for customizing the URL route, e.g. route=”/api/v2/model_a/predict” Default: None (the same as api_name)

  • api_doc – user-facing documentation of the inference API. default to the user-defined callback function’s docstring

  • mb_max_batch_size – The maximum size of requests batch accepted by this inference API. This parameter governs the throughput/latency trade off, and avoids having large batches that exceed some resource constraint (e.g. GPU memory to hold the entire batch’s data). Default: 1000.

  • mb_max_latency – The latency goal of this inference API in milliseconds. Default: 10000.

Example usage:

>>> from bentoml import BentoService, api
>>> from bentoml.adapters import JsonInput, DataframeInput
>>>
>>> class FraudDetectionAndIdentityService(BentoService):
>>>
>>>     @api(input=JsonInput(), batch=True)
>>>     def fraud_detect(self, json_list):
>>>         # user-defined callback function that process inference requests
>>>
>>>     @api(input=DataframeInput(input_json_orient='records'), batch=True)
>>>     def identity(self, df):
>>>         # user-defined callback function that process inference requests

env

bentoml.env(pip_dependencies: Optional[List[str]] = None, pip_packages: Optional[List[str]] = None, pip_index_url: Optional[str] = None, pip_trusted_host: Optional[str] = None, pip_extra_index_url: Optional[str] = None, auto_pip_dependencies: Optional[bool] = None, infer_pip_packages: bool = False, requirements_txt_file: Optional[str] = None, conda_channels: Optional[List[str]] = None, conda_overwrite_channels: bool = False, conda_override_channels: bool = False, conda_dependencies: Optional[List[str]] = None, conda_env_yml_file: Optional[str] = None, setup_sh: Optional[str] = None, docker_base_image: Optional[str] = None, zipimport_archives: Optional[List[str]] = None)

Define environment and dependencies required for the BentoService being created

Parameters

  • pip_packages: – list of pip_packages required: or with specified version {package_name}=={package_version}

  • by package name (specified) – or with specified version {package_name}=={package_version}

  • pip_dependencies – same as pip_packages but deprecated

  • pip_index_url – passing down to pip install –index-url option

  • pip_trusted_host – passing down to pip install –trusted-host option

  • pip_extra_index_url – passing down to pip install –extra-index-url option

  • infer_pip_packages – whether to automatically find all the required pip dependencies and pin their version

  • auto_pip_dependencies – same as infer_pip_packages but deprecated

  • requirements_txt_file – path to the requirements.txt where pip dependencies are explicitly specified, with ideally pinned versions

  • conda_channels – list of extra conda channels other than default channels to be used. This is equivalent to passing the –channels to conda commands

  • conda_override_channels – ensures that conda searches only your specified channel and no other channels, such as default channels. This is equivalent to passing the –override-channels option to conda commands, or adding nodefaults to the channels in the environment.yml

  • conda_overwrite_channels – aliases to override_channels

  • conda_dependencies – list of conda dependencies required

  • conda_env_yml_file – use a pre-defined conda environment yml file

  • setup_sh – user defined setup bash script, it is executed in docker build time

  • docker_base_image – used for customizing the docker container image built with BentoML saved bundle. Base image must either have both bash and conda installed; or have bash, pip, python installed, in which case the user is required to ensure the python version matches the BentoService bundle

  • zipimport_archives – list of zipimport archives paths relative to the module path

artifacts

bentoml.artifacts(artifacts: List[bentoml.service.artifacts.BentoServiceArtifact])

Define artifacts required to be bundled with a BentoService

Parameters

artifacts (list(bentoml.artifact.BentoServiceArtifact)) – A list of desired artifacts required by this BentoService

ver

bentoml.ver(major, minor)

Decorator for specifying the version of a custom BentoService.

Parameters

  • major (int) – Major version number for Bento Service

  • minor (int) – Minor version number for Bento Service

BentoML uses semantic versioning for BentoService distribution:

  • MAJOR is incremented when you make breaking API changes

  • MINOR is incremented when you add new functionality without breaking the existing API or functionality

  • PATCH is incremented when you make backwards-compatible bug fixes

‘Patch’ is provided(or auto generated) when calling BentoService#save, while ‘Major’ and ‘Minor’ can be defined with ‘@ver’ decorator

>>>  from bentoml import ver, artifacts
>>>  from bentoml.service.artifacts.common import PickleArtifact
>>>
>>>  @ver(major=1, minor=4)
>>>  @artifacts([PickleArtifact('model')])
>>>  class MyMLService(BentoService):
>>>     pass
>>>
>>>  svc = MyMLService()
>>>  svc.pack("model", trained_classifier)
>>>  svc.set_version("2019-08.iteration20")
>>>  svc.save()
>>>  # The final produced BentoService bundle will have version:
>>>  # "1.4.2019-08.iteration20"

save

bentoml.save(bento_service, base_path=None, version=None, labels=None)

Save and register the given BentoService via BentoML’s built-in model management system. BentoML by default keeps track of all the SavedBundle’s files and metadata in local file system under the $BENTOML_HOME(~/bentoml) directory. Users can also configure BentoML to save their BentoService to a shared Database and cloud object storage such as AWS S3.

Parameters

  • bento_service – target BentoService instance to be saved

  • base_path – optional - override repository base path

  • version – optional - save with version override

  • labels – optional - user defined labels

Returns

saved_path: file path to where the BentoService is saved

load

bentoml.load(bundle_path)

Load bento service from local file path or s3 path

Parameters

bundle_path (str) – The path that contains saved BentoService bundle, supporting both local file path and s3 path

Returns

a loaded BentoService instance

Return type

bentoml.service.BentoService

save_to_dir

bentoml.save_to_dir(bento_service: BentoService, path: str, version: str = None, silent: bool = False) → None

Save given BentoService along with all its artifacts, source code and dependencies to target file path, assuming path exist and empty. If target path is not empty, this call may override existing files in the given path.

Parameters

  • bento_service (BentoService) – a BentoService instance

  • path (str) – Destination of where the bento service will be saved. The destination can be local path or remote path. The remote path supports both AWS S3(‘s3://bucket/path’) and Google Cloud Storage(‘gs://bucket/path’).

  • version (str, optional) – Override the service version with given version string

  • silent (bool, optional) – whether to hide the log message showing target save path

load_from_dir

bentoml.load_from_dir(bundle_path)

Load bento service from local file path or s3 path

Parameters

bundle_path (str) – The path that contains saved BentoService bundle, supporting both local file path and s3 path

Returns

a loaded BentoService instance

Return type

bentoml.service.BentoService