Core Components

bentoml.Service

class bentoml.Service(name: str, *, runners: list[bentoml._internal.runner.runner.AbstractRunner] | None = None, models: list[bentoml._internal.models.model.Model] | None = None)

The service definition is the manifestation of the Service Oriented Architecture and the core building block in BentoML where users define the service runtime architecture and model serving logic.

A BentoML service is defined via instantiate this Service class. When creating a Service instance, user must provide a Service name and list of runners that are required by this Service. The instance can then be used to define InferenceAPIs via the api decorator.

add_asgi_middleware(middleware_cls: t.Type[ext.AsgiMiddleware], **options: t.Any) → None

api(input: IODescriptor[IOType], output: IODescriptor[IOType]) → _inference_api_wrapper[IOType]

api(input: IODescriptor[IOType], output: IODescriptor[IOType], *, route: str = None) → _inference_api_wrapper[IOType]

Decorator for adding InferenceAPI to this service

apis: t.Dict[str, InferenceAPI[t.Any]]

mount_asgi_app(app: ext.ASGIApp, path: str = '/', name: t.Optional[str] = None) → None

mount_wsgi_app(app: ext.WSGIApp, path: str = '/', name: t.Optional[str] = None) → None

runners: t.List[Runner]

bentoml.load(bento_identifier: str | Tag | Bento, working_dir: t.Optional[str] = None, standalone_load: bool = False) → Service

Load a Service instance by the bento_identifier

Parameters:

• bento_identifier – target Service to import or Bento to load

• working_dir – when importing from service, set the working_dir

• standalone_load – treat target Service as standalone. This will change global current working directory and global model store.

Returns:

The loaded bentoml.Service instance.

The argument bento_identifier can be one of the following forms:

• Tag pointing to a Bento in local Bento store under BENTOML_HOME/bentos

• File path to a Bento directory

• “import_str” for loading a service instance from the working_dir

Example load from Bento usage:

# load from local bento store
load("FraudDetector:latest")
load("FraudDetector:4tht2icroji6zput")

# load from bento directory
load("~/bentoml/bentos/iris_classifier/4tht2icroji6zput")

Example load from working directory by “import_str” usage:

# When multiple service defined in the same module
load("fraud_detector:svc_a")
load("fraud_detector:svc_b")

# Find svc by Python module name or file path
load("fraud_detector:svc")
load("fraud_detector.py:svc")
load("foo.bar.fraud_detector:svc")
load("./def/abc/fraud_detector.py:svc")

# When there's only one Service instance in the target module, the attributes
# part in the svc_import_path can be omitted
load("fraud_detector.py")
load("fraud_detector")

Limitations when standalone_load=False:

• Models used in the Service being imported, if not accessed during module import, must be presented in the global model store

• Files required for the Service to run, if not accessed during module import, must be presented in the current working directory

Todo

Add docstring to the following classes/functions

bentoml.build

bentoml.bentos.build(service: str, *, name: str | None = None, labels: dict[str, str] | None = None, description: str | None = None, include: t.List[str] | None = None, exclude: t.List[str] | None = None, docker: DockerOptions | dict[str, t.Any] | None = None, python: PythonOptions | dict[str, t.Any] | None = None, conda: CondaOptions | dict[str, t.Any] | None = None, models: t.List[ModelSpec | str | dict[str, t.Any]] | None = None, version: str | None = None, build_ctx: str | None = None, _bento_store: BentoStore = <simple_di.providers.SingletonFactory object>) → Bento

User-facing API for building a Bento. The available build options are identical to the keys of a valid ‘bentofile.yaml’ file.

This API will not respect any ‘bentofile.yaml’ files. Build options should instead be provided via function call parameters.

Parameters:

• service – import str for finding the bentoml.Service instance build target

• labels – optional immutable labels for carrying contextual info

• description – optional description string in markdown format

• include – list of file paths and patterns specifying files to include in Bento, default is all files under build_ctx, beside the ones excluded from the exclude parameter or a .bentoignore file for a given directory

• exclude – list of file paths and patterns to exclude from the final Bento archive

• docker – dictionary for configuring Bento’s containerization process, see details in bentoml._internal.bento.build_config.DockerOptions

• python – dictionary for configuring Bento’s python dependencies, see details in bentoml._internal.bento.build_config.PythonOptions

• conda – dictionary for configuring Bento’s conda dependencies, see details in bentoml._internal.bento.build_config.CondaOptions

• version – Override the default auto generated version str

• build_ctx – Build context directory, when used as

• _bento_store – save Bento created to this BentoStore

Returns:

a Bento instance representing the materialized Bento saved in BentoStore

Return type:

Bento

Example

import bentoml

bentoml.build(
    service="fraud_detector.py:svc",
    version="any_version_label",  # override default version generator
    description=open("README.md").read(),
    include=['*'],
    exclude=[], # files to exclude can also be specified with a .bentoignore file
    labels={
        "foo": "bar",
        "team": "abc"
    },
    python=dict(
        packages=["tensorflow", "numpy"],
        # requirements_txt="./requirements.txt",
        index_url="http://<api token>:@mycompany.com/pypi/simple",
        trusted_host=["mycompany.com"],
        find_links=['thirdparty..'],
        extra_index_url=["..."],
        pip_args="ANY ADDITIONAL PIP INSTALL ARGS",
        wheels=["./wheels/*"],
        lock_packages=True,
    ),
    docker=dict(
        distro="amazonlinux2",
        setup_script="setup_docker_container.sh",
        python_version="3.8",
    ),
)

bentoml.bentos.build_bentofile(bentofile: str = 'bentofile.yaml', *, version: str | None = None, build_ctx: str | None = None, _bento_store: BentoStore = <simple_di.providers.SingletonFactory object>) → Bento

Build a Bento base on options specified in a bentofile.yaml file.

By default, this function will look for a bentofile.yaml file in current working directory.

Parameters:

• bentofile – The file path to build config yaml file

• version – Override the default auto generated version str

• build_ctx – Build context directory, when used as

• _bento_store – save Bento created to this BentoStore

bentoml.bentos.containerize(bento_tag: Tag | str, **kwargs: Any) → bool

DEPRECATED: Use bentoml.container.build() instead.

bentoml.Bento

class bentoml.Bento(tag: Tag, bento_fs: FS, info: BentoInfo)

property doc: str

property info: BentoInfo

property path: str

path_of(item: str) → str

property tag: Tag

bentoml.Runner

class bentoml.Runner(runnable_class: type[bentoml._internal.runner.runnable.Runnable], *, runnable_init_params: dict[str, typing.Any] | None = None, name: str | None = None, scheduling_strategy: type[bentoml._internal.runner.strategy.Strategy] = <class 'bentoml._internal.runner.strategy.DefaultStrategy'>, models: list[bentoml._internal.models.model.Model] | None = None, max_batch_size: int | None = None, max_latency_ms: int | None = None, method_configs: dict[str, dict[str, int]] | None = None, embedded: bool = False)

bentoml.Runnable

class bentoml.Runnable

static method(meth: t.Callable[t.Concatenate[T, P], R], *, batchable: bool = False, batch_dim: tuple[int, int] | int = 0, input_spec: AnyType | tuple[AnyType, ...] | None = None, output_spec: AnyType | None = None) → RunnableMethod[T, P, R]

static method(meth: None = None, *, batchable: bool = False, batch_dim: tuple[int, int] | int = 0, input_spec: AnyType | tuple[AnyType, ...] | None = None, output_spec: AnyType | None = None) → t.Callable[[t.Callable[t.Concatenate[T, P], R]], RunnableMethod[T, P, R]]

Tag

class bentoml.Tag(name: str, version: str | None = None)

Model

class bentoml.Model(tag: Tag, model_fs: FS, info: ModelInfo, custom_objects: dict[str, Any] | None = None, *, _internal: bool = False)

property info: ModelInfo

property path: str

path_of(item: str) → str

to_runnable() → t.Type[Runnable]

to_runner(name: str = '', max_batch_size: int | None = None, max_latency_ms: int | None = None, method_configs: dict[str, dict[str, int]] | None = None, embedded: bool = False, scheduling_strategy: type[Strategy] | None = None) → Runner

TODO(chaoyu): add docstring

Parameters:

• name –

• max_batch_size –

• max_latency_ms –

• runnable_method_configs –

Returns:

with_options(**kwargs: Any) → Model

YataiClient

class bentoml.YataiClient