Picklable Model

Users can now save any given python method or object as a loadable model in BentoML with the following API: load, save, and load_runner:

import bentoml

 class MyPicklableModel:
     def predict(self, some_integer: int):
         return some_integer ** 2

# `save` a given model or function
model = MyPicklableModel()
tag = bentoml.picklable_model.save('mypicklablemodel', model, batch=False, method="predict")

# retrieve metadata with `bentoml.models.get`:
metadata = bentoml.models.get(tag)

# load the model back:
loaded = bentoml.picklable_model.load("mypicklablemodel:latest")

# Run a given model under `Runner` abstraction with `load_runner`
runner = bentoml.picklable_model.load_runner(tag)
runner.run(7)

Users can also save models which take advantage of BentoML’s adaptive batching capability using the “batch” option

import bentoml

# Model which takes in a batch of values
class MyPicklableModelBatch:
    def predict(self, some_integers: t.List[int]):
        return list(map(lambda x: x ** 2, some_integers))

model = MyPicklableModelBatch()

# Use the option "batch" in order to save a model which is passed a batch of values to be evaluated
tag = bentoml.picklable_model.save('mypicklablemodel', model, batch=True, method="predict")
runner = bentoml.picklable_model.load_runner(tag)

# runner is sent an array of values in batch mode for inference
runner.run_batch([7, 6, 8])

Note

You can find more examples for picklable-model in our gallery repo.

bentoml.picklable_model.save(name, obj, *, labels=None, custom_objects=None, metadata=None)

Save a model instance to BentoML modelstore.

Parameters

  • name (str) – Name for given model instance. This should pass Python identifier check.

  • obj (Any) – Instance of an object to be saved.

  • labels (Dict[str, str], optional, default to None) – user-defined labels for managing models, e.g. team=nlp, stage=dev

  • custom_objects (Dict[str, Any]], optional, default to None) – user-defined additional python objects to be saved alongside the model, e.g. a tokenizer instance, preprocessor function, model configuration json

  • metadata (Dict[str, Any], optional, default to None) – Custom metadata for given model.

  • model_store (ModelStore, default to BentoMLContainer.model_store) – BentoML modelstore, provided by DI Container.

Returns

A tag with a format name:version where name is the user-defined model’s name, and a generated version by BentoML.

Return type

Tag

Examples:

import bentoml

class MyCoolModel:
    def predict(self, some_integer: int):
        return some_integer**2

model_to_save = MyCoolModel();
tag_info = bentoml.picklable_model.save("test_pickle_model", model_to_save)
runner = bentoml.picklable_model.load_runner(tag_info)
runner.run(3)
bentoml.picklable_model.load(tag, model_store=<simple_di.providers.SingletonFactory object>)

Load a model from BentoML local modelstore with given name.

Parameters

  • tag (Union[str, Tag]) – Tag of a saved model in BentoML local modelstore.

  • model_store (ModelStore, default to BentoMLContainer.model_store) – BentoML modelstore, provided by DI Container.

Returns

obj: an instance of :obj: model from BentoML modelstore.

Return type

Any

Examples:

import bentoml

unpickled_model = bentoml.picklable_model.load('my_model:latest')
bentoml.picklable_model.load_runner(tag, *, name=None, method_name='__call__', batch=False)

Runner represents a unit of serving logic that can be scaled horizontally to maximize throughput. bentoml.picklable_model.load_runner() implements a Runner class that wraps the commands that dump and load a pickled object, which optimizes it for the BentoML runtime.

Parameters

  • tag (Union[str, Tag]) – Tag of a saved model in BentoML local modelstore..

  • method_name (str) – Method to call on the pickled object

  • batch (bool) – Determines whether the model supports batching

Returns

Runner instances for the target bentoml.picklable_model model

Return type

Runner

Examples:

import bentoml

runner = bentoml.picklable_model.load_runner("my_model:latest")
runner.run([[1,2,3,4]])