Frameworks

Here are the all of the supported ML frameworks for BentoML. You can find the official BentoML example projects in the bentoml/gallery repository, group by the ML training frameworks used in the project.

You can download the examples below and run them on your computer. Links to run them on Google Colab are also available, although some of the features demoed in the notebooks does not work in the Colab environment due to its limitations, including running the BentoML API model server, building docker image or creating cloud deployment.

Scikit-Learn

Example Projects:

class bentoml.frameworks.sklearn.SklearnModelArtifact(*args, **kwargs)

Artifact class for saving/loading scikit learn models using sklearn.externals.joblib

Parameters
  • name (str) – Name for the artifact

  • pickle_extension (str) – The extension format for pickled file

Raises

MissingDependencyException – sklearn package is required for SklearnModelArtifact

Example usage:

>>> from sklearn import svm
>>>
>>> model_to_save = svm.SVC(gamma='scale')
>>> # ... training model, etc.
>>>
>>> import bentoml
>>> from bentoml.frameworks.sklearn import SklearnModelArtifact
>>> from bentoml.adapters import DataframeInput
>>>
>>> @bentoml.env(infer_pip_packages=True)
>>> @bentoml.artifacts([SklearnModelArtifact('model')])
>>> class SklearnModelService(bentoml.BentoService):
>>>
>>>     @bentoml.api(input=DataframeInput(), batch=True)
>>>     def predict(self, df):
>>>         result = self.artifacts.model.predict(df)
>>>         return result
>>>
>>> svc = SklearnModelService()
>>>
>>> # Pack directly with sklearn model object
>>> svc.pack('model', model_to_save)
>>> svc.save()

PyTorch

Example Projects:

class bentoml.frameworks.pytorch.PytorchModelArtifact(*args, **kwargs)

Artifact class for saving/loading objects with torch.save and torch.load

Parameters

name (string) – name of the artifact

Raises
  • MissingDependencyException – torch package is required for PytorchModelArtifact

  • InvalidArgument – invalid argument type, model being packed must be instance of torch.nn.Module

Example usage:

>>> import torch.nn as nn
>>>
>>> class Net(nn.Module):
>>>     def __init__(self):
>>>         super(Net, self).__init__()
>>>         ...
>>>
>>>     def forward(self, x):
>>>         ...
>>>
>>> net = Net()
>>> # Train model with data
>>>
>>>
>>> import bentoml
>>> from bentoml.adapters import ImageInput
>>> from bentoml.frameworks.pytorch import PytorchModelArtifact
>>>
>>> @bentoml.env(infer_pip_packages=True)
>>> @bentoml.artifacts([PytorchModelArtifact('net')])
>>> class PytorchModelService(bentoml.BentoService):
>>>
>>>     @bentoml.api(input=ImageInput(), batch=True)
>>>     def predict(self, imgs):
>>>         outputs = self.artifacts.net(imgs)
>>>         return outputs
>>>
>>>
>>> svc = PytorchModelService()
>>>
>>> # Pytorch model can be packed directly.
>>> svc.pack('net', net)
>>>
>>> # Alternatively,
>>>
>>> # Pack a TorchScript Model
>>> # Random input in the format expected by the net
>>> sample_input = ...
>>> traced_net = torch.jit.trace(net, sample_input)
>>> svc.pack('net', traced_net)

Tensorflow 2.0 (Native API)

Example Projects:

class bentoml.frameworks.tensorflow.TensorflowSavedModelArtifact(*args, **kwargs)

Artifact class for saving/loading Tensorflow model in tf.saved_model format

Parameters

name (string) – name of the artifact

Raises

MissingDependencyException – tensorflow package is required for TensorflowSavedModelArtifact

Example usage:

>>> import tensorflow as tf
>>>
>>> # Option 1: custom model with specific method call
>>> class Adder(tf.Module):
>>>     @tf.function(input_signature=[tf.TensorSpec(shape=None, dtype=tf.float32)])
>>>     def add(self, x):
>>>         return x + x + 1.
>>> model_to_save = Adder()
>>> # ... compiling, training, etc
>>>
>>> # Option 2: Sequential model (direct call only)
>>> model_to_save = tf.keras.Sequential([
>>>     tf.keras.layers.Flatten(input_shape=(28, 28)),
>>>     tf.keras.layers.Dense(128, activation='relu'),
>>>     tf.keras.layers.Dense(10, activation='softmax')
>>> ])
>>> # ... compiling, training, etc
>>>
>>> import bentoml
>>> from bentoml.adapters import JsonInput
>>> from bentoml.frameworks.tensorflow import TensorflowSavedModelArtifact
>>>
>>> @bentoml.env(pip_packages=["tensorflow"])
>>> @bentoml.artifacts([TensorflowSavedModelArtifact('model')])
>>> class TfModelService(bentoml.BentoService):
>>>
>>>     @bentoml.api(input=JsonInput(), batch=False)
>>>     def predict(self, json):
>>>         input_data = json['input']
>>>         prediction = self.artifacts.model.add(input_data)
>>>         # prediction = self.artifacts.model(input_data)  # if Sequential mode
>>>         return prediction.numpy()
>>>
>>> svc = TfModelService()
>>>
>>> # Option 1: pack directly with Tensorflow trackable object
>>> svc.pack('model', model_to_save)
>>>
>>> # Option 2: save to file path then pack
>>> tf.saved_model.save(model_to_save, '/tmp/adder/1')
>>> svc.pack('model', '/tmp/adder/1')

Keras (Tensorflow 2.0 as the backend)

Example Projects:

class bentoml.frameworks.keras.KerasModelArtifact(*args, **kwargs)

Artifact class for saving and loading Keras Model

Parameters
  • name (string) – name of the artifact

  • default_custom_objects (dict) – dictionary of Keras custom objects for model

  • store_as_json_and_weights (bool) – flag allowing storage of the Keras model as JSON and weights

Raises
  • MissingDependencyException – keras or tensorflow.keras package is required for KerasModelArtifact

  • InvalidArgument – invalid argument type, model being packed must be instance of keras.engine.network.Network, tf.keras.models.Model, or their aliases

Example usage:

>>> from tensorflow import keras
>>> from tensorflow.keras.models import Sequential
>>> from tensorflow.keras.preprocessing import sequence, text
>>>
>>> model_to_save = Sequential()
>>> # training model
>>> model_to_save.compile(...)
>>> model_to_save.fit(...)
>>>
>>> import bentoml
>>> from bentoml.frameworks.keras import KerasModelArtifact
>>>
>>> @bentoml.env(infer_pip_packages=True)
>>> @bentoml.artifacts([KerasModelArtifact('model')])
>>> class KerasModelService(bentoml.BentoService):
>>>     @bentoml.api(input=JsonInput(), batch=False)
>>>     def predict(self, parsed_json):
>>>         input_data = text.text_to_word_sequence(parsed_json['text'])
>>>         return self.artifacts.model.predict_classes(input_data)
>>>
>>> svc = KerasModelService()
>>> svc.pack('model', model_to_save)

Tensorflow 1.0

Example Projects:

class bentoml.frameworks.tensorflow.TensorflowSavedModelArtifact(*args, **kwargs)

Artifact class for saving/loading Tensorflow model in tf.saved_model format

Parameters

name (string) – name of the artifact

Raises

MissingDependencyException – tensorflow package is required for TensorflowSavedModelArtifact

Example usage:

>>> import tensorflow as tf
>>>
>>> # Option 1: custom model with specific method call
>>> class Adder(tf.Module):
>>>     @tf.function(input_signature=[tf.TensorSpec(shape=None, dtype=tf.float32)])
>>>     def add(self, x):
>>>         return x + x + 1.
>>> model_to_save = Adder()
>>> # ... compiling, training, etc
>>>
>>> # Option 2: Sequential model (direct call only)
>>> model_to_save = tf.keras.Sequential([
>>>     tf.keras.layers.Flatten(input_shape=(28, 28)),
>>>     tf.keras.layers.Dense(128, activation='relu'),
>>>     tf.keras.layers.Dense(10, activation='softmax')
>>> ])
>>> # ... compiling, training, etc
>>>
>>> import bentoml
>>> from bentoml.adapters import JsonInput
>>> from bentoml.frameworks.tensorflow import TensorflowSavedModelArtifact
>>>
>>> @bentoml.env(pip_packages=["tensorflow"])
>>> @bentoml.artifacts([TensorflowSavedModelArtifact('model')])
>>> class TfModelService(bentoml.BentoService):
>>>
>>>     @bentoml.api(input=JsonInput(), batch=False)
>>>     def predict(self, json):
>>>         input_data = json['input']
>>>         prediction = self.artifacts.model.add(input_data)
>>>         # prediction = self.artifacts.model(input_data)  # if Sequential mode
>>>         return prediction.numpy()
>>>
>>> svc = TfModelService()
>>>
>>> # Option 1: pack directly with Tensorflow trackable object
>>> svc.pack('model', model_to_save)
>>>
>>> # Option 2: save to file path then pack
>>> tf.saved_model.save(model_to_save, '/tmp/adder/1')
>>> svc.pack('model', '/tmp/adder/1')

FastAI v1

Example Projects:

class bentoml.frameworks.fastai.Fastai1ModelArtifact(*args, **kwargs)

Artifact class for saving and loading FastAI v1 Model

Parameters

name (str) – Name for the fastai model

Raises
  • MissingDependencyException – Require fastai package to use Fast ai model artifact

  • InvalidArgument – invalid argument type, model being packed must be instance of fastai.basic_train.Learner

Example usage:

>>> from fastai.tabular import *
>>>
>>> # prepare data
>>> data = TabularList.from_df(...)
>>> learn = tabular_learner(data, ...)
>>> # train model
>>>
>>> import bentoml
>>> from bentoml.adapters import DataframeInput
>>> from bentoml.frameworks.fastai import Fastai1ModelArtifact
>>>
>>> @bentoml.artifacts([Fastai1ModelArtifact('model')])
>>> @bentoml.env(infer_pip_packages=True)
>>> class FastaiModelService(bentoml.BentoService):
>>>
>>>     @api(input=DataframeInput(), batch=True)
>>>     def predict(self, df):
>>>         results = []
>>>         for _, row in df.iterrows():
>>>             prediction = self.artifacts.model.predict(row)
>>>             results.append(prediction[0].obj)
>>>         return results
>>>
>>> svc = FastaiModelService()
>>>
>>> # Pack fastai basic_learner directly
>>> svc.pack('model', learn)

FastAI v2

Example Projects:

class bentoml.frameworks.fastai.FastaiModelArtifact(*args, **kwargs)

Artifact class for saving and loading FastAI v2 Model

Parameters

name (str) – Name for the fastai v2 model

Raises
  • MissingDependencyException – Require fastai>=2.0.0 package for Fastai model artifact

  • InvalidArgument – invalid argument type, model being packed must be instance of fastai.basics.Learner

Example usage:

>>> from fastai.vision.learner import cnn_learner
>>>
>>> learner = cnn_learner(...)
>>> # train model
>>> # learner.fit_one_cycle(1)
>>>
>>> import bentoml
>>> from bentoml.frameworks.fastai import FastaiModelArtifact
>>> from bentoml.adapters import FileAdapter
>>>
>>> @bentoml.artifacts([FastaiModelArtifact('learner')])
>>> @bentoml.env(infer_pip_packages=True)
>>> class FastaiImageService(bentoml.BentoService):
>>>
>>>     @bentoml.api(input=FileAdapter(), batch=True)
>>>     def predict(self, files):
>>>         dl = self.artifacts.learner.dls.test_dl(files, ...)
>>>         result = self.artifacts.learner.get_preds(dl=dl)
>>>         return result
>>>
>>> svc = FastaiImageService()
>>> svc.pack('learner', learner)

XGBoost

Example Projects:

class bentoml.frameworks.xgboost.XgboostModelArtifact(*args, **kwargs)

Artifact class for saving and loading Xgboost model

Parameters

name (string) – name of the artifact

Raises
  • ImportError – xgboost package is required for using XgboostModelArtifact

  • TypeError – invalid argument type, model being packed must be instance of xgboost.core.Booster

Example usage:

>>> import xgboost
>>>
>>> # prepare data
>>> params = {... params}
>>> dtrain = xgboost.DMatrix(...)
>>>
>>> # train model
>>> model_to_save = xgboost.train(params=params, dtrain=dtrain)
>>>
>>> import bentoml
>>> from bentoml.frameworks.xgboost import XgboostModelArtifact
>>> from bentoml.adapters import DataframeInput
>>>
>>> @bentoml.env(infer_pip_packages=True)
>>> @bentoml.artifacts(XgboostModelArtifact('model'))
>>> class XGBoostModelService(bentoml.BentoService):
>>>
>>>     @bentoml.api(input=DataframeInput(), batch=True)
>>>     def predict(self, df):
>>>         result = self.artifacts.model.predict(df)
>>>         return result
>>>
>>> svc = XGBoostModelService()
>>> # Pack xgboost model
>>> svc.pack('model', model_to_save)

LightGBM

Example Projects:

class bentoml.frameworks.lightgbm.LightGBMModelArtifact(*args, **kwargs)

Artifact class for saving and loading LightGBM Model

Parameters

name (string) – name of the artifact

Raises
  • MissingDependencyException – lightgbm package is required for using LightGBMModelArtifact

  • InvalidArgument – invalid argument type, model being packed must be instance of lightgbm.Booster

Example usage:

>>> import lightgbm as lgb
>>> # Prepare data
>>> train_data = lgb.Dataset(...)
>>> # train model
>>> model = lgb.train(train_set=train_data, ...)
>>>
>>> import bentoml
>>> from bentoml.frameworks.lightgbm import LightGBMModelArtifact
>>> from bentoml.adapters import DataframeInput
>>>
>>> @bentoml.artifacts([LightGBMModelArtifact('model')])
>>> @bentoml.env(infer_pip_packages=True)
>>> class LgbModelService(bentoml.BentoService):
>>>
>>>     @bentoml.api(input=DataframeInput(), batch=True)
>>>     def predict(self, df):
>>>         return self.artifacts.model.predict(df)
>>>
>>> svc = LgbModelService()
>>> svc.pack('model', model)

FastText

Example Projects:

class bentoml.frameworks.fasttext.FasttextModelArtifact(*args, **kwargs)

Artifact class for saving and loading fasttext models

Parameters

name (str) – Name for the artifact

Raises

MissingDependencyError – fasttext package is required for FasttextModelArtifact

Example usage:

>>> import fasttext
>>> # prepare training data and store to file
>>> training_data_file = 'training-data-file.train'
>>> model = fasttext.train_supervised(input=training_data_file)
>>>
>>> import bentoml
>>> from bentoml.adapters JsonInput
>>> from bentoml.frameworks.fasttext import FasttextModelArtifact
>>>
>>> @bentoml.env(infer_pip_packages=True)
>>> @bentoml.artifacts([FasttextModelArtifact('model')])
>>> class FasttextModelService(bentoml.BentoService):
>>>
>>>     @bentoml.api(input=JsonInput(), batch=False)
>>>     def predict(self, parsed_json):
>>>         # K is the number of labels that successfully were predicted,
>>>         # among all the real labels
>>>         return self.artifacts.model.predict(parsed_json['text'], k=5)
>>>
>>> svc = FasttextModelService()
>>> svc.pack('model', model)

H2O

Example Projects:

class bentoml.frameworks.h2o.H2oModelArtifact(*args, **kwargs)

Artifact class for saving and loading h2o models using h2o.save_model and h2o.load_model

Parameters

name (str) – Name for this h2o artifact..

Raises

MissingDependencyException – h2o package is required to use H2o model artifact

Example usage:

>>> import h2o
>>> h2o.init()
>>>
>>> from h2o.estimators.deeplearning import H2ODeepLearningEstimator
>>> model_to_save = H2ODeepLearningEstimator(...)
>>> # train model with data
>>> data = h2o.import_file(...)
>>> model_to_save.train(...)
>>>
>>> import bentoml
>>> from bentoml.frameworks.h2o import H2oModelArtifact
>>> from bentoml.adapters import DataframeInput
>>>
>>> @bentoml.artifacts([H2oModelArtifact('model')])
>>> @bentoml.env(infer_pip_packages=True)
>>> class H2oModelService(bentoml.BentoService):
>>>
>>>     @bentoml.api(input=DataframeInput(), batch=True)
>>>     def predict(self, df):
>>>         hf = h2o.H2OFrame(df)
>>>         predictions = self.artifacts.model.predict(hf)
>>>         return predictions.as_data_frame()
>>>
>>> svc = H2oModelService()
>>>
>>> svc.pack('model', model_to_save)

CoreML

class bentoml.frameworks.coreml.CoreMLModelArtifact(*args, **kwargs)

Artifact class for saving/loading coreml.models.MLModel objects with coremltools.models.MLModel.save and coremltools.models.MLModel(path).

Parameters

name (string) – name of the artifact

Raises
  • MissingDependencyException – coremltools package required for CoreMLModelArtifact

  • InvalidArgument – invalid argument type, model being packed must be instance of coremltools.models.MLModel

Example usage:

>>> import coremltools as ct
>>> import torch.nn as nn
>>>
>>> class Net(nn.Module):
>>>     def __init__(self):
>>>         super(Net, self).__init__()
>>>         ...
>>>
>>>     def forward(self, x):
>>>         ...
>>>
>>> net = Net()
>>> # Train model with data, then convert to CoreML.
>>> model = ct.convert(net, ...)
>>>
>>> import bentoml
>>> import PIL.Image
>>> from bentoml.adapters import ImageInput
>>> from bentoml.frameworks.coreml import CoreMLModelArtifact
>>>
>>> @bentoml.env(infer_pip_packages=True)
>>> @bentoml.artifacts([CoreMLModelArtifact('model')])
>>> class CoreMLModelService(bentoml.BentoService):
>>>
>>>     @bentoml.api(input=ImageInput(), batch=True)
>>>     def predict(self, imgs):
>>>         outputs = [self.artifacts.model(PIL.Image.fromarray(_.astype("uint8")))
>>>                    for img in imgs]
>>>         return outputs
>>>
>>>
>>> svc = CoreMLModelService()
>>>
>>> # Pytorch model can be packed directly.
>>> svc.pack('model', model)

ONNX

Example Projects:

class bentoml.frameworks.onnx.OnnxModelArtifact(*args, **kwargs)

Artifact class for saving and loading ONNX Model

Parameters
  • name (string) – Name of the artifact

  • backend (string) – Name of ONNX inference runtime. [“onnxruntime”, “onnxruntime-gpu”]

Raises
  • MissingDependencyException – onnx package is required for packing a ModelProto object

  • NotImplementedError – {backend} as onnx runtime is not supported at the moment

Example usage:

>>> # Train a model.
>>> from sklearn.datasets import load_iris
>>> from sklearn.model_selection import train_test_split
>>> from sklearn.ensemble import RandomForestClassifier
>>> iris = load_iris()
>>> X, y = iris.data, iris.target
>>> X_train, X_test, y_train, y_test = train_test_split(X, y)
>>> clr = RandomForestClassifier()
>>> clr.fit(X_train, y_train)
>>> # Convert into ONNX format
>>> from skl2onnx import convert_sklearn
>>> from skl2onnx.common.data_types import FloatTensorType
>>> initial_type = [('float_input', FloatTensorType([None, 4]))]
>>>
>>> onnx_model = convert_sklearn(clr, initial_types=initial_type)
>>> with open("rf_iris.onnx", "wb") as f:
>>>     f.write(onnx_model.SerializeToString())
>>>
>>>
>>> import numpy
>>> import bentoml
>>> from bentoml.frameworks.onnx import OnnxModelArtifact
>>> from bentoml.adapters import DataframeInput
>>>
>>> @bentoml.env(infer_pip_packages=True)
>>> @bentoml.artifacts([OnnxModelArtifact('model', backend='onnxruntime')])
>>> class OnnxIrisClassifierService(bentoml.BentoService):
>>>     @bentoml.api(input=DataframeInput(), batch=True)
>>>     def predict(self, df):
>>>         input_data = df.to_numpy().astype(numpy.float32
>>>         input_name = self.artifacts.model.get_inputs()[0].name
>>>         output_name = self.artifacts.model.get_outputs()[0].name
>>>         return self.artifacts.model.run(
>>>                     [output_name], {input_name: input_data}
>>>                )[0]
>>>
>>> svc = OnnxIrisClassifierService()
>>>
>>> # Option one: pack with path to model on local system
>>> svc.pack('model', './rf_iris.onnx')
>>>
>>> # Option two: pack with ONNX model object
>>> # svc.pack('model', onnx_model)
>>>
>>> # Save BentoService
>>> svc.save()

ONNX-MLIR

class bentoml.frameworks.onnxmlir.OnnxMlirModelArtifact(*args, **kwargs)

Artifact class for saving/loading onnx-mlir compiled model and operationalized using pyruntime wrapper

onnx-mlir is a compiler technology that can take an onnx model and lower it (using llvm) to an inference library that is optimized and has little external dependencies.

The PyRuntime interface is created during the build of onnx-mlir using pybind. See the onnx-mlir supporting documentation for detail.

Parameters

name (string) – Name of the artifact

Raises

MissingDependencyException – PyRuntime must be accessible in path.

Example usage:

>>> import bentoml
>>> from bentoml import env, artifacts, api, BentoService
>>> from bentoml.adapters import ImageInput
>>> from bentoml.frameworks.onnxmlir import OnnxMlirModelArtifact
>>> from bentoml.types import JsonSerializable, InferenceTask, InferenceError
>>> from bentoml.service.artifacts.common import PickleArtifact
>>> from typing import List
>>> import numpy as np
>>> import pandas as pd
>>> import sys
>>>
>>> @bentoml.env(infer_pip_packages=True)
>>> @bentoml.artifacts([OnnxMlirModelArtifact('model'), PickleArtifact('labels')])
>>> class ResNetPredict(BentoService):
>>>
>>>     def preprocess(self, input_data):
>>>         # convert the input data into the float32 input
>>>         img_data = np.stack(input_data).transpose(0, 3, 1, 2)
>>>
>>>         #normalize
>>>         mean_vec = np.array([0.485, 0.456, 0.406])
>>>         stddev_vec = np.array([0.229, 0.224, 0.225])
>>>
>>>
>>>         norm_img_data = np.zeros(img_data.shape).astype('float32')
>>>
>>>
>>>         for i in range(img_data.shape[0]):
>>>             for j in range(img_data.shape[1]):
>>>                 norm_img_data[i,j,:,:] =
>>>                 (img_data[i,j,:,:]/255 - mean_vec[j]) / stddev_vec[j]
>>>
>>>         #add batch channel
>>>         norm_img_data = norm_img_data.reshape(-1, 3, 224, 224).astype('float32')
>>>         return norm_img_data
>>>
>>>
>>>     def softmax(self, x):
>>>         x = x.reshape(-1)
>>>         e_x = np.exp(x - np.max(x))
>>>         return e_x / e_x.sum(axis=0)
>>>
>>>     def post_process(self, raw_result):
>>>         return self.softmax(np.array(raw_result)).tolist()
>>>
>>>     @bentoml.api(input=ImageInput(), batch=True)
>>>     def predict(self, image_ndarrays: List[np.ndarray]) -> List[str]:
>>>         input_datas = self.preprocess(image_ndarrays)
>>>
>>>         outputs = []
>>>         for i in range(input_datas.shape[0]):
>>>             raw_result = self.artifacts.model.run(input_datas[i:i+1])
>>>             result = self.post_process(raw_result)
>>>             idx = np.argmax(result)
>>>             sort_idx = np.flip(np.squeeze(np.argsort(result)))
>>>
>>>             # return top 5 labels
>>>             outputs.append(self.artifacts.labels[sort_idx[:5]])
>>>             return outputs
>>>

Spacy

class bentoml.frameworks.spacy.SpacyModelArtifact(*args, **kwargs)

Artifact class for saving/loading spacy models with Language.to_disk and spacy.util.load_model methods

Parameters

name (string) – name of the artifact

Raises
  • MissingDependencyException – spacy package is required for SpacyModelArtifact

  • InvalidArgument – invalid argument type, model being packed must be instance of spacy.language.Language

Example usage:

>>> import spacy
>>> # TRAIN MODEL WITH DATA
>>> TRAIN_DATA = [
>>>     ("Uber blew through $1 million a week", {"entities": [(0, 4, "ORG")]}),
>>>     ("Google rebrands its business apps", {"entities": [(0, 6, "ORG")]})]
>>>
>>> nlp = spacy.blank("en")
>>> optimizer = nlp.begin_training()
>>> for i in range(20):
>>>     random.shuffle(TRAIN_DATA)
>>>     for text, annotations in TRAIN_DATA:
>>>         nlp.update([text], [annotations], sgd=optimizer)
>>> nlp.to_disk("/model")
>>>
>>> import bentoml
>>> from bentoml.adapters import JsonInput
>>> from bentoml.frameworks.spacy import SpacyModelArtifact
>>>
>>> @bentoml.env(infer_pip_packages=True)
>>> @bentoml.artifacts([SpacyModelArtifact('nlp')])
>>> class SpacyModelService(bentoml.BentoService):
>>>
>>>     @bentoml.api(input=JsonInput(), batch=False)
>>>     def predict(self, parsed_json):
>>>         output = self.artifacts.nlp(parsed_json['text'])
>>>         return output
>>>
>>>
>>> svc = SpacyModelService()
>>>
>>> # Spacy model can be packed directly.
>>> svc.pack('nlp', nlp)

Transformers

class bentoml.frameworks.transformers.TransformersModelArtifact(*args, **kwargs)

Artifact class for saving/loading Transformers models

Parameters

name (string) – name of the artifact

Raises
  • MissingDependencyException – transformers package is required for TransformersModelArtifact

  • InvalidArgument – invalid argument type, model being packed must be either a dictionary of format {‘model’:transformers model object, ‘tokenizer’:transformers tokenizer object} or a directory path where the model is saved or a pre-trained model provided by transformers which can be loaded using transformers.AutoModelWithLMHead

  • NotFound – if the provided model name or model path is not found

Example usage:

>>> import bentoml
>>> from transformers import AutoModelWithLMHead, AutoTokenizer
>>> from bentoml.adapters import JsonInput
>>>
>>> @bentoml.env(pip_packages=["transformers==3.1.0", "torch==1.6.0"])
>>> @bentoml.artifacts([TransformersModelArtifact("gptModel")])
>>> class TransformerService(bentoml.BentoService):
>>>     @bentoml.api(input=JsonInput(), batch=False)
>>>     def predict(self, parsed_json):
>>>         src_text = parsed_json.get("text")
>>>         model = self.artifacts.gptModel.get("model")
>>>         tokenizer = self.artifacts.gptModel.get("tokenizer")
>>>         input_ids = tokenizer.encode(src_text, return_tensors="pt")
>>>         output = model.generate(input_ids, max_length=50)
>>>         output = tokenizer.decode(output[0], skip_special_tokens=True)
>>>         return output
>>>
>>>
>>> ts = TransformerService()
>>>
>>> model_name = "gpt2"
>>> model = AutoModelWithLMHead.from_pretrained("gpt2")
>>> tokenizer = AutoTokenizer.from_pretrained("gpt2")
>>> # Option 1: Pack using dictionary (recommended)
>>> artifact = {"model": model, "tokenizer": tokenizer}
>>> ts.pack("gptModel", artifact)
>>> # Option 2: pack using the name of the model
>>> # ts.pack("gptModel","gpt2")
>>> # Note that while packing using the name of the model,
>>> # ensure that the model can be loaded using
>>> # transformers.AutoModelWithLMHead (eg GPT, Bert, Roberta etc.)
>>> # If this is not the case (eg AutoModelForQuestionAnswering, BartModel etc)
>>> # then pack the model by passing a dictionary
>>> # with the model and tokenizer declared explicitly
>>> saved_path = ts.save()

Statsmodels

For statsmodels, we recommend using PickleArtifact:

class bentoml.service.artifacts.common.PickleArtifact(*args, **kwargs)

Abstraction for saving/loading python objects with pickle serialization

Parameters
  • name (str) – Name for the artifact

  • pickle_module (module|str) – The python module will be used for pickle and unpickle artifact, default pickle module in BentoML’s fork of cloudpickle, which is identical to the Apache Spark fork

  • pickle_extension (str) – The extension format for pickled file.

Example Projects:

Gluon

class bentoml.frameworks.gluon.GluonModelArtifact(*args, **kwargs)

Abstraction for saving/loading gluon models

Parameters

name (str) – Name for the artifact

Raises

MissingDependencyError – mxnet package is required for GluonModelArtifact

Example usage:

>>> from bentoml import env, artifacts, api, BentoService
>>> from bentoml.adapters import JsonInput
>>> from bentoml.frameworks.gluon import GluonModelArtifact
>>> import mxnet as mx
>>>
>>> @env(infer_pip_packages=True)
>>> @artifacts([GluonModelArtifact('model')])
>>> class GluonClassifier(BentoService):
>>>     @api(input=JsonInput(), batch=False)
>>>     def predict(self, request):
>>>         nd_input = mx.nd.array(request['input'])
>>>     return self.artifacts.model(nd_input).asnumpy()
>>>
>>> svc = GluonClassifier()
>>> svc.pack('model', model_to_save)
>>> svc.save()

Pytorch Lightning

class bentoml.frameworks.pytorch.PytorchLightningModelArtifact(*args, **kwargs)

Artifact class for saving and loading pytorch lightning model

Parameters

name (string) – Name of the pytorch model

Raises

MissingDependencyException – torch and pytorch_lightning package is required.

Example usage:

>>>
>>> # Train pytorch lightning model
>>> from pytorch_lightning.core.lightning import LightningModule
>>>
>>> class SimpleModel(LightningModule):
>>>     def forward(self, x):
>>>         return x.add(1)
>>>
>>> model = SimpleModel()
>>>
>>> import bentoml
>>> import torch
>>> from bentoml.adapters import JsonInput
>>> from bentoml.frameworks.pytorch import PytorchLightningModelArtifact
>>>
>>> @bentoml.env(infer_pip_packages=True)
>>> @bentoml.artifacts([PytorchLightningModelArtifact('model')])
>>> class PytorchLightingService(bentoml.BentoService):
>>>     @bentoml.api(input=DataframeInput(), batch=True)
>>>     def predict(self, df):
>>>         input_tensor = torch.from_numpy(df.to_numpy())
>>>         return self.artifacts.model(input).numpy()
>>>
>>> svc = PytorchLightingService()
>>> # Pack Pytorch Lightning model instance
>>> svc.pack('model', model)
>>>
>>> # Pack saved Pytorch Lightning model
>>> # import torch
>>> # script = model.to_torchscript()
>>> # saved_model_path = 'path/to/model.pt'
>>> # torch.jit.save(script, saved_model_path)
>>> # svc.pack('model', saved_model_path)
>>>
>>> svc.save()

Detectron

class bentoml.frameworks.detectron.DetectronModelArtifact(*args, **kwargs)

Artifact class for saving/loading Detectron2 models, in the form of detectron2.checkpoint.DetectionCheckpointer

Parameters

name (string) – name of the artifact

Raises
  • MissingDependencyException – detectron2 package is required for

  • DetectronModelArtifact

  • InvalidArgument – invalid argument type, model being packed must be instance of torch.nn.Module

Example usage:

>>> # Train model with data
>>>
>>>
>>> import bentoml
>>> from bentoml.adapters import ImageInput
>>> from bentoml.frameworks.detectron import DetectronModelArtifact
>>> from detectron2.data import transforms as T
>>>
>>> @bentoml.env(infer_pip_packages=True)
>>> @bentoml.artifacts([DetectronModelArtifact('model')])
>>> class CocoDetectronService(bentoml.BentoService):
>>>
>>>     @bentoml.api(input=ImageInput(), batch=False)
>>>     def predict(self, img: np.ndarray) -> Dict:
>>>         _aug = T.ResizeShortestEdge(
>>>            [800, 800], 1333
>>>        )
>>>        boxes = None
>>>        scores = None
>>>        pred_classes = None
>>>        pred_masks= None
>>>        try:
>>>            original_image = img[:, :, ::-1]
>>>            height, width = original_image.shape[:2]
>>>            image = _aug.get_transform(original_image).
>>>                             apply_image(original_image)
>>>            image = torch.as_tensor(image.astype("float32").
>>>                             transpose(2, 0, 1))
>>>            inputs = {"image": image, "height": height, "width": width}
>>>            predictions = self.artifacts.model([inputs])[0]
>>>
>>>
>>> cfg = get_cfg()
>>> cfg.merge_from_file("input_model.yaml")
>>> meta_arch = META_ARCH_REGISTRY.get(cfg.MODEL.META_ARCHITECTURE)
>>> model = meta_arch(cfg)
>>> model.eval()
>>> device = "cuda:{}".format(0)
>>> model.to(device)
>>> checkpointer = DetectionCheckpointer(model)
>>> checkpointer.load("output/model.pth")
>>> metadata = {
>>>     'device' : device
>>> }
>>> bento_svc = CocoDetectronService()
>>> bento_svc.pack('model', model, metadata, "input_model.yaml")
>>> saved_path = bento_svc.save()

Paddle

Example Projects:

class bentoml.frameworks.paddle.PaddlePaddleModelArtifact(*args, **kwargs)

Artifact class for saving and loading PaddlePaddle’s PaddleInference model

Parameters

name (string) – name of the artifact

Raises

MissingDependencyException – paddle package is required for PaddlePaddleModelArtifact

Example usage:

>>> import pandas as pd
>>> import numpy as np
>>>
>>> from bentoml import env, artifacts, api, BentoService
>>> from bentoml.adapters import DataframeInput
>>> from bentoml.frameworks.paddle import PaddlePaddleModelArtifact
>>>
>>> @env(infer_pip_packages=True)
>>> @artifacts([PaddlePaddleModelArtifact('model')])
>>> class PaddleService(BentoService):
>>>    @api(input=DataframeInput(), batch=True)
>>>    def predict(self, df: pd.DataFrame):
>>>        input_data = df.to_numpy().astype(np.float32)
>>>        predictor = self.artifacts.model
>>>
>>>        input_names = predictor.get_input_names()
>>>        input_handle = predictor.get_input_handle(input_names[0])
>>>        input_handle.reshape(input_data.shape)
>>>        input_handle.copy_from_cpu(input_data)
>>>
>>>        predictor.run()
>>>
>>>        output_names = predictor.get_output_names()
>>>        output_handle = predictor.get_output_handle(output_names[0])
>>>        output_data = output_handle.copy_to_cpu()
>>>        return output_data
>>>
>>> service = PaddleService()
>>>
>>> service.pack('model', model_to_save)

EasyOCR

class bentoml.frameworks.easyocr.EasyOCRArtifact(*args, **kwargs)

Artifact class for saving/loading EasyOCR models

Parameters

name (str) – Name for the artifact

Raises
  • MissingDependencyError – easyocr>=1.3 package is required for EasyOCRArtifact

  • InvalidArgument – invalid argument type, model being packed must be easyocr.easyocr.Reader

Example usage:

>>> import bentoml
>>> from bentoml.frameworks.easyocr import EasyOCRArtifact
>>> from bentoml.adapters import ImageInput
>>>
>>> @bentoml.env(pip_packages=["easyocr>=1.3.0"])
>>> @bentoml.artifacts([EasyOCRArtifact("chinese_small")])
>>> class EasyOCRService(bentoml.BentoService):
>>>     @bentoml.api(input=ImageInput(), batch=False)
>>>     def predict(self, image):
>>>         reader = self.artifacts.chinese_small
>>>         raw_results = reader.readtext(np.array(image))
>>>         text_instances = [x[1] for x in raw_results]
>>>         return {"text" : text_instances}
>>>
>>> import easyocr
>>> service = EasyOCRService()
>>>
>>> lang_list = ['ch_sim', 'en']
>>> recog_network = "zh_sim_g2"
>>>
>>> model = easyocr.Reader(lang_list=lang_list, download_enabled=True,     >>>                        recog_network=recog_network)
>>> service.pack('chinese_small', model, lang_list=lang_list,     >>>              recog_network= recog_network)
>>>
>>> saved_path = service.save()

EvalML

class bentoml.frameworks.evalml.EvalMLModelArtifact(*args, **kwargs)

Artifact class for saving/loading EvalML models

Parameters

name (str) – Name for the artifact

Raises

MissingDependencyException – evalml package is required for EvalMLModelArtifact

Example usage:

>>> # import the EvalML base pipeline class corresponding to your ML task
>>> from evalml.pipelines import BinaryClassificationPipeline
>>> # create an EvalML pipeline instance with desired components
>>> model_to_save = BinaryClassificationPipeline(
>>>     ['Imputer', 'One Hot Encoder', 'Random Forest Classifier'])
>>> # Train model with data using model_to_save.fit(X, y)
>>>
>>> import bentoml
>>> from bentoml.frameworks.sklearn import EvalMLModelArtifact
>>> from bentoml.adapters import DataframeInput
>>>
>>> @bentoml.env(infer_pip_packages=True)
>>> @bentoml.artifacts([EvalMLModelArtifact('EvalML pipeline')])
>>> class EvalMLModelService(bentoml.BentoService):
>>>
>>>     @bentoml.api(input=DataframeInput(), batch=True)
>>>     def predict(self, df):
>>>         result = self.artifacts.model.predict(df)
>>>         return result
>>>
>>> svc = EvalMLModelService()
>>>
>>> # Pack directly with EvalML model object
>>> svc.pack('model', model_to_save)
>>> svc.save()