Frameworks

https://static.scarf.sh/a.png?x-pxid=0beb35eb-7742-4dfb-b183-2228e8caf04c

Here are the all of the supported ML frameworks for BentoML. You can find the official BentoML example projects in the bentoml/gallery repository, group by the ML training frameworks used in the project.

You can download the examples below and run them on your computer. Links to run them on Google Colab are also available, although some of the features demo’d in the notebooks does not work in the Colab environment due to its limitations, including running the BentoML API model server, building docker image or creating cloud deployment.

Scikit-Learn

Example Projects:

class bentoml.frameworks.sklearn.SklearnModelArtifact(name, pickle_extension='.pkl')

Abstraction for saving/loading scikit learn models using sklearn.externals.joblib

Parameters
  • name (str) – Name for the artifact

  • pickle_extension (str) – The extension format for pickled file

Raises

MissingDependencyException – sklean package is required for SklearnModelArtifact

Example usage:

>>> from sklearn import svm
>>>
>>> model_to_save = svm.SVC(gamma='scale')
>>> # ... training model, etc.
>>>
>>> import bentoml
>>> from bentoml.frameworks.sklearn import SklearnModelArtifact
>>> from bentoml.adapters import DataframeInput
>>>
>>> @bentoml.env(infer_pip_packages=True)
>>> @bentoml.artifacts([SklearnModelArtifact('model')])
>>> class SklearnModelService(bentoml.BentoService):
>>>
>>>     @bentoml.api(input=DataframeInput(), batch=True)
>>>     def predict(self, df):
>>>         result = self.artifacts.model.predict(df)
>>>         return result
>>>
>>> svc = SklearnModelService()
>>>
>>> # Pack directly with sklearn model object
>>> svc.pack('model', model_to_save)

PyTorch

Example Projects:

class bentoml.frameworks.pytorch.PytorchModelArtifact(name, file_extension='.pt')

Abstraction for saving/loading objects with torch.save and torch.load

Parameters

name (string) – name of the artifact

Raises
  • MissingDependencyException – torch package is required for PytorchModelArtifact

  • InvalidArgument – invalid argument type, model being packed must be instance of torch.nn.Module

Example usage:

>>> import torch.nn as nn
>>>
>>> class Net(nn.Module):
>>>     def __init__(self):
>>>         super(Net, self).__init__()
>>>         ...
>>>
>>>     def forward(self, x):
>>>         ...
>>>
>>> net = Net()
>>> # Train model with data
>>>
>>>
>>> import bentoml
>>> from bentoml.adapters import ImageInput
>>> from bentoml.frameworks.pytorch import PytorchModelArtifact
>>>
>>> @bentoml.env(infer_pip_packages=True)
>>> @bentoml.artifacts([PytorchModelArtifact('net')])
>>> class PytorchModelService(bentoml.BentoService):
>>>
>>>     @bentoml.api(input=ImageInput(), batch=True)
>>>     def predict(self, imgs):
>>>         outputs = self.artifacts.net(imgs)
>>>         return outputs
>>>
>>>
>>> svc = PytorchModelService()
>>>
>>> # Pytorch model can be packed directly.
>>> svc.pack('net', net)

Tensorflow 2.0

Example Projects:

class bentoml.frameworks.tensorflow.TensorflowSavedModelArtifact(name)

Abstraction for saving/loading Tensorflow model in tf.saved_model format

Parameters

name (string) – name of the artifact

Raises

MissingDependencyException – tensorflow package is required for TensorflowSavedModelArtifact

Example usage:

>>> import tensorflow as tf
>>>
>>> # Option 1: custom model with specific method call
>>> class Adder(tf.Module):
>>>     @tf.function(input_signature=[tf.TensorSpec(shape=None, dtype=tf.float32)])
>>>     def add(self, x):
>>>         return x + x + 1.
>>> model_to_save = Adder()
>>> # ... compiling, training, etc
>>>
>>> # Option 2: Sequential model (direct call only)
>>> model_to_save = tf.keras.Sequential([
>>>     tf.keras.layers.Flatten(input_shape=(28, 28)),
>>>     tf.keras.layers.Dense(128, activation='relu'),
>>>     tf.keras.layers.Dense(10, activation='softmax')
>>> ])
>>> # ... compiling, training, etc
>>>
>>> import bentoml
>>> from bentoml.adapters import JsonInput
>>> from bentoml.frameworks.tensorflow import TensorflowSavedModelArtifact
>>>
>>> @bentoml.env(pip_packages=["tensorflow"])
>>> @bentoml.artifacts([TensorflowSavedModelArtifact('model')])
>>> class TfModelService(bentoml.BentoService):
>>>
>>>     @bentoml.api(input=JsonInput(), batch=False)
>>>     def predict(self, json):
>>>         input_data = json['input']
>>>         prediction = self.artifacts.model.add(input_data)
>>>         # prediction = self.artifacts.model(input_data)  # if Sequential mode
>>>         return prediction.numpy()
>>>
>>> svc = TfModelService()
>>>
>>> # Option 1: pack directly with Tensorflow trackable object
>>> svc.pack('model', model_to_save)
>>>
>>> # Option 2: save to file path then pack
>>> tf.saved_model.save(model_to_save, '/tmp/adder/1')
>>> svc.pack('model', '/tmp/adder/1')

Tensorflow Keras

Example Projects:

class bentoml.frameworks.keras.KerasModelArtifact(name, custom_objects=None, model_extension='.h5', store_as_json_and_weights=False)

Abstraction for saving/loading Keras model

Parameters
  • name (string) – name of the artifact

  • custom_objects (dict) – dictionary of Keras custom objects for model

  • store_as_json_and_weights (bool) – flag allowing storage of the Keras model as JSON and weights

Raises
  • MissingDependencyException – keras or tensorflow.keras package is required for KerasModelArtifact

  • InvalidArgument – invalid argument type, model being packed must be instance of keras.engine.network.Network, tf.keras.models.Model, or their aliases

Example usage:

>>> from tensorflow import keras
>>> from tensorflow.keras.models import Sequential
>>> from tensorflow.keras.preprocessing import sequence, text
>>>
>>> model_to_save = Sequential()
>>> # training model
>>> model_to_save.compile(...)
>>> model_to_save.fit(...)
>>>
>>> import bentoml
>>> from bentoml.frameworks.keras import KerasModelArtifact
>>>
>>> @bentoml.env(infer_pip_packages=True)
>>> @bentoml.artifacts([KerasModelArtifact('model')])
>>> class KerasModelService(bentoml.BentoService):
>>>     @bentoml.api(input=JsonInput(), batch=False)
>>>     def predict(self, parsed_json):
>>>         input_data = text.text_to_word_sequence(parsed_json['text'])
>>>         return self.artifacts.model.predict_classes(input_data)
>>>
>>> svc = KerasModelService()
>>> svc.pack('model', model_to_save)

FastAI

Example Projects:

class bentoml.frameworks.fastai.FastaiModelArtifact(name)

Saving and Loading FastAI Model

Parameters

name (str) – Name for the fastai model

Raises
  • MissingDependencyException – Require fastai package to use Fast ai model artifact

  • InvalidArgument – invalid argument type, model being packed must be instance of fastai.basic_train.Learner

Example usage:

>>> from fastai.tabular import *
>>>
>>> # prepare data
>>> data = TabularList.from_df(...)
>>> learn = tabular_learner(data, ...)
>>> # train model
>>>
>>> import bentoml
>>> from bentoml.adapters import DataframeInput
>>> from bentoml.frameworks.fastai import FastaiModelArtifact
>>>
>>> @bentoml.artifacts([FastaiModelArtifact('model')])
>>> @bentoml.env(infer_pip_packages=True)
>>> class FastaiModelService(bentoml.BentoService):
>>>
>>>     @api(input=DataframeInput(), batch=True)
>>>     def predict(self, df):
>>>         results = []
>>>         for _, row in df.iterrows():
>>>             prediction = self.artifacts.model.predict(row)
>>>             results.append(prediction[0].obj)
>>>         return results
>>>
>>> svc = FastaiModelService()
>>>
>>> # Pack fastai basic_learner directly
>>> svc.pack('model', learn)

FastAI v2

Example Projects:

class bentoml.frameworks.fastai2.Fastai2ModelArtifact(name)

Saving and Loading FastAI v2 Model

Parameters

name (str) – Name for the fastai v2 model

Raises
  • MissingDependencyException – Require fastai>=2.0.0 package for Fastai model artifact

  • InvalidArgument – invalid argument type, model being packed must be instance of fastai.basics.Learner

XGBoost

Example Projects:

class bentoml.frameworks.xgboost.XgboostModelArtifact(name, model_extension='.model')

Abstraction for save/load object with Xgboost.

Parameters
  • name (string) – name of the artifact

  • model_extension (string) – Extension name for saved xgboost model

Raises
  • ImportError – xgboost package is required for using XgboostModelArtifact

  • TypeError – invalid argument type, model being packed must be instance of xgboost.core.Booster

Example usage:

>>> import xgboost
>>>
>>> # prepare data
>>> params = {... params}
>>> dtrain = xgboost.DMatrix(...)
>>>
>>> # train model
>>> model_to_save = xgboost.train(params=params, dtrain=dtrain)
>>>
>>> import bentoml
>>> from bentoml.frameworks.xgboost import XgboostModelArtifact
>>> from bentoml.adapters import DataframeInput
>>>
>>> @bentoml.env(infer_pip_packages=True)
>>> @bentoml.artifacts(XgboostModelArtifact('model'))
>>> class XGBoostModelService(bentoml.BentoService):
>>>
>>>     @bentoml.api(input=DataframeInput(), batch=True)
>>>     def predict(self, df):
>>>         result = self.artifacts.model.predict(df)
>>>         return result
>>>
>>> svc = XGBoostModelService()
>>> # Pack xgboost model
>>> svc.pack('model', model_to_save)

LightGBM

Example Projects:

class bentoml.frameworks.lightgbm.LightGBMModelArtifact(name, model_extension='.txt')

Abstraction for save/load object with LightGBM.

Parameters
  • name (string) – name of the artifact

  • model_extension (string) – Extension name for saved xgboost model

Raises
  • MissingDependencyException – lightgbm package is required for using LightGBMModelArtifact

  • InvalidArgument – invalid argument type, model being packed must be instance of lightgbm.Booster

Example usage:

>>> import lightgbm as lgb
>>> # Prepare data
>>> train_data = lgb.Dataset(...)
>>> # train model
>>> model = lgb.train(train_set=train_data, ...)
>>>
>>> import bentoml
>>> from bentoml.frameworks.lightgbm import LightGBMModelArtifact
>>> from bentoml.adapters import DataframeInput
>>>
>>> @bentoml.artifacts([LightGBMModelArtifact('model')])
>>> @bentoml.env(infer_pip_packages=True)
>>> class LgbModelService(bentoml.BentoService):
>>>
>>>     @bentoml.api(input=DataframeInput(), batch=True)
>>>     def predict(self, df):
>>>         return self.artifacts.model.predict(df)
>>>
>>> svc = LgbModelService()
>>> svc.pack('model', model)

FastText

Example Projects:

class bentoml.frameworks.fasttext.FasttextModelArtifact(name)

Abstraction for saving/loading fasttext models

Parameters

name (str) – Name for the artifact

Raises

MissingDependencyError – fasttext package is required for FasttextModelArtifact

Example usage:

>>> import fasttext
>>> # prepare training data and store to file
>>> training_data_file = 'training-data-file.train'
>>> model = fasttext.train_supervised(input=training_data_file)
>>>
>>> import bentoml
>>> from bentoml.adapters JsonInput
>>> from bentoml.frameworks.fasttext import FasttextModelArtifact
>>>
>>> @bentoml.env(infer_pip_packages=True)
>>> @bentoml.artifacts([FasttextModelArtifact('model')])
>>> class FasttextModelService(bentoml.BentoService):
>>>
>>>     @bentoml.api(input=JsonInput(), batch=False)
>>>     def predict(self, parsed_json):
>>>         # K is the number of labels that successfully were predicted,
>>>         # among all the real labels
>>>         return self.artifacts.model.predict(parsed_json['text'], k=5)
>>>
>>> svc = FasttextModelService()
>>> svc.pack('model', model)

Statsmodels

Example Projects:

H2O

Example Projects:

class bentoml.frameworks.h2o.H2oModelArtifact(name)

Abstraction for saving/loading objects with h2o.save_model and h2o.load_model

Parameters

name (str) – Name for this h2o artifact..

Raises

MissingDependencyException – h2o package is required to use H2o model artifact

Example usage:

>>> import h2o
>>> h2o.init()
>>>
>>> from h2o.estimators.deeplearning import H2ODeepLearningEstimator
>>> model_to_save = H2ODeepLearningEstimator(...)
>>> # train model with data
>>> data = h2o.import_file(...)
>>> model_to_save.train(...)
>>>
>>> import bentoml
>>> from bentoml.frameworks.h2o import H2oModelArtifact
>>> from bentoml.adapters import DataframeInput
>>>
>>> @bentoml.artifacts([H2oModelArtifact('model')])
>>> @bentoml.env(infer_pip_packages=True)
>>> class H2oModelService(bentoml.BentoService):
>>>
>>>     @bentoml.api(input=DataframeInput(), batch=True)
>>>     def predict(self, df):
>>>         hf = h2o.H2OFrame(df)
>>>         predictions = self.artifacts.model.predict(hf)
>>>         return predictions.as_data_frame()
>>>
>>> svc = H2oModelService()
>>>
>>> svc.pack('model', model_to_save)

CoreML

class bentoml.frameworks.coreml.CoreMLModelArtifact(name, file_extension='.mlmodel')

Abstraction for saving/loading coreml.models.MLModel objects with coremltools.models.MLModel.save and coremltools.models.MLModel(path).

Parameters

name (string) – name of the artifact

Raises
  • MissingDependencyException – coremltools package required for CoreMLModelArtifact

  • InvalidArgument – invalid argument type, model being packed must be instance of coremltools.models.MLModel

Example usage:

>>> import coremltools as ct
>>> import torch.nn as nn
>>>
>>> class Net(nn.Module):
>>>     def __init__(self):
>>>         super(Net, self).__init__()
>>>         ...
>>>
>>>     def forward(self, x):
>>>         ...
>>>
>>> net = Net()
>>> # Train model with data, then convert to CoreML.
>>> model = ct.convert(net, ...)
>>>
>>> import bentoml
>>> import PIL.Image
>>> from bentoml.adapters import ImageInput
>>> from bentoml.frameworks.coreml import CoreMLModelArtifact
>>>
>>> @bentoml.env(infer_pip_packages=True)
>>> @bentoml.artifacts([CoreMLModelArtifact('model')])
>>> class CoreMLModelService(bentoml.BentoService):
>>>
>>>     @bentoml.api(input=ImageInput(), batch=True)
>>>     def predict(self, imgs):
>>>         outputs = [self.artifacts.model(PIL.Image.fromarray(_.astype("uint8")))
>>>                    for img in imgs]
>>>         return outputs
>>>
>>>
>>> svc = CoreMLModelService()
>>>
>>> # Pytorch model can be packed directly.
>>> svc.pack('model', model)

ONNX

Example Projects:

class bentoml.frameworks.onnx.OnnxModelArtifact(name, backend='onnxruntime')

Abstraction for saving/loading onnx model

Parameters
  • name (string) – Name of the artifact

  • backend (string) – Name of ONNX inference runtime. [onnx]

Raises
  • MissingDependencyException – onnx package is required for packing a ModelProto object

  • NotImplementedError – {backend} as onnx runtime is not supported at the moment

Example usage:

>>>
>>> # Train a model.
>>> from sklearn.datasets import load_iris
>>> from sklearn.model_selection import train_test_split
>>> from sklearn.ensemble import RandomForestClassifier
>>> iris = load_iris()
>>> X, y = iris.data, iris.target
>>> X_train, X_test, y_train, y_test = train_test_split(X, y)
>>> clr = RandomForestClassifier()
>>> clr.fit(X_train, y_train)
>>> # Convert into ONNX format
>>> from skl2onnx import convert_sklearn
>>> from skl2onnx.common.data_types import FloatTensorType
>>> initial_type = [('float_input', FloatTensorType([None, 4]))]
>>>
>>> onnx_model = convert_sklearn(clr, initial_types=initial_type)
>>> with open("rf_iris.onnx", "wb") as f:
>>>     f.write(onnx_model.SerializeToString())
>>>
>>>
>>> import numpy
>>> import bentoml
>>> from bentoml.frameworks.onnx import OnnxModelArtifact
>>> from bentoml.adapters import DataframeInput
>>>
>>> @bentoml.env(infer_pip_packages=True)
>>> @bentoml.artifacts([OnnxModelArtifact('model', backend='onnxruntime')])
>>> class OnnxIrisClassifierService(bentoml.BentoService):
>>>     @bentoml.api(input=DataframeInput(), batch=True)
>>>     def predict(self, df):
>>>         input_data = df.to_numpy().astype(numpy.float32
>>>         input_name = self.artifacts.model.get_inputs()[0].name
>>>         output_name = self.artifacts.model.get_outputs()[0].name
>>>         return self.artifacts.model.run(
>>>                     [output_name], {input_name: input_data}
>>>                )[0]
>>>
>>> svc = OnnxIrisClassifierService()
>>>
>>> # Option one: pack with path to model on local system
>>> svc.pack('model', './rf_iris.onnx')
>>>
>>> # Option two: pack with ONNX model object
>>> # svc.pack('model', onnx_model)
>>>
>>> # Save BentoService
>>> svc.save()

Spacy

class bentoml.frameworks.spacy.SpacyModelArtifact(name)

Abstraction for saving/loading spacy models with to_disk and spacy.util.load_model methods. :param name: name of the artifact :type name: string

Raises
  • MissingDependencyException – spacy package is required for SpacyModelArtifact

  • InvalidArgument – invalid argument type, model being packed must be instance of spacy.language.Language

Example usage:

>>> import spacy
>>> # TRAIN MODEL WITH DATA
>>> TRAIN_DATA = [
>>>     ("Uber blew through $1 million a week", {"entities": [(0, 4, "ORG")]}),
>>>     ("Google rebrands its business apps", {"entities": [(0, 6, "ORG")]})]
>>>
>>> nlp = spacy.blank("en")
>>> optimizer = nlp.begin_training()
>>> for i in range(20):
>>>     random.shuffle(TRAIN_DATA)
>>>     for text, annotations in TRAIN_DATA:
>>>         nlp.update([text], [annotations], sgd=optimizer)
>>> nlp.to_disk("/model")
>>>
>>> import bentoml
>>> from bentoml.adapters import JsonInput
>>> from bentoml.frameworks.spacy import SpacyModelArtifact
>>>
>>> @bentoml.env(infer_pip_packages=True)
>>> @bentoml.artifacts([SpacyModelArtifact('nlp')])
>>> class SpacyModelService(bentoml.BentoService):
>>>
>>>     @bentoml.api(input=JsonInput(), batch=False)
>>>     def predict(self, parsed_json):
>>>         output = self.artifacts.nlp(parsed_json['text'])
>>>         return output
>>>
>>>
>>> svc = SpacyModelService()
>>>
>>> # Spacy model can be packed directly.
>>> svc.pack('nlp', nlp)

Transformers

class bentoml.frameworks.transformers.TransformersModelArtifact(name)

Abstraction for saving/loading Transformers models

Parameters

name (string) – name of the artifact

Raises
  • MissingDependencyException – transformers package is required for TransformersModelArtifact

  • InvalidArgument – invalid argument type, model being packed must be either a dictionary of format {‘model’:transformers model object, ‘tokenizer’:transformers tokenizer object} or a directory path where the model is saved or a pretrained model provided by transformers which can be loaded using transformers.AutoModelWithLMHead

  • NotFound – if the provided model name or model path is not found

Example usage:

>>> import bentoml
>>> from transformers import AutoModelWithLMHead, AutoTokenizer
>>> from bentoml.adapters import JsonInput
>>>
>>> @bentoml.env(pip_packages=["transformers==3.1.0", "torch==1.6.0"])
>>> @bentoml.artifacts([TransformersModelArtifact("gptModel")])
>>> class TransformerService(bentoml.BentoService):
>>>     @bentoml.api(input=JsonInput(), batch=False)
>>>     def predict(self, parsed_json):
>>>         src_text = parsed_json.get("text")
>>>         model = self.artifacts.gptModel.get("model")
>>>         tokenizer = self.artifacts.gptModel.get("tokenizer")
>>>         input_ids = tokenizer.encode(src_text, return_tensors="pt")
>>>         output = model.generate(input_ids, max_length=50)
>>>         output = tokenizer.decode(output[0], skip_special_tokens=True)
>>>         return output
>>>
>>>
>>> ts = TransformerService()
>>>
>>> model_name = "gpt2"
>>> model = AutoModelWithLMHead.from_pretrained("gpt2")
>>> tokenizer = AutoTokenizer.from_pretrained("gpt2")
>>> # Option 1: Pack using dictionary (recommended)
>>> artifact = {"model": model, "tokenizer": tokenizer}
>>> ts.pack("gptModel", artifact)
>>> # Option 2: pack using the name of the model
>>> # ts.pack("gptModel","gpt2")
>>> # Note that while packing using the name of the model,
>>> # ensure that the model can be loaded using
>>> # transformers.AutoModelWithLMHead (eg GPT, Bert, Roberta etc.)
>>> # If this is not the case (eg AutoModelForQuestionAnswering, BartModel etc)
>>> # then pack the model by passing a dictionary
>>> # with the model and tokenizer declared explicitly
>>> saved_path = ts.save()

Statsmodels

For statsmodels, we recommend using PickleArtifact:

class bentoml.service.artifacts.common.PickleArtifact(name, pickle_module=<module 'bentoml.utils.cloudpickle' from '/home/docs/checkouts/readthedocs.org/user_builds/bentoml/checkouts/latest/bentoml/utils/cloudpickle.py'>, pickle_extension='.pkl')

Abstraction for saving/loading python objects with pickle serialization

Parameters
  • name (str) – Name for the artifact

  • pickle_module (module|str) – The python module will be used for pickle and unpickle artifact, default pickle module in BentoML’s fork of cloudpickle, which is identical to the Apache Spark fork

  • pickle_extension (str) – The extension format for pickled file.

Example Projects: