API InputAdapters

BaseInputAdapter

class bentoml.adapters.BaseInputAdapter(http_input_example=None, **base_config)

InputAdapter is an abstraction layer between user defined API callback function and prediction request input in a variety of different forms, such as HTTP request body, command line arguments or AWS Lambda event object.

DataframeInput

class bentoml.adapters.DataframeInput(typ: str = 'frame', orient: Optional[str] = None, columns: Sequence[str] = None, dtype: Mapping[str, object] = None, input_dtypes: Mapping[str, object] = None, **base_kwargs)

Convert various inputs(HTTP, Aws Lambda or CLI) to pandas dataframe, passing it to API functions.

Parameters
  • orient (str) –

    Indication of expected JSON string format. Compatible JSON strings can be produced by to_json() with a corresponding orient value. The set of possible orients is:

    • 'split' : dict like {index -> [index], columns -> [columns], data -> [values]}

    • 'records' : list like [{column -> value}, ... , {column -> value}]

    • 'index' : dict like {index -> {column -> value}}

    • 'columns' : dict like {column -> {index -> value}}

    • 'values' : just the values array

    The allowed and default values depend on the value of the typ parameter.

    • when typ == 'series' (not available now),

      • allowed orients are {'split','records','index'}

      • default is 'index'

      • The Series index must be unique for orient 'index'.

    • when typ == 'frame',

      • allowed orients are {'split','records','index', 'columns','values'}

      • default is 'columns'

      • The DataFrame index must be unique for orients 'index' and 'columns'.

      • The DataFrame columns must be unique for orients 'index', 'columns', and 'records'.

  • typ ({'frame', 'series'}, default 'frame') – ** Note: ‘series’ is not supported now. ** The type of object to recover.

  • dtype (dict, default None) – If is None, infer dtypes; if a dict of column to dtype, then use those. Not applicable for orient='table'.

  • input_dtypes (dict, default None) – ** Deprecated ** The same as the dtype

Raises
  • ValueError – Incoming data is missing required columns in dtype:

  • ValueError – Incoming data format can not be handled. Only json and csv:

Examples

Example Service:

from bentoml import env, artifacts, api, BentoService
from bentoml.adapters import DataframeInput
from bentoml.frameworks.sklearn import SklearnModelArtifact

@env(infer_pip_packages=True)
@artifacts([SklearnModelArtifact('model')])
class IrisClassifier(BentoService):

    @api(
        input=DataframeInput(
            orient="records",
            columns=["sw", "sl", "pw", "pl"],
            dtype={"sw": "float", "sl": "float", "pw": "float", "pl": "float"},
        ),
        batch=True,
    )
    def predict(self, df):
        # Optional pre-processing, post-processing code goes here
        return self.artifacts.model.predict(df)

Query with HTTP request:

curl -i \
  --header "Content-Type: application/json" \
  --request POST \
  --data '[{"sw": 1, "sl": 2, "pw": 1, "pl": 2}]' \
  localhost:5000/predict

OR:

curl -i \
  --header "Content-Type: text/csv" \
  --request POST \
  --data @file.csv \
  localhost:5000/predict

Query with CLI command:

bentoml run IrisClassifier:latest predict --input \
  '[{"sw": 1, "sl": 2, "pw": 1, "pl": 2}]'

OR:

bentoml run IrisClassifier:latest predict --format csv --input-file test.csv

JsonInput

class bentoml.adapters.JsonInput(http_input_example=None, **base_config)

JsonInput parses REST API request or CLI command into parsed_jsons(a list of json serializable object in python) and pass down to user defined API function

Examples

Example services:

Use JsonInput with batch=True

from typings import List
from bentoml.types import JsonSerializable

@bentoml.api(input=JsonInput(), batch=True)
def predict(self, parsed_json_list: List[JsonSerializable]):
    results = self.artifacts.classifier([j['text'] for j in parsed_json_list])
    return results

OR use JsonInput with batch=False (the default)

@bentoml.api(input=JsonInput())
def predict(self, parsed_json):
    results = self.artifacts.classifier([parsed_json['text']])
    return results[0]

For client prediction request, it is the same for both batch and non-batch API, the request should contain only one single input item:

curl -i \
  --header "Content-Type: application/json" \
  --request POST \
  --data '{"text": "best movie ever"}' \
  localhost:5000/predict

TfTensorInput

class bentoml.adapters.TfTensorInput(method='predict', **base_kwargs)

Tensor input adapter for Tensorflow models. Transform incoming tf tensor data from http request, cli or lambda event into tf tensor. The behaviour should be compatible with tensorflow serving REST API: * https://www.tensorflow.org/tfx/serving/api_rest#classify_and_regress_api * https://www.tensorflow.org/tfx/serving/api_rest#predict_api

Parameters

method (*) – equivalence of serving API methods: (predict, classify, regress)

Raises

BentoMLException – BentoML currently doesn’t support Content-Type

ImageInput

class bentoml.adapters.ImageInput(accept_image_formats=None, pilmode='RGB', **base_kwargs)

Convert incoming image data from http request, cli or lambda event into numpy array and pass down to user defined API functions

** To operate raw files or PIL.Image obj, use the lowlevel FileInput. **

Parameters
  • accept_image_formats (List[str]) – A list of acceptable image formats. Default value is loaded from bentoml config ‘apiserver/default_image_input_accept_file_extensions’, which is set to [‘.jpg’, ‘.png’, ‘.jpeg’, ‘.tiff’, ‘.webp’, ‘.bmp’] by default. List of all supported format can be found here: https://imageio.readthedocs.io/en/stable/formats.html

  • pilmode (str) – The pilmode to be used for reading image file into numpy array. Default value is ‘RGB’. Find more information at: https://imageio.readthedocs.io/en/stable/format_png-pil.html

Raises

ImportError – imageio package is required to use ImageInput:

Examples

Service using ImageInput:

from typing import List

import numpy as np
from bentoml import BentoService, api, artifacts
from bentoml.frameworks.tensorflow import TensorflowSavedModelArtifact
from bentoml.adapters import ImageInput

CLASS_NAMES = ['cat', 'dog']

@artifacts([TensorflowSavedModelArtifact('classifier')])
class PetClassification(BentoService):
    @api(input=ImageInput(), batch=True)
    def predict(self, image_ndarrays: List[np.ndarray]) -> List[str]:
        results = self.artifacts.classifer.predict(image_ndarrays)
        return [CLASS_NAMES[r] for r in results]

OR use ImageInput with batch=False (the default):

@api(input=ImageInput(), batch=False)
def predict(self, image_ndarray: np.ndarray) -> str:
    results = self.artifacts.classifer.predict([image_ndarray])
    return CLASS_NAMES[results[0]]

Query with HTTP request:

curl -i \
  --header "Content-Type: image/jpeg" \
  --request POST \
  --data @test.jpg \
  localhost:5000/predict

OR:

curl -i \
  -F image=@test.jpg \
  localhost:5000/predict

OR by an HTML form that sends multipart data:

<form action="http://localhost:8000" method="POST"
      enctype="multipart/form-data">
    <input name="image" type="file">
    <input type="submit">
</form>

Query with CLI command:

bentoml run PyTorchFashionClassifier:latest predict --input-file test.jpg

OR infer all images under a folder with ten images each batch:

bentoml run PyTorchFashionClassifier:latest predict \
  --input-file folder/*.jpg --max-batch-size 10

MultiImageInput

class bentoml.adapters.MultiImageInput(input_names='image', pilmode='RGB', accept_image_formats=None, **base_kwargs)
Transform incoming image datas from http request, cli or lambda event into numpy

arrays.

Parameters
  • input_names (List[str]) – A tuple of acceptable input name for HTTP request. Default value is (image,)

  • accept_image_formats (List[str]) – A list of acceptable image formats. Default value is loaded from bentoml config ‘apiserver/default_image_input_accept_file_extensions’, which is set to [‘.jpg’, ‘.png’, ‘.jpeg’, ‘.tiff’, ‘.webp’, ‘.bmp’] by default. List of all supported format can be found here: https://imageio.readthedocs.io/en/stable/formats.html

  • pilmode (str) – The pilmode to be used for reading image file into numpy array. Default value is ‘RGB’. Find more information at: https://imageio.readthedocs.io/en/stable/format_png-pil.html

Raises

ImportError – imageio package is required to use MultiImageInput:

Examples

Service using MultiImageInput:

from bentoml import BentoService
import bentoml

class MyService(BentoService):
    @bentoml.api(
        input=MultiImageInput(input_names=('imageX', 'imageY')), batch=True)
    def predict(self, image_groups):
        for image_group in image_groups:
            image_array_x = image_group['imageX']
            image_array_y = image_group['imageY']

Query the endpoint with HTTP request performed by cURL:

curl -i \
  -F imageX=@testx.jpg \
  -F imageY=@testy.jpg \
  localhost:5000/predict

OR by an HTML form that sends multipart data:

<form action="http://localhost:8000" method="POST"
      enctype="multipart/form-data">
    <input name="imageX" type="file">
    <input name="imageY" type="file">
    <input type="submit">
</form>

Query with CLI command:

bentoml run PyTorchFashionClassifier:latest predict \
  --input-file-imageX testx.jpg \
  --input-file-imageY testy.jpg

OR infer all file pairs under a folder with ten pairs each batch:

bentoml run PyTorchFashionClassifier:latest predict --max-batch-size 10 \
  --input-file-imageX folderx/*.jpg \
  --input-file-imageY foldery/*.jpg

Note: jpg files and json files should be in same prefix like this:

folderx:
    - 1.jpg
    - 2.jpg
    ...
foldery:
    - 1.jpg
    - 2.jpg
    ...

Or the following cURL command

>>> curl -F imageX=@image_file_x.png
>>>      -F imageY=@image_file_y.jpg
>>>      http://localhost:8000

AnnotatedImageInput

class bentoml.adapters.AnnotatedImageInput(image_input_name='image', annotation_input_name='annotations', pilmode='RGB', accept_image_formats=None, **base_kwargs)

Transform incoming image data from http request, cli or lambda event into a numpy array, while allowing an optional JSON file for image annotations (such as object bounding boxes, class labels, etc.)

Transforms input image file into a numpy array, and loads JSON file as a JSON serializable Python object, providing them to user-defined API functions.

Parameters
  • image_input_name (str) – An acceptable input name for HTTP request and function keyword argument. Default value is “image”

  • annotation_input_name (str) – An acceptable input name for HTTP request and function keyword argument. Default value is “annotations”

  • accept_image_formats (List[str]) – A list of acceptable image formats. Default value is loaded from bentoml config ‘apiserver/default_image_input_accept_file_extensions’, which is set to [‘.jpg’, ‘.png’, ‘.jpeg’, ‘.tiff’, ‘.webp’, ‘.bmp’] by default. List of all supported format can be found here: https://imageio.readthedocs.io/en/stable/formats.html

  • pilmode (str) – The pilmode to be used for reading image file into numpy array. Default value is ‘RGB’. Find more information at: https://imageio.readthedocs.io/en/stable/format_png-pil.html

Raises

ImportError – imageio package is required to use AnnotatedImageInput:

Examples

from typing import Sequence

from bentoml import BentoService, api, artifacts
from bentoml.frameworks.tensorflow import TensorflowSavedModelArtifact
from bentoml.adapters import AnnotatedImageInput

CLASS_NAMES = ['cat', 'dog']

@artifacts([TensorflowSavedModelArtifact('classifier')])
class PetClassification(BentoService):
   @api(input=AnnotatedImageInput(), batch=True)
   def predict(
           self,
           image_list: 'Sequence[numpy.ndarray]',
           annotations_list: 'Sequence[JsonSerializable]',
       ) -> Sequence[str]:
       cropped_pets = some_pet_finder(image_list, annotations_list)
       results = self.artifacts.classifer.predict(cropped_pets)
       return [CLASS_NAMES[r] for r in results]

Query with HTTP request performed by cURL:

curl -i \
  -F image=@test.jpg \
  -F annotations=@test.json \
  localhost:5000/predict

OR by an HTML form that sends multipart data:

<form action="http://localhost:8000" method="POST"
      enctype="multipart/form-data">
    <input name="image" type="file">
    <input name="annotations" type="file">
    <input type="submit">
</form>

Query with CLI command:

bentoml run PyTorchFashionClassifier:latest predict \
  --input-file-image test.jpg \
  --input-file-annotations test.json

OR infer all file pairs under a folder with ten pairs each batch:

bentoml run PyTorchFashionClassifier:latest predict --max-batch-size 10 \
  --input-file-image folder/*.jpg \
  --input-file-annotations folder/*.json

Note: jpg files and json files should be in same prefix like this:

folder:
    - apple.jpg
    - apple.json
    - banana.jpg
    - banana.json
    ...

FileInput

class bentoml.adapters.FileInput(http_input_example=None, **base_config)

Convert incoming file data from http request, cli or lambda event into file stream object and pass down to user defined API functions

Parameters

None

Examples

Service using FileInput:

import bentoml
from PIL import Image
import numpy as np

from bentoml.frameworks.pytorch import PytorchModelArtifact
from bentoml.adapters import FileInput

FASHION_MNIST_CLASSES = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat',
                         'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']

@bentoml.env(pip_packages=['torch', 'pillow', 'numpy'])
@bentoml.artifacts([PytorchModelArtifact('classifier')])
class PyTorchFashionClassifier(bentoml.BentoService):

    @bentoml.api(input=FileInput(), batch=True)
    def predict(self, file_streams):
        img_arrays = []
        for fs in file_streams:
            im = Image.open(fs).convert(mode="L").resize((28, 28))
            img_array = np.array(im)
            img_arrays.append(img_array)

        inputs = np.stack(img_arrays, axis=0)

        outputs = self.artifacts.classifier(inputs)
        return [FASHION_MNIST_CLASSES[c] for c in outputs]

OR use FileInput with batch=False (the default):

@bentoml.api(input=FileInput(), batch=False)
def predict(self, file_stream):
    im = Image.open(file_stream).convert(mode="L").resize((28, 28))
    img_array = np.array(im)
    inputs = np.stack([img_array], axis=0)
    outputs = self.artifacts.classifier(inputs)
    return FASHION_MNIST_CLASSES[outputs[0]]

Query with HTTP request performed by cURL:

curl -i \
  --header "Content-Type: image/jpeg" \
  --request POST \
  --data @test.jpg \
  localhost:5000/predict

OR:

curl -i \
  -F image=@test.jpg \
  localhost:5000/predict

OR by an HTML form that sends multipart data:

<form action="http://localhost:8000" method="POST"
      enctype="multipart/form-data">
    <input name="image" type="file">
    <input type="submit">
</form>

Query with CLI command:

bentoml run PyTorchFashionClassifier:latest predict --input-file test.jpg

OR infer all images under a folder with ten images each batch:

bentoml run PyTorchFashionClassifier:latest predict \
  --input-file folder/*.jpg --max-batch-size 10

ClipperInput

A special group of adapters that are designed to be used when deploying with Clipper.

class bentoml.adapters.ClipperBytesInput(http_input_example=None, **base_config)

ClipperInput that deals with input type Bytes

class bentoml.adapters.ClipperFloatsInput(http_input_example=None, **base_config)

ClipperInput that deals with input type Floats

class bentoml.adapters.ClipperIntsInput(http_input_example=None, **base_config)

ClipperInput that deals with input type Ints

class bentoml.adapters.ClipperDoublesInput(http_input_example=None, **base_config)

ClipperInput that deals with input type Doubles

class bentoml.adapters.ClipperStringsInput(http_input_example=None, **base_config)

ClipperInput that deals with input type String

API OutputAdapters

BaseOutputAdapter

class bentoml.adapters.BaseOutputAdapter(cors='*')

OutputAdapter is an layer between result of user defined API callback function and final output in a variety of different forms, such as HTTP response, command line stdout or AWS Lambda event object.

DefaultOutput

The default output adapter is used when there’s no output specified in an Inference API. It ensembles multiple common output adapter and dynamically choose one base on the return value of the user-defined inference API callback function.

class bentoml.adapters.DefaultOutput(**kwargs)

Detect suitable output adapter automatically and converts result of use defined API function into specific output.

Parameters

cors (str) – The value of the Access-Control-Allow-Origin header set in the AWS Lambda response object. Default is “*”. If set to None, the header will not be set.

DataframeOutput

class bentoml.adapters.DataframeOutput(output_orient='records', **kwargs)

Converts result of user defined API function into specific output.

Parameters

cors (str) – The value of the Access-Control-Allow-Origin header set in the AWS Lambda response object. Default is “*”. If set to None, the header will not be set.

TfTensorOutput

class bentoml.adapters.TfTensorOutput(cors='*')

Converts result of user defined API function into specific output.

Parameters

cors (str) – The value of the Access-Control-Allow-Origin header set in the AWS Lambda response object. Default is “*”. If set to None, the header will not be set.

JsonOutput

class bentoml.adapters.JsonOutput(cors='*')

Converts result of user defined API function into specific output.

Parameters

cors (str) – The value of the Access-Control-Allow-Origin header set in the AWS Lambda response object. Default is “*”. If set to None, the header will not be set.

Adapter Roadmap

The following adapter types are on our roadmap:

  • NpNdarrayInputAdapter

  • NpNdarrayOutputAdapter

  • FileOutputAdapter

  • ImageOutputAdapter

  • MultiFileOutputAdapter