API InputAdapters¶

BaseInputAdapter¶

class bentoml.adapters.BaseInputAdapter(http_input_example=None, **base_config)¶: InputAdapter is an abstraction layer between user defined API callback function and prediction request input in a variety of different forms, such as HTTP request body, command line arguments or AWS Lambda event object.

DataframeInput¶

class bentoml.adapters.DataframeInput(typ: str = 'frame', orient: Optional[str] = None, columns: Optional[Sequence[str]] = None, dtype: Optional[Mapping[str, object]] = None, input_dtypes: Optional[Mapping[str, object]] = None, **base_kwargs)¶

Convert various inputs(HTTP, Aws Lambda or CLI) to pandas dataframe, passing it to API functions.

Parameters

orient (str) –
Indication of expected JSON string format. Compatible JSON strings can be produced by to_json() with a corresponding orient value. The set of possible orients is:
- 'split' : dict like {index -> [index], columns -> [columns], data -> [values]}
- 'records' : list like [{column -> value}, ... , {column -> value}]
- 'index' : dict like {index -> {column -> value}}
- 'columns' : dict like {column -> {index -> value}}
- 'values' : just the values array
The allowed and default values depend on the value of the typ parameter.
- when typ == 'series' (not available now),
  - allowed orients are {'split','records','index'}
  - default is 'index'
  - The Series index must be unique for orient 'index'.
- when typ == 'frame',
  - allowed orients are {'split','records','index', 'columns','values'}
  - default is 'columns'
  - The DataFrame index must be unique for orients 'index' and 'columns'.
  - The DataFrame columns must be unique for orients 'index', 'columns', and 'records'.
typ ({'frame', 'series'}, default 'frame') – ** Note: ‘series’ is not supported now. ** The type of object to recover.
dtype (dict, default None) – If is None, infer dtypes; if a dict of column to dtype, then use those. Not applicable for orient='table'.
input_dtypes (dict, default None) – ** Deprecated ** The same as the dtype

Raises

ValueError – Incoming data is missing required columns in dtype:
ValueError – Incoming data format can not be handled. Only json and csv:

Examples

Example Service:

from bentoml import env, artifacts, api, BentoService
from bentoml.adapters import DataframeInput
from bentoml.frameworks.sklearn import SklearnModelArtifact

@env(infer_pip_packages=True)
@artifacts([SklearnModelArtifact('model')])
class IrisClassifier(BentoService):

    @api(
        input=DataframeInput(
            orient="records",
            columns=["sw", "sl", "pw", "pl"],
            dtype={"sw": "float", "sl": "float", "pw": "float", "pl": "float"},
        ),
        batch=True,
    )
    def predict(self, df):
        # Optional pre-processing, post-processing code goes here
        return self.artifacts.model.predict(df)

Query with HTTP request:

curl -i \
  --header "Content-Type: application/json" \
  --request POST \
  --data '[{"sw": 1, "sl": 2, "pw": 1, "pl": 2}]' \
  localhost:5000/predict

OR:

curl -i \
  --header "Content-Type: text/csv" \
  --request POST \
  --data-binary @file.csv \
  localhost:5000/predict

Query with CLI command:

bentoml run IrisClassifier:latest predict --input \
  '[{"sw": 1, "sl": 2, "pw": 1, "pl": 2}]'

OR:

bentoml run IrisClassifier:latest predict --format csv --input-file test.csv

JsonInput¶

class bentoml.adapters.JsonInput(http_input_example=None, **base_config)¶

JsonInput parses REST API request or CLI command into parsed_jsons(a list of json serializable object in python) and pass down to user defined API function

Examples

Example services:

Use JsonInput with batch=True

from typings import List
from bentoml.types import JsonSerializable

@bentoml.api(input=JsonInput(), batch=True)
def predict(self, parsed_json_list: List[JsonSerializable]):
    results = self.artifacts.classifier([j['text'] for j in parsed_json_list])
    return results

OR use JsonInput with batch=False (the default)

@bentoml.api(input=JsonInput())
def predict(self, parsed_json):
    results = self.artifacts.classifier([parsed_json['text']])
    return results[0]

For client prediction request, it is the same for both batch and non-batch API, the request should contain only one single input item:

curl -i \
  --header "Content-Type: application/json" \
  --request POST \
  --data '{"text": "best movie ever"}' \
  localhost:5000/predict

TfTensorInput¶

class bentoml.adapters.TfTensorInput(method='predict', **base_kwargs)¶

Tensor input adapter for Tensorflow models. Transform incoming tf tensor data from http request, cli or lambda event into tf tensor. The behavior should be compatible with tensorflow serving REST API: * https://www.tensorflow.org/tfx/serving/api_rest#classify_and_regress_api * https://www.tensorflow.org/tfx/serving/api_rest#predict_api

Parameters: method (*) – equivalence of serving API methods: (predict, classify, regress)
Raises: BentoMLException – BentoML currently doesn’t support Content-Type

Examples

Example Service:

import tensorflow as tf
import bentoml
from bentoml.adapters import TfTensorInput
from bentoml.frameworks.tensorflow import TensorflowSavedModelArtifact

@bentoml.env(infer_pip_packages=True)
@bentoml.artifacts([TensorflowSavedModelArtifact('model')])
class MyService(bentoml.BentoService):

    @bentoml.api(input=TfTensorInput(), batch=True)
    def predict(self, input: tf.Tensor):
        result = self.artifacts.model.predict(input)
        return result

Query with Http request:

curl -i –header “Content-Type: application/json” –request POST –data ‘{“instances”: [1]}’ localhost:5000/predict

Query with CLI command:

bentoml run MyService:latest predict --input \
  '{"instances": [1]}'

ImageInput¶

class bentoml.adapters.ImageInput(accept_image_formats=None, pilmode='RGB', **base_kwargs)¶

Convert incoming image data from http request, cli or lambda event into imageio array (a subclass of numpy.ndarray that has a meta attribute) and pass down to user defined API functions.

** To operate raw files or PIL.Image obj, use the low-level FileInput. **

Parameters

accept_image_formats (List[str]) – A list of acceptable image formats. Default value is loaded from bentoml config ‘apiserver/default_image_input_accept_file_extensions’, which is set to [‘.jpg’, ‘.png’, ‘.jpeg’, ‘.tiff’, ‘.webp’, ‘.bmp’] by default. List of all supported format can be found here: https://imageio.readthedocs.io/en/stable/formats.html
pilmode (str) – The pilmode to be used for reading image file into numpy array. Default value is ‘RGB’. Find more information at: https://imageio.readthedocs.io/en/stable/format_png-pil.html

Raises

ImportError – imageio package is required to use ImageInput:

Examples

Service using ImageInput:

from typing import List

import numpy as np
from bentoml import BentoService, api, artifacts
from bentoml.frameworks.tensorflow import TensorflowSavedModelArtifact
from bentoml.adapters import ImageInput

CLASS_NAMES = ['cat', 'dog']

@artifacts([TensorflowSavedModelArtifact('classifier')])
class PetClassification(BentoService):
    @api(input=ImageInput(), batch=True)
    def predict(
        self, image_arrays: List[imageio.core.utils.Array]
    ) -> List[str]:
        results = self.artifacts.classifer.predict(image_arrays)
        return [CLASS_NAMES[r] for r in results]

OR use ImageInput with batch=False (the default):

@api(input=ImageInput(), batch=False)
def predict(self, image_array: imageio.core.utils.Array) -> str:
    results = self.artifacts.classifer.predict([image_array])
    return CLASS_NAMES[results[0]]

Query with HTTP request:

curl -i \
  --header "Content-Type: image/jpeg" \
  --request POST \
  --data-binary @test.jpg \
  localhost:5000/predict

OR:

curl -i \
  -F image=@test.jpg \
  localhost:5000/predict

OR by an HTML form that sends multipart data:

<form action="http://localhost:8000" method="POST"
      enctype="multipart/form-data">
    <input name="image" type="file">
    <input type="submit">
</form>

OR by python requests:

import requests

with open("test.jpg", "rb") as f:
    image_bytes = f.read()  # from file path

files = {
    "image": ("test.jpg", image_bytes),
}
response = requests.post(your_url, files=files)

import requests
import PIL

pil_image = PIL.Image.open('test.jpg')

image_bytes = pil_image.tobytes()  # from PIL.Image

files = {
    "image": ("test.jpg", image_bytes),
}
response = requests.post(your_url, files=files)

Query with CLI command:

bentoml run PyTorchFashionClassifier:latest predict --input-file test.jpg

OR infer all images under a folder with ten images each batch:

bentoml run PyTorchFashionClassifier:latest predict \
  --input-file folder/*.jpg --max-batch-size 10

StringInput¶

class bentoml.adapters.StringInput(http_input_example=None, **base_config)¶

Convert various inputs(HTTP, Aws Lambda or CLI) to strings(list of str), passing it to API functions.

Parameters

none –
Errors –
------- – 400 UnicodeDecodeError 400 Unsupported charset
Request (Example) –
------- –
` curl -i --header "Content-Type: text/plain; charset=utf-8" --request POST --data 'best movie ever' localhost:5000/predict `

MultiImageInput¶

class bentoml.adapters.MultiImageInput(input_names=('image'), pilmode='RGB', accept_image_formats=None, **base_kwargs)¶

Transform incoming image data from http request, cli or lambda event into imageio arrays (a subclass of numpy.ndarray that has a meta attribute) and pass down to user defined API functions.

Parameters

input_names (List[str]) – A tuple of acceptable input name for HTTP request. Default value is (image,)
accept_image_formats (List[str]) – A list of acceptable image formats. Default value is loaded from bentoml config ‘apiserver/default_image_input_accept_file_extensions’, which is set to [‘.jpg’, ‘.png’, ‘.jpeg’, ‘.tiff’, ‘.webp’, ‘.bmp’] by default. List of all supported format can be found here: https://imageio.readthedocs.io/en/stable/formats.html
pilmode (str) – The pilmode to be used for reading image file into numpy array. Default value is ‘RGB’. Find more information at: https://imageio.readthedocs.io/en/stable/format_png-pil.html

Raises

ImportError – imageio package is required to use MultiImageInput:

Examples

Service using MultiImageInput:

from bentoml import BentoService
import bentoml

class MyService(BentoService):
    @bentoml.api(
        input=MultiImageInput(input_names=('imageX', 'imageY')), batch=True)
    def predict(self, image_groups):
        for image_group in image_groups:
            image_array_x = image_group['imageX']
            image_array_y = image_group['imageY']

Query the endpoint with HTTP request performed by cURL:

curl -i \
  -F imageX=@testx.jpg \
  -F imageY=@testy.jpg \
  localhost:5000/predict

OR by an HTML form that sends multipart data:

<form action="http://localhost:8000" method="POST"
      enctype="multipart/form-data">
    <input name="imageX" type="file">
    <input name="imageY" type="file">
    <input type="submit">
</form>

OR by Python requests:

import requests

with open("testx.jpg", "rb") as f:
    image_bytes1 = f.read()
with open("testy.jpg", "rb") as f:
    image_bytes2 = f.read()

files = {
    "imageX": ("testx.jpg", image_bytes1),
    "imageY": ("testy.jpg", image_bytes2),
}
response = requests.post(your_url, files=files)

import requests
import PIL

pil_image_x = PIL.Image.open('testx.jpg')
pil_image_y = PIL.Image.open('testy.jpg')

image_bytes1 = pil_image_x.tobytes()  # from PIL.Image
image_bytes1 = pil_image_y.tobytes()

files = {
    "imageX": ("testx.jpg", image_bytes1),
    "imageY": ("testy.jpg", image_bytes2),
}
response = requests.post(your_url, files=files)

Query with CLI command:

bentoml run PyTorchFashionClassifier:latest predict \
  --input-file-imageX testx.jpg \
  --input-file-imageY testy.jpg

OR infer all file pairs under a folder with ten pairs each batch:

bentoml run PyTorchFashionClassifier:latest predict --max-batch-size 10 \
  --input-file-imageX folderx/*.jpg \
  --input-file-imageY foldery/*.jpg

Note: jpg files and json files should be in same prefix like this:

folderx:
    - 1.jpg
    - 2.jpg
    ...
foldery:
    - 1.jpg
    - 2.jpg
    ...

Or the following cURL command

>>> curl -F imageX=@image_file_x.png
>>>      -F imageY=@image_file_y.jpg
>>>      http://localhost:8000

AnnotatedImageInput¶

class bentoml.adapters.AnnotatedImageInput(image_input_name='image', annotation_input_name='annotations', pilmode='RGB', accept_image_formats=None, **base_kwargs)¶

Transform incoming image data from http request, cli or lambda event into a imageio array (a subclass of numpy.ndarray that has a meta attribute), while allowing an optional JSON file for image annotations (such as object bounding boxes, class labels, etc.)

Transforms input image file into a numpy array, and loads JSON file as a JSON serializable Python object, providing them to user-defined API functions.

Parameters

image_input_name (str) – An acceptable input name for HTTP request and function keyword argument. Default value is “image”
annotation_input_name (str) – An acceptable input name for HTTP request and function keyword argument. Default value is “annotations”
accept_image_formats (List[str]) – A list of acceptable image formats. Default value is loaded from bentoml config ‘apiserver/default_image_input_accept_file_extensions’, which is set to [‘.jpg’, ‘.png’, ‘.jpeg’, ‘.tiff’, ‘.webp’, ‘.bmp’] by default. List of all supported format can be found here: https://imageio.readthedocs.io/en/stable/formats.html
pilmode (str) – The pilmode to be used for reading image file into numpy array. Default value is ‘RGB’. Find more information at: https://imageio.readthedocs.io/en/stable/format_png-pil.html

Raises

ImportError – imageio package is required to use AnnotatedImageInput:

Examples

from typing import Sequence

from bentoml import BentoService, api, artifacts
from bentoml.frameworks.tensorflow import TensorflowSavedModelArtifact
from bentoml.adapters import AnnotatedImageInput

CLASS_NAMES = ['cat', 'dog']

@artifacts([TensorflowSavedModelArtifact('classifier')])
class PetClassification(BentoService):
   @api(input=AnnotatedImageInput(), batch=True)
   def predict(
           self,
           image_list: 'Sequence[imageio.core.utils.Array]',
           annotations_list: 'Sequence[JsonSerializable]',
       ) -> Sequence[str]:
       cropped_pets = some_pet_finder(image_list, annotations_list)
       results = self.artifacts.classifer.predict(cropped_pets)
       return [CLASS_NAMES[r] for r in results]

Query with HTTP request performed by cURL:

curl -i \
  -F image=@test.jpg \
  -F annotations=@test.json \
  localhost:5000/predict

OR by an HTML form that sends multipart data:

<form action="http://localhost:8000" method="POST"
      enctype="multipart/form-data">
    <input name="image" type="file">
    <input name="annotations" type="file">
    <input type="submit">
</form>

OR by python requests:

import requests

with open("test.jpg", "rb") as f:
    image_bytes = f.read()
with open("anno.json", "rb") as f:
    anno_bytes = f.read()

files = {
    "image": ("test.jpg", image_bytes),
    "annotations": ("test.json", anno_bytes),
}
response = requests.post(your_url, files=files)

import requests
import PIL

pil_image = PIL.Image.open('test.jpg')
annotations = { "age": 10, "bar": "foo" }

image_bytes = pil_image.tobytes()
anno_bytes = json.dumps(annotations).encode('utf-8')

files = {
    "image": ("test.jpg", image_bytes),
    "annotations": ("test.json", anno_bytes),
}
response = requests.post(your_url, files=files)

Query with CLI command:

bentoml run PyTorchFashionClassifier:latest predict \
  --input-file-image test.jpg \
  --input-file-annotations test.json

OR infer all file pairs under a folder with ten pairs each batch:

bentoml run PyTorchFashionClassifier:latest predict --max-batch-size 10 \
  --input-file-image folder/*.jpg \
  --input-file-annotations folder/*.json

Note: jpg files and json files should be in same prefix like this:

folder:
    - apple.jpg
    - apple.json
    - banana.jpg
    - banana.json
    ...

FileInput¶

class bentoml.adapters.FileInput(http_input_example=None, **base_config)¶

Convert incoming file data from http request, cli or lambda event into file stream object and pass down to user defined API functions

Parameters: None –

Examples

Service using FileInput:

import bentoml
from PIL import Image
import numpy as np

from bentoml.frameworks.pytorch import PytorchModelArtifact
from bentoml.adapters import FileInput

FASHION_MNIST_CLASSES = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat',
                         'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']

@bentoml.env(pip_packages=['torch', 'pillow', 'numpy'])
@bentoml.artifacts([PytorchModelArtifact('classifier')])
class PyTorchFashionClassifier(bentoml.BentoService):

    @bentoml.api(input=FileInput(), batch=True)
    def predict(self, file_streams):
        img_arrays = []
        for fs in file_streams:
            im = Image.open(fs).convert(mode="L").resize((28, 28))
            img_array = np.array(im)
            img_arrays.append(img_array)

        inputs = np.stack(img_arrays, axis=0)

        outputs = self.artifacts.classifier(inputs)
        return [FASHION_MNIST_CLASSES[c] for c in outputs]

OR use FileInput with batch=False (the default):

@bentoml.api(input=FileInput(), batch=False)
def predict(self, file_stream):
    im = Image.open(file_stream).convert(mode="L").resize((28, 28))
    img_array = np.array(im)
    inputs = np.stack([img_array], axis=0)
    outputs = self.artifacts.classifier(inputs)
    return FASHION_MNIST_CLASSES[outputs[0]]

Query with HTTP request performed by cURL:

curl -i \
  --header "Content-Type: image/jpeg" \
  --request POST \
  --data-binary @test.jpg \
  localhost:5000/predict

OR:

curl -i \
  -F image=@test.jpg \
  localhost:5000/predict

OR by an HTML form that sends multipart data:

<form action="http://localhost:8000" method="POST"
      enctype="multipart/form-data">
    <input name="image" type="file">
    <input type="submit">
</form>

OR by python requests:

import requests

with open("test.jpg", "rb") as f:
    image_bytes = f.read()

files = {
    "image": ("test.jpg", image_bytes),
}
response = requests.post(your_url, files=files)

Query with CLI command:

bentoml run PyTorchFashionClassifier:latest predict --input-file test.jpg

OR infer all images under a folder with ten images each batch:

bentoml run PyTorchFashionClassifier:latest predict \
  --input-file folder/*.jpg --max-batch-size 10

MultiFileInput¶

class bentoml.adapters.MultiFileInput(input_names: Sequence[str], allow_none: bool = False, **base_kwargs)¶

Low level input adapters that transform incoming files data from http request, CLI or AWS lambda event into binary stream objects, then pass down to user defined API functions.

Parameters

input_names (List[str]) – list of input names. For HTTP they are form input names. For CLI they are CLI args –input-<name1> or –input-file-<name1>
allow_none (bool) – accept HTTP requests or AWS Lambda events without all files provided. Does not take effect on CLI.

Examples

Service using MultiFileInput:

from typing import List

from PIL import Image
import numpy as np
import bentoml
from bentoml.types import FileLike
from bentoml.framework.pytroch import PytorchModelArtifact
from bentoml.adapters import MultiFileInput

@bentoml.env(pip_packages=['torch', 'pillow', 'numpy'])
@bentoml.artifacts([PytorchModelArtifact('classifier')])
class PyTorchFashionClassifier(bentoml.BentoService):
    @bentoml.api(
        input=MultiFileInput(input_names=['image', 'json']), batch=True)
    def predict(self, image_list: List[FileLike], json_list: List[FileLike]):
        inputs = []
        for img_io, json_io in zip(image_list, json_list):
            img = Image.open(img_io)
            json_obj = json.load(json_io)
            inputs.append([img, json_obj])
        outputs = self.artifacts.classifier(inputs)
        return outputs

Query with HTTP request performed by cURL:

curl -i \
  -F image=@test.jpg \
  -F json=@test.json \
  localhost:5000/predict

OR by an HTML form that sends multipart data:

<form action="http://localhost:8000" method="POST"
      enctype="multipart/form-data">
    <input name="image" type="file">
    <input name="json" type="file">
    <input type="submit">
</form>

OR by python requests:

import requests

with open("test.jpg", "rb") as f:
    image_bytes = f.read()
with open("anno.json", "rb") as f:
    json_bytes = f.read()

files = {
    "image": ("test.jpg", image_bytes),
    "json": ("test.json", json_bytes),
}
response = requests.post(your_url, files=files)

Query with CLI command:

bentoml run PyTorchFashionClassifier:latest predict \
  --input-file-image test.jpg \
  --input-file-json test.json

OR infer all file pairs under a folder with ten pairs each batch:

bentoml run PyTorchFashionClassifier:latest predict --max-batch-size 10 \
  --input-file-image folder/*.jpg \
  --input-file-json folder/*.json

Note: jpg files and json files should be in same prefix like this:

folder:
    - apple.jpg
    - apple.json
    - banana.jpg
    - banana.json
    ...

ClipperInput¶

A special group of adapters that are designed to be used when deploying with Clipper.

class bentoml.adapters.ClipperBytesInput(http_input_example=None, **base_config)¶: ClipperInput that deals with input type Bytes

class bentoml.adapters.ClipperFloatsInput(http_input_example=None, **base_config)¶: ClipperInput that deals with input type Floats

class bentoml.adapters.ClipperIntsInput(http_input_example=None, **base_config)¶: ClipperInput that deals with input type Ints

class bentoml.adapters.ClipperDoublesInput(http_input_example=None, **base_config)¶: ClipperInput that deals with input type Doubles

class bentoml.adapters.ClipperStringsInput(http_input_example=None, **base_config)¶: ClipperInput that deals with input type String

API OutputAdapters¶

BaseOutputAdapter¶

class bentoml.adapters.BaseOutputAdapter(cors='*')¶: OutputAdapter is an layer between result of user defined API callback function and final output in a variety of different forms, such as HTTP response, command line stdout or AWS Lambda event object.

DefaultOutput¶

The default output adapter is used when there’s no output specified in an Inference API. It ensembles multiple common output adapter and dynamically choose one base on the return value of the user-defined inference API callback function.

class bentoml.adapters.DefaultOutput(**kwargs)¶

Detect suitable output adapter automatically and converts result of use defined API function into specific output.

Parameters: cors (str) – The value of the Access-Control-Allow-Origin header set in the AWS Lambda response object. Default is “*”. If set to None, the header will not be set.

DataframeOutput¶

class bentoml.adapters.DataframeOutput(output_orient='records', **kwargs)¶

Converts result of user defined API function into specific output.

Parameters: cors (str) – The value of the Access-Control-Allow-Origin header set in the AWS Lambda response object. Default is “*”. If set to None, the header will not be set.

TfTensorOutput¶

class bentoml.adapters.TfTensorOutput(cors='*')¶

Converts result of user defined API function into specific output.

Parameters: cors (str) – The value of the Access-Control-Allow-Origin header set in the AWS Lambda response object. Default is “*”. If set to None, the header will not be set.

JsonOutput¶

class bentoml.adapters.JsonOutput(cors='*')¶

Converts result of user defined API function into specific output.

Parameters: cors (str) – The value of the Access-Control-Allow-Origin header set in the AWS Lambda response object. Default is “*”. If set to None, the header will not be set.

Adapter Roadmap¶

The following adapter types are on our roadmap:

NpNdarrayInputAdapter
NpNdarrayOutputAdapter
FileOutputAdapter
ImageOutputAdapter
MultiFileOutputAdapter