API InputAdapters¶
BaseInputAdapter¶
-
class
bentoml.adapters.
BaseInputAdapter
(http_input_example=None, **base_config)¶ InputAdapter is an abstraction layer between user defined API callback function and prediction request input in a variety of different forms, such as HTTP request body, command line arguments or AWS Lambda event object.
DataframeInput¶
-
class
bentoml.adapters.
DataframeInput
(typ: str = 'frame', orient: Optional[str] = None, columns: Optional[Sequence[str]] = None, dtype: Optional[Mapping[str, object]] = None, input_dtypes: Optional[Mapping[str, object]] = None, **base_kwargs)¶ Convert various inputs(HTTP, Aws Lambda or CLI) to pandas dataframe, passing it to API functions.
- Parameters
orient (str) –
Indication of expected JSON string format. Compatible JSON strings can be produced by
to_json()
with a corresponding orient value. The set of possible orients is:'split'
: dict like{index -> [index], columns -> [columns], data -> [values]}
'records'
: list like[{column -> value}, ... , {column -> value}]
'index'
: dict like{index -> {column -> value}}
'columns'
: dict like{column -> {index -> value}}
'values'
: just the values array
The allowed and default values depend on the value of the typ parameter.
when
typ == 'series'
(not available now),allowed orients are
{'split','records','index'}
default is
'index'
The Series index must be unique for orient
'index'
.
when
typ == 'frame'
,allowed orients are
{'split','records','index', 'columns','values'}
default is
'columns'
The DataFrame index must be unique for orients
'index'
and'columns'
.The DataFrame columns must be unique for orients
'index'
,'columns'
, and'records'
.
typ ({'frame', 'series'}, default 'frame') – ** Note: ‘series’ is not supported now. ** The type of object to recover.
dtype (dict, default None) – If is None, infer dtypes; if a dict of column to dtype, then use those. Not applicable for
orient='table'
.input_dtypes (dict, default None) – ** Deprecated ** The same as the dtype
- Raises
ValueError – Incoming data is missing required columns in dtype:
ValueError – Incoming data format can not be handled. Only json and csv:
Examples
Example Service:
from bentoml import env, artifacts, api, BentoService from bentoml.adapters import DataframeInput from bentoml.frameworks.sklearn import SklearnModelArtifact @env(infer_pip_packages=True) @artifacts([SklearnModelArtifact('model')]) class IrisClassifier(BentoService): @api( input=DataframeInput( orient="records", columns=["sw", "sl", "pw", "pl"], dtype={"sw": "float", "sl": "float", "pw": "float", "pl": "float"}, ), batch=True, ) def predict(self, df): # Optional pre-processing, post-processing code goes here return self.artifacts.model.predict(df)
Query with HTTP request:
curl -i \ --header "Content-Type: application/json" \ --request POST \ --data '[{"sw": 1, "sl": 2, "pw": 1, "pl": 2}]' \ localhost:5000/predict
OR:
curl -i \ --header "Content-Type: text/csv" \ --request POST \ --data-binary @file.csv \ localhost:5000/predict
Query with CLI command:
bentoml run IrisClassifier:latest predict --input \ '[{"sw": 1, "sl": 2, "pw": 1, "pl": 2}]'
OR:
bentoml run IrisClassifier:latest predict --format csv --input-file test.csv
JsonInput¶
-
class
bentoml.adapters.
JsonInput
(http_input_example=None, **base_config)¶ JsonInput parses REST API request or CLI command into parsed_jsons(a list of json serializable object in python) and pass down to user defined API function
Examples
Example services:
Use JsonInput with
batch=True
from typings import List from bentoml.types import JsonSerializable @bentoml.api(input=JsonInput(), batch=True) def predict(self, parsed_json_list: List[JsonSerializable]): results = self.artifacts.classifier([j['text'] for j in parsed_json_list]) return results
OR use JsonInput with
batch=False
(the default)@bentoml.api(input=JsonInput()) def predict(self, parsed_json): results = self.artifacts.classifier([parsed_json['text']]) return results[0]
For client prediction request, it is the same for both batch and non-batch API, the request should contain only one single input item:
curl -i \ --header "Content-Type: application/json" \ --request POST \ --data '{"text": "best movie ever"}' \ localhost:5000/predict
TfTensorInput¶
-
class
bentoml.adapters.
TfTensorInput
(method='predict', **base_kwargs)¶ Tensor input adapter for Tensorflow models. Transform incoming tf tensor data from http request, cli or lambda event into tf tensor. The behavior should be compatible with tensorflow serving REST API: * https://www.tensorflow.org/tfx/serving/api_rest#classify_and_regress_api * https://www.tensorflow.org/tfx/serving/api_rest#predict_api
- Parameters
method (*) – equivalence of serving API methods: (predict, classify, regress)
- Raises
BentoMLException – BentoML currently doesn’t support Content-Type
Examples
Example Service:
import tensorflow as tf import bentoml from bentoml.adapters import TfTensorInput from bentoml.frameworks.tensorflow import TensorflowSavedModelArtifact @bentoml.env(infer_pip_packages=True) @bentoml.artifacts([TensorflowSavedModelArtifact('model')]) class MyService(bentoml.BentoService): @bentoml.api(input=TfTensorInput(), batch=True) def predict(self, input: tf.Tensor): result = self.artifacts.model.predict(input) return result
Query with Http request:
curl -i –header “Content-Type: application/json” –request POST –data ‘{“instances”: [1]}’ localhost:5000/predict
Query with CLI command:
bentoml run MyService:latest predict --input \ '{"instances": [1]}'
ImageInput¶
-
class
bentoml.adapters.
ImageInput
(accept_image_formats=None, pilmode='RGB', **base_kwargs)¶ Convert incoming image data from http request, cli or lambda event into imageio array (a subclass of numpy.ndarray that has a meta attribute) and pass down to user defined API functions.
** To operate raw files or PIL.Image obj, use the low-level
FileInput
. **- Parameters
accept_image_formats (List[str]) – A list of acceptable image formats. Default value is loaded from bentoml config ‘apiserver/default_image_input_accept_file_extensions’, which is set to [‘.jpg’, ‘.png’, ‘.jpeg’, ‘.tiff’, ‘.webp’, ‘.bmp’] by default. List of all supported format can be found here: https://imageio.readthedocs.io/en/stable/formats.html
pilmode (str) – The pilmode to be used for reading image file into numpy array. Default value is ‘RGB’. Find more information at: https://imageio.readthedocs.io/en/stable/format_png-pil.html
- Raises
ImportError – imageio package is required to use ImageInput:
Examples
Service using ImageInput:
from typing import List import numpy as np from bentoml import BentoService, api, artifacts from bentoml.frameworks.tensorflow import TensorflowSavedModelArtifact from bentoml.adapters import ImageInput CLASS_NAMES = ['cat', 'dog'] @artifacts([TensorflowSavedModelArtifact('classifier')]) class PetClassification(BentoService): @api(input=ImageInput(), batch=True) def predict( self, image_arrays: List[imageio.core.utils.Array] ) -> List[str]: results = self.artifacts.classifer.predict(image_arrays) return [CLASS_NAMES[r] for r in results]
OR use ImageInput with
batch=False
(the default):@api(input=ImageInput(), batch=False) def predict(self, image_array: imageio.core.utils.Array) -> str: results = self.artifacts.classifer.predict([image_array]) return CLASS_NAMES[results[0]]
Query with HTTP request:
curl -i \ --header "Content-Type: image/jpeg" \ --request POST \ --data-binary @test.jpg \ localhost:5000/predict
OR:
curl -i \ -F image=@test.jpg \ localhost:5000/predict
OR by an HTML form that sends multipart data:
<form action="http://localhost:8000" method="POST" enctype="multipart/form-data"> <input name="image" type="file"> <input type="submit"> </form>
OR by python requests:
import requests with open("test.jpg", "rb") as f: image_bytes = f.read() # from file path files = { "image": ("test.jpg", image_bytes), } response = requests.post(your_url, files=files)
import requests import PIL pil_image = PIL.Image.open('test.jpg') image_bytes = pil_image.tobytes() # from PIL.Image files = { "image": ("test.jpg", image_bytes), } response = requests.post(your_url, files=files)
Query with CLI command:
bentoml run PyTorchFashionClassifier:latest predict --input-file test.jpg
OR infer all images under a folder with ten images each batch:
bentoml run PyTorchFashionClassifier:latest predict \ --input-file folder/*.jpg --max-batch-size 10
StringInput¶
-
class
bentoml.adapters.
StringInput
(http_input_example=None, **base_config)¶ Convert various inputs(HTTP, Aws Lambda or CLI) to strings(list of str), passing it to API functions.
- Errors
400 UnicodeDecodeError
400 Unsupported charset
Example Request
>>> curl -i \ >>> --header "Content-Type: text/plain; charset=utf-8" \ >>> --request POST \ >>> --data 'best movie ever' \ >>> localhost:5000/predict
MultiImageInput¶
-
class
bentoml.adapters.
MultiImageInput
(input_names=('image'), pilmode='RGB', accept_image_formats=None, **base_kwargs)¶ Transform incoming image data from http request, cli or lambda event into imageio arrays (a subclass of numpy.ndarray that has a meta attribute) and pass down to user defined API functions.
- Parameters
input_names (List[str]) – A tuple of acceptable input name for HTTP request. Default value is (image,)
accept_image_formats (List[str]) – A list of acceptable image formats. Default value is loaded from bentoml config ‘apiserver/default_image_input_accept_file_extensions’, which is set to [‘.jpg’, ‘.png’, ‘.jpeg’, ‘.tiff’, ‘.webp’, ‘.bmp’] by default. List of all supported format can be found here: https://imageio.readthedocs.io/en/stable/formats.html
pilmode (str) – The pilmode to be used for reading image file into numpy array. Default value is ‘RGB’. Find more information at: https://imageio.readthedocs.io/en/stable/format_png-pil.html
- Raises
ImportError – imageio package is required to use MultiImageInput:
Examples
Service using MultiImageInput:
from bentoml import BentoService import bentoml class MyService(BentoService): @bentoml.api( input=MultiImageInput(input_names=('imageX', 'imageY')), batch=True) def predict(self, image_groups): for image_group in image_groups: image_array_x = image_group['imageX'] image_array_y = image_group['imageY']
Query the endpoint with HTTP request performed by cURL:
curl -i \ -F imageX=@testx.jpg \ -F imageY=@testy.jpg \ localhost:5000/predict
OR by an HTML form that sends multipart data:
<form action="http://localhost:8000" method="POST" enctype="multipart/form-data"> <input name="imageX" type="file"> <input name="imageY" type="file"> <input type="submit"> </form>
OR by Python requests:
import requests with open("testx.jpg", "rb") as f: image_bytes1 = f.read() with open("testy.jpg", "rb") as f: image_bytes2 = f.read() files = { "imageX": ("testx.jpg", image_bytes1), "imageY": ("testy.jpg", image_bytes2), } response = requests.post(your_url, files=files)
import requests import PIL pil_image_x = PIL.Image.open('testx.jpg') pil_image_y = PIL.Image.open('testy.jpg') image_bytes1 = pil_image_x.tobytes() # from PIL.Image image_bytes1 = pil_image_y.tobytes() files = { "imageX": ("testx.jpg", image_bytes1), "imageY": ("testy.jpg", image_bytes2), } response = requests.post(your_url, files=files)
Query with CLI command:
bentoml run PyTorchFashionClassifier:latest predict \ --input-file-imageX testx.jpg \ --input-file-imageY testy.jpg
OR infer all file pairs under a folder with ten pairs each batch:
bentoml run PyTorchFashionClassifier:latest predict --max-batch-size 10 \ --input-file-imageX folderx/*.jpg \ --input-file-imageY foldery/*.jpg
Note: jpg files and json files should be in same prefix like this:
folderx: - 1.jpg - 2.jpg ... foldery: - 1.jpg - 2.jpg ...
Or the following cURL command
>>> curl -F imageX=@image_file_x.png >>> -F imageY=@image_file_y.jpg >>> http://localhost:8000
AnnotatedImageInput¶
-
class
bentoml.adapters.
AnnotatedImageInput
(image_input_name='image', annotation_input_name='annotations', pilmode='RGB', accept_image_formats=None, **base_kwargs)¶ Transform incoming image data from http request, cli or lambda event into a imageio array (a subclass of numpy.ndarray that has a meta attribute), while allowing an optional JSON file for image annotations (such as object bounding boxes, class labels, etc.)
Transforms input image file into a numpy array, and loads JSON file as a JSON serializable Python object, providing them to user-defined API functions.
- Parameters
image_input_name (str) – An acceptable input name for HTTP request and function keyword argument. Default value is “image”
annotation_input_name (str) – An acceptable input name for HTTP request and function keyword argument. Default value is “annotations”
accept_image_formats (List[str]) – A list of acceptable image formats. Default value is loaded from bentoml config ‘apiserver/default_image_input_accept_file_extensions’, which is set to [‘.jpg’, ‘.png’, ‘.jpeg’, ‘.tiff’, ‘.webp’, ‘.bmp’] by default. List of all supported format can be found here: https://imageio.readthedocs.io/en/stable/formats.html
pilmode (str) – The pilmode to be used for reading image file into numpy array. Default value is ‘RGB’. Find more information at: https://imageio.readthedocs.io/en/stable/format_png-pil.html
- Raises
ImportError – imageio package is required to use AnnotatedImageInput:
Examples
from typing import Sequence from bentoml import BentoService, api, artifacts from bentoml.frameworks.tensorflow import TensorflowSavedModelArtifact from bentoml.adapters import AnnotatedImageInput CLASS_NAMES = ['cat', 'dog'] @artifacts([TensorflowSavedModelArtifact('classifier')]) class PetClassification(BentoService): @api(input=AnnotatedImageInput(), batch=True) def predict( self, image_list: 'Sequence[imageio.core.utils.Array]', annotations_list: 'Sequence[JsonSerializable]', ) -> Sequence[str]: cropped_pets = some_pet_finder(image_list, annotations_list) results = self.artifacts.classifer.predict(cropped_pets) return [CLASS_NAMES[r] for r in results]
Query with HTTP request performed by cURL:
curl -i \ -F image=@test.jpg \ -F annotations=@test.json \ localhost:5000/predict
OR by an HTML form that sends multipart data:
<form action="http://localhost:8000" method="POST" enctype="multipart/form-data"> <input name="image" type="file"> <input name="annotations" type="file"> <input type="submit"> </form>
OR by python requests:
import requests with open("test.jpg", "rb") as f: image_bytes = f.read() with open("anno.json", "rb") as f: anno_bytes = f.read() files = { "image": ("test.jpg", image_bytes), "annotations": ("test.json", anno_bytes), } response = requests.post(your_url, files=files)
import requests import PIL pil_image = PIL.Image.open('test.jpg') annotations = { "age": 10, "bar": "foo" } image_bytes = pil_image.tobytes() anno_bytes = json.dumps(annotations).encode('utf-8') files = { "image": ("test.jpg", image_bytes), "annotations": ("test.json", anno_bytes), } response = requests.post(your_url, files=files)
Query with CLI command:
bentoml run PyTorchFashionClassifier:latest predict \ --input-file-image test.jpg \ --input-file-annotations test.json
OR infer all file pairs under a folder with ten pairs each batch:
bentoml run PyTorchFashionClassifier:latest predict --max-batch-size 10 \ --input-file-image folder/*.jpg \ --input-file-annotations folder/*.json
Note: jpg files and json files should be in same prefix like this:
folder: - apple.jpg - apple.json - banana.jpg - banana.json ...
FileInput¶
-
class
bentoml.adapters.
FileInput
(http_input_example=None, **base_config)¶ Convert incoming file data from http request, cli or lambda event into file stream object and pass down to user defined API functions
- Parameters
None –
Examples
Service using FileInput:
import bentoml from PIL import Image import numpy as np from bentoml.frameworks.pytorch import PytorchModelArtifact from bentoml.adapters import FileInput FASHION_MNIST_CLASSES = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat', 'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot'] @bentoml.env(pip_packages=['torch', 'pillow', 'numpy']) @bentoml.artifacts([PytorchModelArtifact('classifier')]) class PyTorchFashionClassifier(bentoml.BentoService): @bentoml.api(input=FileInput(), batch=True) def predict(self, file_streams): img_arrays = [] for fs in file_streams: im = Image.open(fs).convert(mode="L").resize((28, 28)) img_array = np.array(im) img_arrays.append(img_array) inputs = np.stack(img_arrays, axis=0) outputs = self.artifacts.classifier(inputs) return [FASHION_MNIST_CLASSES[c] for c in outputs]
OR use FileInput with
batch=False
(the default):@bentoml.api(input=FileInput(), batch=False) def predict(self, file_stream): im = Image.open(file_stream).convert(mode="L").resize((28, 28)) img_array = np.array(im) inputs = np.stack([img_array], axis=0) outputs = self.artifacts.classifier(inputs) return FASHION_MNIST_CLASSES[outputs[0]]
Query with HTTP request performed by cURL:
curl -i \ --header "Content-Type: image/jpeg" \ --request POST \ --data-binary @test.jpg \ localhost:5000/predict
OR:
curl -i \ -F image=@test.jpg \ localhost:5000/predict
OR by an HTML form that sends multipart data:
<form action="http://localhost:8000" method="POST" enctype="multipart/form-data"> <input name="image" type="file"> <input type="submit"> </form>
OR by python requests:
import requests with open("test.jpg", "rb") as f: image_bytes = f.read() files = { "image": ("test.jpg", image_bytes), } response = requests.post(your_url, files=files)
Query with CLI command:
bentoml run PyTorchFashionClassifier:latest predict --input-file test.jpg
OR infer all images under a folder with ten images each batch:
bentoml run PyTorchFashionClassifier:latest predict \ --input-file folder/*.jpg --max-batch-size 10
MultiFileInput¶
-
class
bentoml.adapters.
MultiFileInput
(input_names: Sequence[str], allow_none: bool = False, **base_kwargs)¶ Low level input adapters that transform incoming files data from http request, CLI or AWS lambda event into binary stream objects, then pass down to user defined API functions.
- Parameters
input_names (List[str]) – list of input names. For HTTP they are form input names. For CLI they are CLI args –input-<name1> or –input-file-<name1>
allow_none (bool) – accept HTTP requests or AWS Lambda events without all files provided. Does not take effect on CLI.
Examples
Service using MultiFileInput:
from typing import List from PIL import Image import numpy as np import bentoml from bentoml.types import FileLike from bentoml.framework.pytroch import PytorchModelArtifact from bentoml.adapters import MultiFileInput @bentoml.env(pip_packages=['torch', 'pillow', 'numpy']) @bentoml.artifacts([PytorchModelArtifact('classifier')]) class PyTorchFashionClassifier(bentoml.BentoService): @bentoml.api( input=MultiFileInput(input_names=['image', 'json']), batch=True) def predict(self, image_list: List[FileLike], json_list: List[FileLike]): inputs = [] for img_io, json_io in zip(image_list, json_list): img = Image.open(img_io) json_obj = json.load(json_io) inputs.append([img, json_obj]) outputs = self.artifacts.classifier(inputs) return outputs
Query with HTTP request performed by cURL:
curl -i \ -F image=@test.jpg \ -F json=@test.json \ localhost:5000/predict
OR by an HTML form that sends multipart data:
<form action="http://localhost:8000" method="POST" enctype="multipart/form-data"> <input name="image" type="file"> <input name="json" type="file"> <input type="submit"> </form>
OR by python requests:
import requests with open("test.jpg", "rb") as f: image_bytes = f.read() with open("anno.json", "rb") as f: json_bytes = f.read() files = { "image": ("test.jpg", image_bytes), "json": ("test.json", json_bytes), } response = requests.post(your_url, files=files)
Query with CLI command:
bentoml run PyTorchFashionClassifier:latest predict \ --input-file-image test.jpg \ --input-file-json test.json
OR infer all file pairs under a folder with ten pairs each batch:
bentoml run PyTorchFashionClassifier:latest predict --max-batch-size 10 \ --input-file-image folder/*.jpg \ --input-file-json folder/*.json
Note: jpg files and json files should be in same prefix like this:
folder: - apple.jpg - apple.json - banana.jpg - banana.json ...
ClipperInput¶
A special group of adapters that are designed to be used when deploying with Clipper.
-
class
bentoml.adapters.
ClipperBytesInput
(http_input_example=None, **base_config)¶ ClipperInput that deals with input type Bytes
-
class
bentoml.adapters.
ClipperFloatsInput
(http_input_example=None, **base_config)¶ ClipperInput that deals with input type Floats
-
class
bentoml.adapters.
ClipperIntsInput
(http_input_example=None, **base_config)¶ ClipperInput that deals with input type Ints
-
class
bentoml.adapters.
ClipperDoublesInput
(http_input_example=None, **base_config)¶ ClipperInput that deals with input type Doubles
-
class
bentoml.adapters.
ClipperStringsInput
(http_input_example=None, **base_config)¶ ClipperInput that deals with input type String
API OutputAdapters¶
BaseOutputAdapter¶
-
class
bentoml.adapters.
BaseOutputAdapter
(cors='*')¶ OutputAdapter is an layer between result of user defined API callback function and final output in a variety of different forms, such as HTTP response, command line stdout or AWS Lambda event object.
DefaultOutput¶
The default output adapter is used when there’s no output specified in an Inference API. It ensembles multiple common output adapter and dynamically choose one base on the return value of the user-defined inference API callback function.
-
class
bentoml.adapters.
DefaultOutput
(**kwargs)¶ Detect suitable output adapter automatically and converts result of use defined API function into specific output.
- Parameters
cors (str) – The value of the Access-Control-Allow-Origin header set in the AWS Lambda response object. Default is “*”. If set to None, the header will not be set.
DataframeOutput¶
-
class
bentoml.adapters.
DataframeOutput
(output_orient='records', **kwargs)¶ Converts result of user defined API function into specific output.
- Parameters
cors (str) – The value of the Access-Control-Allow-Origin header set in the AWS Lambda response object. Default is “*”. If set to None, the header will not be set.
TfTensorOutput¶
-
class
bentoml.adapters.
TfTensorOutput
(ensure_ascii=False, **kwargs)¶ Converts result of user defined API function into specific output.
- Parameters
cors (str) – The value of the Access-Control-Allow-Origin header set in the AWS Lambda response object. Default is “*”. If set to None, the header will not be set.
JsonOutput¶
-
class
bentoml.adapters.
JsonOutput
(ensure_ascii=False, **kwargs)¶ Converts result of user defined API function into specific output.
- Parameters
cors (str) – The value of the Access-Control-Allow-Origin header set in the AWS Lambda response object. Default is “*”. If set to None, the header will not be set.
ensure_ascii (bool) – Escape all non-ASCII characters. Default False.
Adapter Roadmap¶
The following adapter types are on our roadmap:
NpNdarrayInputAdapter
NpNdarrayOutputAdapter
FileOutputAdapter
ImageOutputAdapter
MultiFileOutputAdapter