Advanced Containerization#
time expected: 12 minutes
This guide describes advanced containerization options provided by BentoML:
This is an advanced feature for user to customize container environment that are not directly supported in BentoML. For basic containerizing options, see Docker Options.
Why you may need this?#
If you want to customize the containerization process of your Bento.
If you need a certain tools, configs, prebuilt binaries that is available across all your Bento generated container images.
A big difference with base image features is that you donât have to setup a custom base image and then push it to a remote registry.
Custom Base Image#
If none of the provided distros work for your use case, e.g. if your infrastructure requires all docker images to be derived from the same base image with certain security fixes and libraries, you can config BentoML to use your base image instead:
docker:
base_image: "my_custom_image:latest"
When a base_image
is provided, all other docker options will be ignored,
(distro, cuda_version, system_packages, python_version). bentoml containerize
will build a new image on top of the base_image with the following steps:
setup env vars
run the
setup_script
if providedinstall the required Python packages
copy over the Bento file
setup the entrypoint command for serving.
Note
Warning: user must ensure that the provided base image has desired
Python version installed. If the base image you have doesnât have Python, you may
install python via a setup_script
. The implementation of the script depends
on the base image distro or the package manager available.
docker:
base_image: "my_custom_image:latest"
setup_script: "./setup.sh"
Warning
By default, BentoML supports multi-platform docker image build out-of-the-box.
However, when a custom base_image
is provided, the generated Dockerfile can
only be used for building linux/amd64 platform docker images.
If you are running BentoML from an Apple M1 device or an ARM based computer, make
sure to pass the --opt platform=linux/amd64
parameter when containerizing a Bento. e.g.:
bentoml containerize iris_classifier:latest --opt platform=linux/amd64
Dockerfile Template#
The dockerfile_template
field gives the user full control over how the
Dockerfile
is generated for a Bento by extending the template used by
BentoML.
First, create a Dockerfile.template
file next to your bentofile.yaml
build file. This file should follow the
Jinja2 template language, and extend
BentoMLâs base template and blocks. The template should render a valid
Dockerfile. For example:
{% extends bento_base_template %}
{% block SETUP_BENTO_COMPONENTS %}
{{ super() }}
RUN echo "We are running this during bentoml containerize!"
{% endblock %}
Then add the path to your template file to the dockerfile_template
field in
your :code: bentofile.yaml:
docker:
dockerfile_template: "./Dockerfile.template"
Now run bentoml build
to build a new Bento. It will contain a Dockerfile
generated with the custom template. To confirm the generated Dockerfile works as
expected, run bentoml containerize <bento>
to build a docker image with it.
During development and debugging, you may want to see the generated Dockerfile. Hereâs shortcut for that:
cat "$(bentoml get <bento>:<tag> -o path)/env/docker/Dockerfile"
Examples#
Building TensorFlow custom op#
Letâs start with an example that builds a custom TensorFlow op binary into a Bento, which is based on zero_out.cc
implementation details:
Define the following Dockerfile.template
:
{% extends bento_base_template %}
{% block SETUP_BENTO_BASE_IMAGE %}
{{ super() }}
WORKDIR /tmp
COPY ./src/tfops/zero_out.cc .
RUN pip3 install tensorflow
RUN set -ex && \
TF_CFLAGS=( $(python3 -c 'import tensorflow as tf; print(" ".join(tf.sysconfig.get_compile_flags()))') ) && \
TF_LFLAGS=( $(python3 -c 'import tensorflow as tf; print(" ".join(tf.sysconfig.get_link_flags()))') ) && \
g++ --std=c++14 -shared zero_out.cc -o zero_out.so -fPIC ${TF_CFLAGS[@]} ${TF_LFLAGS[@]} -I$(python -c 'import tensorflow as tf; print(tf.sysconfig.get_include());') -D_GLIBCXX_USE_CXX11_ABI=0 -O2
{% endblock %}
{% block SETUP_BENTO_COMPONENTS %}
{{ super() }}
RUN stat /usr/lib/zero_out.so
{% endblock %}
Then add the following to your bentofile.yaml
:
include:
- "zero_out.cc"
python:
packages:
- tensorflow
docker:
dockerfile_template: ./Dockerfile.template
Proceed to build your Bento with bentoml build
and containerize with bentoml containerize
:
bentoml build
bentoml containerize <bento>:<tag>
Tip
You can also provide --progress plain
to see the progress from
buildkit in plain text
bentoml containerize --progress plain <bento>:<tag>
Access AWS credentials during image build#
We will now demonstrate how to provide AWS credentials to a Bento via two approaches:
Note
Remarks: We recommend for most cases to use the second option (Mount credentials from host) as it prevents any securities leak.
By default BentoML uses the latest dockerfile frontend which allows mounting secrets to container.
For both examples, you will need to add the following to your bentofile.yaml
:
python:
packages:
- awscli
docker:
dockerfile_template: ./Dockerfile.template
Using environment variables#
Define the following Dockerfile.template
:
{% extends bento_base_template %}
{% block SETUP_BENTO_BASE_IMAGE %}
ARG AWS_SECRET_ACCESS_KEY
ARG AWS_ACCESS_KEY_ID
{{ super() }}
ARG AWS_SECRET_ACCESS_KEY
ARG AWS_ACCESS_KEY_ID
ENV AWS_SECRET_ACCESS_KEY=$ARG AWS_SECRET_ACCESS_KEY
ENV AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
{% endblock %}
{% block SETUP_BENTO_COMPONENTS %}
{{ super() }}
RUN aws s3 cp s3://path/to/file {{ bento__path }}
{% endblock %}
After building the bento with bentoml build
, you can then
pass AWS_SECRET_ACCESS_KEY
and AWS_ACCESS_KEY_ID
as arguments to bentoml containerize
:
bentoml containerize --build-arg AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \
--build-arg AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
<bento>:<tag>
Mount credentials from host#
Define the following Dockerfile.template
:
{% extends bento_base_template %}
{% block SETUP_BENTO_COMPONENTS %}
{{ super() }}
RUN --mount=type=secret,id=aws,target=/root/.aws/credentials \
aws s3 cp s3://path/to/file {{ bento__path }}
{% endblock %}
Follow the above addition to bentofile.yaml
to include awscli
and
the custom dockerfile template.
To pass in secrets to the Bento, pass it via --secret
to bentoml
containerize
:
bentoml containerize --secret id=aws,src=$HOME/.aws/credentials <bento>:<tag>
See also
Writing dockerfile_template
#
BentoML utilize Jinja2 to
structure a Dockerfile.template
.
The Dockerfile template is a mix between Jinja2
syntax and Dockerfile
syntax. BentoML set both trim_blocks and lstrip_blocks in Jinja
templates environment to True
.
Note
Make sure that your Dockerfile instruction is unindented as if you are writting a normal Dockerfile.
See also
An example of a Dockerfile template takes advantage of multi-stage build to
isolate the installation of a local library mypackage
:
{% extends bento_base_template %}
{% block SETUP_BENTO_BASE_IMAGE %}
FROM --platform=$BUILDPLATFORM python:3.7-slim as buildstage
RUN mkdir /tmp/mypackage
WORKDIR /tmp/mypackage/
COPY mypackage .
RUN python setup.py sdist && mv dist/mypackage-0.0.1.tar.gz mypackage.tar.gz
{{ super() }}
{% endblock %}
{% block SETUP_BENTO_COMPONENTS %}
{{ super() }}
COPY --from=buildstage mypackage.tar.gz /tmp/wheels/
RUN --network=none pip install --find-links /tmp/wheels mypackage
{% endblock %}
Note
Notice how for all Dockerfile instruction, we consider as if the Jinja logics arenât there đ.
Jinja templates#
One of the powerful features Jinja offers is its template inheritance. This allows BentoML to enable users to fully customize how to structure a Bentoâs Dockerfile.
Note
To use a custom Dockerfile template, users have to provide a file with a format
that follows the Jinja2 template syntax. The template file should have
extensions of .j2
, .template
, .jinja
.
Note
This section is not meant to be a complete reference on Jinja2. For any advanced features from on Jinja2, please refers to their Templates Design Documentation.
To construct a custom Dockerfile
template, users have to provide an extends block at the beginning of the Dockerfile template Dockerfile.template
followed by the given base template name bento_base_template
:
{% extends bento_base_template %}
Tip
Warning: If you pass in a generic Dockerfile
file, and then run bentoml build
to build a Bento and it doesnât throw any errors.
However, when you try to run bentoml containerize
, this wonât work.
This is an expected behaviour from Jinja2, where Jinja2 accepts any file as a template.
We decided not to put any restrictions to validate the template file, simply because we want to enable users to customize to their own needs.
{{ super() }}
#
As you can notice throughout this guides, we use a special function {{ super() }}
. This is a Jinja
features that allow users to call content of parent block. This
enables users to fully extend base templates provided by BentoML to ensure that
the result Bentos can be containerized.
See also
{{ super() }}
Syntax for more information on template inheritance.
Blocks#
BentoML defines a sets of Blocks under the object bento_base_template
.
All exported blocks that users can use to extend are as follow:
Blocks |
Definition |
---|---|
|
Instructions to set up multi architecture supports, base images as well as installing system packages that is defined by users. |
|
Setup bento users with correct UID, GID and directory for a đ±. |
|
Add users environment variables (if specified) and other required variables from BentoML. |
|
Setup components for a đ± , including installing pip packages, running setup scripts, installing bentoml, etc. |
|
Finalize ports and set |
Note
All the defined blocks are prefixed with SETUP_BENTO_*
. This is to
ensure that users can extend blocks defined by BentoML without sacrificing
the flexibility of a Jinja template.
To extend any given block, users can do so by adding {{ super() }}
at
any point inside block.
Dockerfile instruction#
See also
Dockerfile reference for writing a Dockerfile.
We recommend that users should use the following Dockerfile instructions in
their custom Dockerfile templates: ENV
, RUN
, ARG
. These
instructions are mostly used and often times will get the jobs done.
The use of the following instructions can be potentially harmful. They should be reserved for specialized advanced use cases.
Instruction |
Reasons not to use |
---|---|
|
Since the containerized Bento is a multi-stage builds container, adding |
|
BentoML uses heredoc syntax and using |
|
Changing |
The following instructions should be used with caution:
WORKDIR
#
See also
Since WORKDIR
determines the working directory for any RUN
, CMD
, ENTRYPOINT
, COPY
and ADD
instructions that follow it in the Dockerfile,
make sure that your instructions define the correct path to any working files.
Note
By default, all paths for Bento-related files will be generated to its
fspath, which ensures that Bento will work regardless of WORKDIR
ENTRYPOINT
#
See also
The flexibility of a Jinja template also brings up the flexibility of setting up ENTRYPOINT
and CMD
.
From Dockerfile documentation:
Only the last
ENTRYPOINT
instruction in the Dockerfile will have an effect.
By default, a Bento sets:
ENTRYPOINT [ "{{ bento__entrypoint }}" ]
CMD ["bentoml", "serve", "{{ bento__path }}"]
This aboved instructions ensure that whenever docker run
is invoked on the đ± container, bentoml
is called correctly.
In scenarios where one needs to setup a custom ENTRYPOINT
, make sure to use
the ENTRYPOINT
instruction under the SETUP_BENTO_ENTRYPOINT
block as follows:
{% extends bento_base_template %}
{% block SETUP_BENTO_ENTRYPOINT %}
{{ super() }}
...
ENTRYPOINT [ "{{ bento__entrypoint }}", "python", "-m", "awslambdaric" ]
{% endblock %}
Tip
{{ bento__entrypoint }}
is the path the BentoML entrypoint,
nothinig special here đ.
Read more about CMD
and ENTRYPOINT
interaction here.
Advanced Options#
The next part goes into advanced options. Skip this part if you are not comfortable with using it.
Dockerfile variables#
BentoML does expose some variables that user can modify to fit their needs.
The following are the variables that users can set in their custom Dockerfile template:
Variables |
Description |
---|---|
|
Setup bento home, default to |
|
Setup bento user, default to |
|
Setup UID and GID for the user, default to |
|
Setup bento path, default to |
If any of the aforementioned fields are set with {% set ... %}
, then we
will use your value instead, otherwise a default value will be used.
Adding conda
to CUDA-enabled Bento#
Tip
Warning: miniconda install scripts provided by ContinuumIO (the parent company of Anaconda) supports Python 3.7 to 3.9. Make sure that you are using the correct python version under docker.python_version
.
If you need to use conda for CUDA images, use the following template ( partially extracted from ContinuumIO/docker-images
):
{% import '_macros.j2' as common %}
{% extends bento_base_template %}
{# Make sure to change the correct python_version and conda version accordingly. #}
{# example: py38_4.10.3 #}
{# refers to https://repo.anaconda.com/miniconda/ for miniconda3 base #}
{% set conda_version="py39_4.11.0" %}
{% set conda_path="/opt/conda" %}
{% set conda_exec=[conda_path, "bin", "conda"] | join("/") %}
{% block SETUP_BENTO_BASE_IMAGE %}
FROM debian:bullseye-slim as conda-build
RUN --mount=type=cache,from=cached,sharing=shared,target=/var/cache/apt \
--mount=type=cache,from=cached,sharing=shared,target=/var/lib/apt \
apt-get update -y && \
apt-get install -y --no-install-recommends --allow-remove-essential \
software-properties-common \
bzip2 \
ca-certificates \
git \
libglib2.0-0 \
libsm6 \
libxext6 \
libxrender1 \
mercurial \
openssh-client \
procps \
subversion \
wget && \
apt-get clean
ENV PATH {{ conda_path }}/bin:$PATH
ARG CONDA_VERSION={{ conda_version }}
RUN set -ex && \
UNAME_M=$(uname -m) && \
if [ "${UNAME_M}" = "x86_64" ]; then \
MINICONDA_URL="https://repo.anaconda.com/miniconda/Miniconda3-${CONDA_VERSION}-Linux-x86_64.sh"; \
SHA256SUM="4ee9c3aa53329cd7a63b49877c0babb49b19b7e5af29807b793a76bdb1d362b4"; \
elif [ "${UNAME_M}" = "s390x" ]; then \
MINICONDA_URL="https://repo.anaconda.com/miniconda/Miniconda3-${CONDA_VERSION}-Linux-s390x.sh"; \
SHA256SUM="e5e5e89cdcef9332fe632cd25d318cf71f681eef029a24495c713b18e66a8018"; \
elif [ "${UNAME_M}" = "aarch64" ]; then \
MINICONDA_URL="https://repo.anaconda.com/miniconda/Miniconda3-${CONDA_VERSION}-Linux-aarch64.sh"; \
SHA256SUM="00c7127a8a8d3f4b9c2ab3391c661239d5b9a88eafe895fd0f3f2a8d9c0f4556"; \
elif [ "${UNAME_M}" = "ppc64le" ]; then \
MINICONDA_URL="https://repo.anaconda.com/miniconda/Miniconda3-${CONDA_VERSION}-Linux-ppc64le.sh"; \
SHA256SUM="8ee1f8d17ef7c8cb08a85f7d858b1cb55866c06fcf7545b98c3b82e4d0277e66"; \
fi && \
wget "${MINICONDA_URL}" -O miniconda.sh -q && echo "${SHA256SUM} miniconda.sh" > shasum && \
if [ "${CONDA_VERSION}" != "latest" ]; then \
sha256sum --check --status shasum; \
fi && \
mkdir -p /opt && \
sh miniconda.sh -b -p {{ conda_path }} && rm miniconda.sh shasum && \
find {{ conda_path }}/ -follow -type f -name '*.a' -delete && \
find {{ conda_path }}/ -follow -type f -name '*.js.map' -delete && \
{{ conda_exec }} clean -afy
{{ super() }}
ENV PATH {{ conda_path }}/bin:$PATH
COPY --from=conda-build {{ conda_path }} {{ conda_path }}
RUN set -ex && \
ln -s {{ conda_path }}/etc/profile.d/conda.sh /etc/profile.d/conda.sh && \
echo ". {{ conda_path }}/etc/profile.d/conda.sh" >> ~/.bashrc && \
echo "{{ conda_exec }} activate base" >> ~/.bashrc
{% endblock %}
{% block SETUP_BENTO_ENVARS %}
SHELL [ "/bin/bash", "-eo", "pipefail", "-c" ]
{{ super() }}
{{ common.setup_conda(__python_version__, bento__path, conda_path=conda_path) }}
{% endblock %}
Containerization with different container engines.#
In BentoML version 1.0.11 [1], we support different container engines aside from docker.
BentoML-generated Dockerfiles from version 1.0.11 onward will be OCI-compliant and can be built with:
To use any of the aforementioned backends, they must be installed on your system. Refer to their documentation for installation and setup.
Note
By default, BentoML will use Docker as the container backend.
To use other container engines, please set the environment variable BENTOML_CONTAINERIZE_BACKEND
or
pass in --backend
to bentoml containerize:
# set environment variable
BENTOML_CONTAINERIZE_BACKEND=buildah bentoml containerize pytorch-mnist
# or pass in --backend
bentoml containerize pytorch-mnist:latest --backend buildah
To build a BentoContainer in Python, you can use the Container SDK method bentoml.container.build()
:
import bentoml
bentoml.container.build(
"pytorch-mnist:latest",
backend="podman",
features=["grpc","grpc-reflection"],
cache_from="registry.com/my_cache:v1",
)
Register custom backend#
To register a new backend, there are two functions that need to be implemented:
arg_parser_func
: a function that takes in keyword arguments that represents the builder commandline arguments and returns alist[str]
:def arg_parser_func( *, context_path: str = ".", cache_from: Optional[str] = None, **kwargs, ) -> list[str]: if cache_from: args.extend(["--cache-from", cache_from]) args.append(context_path) return args
health_func
: a function that returns abool
to indicate if the backend is available:import shutil def health_func() -> bool: return shutil.which("limactl") is not None
To register a new backend, use bentoml.container.register_backend()
:
from bentoml.container import register_backend
register_backend(
"lima",
binary="/usr/bin/limactl",
buildkit_support=True,
health=health_func,
construct_build_args=arg_parser_func,
env={"DOCKER_BUILDKIT": "1"},
)
Backward compatibility with bentoml.bentos.containerize
Before 1.0.11, BentoML uses bentoml.bentos.containerize()
to containerize Bento. This method is now deprecated and will be removed in the future.
BuildKit interop#
BentoML leverages BuildKit for a more extensive feature set. However, we recognise that BuildKit has come with a lot of friction for migration purposes as well as restrictions to use with other build tools (such as podman, buildah, kaniko).
Therefore, since BentoML version 1.0.11, BuildKit will be an opt-out. To disable BuildKit, pass DOCKER_BUILDKIT=0
to
bentoml containerize, which aligns with the behaviour of docker build
:
$ DOCKER_BUILDKIT=0 bentoml containerize ...
Note
All Bento container will now be following OCI spec instead of Docker spec. The difference is that in OCI spec, there is no SHELL argument.
Note
The generated Dockerfile included inside the Bento will be a minimal Dockerfile, which ensures compatibility among build tools. We encourage users to always use bentoml containerize.
If you wish to use the generated Dockerfile, make sure that you know what you are doing!
CLI enhancement#
To better support different backends, bentoml containerize will be more agnostic when it comes to parsing options.
One can pass in options for specific backend with --opt
:
$ bentoml containerize pytorch-mnist:latest --backend buildx --opt platform=linux/arm64
--opt
also accepts parsing :
$ bentoml containerize pytorch-mnist:latest --backend buildx --opt platform:linux/arm64
Note
If you are seeing a warning message like:
'--platform=linux/arm64' is now deprecated, use the equivalent '--opt platform=linux/arm64' instead.
BentoML used to depends on Docker buildx. These options are now backward compatible with --opt
. You can safely ignore this warning and use
--opt
to pass options for --backend=buildx
.
Notes