Advanced Containerization#

This guides describes advanced containerization options provided by BentoML:

This is an advanced feature for user to customize container environment that are not directly supported in BentoML. For basic containerizing options, see Docker Options.

Why you may need this?#

  • If you want to customize the containerization process of your Bento.

  • If you need a certain tools, configs, prebuilt binaries that is available across all your Bento generated container images.

  • A big difference with base image features is that you don’t have to setup a custom base image and then push it to a remote registry.

Custom Base Image#

If none of the provided distros work for your use case, e.g. if your infrastructure requires all docker images to be derived from the same base image with certain security fixes and libraries, you can config BentoML to use your base image instead:

docker:
    base_image: "my_custom_image:latest"

When a base_image is provided, all other docker options will be ignored, (distro, cuda_version, system_packages, python_version). bentoml containerize will build a new image on top of the base_image with the following steps:

  • setup env vars

  • run the setup_script if provided

  • install the required Python packages

  • copy over the Bento file

  • setup the entrypoint command for serving.

Note

Warning: user must ensure that the provided base image has desired Python version installed. If the base image you have doesn’t have Python, you may install python via a setup_script. The implementation of the script depends on the base image distro or the package manager available.

docker:
    base_image: "my_custom_image:latest"
    setup_script: "./setup.sh"

Warning

By default, BentoML supports multi-platform docker image build out-of-the-box. However, when a custom base_image is provided, the generated Dockerfile can only be used for building linux/amd64 platform docker images.

If you are running BentoML from an Apple M1 device or an ARM based computer, make sure to pass the --platform parameter when containerizing a Bento. e.g.:

bentoml containerize iris_classifier:latest --platform=linux/amd64

Dockerfile Template#

The dockerfile_template field gives the user full control over how the Dockerfile is generated for a Bento by extending the template used by BentoML.

First, create a Dockerfile.template file next to your bentofile.yaml build file. This file should follow the Jinja2 template language, and extend BentoML’s base template and blocks. The template should render a valid Dockerfile. For example:

{% extends bento_base_template %}
{% block SETUP_BENTO_COMPONENTS %}
{{ super() }}
RUN echo "We are running this during bentoml containerize!"
{% endblock %}

Then add the path to your template file to the dockerfile_template field in your :code: bentofile.yaml:

docker:
    dockerfile_template: "./Dockerfile.template"

Now run bentoml build to build a new Bento. It will contain a Dockerfile generated with the custom template. To confirm the generated Dockerfile works as expected, run bentoml containerize <bento> to build a docker image with it.

View the generated Dockerfile content

During development and debugging, you may want to see the generated Dockerfile. Here’s shortcut for that:

cat "$(bentoml get <bento>:<tag> -o path)/env/docker/Dockerfile"

Examples#

  1. Building TensorFlow custom op

  2. Access AWS credentials during image build

Building TensorFlow custom op#

Let’s start with an example that builds a custom TensorFlow op binary into a Bento, which is based on zero_out.cc implementation details:

Define the following Dockerfile.template:

{% extends bento_base_template %}
{% block SETUP_BENTO_BASE_IMAGE %}

{{ super() }}

WORKDIR /tmp

SHELL [ "bash", "-exo", "pipefail", "-c" ]

COPY ./src/tfops/zero_out.cc .

RUN pip3 install tensorflow
RUN bash <<EOF
set -ex

TF_CFLAGS=( $(python3 -c 'import tensorflow as tf; print(" ".join(tf.sysconfig.get_compile_flags()))') )
TF_LFLAGS=( $(python3 -c 'import tensorflow as tf; print(" ".join(tf.sysconfig.get_link_flags()))') )

g++ --std=c++14 -shared zero_out.cc -o zero_out.so -fPIC ${TF_CFLAGS[@]} ${TF_LFLAGS[@]} -I$(python -c 'import tensorflow as tf; print(tf.sysconfig.get_include());') -D_GLIBCXX_USE_CXX11_ABI=0 -O2
EOF

{% endblock %}
{% block SETUP_BENTO_COMPONENTS %}
{{ super() }}
RUN stat /usr/lib/zero_out.so
{% endblock %}

Then add the following to your bentofile.yaml:

include:
  - "zero_out.cc"
python:
  packages:
  - tensorflow
docker:
  dockerfile_template: ./Dockerfile.template

Proceed to build your Bento with bentoml build and containerize with bentoml containerize:

bentoml build

bentoml containerize <bento>:<tag>

Tip

You can also provide --progress plain to see the progress from buildkit in plain text

bentoml containerize --progress plain <bento>:<tag>

Access AWS credentials during image build#

We will now demonstrate how to provide AWS credentials to a Bento via two approaches:

  1. Using environment variables.

  2. Mount credentials from host.

Note

Remarks: We recommend for most cases to use the second option (Mount credentials from host) as it prevents any securities leak.

By default BentoML uses the latest dockerfile frontend which allows mounting secrets to container.

For both examples, you will need to add the following to your bentofile.yaml:

python:
  packages:
  - awscli
docker:
  dockerfile_template: ./Dockerfile.template

Using environment variables#

Define the following Dockerfile.template:

{% extends bento_base_template %}
{% block SETUP_BENTO_BASE_IMAGE %}
ARG AWS_SECRET_ACCESS_KEY
ARG AWS_ACCESS_KEY_ID
{{ super() }}

ARG AWS_SECRET_ACCESS_KEY
ARG AWS_ACCESS_KEY_ID
{% endblock %}
{% block SETUP_BENTO_COMPONENTS %}
{{ super() }}

RUN aws s3 cp s3://path/to/file {{ bento__path }}

{% endblock %}

After building the bento with bentoml build, you can then pass AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID as arguments to bentoml containerize:

bentoml containerize --build-arg AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \
                     --build-arg AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
                     <bento>:<tag>

Mount credentials from host#

Define the following Dockerfile.template:

{% extends bento_base_template %}
{% block SETUP_BENTO_COMPONENTS %}
{{ super() }}

RUN --mount=type=secret,id=aws,target=/root/.aws/credentials \
     aws s3 cp s3://path/to/file {{ bento__path }}

{% endblock %}

Follow the above addition to bentofile.yaml to include awscli and the custom dockerfile template.

To pass in secrets to the Bento, pass it via --secret to bentoml containerize:

bentoml containerize --secret id=aws,src=$HOME/.aws/credentials <bento>:<tag>

See also

Mounting Secrets

Writing dockerfile_template#

BentoML utilize Jinja2 to structure a Dockerfile.template.

The Dockerfile template is a mix between Jinja2 syntax and Dockerfile syntax. BentoML set both trim_blocks and lstrip_blocks in Jinja templates environment to True.

Note

Make sure that your Dockerfile instruction is unindented as if you are writting a normal Dockerfile.

An example of a Dockerfile template takes advantage of multi-stage build to isolate the installation of a local library mypackage:

{% extends bento_base_template %}
{% block SETUP_BENTO_BASE_IMAGE %}
FROM --platform=$BUILDPLATFORM python:3.7-slim as buildstage
RUN mkdir /tmp/mypackage

WORKDIR /tmp/mypackage/
COPY mypackage .
RUN python setup.py sdist && mv dist/mypackage-0.0.1.tar.gz mypackage.tar.gz

{{ super() }}
{% endblock %}
{% block SETUP_BENTO_COMPONENTS %}
{{ super() }}
COPY --from=buildstage mypackage.tar.gz /tmp/wheels/
RUN --network=none pip install --find-links /tmp/wheels mypackage
{% endblock %}

Note

Notice how for all Dockerfile instruction, we consider as if the Jinja logics aren’t there 🚀.

Jinja templates#

One of the powerful features Jinja offers is its template inheritance. This allows BentoML to enable users to fully customize how to structure a Bento’s Dockerfile.

Note

To use a custom Dockerfile template, users have to provide a file with a format that follows the Jinja2 template syntax. The template file should have extensions of .j2, .template, .jinja.

Note

This section is not meant to be a complete reference on Jinja2. For any advanced features from on Jinja2, please refers to their Templates Design Documentation.

To construct a custom Dockerfile template, users have to provide an extends block at the beginning of the Dockerfile template Dockerfile.template followed by the given base template name bento_base_template:

{% extends bento_base_template %}

Tip

Warning: If you pass in a generic Dockerfile file, and then run bentoml build to build a Bento and it doesn’t throw any errors.

However, when you try to run bentoml containerize, this won’t work.

This is an expected behaviour from Jinja2, where Jinja2 accepts any file as a template.

We decided not to put any restrictions to validate the template file, simply because we want to enable users to customize to their own needs.

{{ super() }}#

As you can notice throughout this guides, we use a special function {{ super() }}. This is a Jinja features that allow users to call content of parent block. This enables users to fully extend base templates provided by BentoML to ensure that the result Bentos can be containerized.

See also

{{ super() }} Syntax for more information on template inheritance.

Blocks#

BentoML defines a sets of Blocks under the object bento_base_template.

All exported blocks that users can use to extend are as follow:

Blocks

Definition

SETUP_BENTO_BASE_IMAGE

Instructions to set up multi architecture supports, base images as well as installing system packages that is defined by users.

SETUP_BENTO_USER

Setup bento users with correct UID, GID and directory for a 🍱.

SETUP_BENTO_ENVARS

Add users environment variables (if specified) and other required variables from BentoML.

SETUP_BENTO_COMPONENTS

Setup components for a 🍱 , including installing pip packages, running setup scripts, installing bentoml, etc.

SETUP_BENTO_ENTRYPOINT

Finalize ports and set ENTRYPOINT and CMD for the 🍱.

Note

All the defined blocks are prefixed with SETUP_BENTO_*. This is to ensure that users can extend blocks defined by BentoML without sacrificing the flexibility of a Jinja template.

To extend any given block, users can do so by adding {{ super() }} at any point inside block.

Dockerfile instruction#

See also

Dockerfile reference for writing a Dockerfile.

We recommend that users should use the following Dockerfile instructions in their custom Dockerfile templates: ENV, RUN, ARG. These instructions are mostly used and often times will get the jobs done.

The use of the following instructions can be potentially harmful. They should be reserved for specialized advanced use cases.

Instruction

Reasons not to use

FROM

Since the containerized Bento is a multi-stage builds container, adding FROM statement will result in failure to containerize the given Bento.

SHELL

BentoML uses heredoc syntax and using bash in our containerization process. Hence changing SHELL will result in failure.

CMD

Changing CMD will inherently modify the behaviour of the bento container where docker won’t be able to run the bento inside the container. More below

The following instructions should be used with caution:

WORKDIR#

Since WORKDIR determines the working directory for any RUN, CMD, ENTRYPOINT, COPY and ADD instructions that follow it in the Dockerfile, make sure that your instructions define the correct path to any working files.

Note

By default, all paths for Bento-related files will be generated to its fspath, which ensures that Bento will work regardless of WORKDIR

ENTRYPOINT#

The flexibility of a Jinja template also brings up the flexibility of setting up ENTRYPOINT and CMD.

From Dockerfile documentation:

Only the last ENTRYPOINT instruction in the Dockerfile will have an effect.

By default, a Bento sets:

ENTRYPOINT [ "{{ bento__entrypoint }}" ]

CMD ["bentoml", "serve", "{{ bento__path }}", "--production"]

This aboved instructions ensure that whenever docker run is invoked on the 🍱 container, bentoml is called correctly.

In scenarios where one needs to setup a custom ENTRYPOINT, make sure to use the ENTRYPOINT instruction under the SETUP_BENTO_ENTRYPOINT block as follows:

{% extends bento_base_template %}
{% block SETUP_BENTO_ENTRYPOINT %}
{{ super() }}

...
ENTRYPOINT [ "{{ bento__entrypoint }}", "python", "-m", "awslambdaric" ]
{% endblock %}

Tip

{{ bento__entrypoint }} is the path the BentoML entrypoint, nothinig special here 😏.

Read more about CMD and ENTRYPOINT interaction here.

Advanced Options#

The next part goes into advanced options. Skip this part if you are not comfortable with using it.

Dockerfile variables#

BentoML does expose some variables that user can modify to fit their needs.

The following are the variables that users can set in their custom Dockerfile template:

Variables

Description

bento__home

Setup bento home, default to /home/{{ bento__user }}

bento__user

Setup bento user, default to bentoml

bento__uid_gid

Setup UID and GID for the user, default to 1034:1034

bento__path

Setup bento path, default to /home/{{ bento__user }}/bento

If any of the aforementioned fields are set with {% set ... %}, then we will use your value instead, otherwise a default value will be used.

Adding conda to CUDA-enabled Bento#

Tip

Warning: miniconda install scripts provided by ContinuumIO (the parent company of Anaconda) supports Python 3.7 to 3.9. Make sure that you are using the correct python version under docker.python_version.

If you need to use conda for CUDA images, use the following template ( partially extracted from ContinuumIO/docker-images ):

Expands me
{% import '_macros.j2' as common %}
{% extends bento_base_template %}
{# Make sure to change the correct python_version and conda version accordingly. #}
{# example: py38_4.10.3 #}
{# refers to https://repo.anaconda.com/miniconda/ for miniconda3 base #}
{% set conda_version="py39_4.11.0" %}
{% set conda_path="/opt/conda" %}
{% set conda_exec=[conda_path, "bin", "conda"] | join("/") %}
{% block SETUP_BENTO_BASE_IMAGE %}
FROM debian:bullseye-slim as conda-build

RUN --mount=type=cache,target=/var/cache/apt \
    --mount=type=cache,target=/var/lib/apt \
    apt-get update -y && \
    apt-get install -y --no-install-recommends --allow-remove-essential \
                software-properties-common \
        bzip2 \
        ca-certificates \
        git \
        libglib2.0-0 \
        libsm6 \
        libxext6 \
        libxrender1 \
        mercurial \
        openssh-client \
        procps \
        subversion \
        wget && \
    apt-get clean

ENV PATH {{ conda_path }}/bin:$PATH

SHELL [ "/bin/bash", "-eo", "pipefail", "-c" ]

ARG CONDA_VERSION={{ conda_version }}

RUN bash <<EOF
set -ex

UNAME_M=$(uname -m)

if [ "${UNAME_M}" = "x86_64" ]; then
    MINICONDA_URL="https://repo.anaconda.com/miniconda/Miniconda3-${CONDA_VERSION}-Linux-x86_64.sh";
    SHA256SUM="4ee9c3aa53329cd7a63b49877c0babb49b19b7e5af29807b793a76bdb1d362b4";
elif [ "${UNAME_M}" = "s390x" ]; then
    MINICONDA_URL="https://repo.anaconda.com/miniconda/Miniconda3-${CONDA_VERSION}-Linux-s390x.sh";
    SHA256SUM="e5e5e89cdcef9332fe632cd25d318cf71f681eef029a24495c713b18e66a8018";
elif [ "${UNAME_M}" = "aarch64" ]; then
    MINICONDA_URL="https://repo.anaconda.com/miniconda/Miniconda3-${CONDA_VERSION}-Linux-aarch64.sh";
    SHA256SUM="00c7127a8a8d3f4b9c2ab3391c661239d5b9a88eafe895fd0f3f2a8d9c0f4556";
elif [ "${UNAME_M}" = "ppc64le" ]; then
    MINICONDA_URL="https://repo.anaconda.com/miniconda/Miniconda3-${CONDA_VERSION}-Linux-ppc64le.sh";
    SHA256SUM="8ee1f8d17ef7c8cb08a85f7d858b1cb55866c06fcf7545b98c3b82e4d0277e66";
fi

wget "${MINICONDA_URL}" -O miniconda.sh -q && echo "${SHA256SUM} miniconda.sh" > shasum

if [ "${CONDA_VERSION}" != "latest" ]; then
    sha256sum --check --status shasum;
fi

mkdir -p /opt
sh miniconda.sh -b -p {{ conda_path }} && rm miniconda.sh shasum

find {{ conda_path }}/ -follow -type f -name '*.a' -delete
find {{ conda_path }}/ -follow -type f -name '*.js.map' -delete
{{ conda_exec }} clean -afy
EOF

{{ super() }}

ENV PATH {{ conda_path }}/bin:$PATH

COPY --from=conda-build {{ conda_path }} {{ conda_path }}

RUN bash <<EOF
ln -s {{ conda_path }}/etc/profile.d/conda.sh /etc/profile.d/conda.sh
echo ". {{ conda_path }}/etc/profile.d/conda.sh" >> ~/.bashrc
echo "{{ conda_exec }} activate base" >> ~/.bashrc
EOF

{% endblock %}
{% block SETUP_BENTO_ENVARS %}

SHELL [ "/bin/bash", "-eo", "pipefail", "-c" ]
{{ super() }}
{{ common.setup_conda(__python_version__, bento__path, conda_path=conda_path) }}
{% endblock %}