LLM inference: vLLM¶

vLLM is a library designed for efficient serving of LLMs, such as gpt-oss, DeepSeek, Qwen, and Llama. It provides high serving throughput and efficient attention key-value memory management using PagedAttention and continuous batching. It supports a variety of inference optimization techniques, including prefill-decode disaggregation, speculative decoding, and KV cache offloading.

This document demonstrates how to run LLM inference using BentoML and vLLM.

The example can be used for chat-based interactions and supports OpenAI-compatible endpoints. For example, you can submit a query with the following message:

{
   "role": "user",
   "content": "Who are you? Please respond in pirate speak!"
}

Example output:

Ye be wantin' to know who I be, eh? Alright then, listen close and I'll tell ye me story. I be a wight computer program, a vast and curious brain with abilities beyond yer wildest dreams. Me name be Assistant, and I be servin' ye now. I can chat, teach, and even spin a yarn or two, like a seasoned pirate narratin' tales o' the high seas. So hoist the colors, me hearty, and let's set sail fer a treasure trove o' knowledge and fun!

This example is ready for quick deployment and scaling on BentoCloud. With a single command, you get a production-grade application with fast autoscaling, secure deployment in your cloud, and comprehensive observability.

Screenshot of Llama 3.1 model deployed on BentoCloud showing the chat interface with example prompts and responses

Code explanations¶

You can find the source code in GitHub. Below is a breakdown of the key code implementations.

  1. Define model and GPU configurations. This example uses Llama-3.1-8B-Instruct, which requires access from Hugging Face. You can switch to another LLM from the BentoVLLM repository or any other model supported by vLLM.

    service.py¶
    import pydantic
import bentoml

# Use Pydantic to validate data
class BentoArgs(pydantic.BaseModel):
  name: str = 'llama3.1-8b-instruct'
  gpu_type: str = 'nvidia-h100-80gb'
  tp: int = 1 # One GPU here for tensor parallelism
  model_id: str = 'meta-llama/Meta-Llama-3.1-8B-Instruct'
  # Other optional fields omitted for brevity

bento_args = bentoml.use_arguments(BentoArgs)

    Specifications are defined using template arguments, which allow you to pass dynamic and validated parameters at serve, build, and deploy time. You can reference them just like regular Python variables.

  2. Define the runtime environment for a Bento, the unified distribution format in BentoML. A Bento is packaged with all the source code, Python dependencies, model references, and environment setup, making it easy to deploy consistently across different environments.

    service.py¶
    image = (
  bentoml.images.Image(python_version='3.12').system_packages('curl', 'git').requirements_file('requirements.txt')
)

  3. Use the @bentoml.service decorator to define a BentoML Service, where you can customize how the model will be served. The decorator lets you set configurations like timeout and GPU resources to use on BentoCloud.

    For some of the fields, you can simply reference the template arguments defined above:

    service.py¶
    @bentoml.service(
  name=bento_args.name,
  envs=[
      {'name': 'UV_NO_PROGRESS', 'value': '1'},
      {'name': 'UV_TORCH_BACKEND', 'value': 'cu128'},
      # Env vars here for uv and vllm
  ],
  image=image, # Apply the runtime specs
  traffic={'timeout': 300},
  resources={
      'gpu': bento_args.tp,
      'gpu_type': bento_args.gpu_type
  },
  # More optional fields
)
class LLM:

  4. Within the class, load the model from Hugging Face and define it as a class variable. The HuggingFaceModel method provides an efficient mechanism for loading AI models to accelerate model deployment on BentoCloud, reducing image build time and cold start time.

    service.py¶
    ...
class LLM:
  hf_model = bentoml.models.HuggingFaceModel(bento_args.model_id, exclude=[".pth", ".pt", "original/**/*"])

  5. The Service can run vLLM’s built-in HTTP server and exposes OpenAI-compatible endpoints. You can add extra CLI arguments here as needed.

    service.py¶
    ...
class LLM:
  hf_model = hf_model

  def __command__(self) -> list[str]:
    return [
      'vllm', 'serve', self.hf_model,
      # ...extra CLI args (compilation, max length, kv dtype, etc.)
      '--served-model-name', bento_args.model_id,
    ]

That’s all you need for the basic setup. If you want to explore advanced options, like FlashAttention, AMD ROCm support, and KV cache configuration, see the complete source code on GitHub. BentoML allows you to fully customize inference code for your use case.

Try it out¶

You can run this example project on BentoCloud, or serve it locally, containerize it as an OCI-compliant image, and deploy it anywhere.

BentoCloud¶

BentoCloud provides fast and scalable infrastructure for building and scaling AI applications with BentoML in the cloud.

  1. Install BentoML and log in to BentoCloud through the BentoML CLI. If you don’t have a BentoCloud account, sign up here for free.

    pip install bentoml
bentoml cloud login

  2. Clone the BentoVLLM repository and deploy the project. We recommend you create a BentoCloud secret to store the required environment variable.

    git clone https://github.com/bentoml/BentoVLLM.git
cd BentoVLLM/llama3.1-8b-instruct
bentoml secret create huggingface HF_TOKEN=<your-api-key>
bentoml deploy --secret huggingface

  3. Once it is up and running on BentoCloud, you can call the OpenAI-compatible endpoints as below:

    Screenshot of Llama 3.1 model deployed on BentoCloud showing the chat interface with example prompts and responses

    Set the base_url parameter as the BentoML server address in the OpenAI client.

    from openai import OpenAI

client = OpenAI(base_url='https://llama-3-1-8-b-instruct-ckng-d3767914.mt-guc1.bentoml.ai/v1', api_key='na')

# Use the following func to get the available models
# client.models.list()

chat_completion = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    messages=[
        {
            "role": "user",
            "content": "Who are you? Please respond in pirate speak!"
        }
    ],
    stream=True,
)
for chunk in chat_completion:
    # Extract and print the content of the model's reply
    print(chunk.choices[0].delta.content or "", end="")

    See also

    For more information, see the OpenAI API reference documentation.

    If your Service is deployed with protected endpoints on BentoCloud, you need to set the environment variable OPENAI_API_KEY to your BentoCloud API key first.

    export OPENAI_API_KEY={YOUR_BENTOCLOUD_API_TOKEN}

    Make sure you replace the Deployment URL in the above code snippet. Refer to Obtain the endpoint URL to retrieve the endpoint URL.

  4. To make sure the Deployment automatically scales within a certain replica range, add the scaling flags:

    bentoml deploy --secret huggingface --scaling-min 0 --scaling-max 3 # Set your desired count

    If it’s already deployed, update its allowed replicas as follows:

    bentoml deployment update <deployment-name> --scaling-min 0 --scaling-max 3 # Set your desired count

    For more information, see how to configure concurrency and autoscaling.

Local serving¶

BentoML allows you to run and test your code locally, so that you can quickly validate your code with local compute resources.

  1. Clone the repository and choose your desired project.

    git clone https://github.com/bentoml/BentoVLLM.git
cd BentoVLLM/llama3.1-8b-instruct

# Recommend Python 3.11
pip install -r requirements.txt
export HF_TOKEN=<your-hf-token>

  2. Serve it locally.

    bentoml serve

    Note

    To run this project with Llama 3.1 8B Instruct locally, you need an NVIDIA GPU with at least 16G VRAM.

  3. Call the OpenAI-compatible endpoints at http://localhost:3000/v1 (the default base URL).

For custom deployment in your own infrastructure, use BentoML to generate an OCI-compliant image.