Advanced Guides

This part of the BentoML documentation is a work in progress. If you have any questions related to this, please join the BentoML Slack community and ask in the bentoml-users channel.

• Configuration
  • Docker Deployment
  • Configuration Priority
  • Legacy Property Mapping
    • Repository Base URL
• Logging
  • 1. Basic Logging Configuration
  • 2. Advanced Logging Configuration
• Offline Batch Serving
  • StringInput
  • FileInput
  • MultiFileInput
• Monitoring with Prometheus
  • Preface
  • Local Deployment
    • Setting up Prometheus
    • Setting up Grafana
    • (Optional) Prometheus - Grafana - docker-compose stack
  • Deploy on Kubernetes
    • Deploy Prometheus on K8s
    • Using Grafana
      • Patching Grafana Service
      • Port Forwarding
    • Setting up your BentoService
    • (Optional) Exposing GPU Metrics on Kubernetes
• Adaptive Micro Batching
  • 1. The overall architecture of BentoML’s micro-batching server
    • 1.1 Why micro batching matters
    • 1.2 Architecture & Data Flow
    • 1.3 Parameters & Concepts of micro batching
    • 1.4 Sequence & How it works
    • 1.5 The main design decisions and trade-offs
  • 2. parameter tuning best practices & recommendations
  • 3. How to implement batch mode for custom input adapters
  • 4. Comparisons
    • 4.1 TensorFlow Serving
    • 4.2 Clipper
• Adding Custom Model Artifact
  • 1. Create custom Artifact
  • 2. Define and save BentoService with the custom Artifact
  • 3. Test with example data
• Customizing InputAdapter
  • 1. Create custom InputAdapter
  • 2. Define and save BentoService with the customized InputAdapter
  • 3. Test with example data
• Deploy Yatai server behind NGINX
  • 1. Make NGINX proxying HTTP/2 from h2 to h2c
  • 2. Config yatai client to use h2
  • 3. (Optional) Using self-signed certificates
  • 4. (Optional) Using TLS client certificates
• Performance Tracing
• Install Yatai on K8s with Helm
  • 1. Configuration
    • 1.1 Persistent Storage
    • 1.2 Ingress
    • 1.3 TLS
  • 2. Deploying
    • 2.1 Minikube (Local)
      • Dry Run
      • Custom Values
    • 2.2 Cloud Providers
• GPU Serving with BentoML
  • Prerequisite
    • NVIDIA Drivers
    • NVIDIA Container Toolkit
      • General workaround (Recommended)
      • Debian-based OS
      • Other OS
      • docker-compose
  • Framework Support for GPU Inference with Implementation
    • Preface
    • Docker Images Options
    • Tensorflow
      • Tensorflow Implementation
    • PyTorch
      • PyTorch Implementation
    • ONNX
      • ONNX Implementation