Advanced Guides

This part of the BentoML documentation is working in progress. If you have any questions related to this, please join the BentoML Slack community and ask in the bentoml-users channel.

• Offline Batch Serving
• Monitoring with Prometheus
• Request Logging
• Understanding BentoML adaptive micro batching
• 1. The overall architecture of BentoML’s micro-batching server
  • 1.1 Why micro batching matters
  • 1.2 Architecture & Data Flow
  • 1.3 Parameters & Concepts of micro batching
  • 1.4 Sequence & How it works
  • 1.5 The main design decisions and tradeoffs
• 2. parameter tuning best practices & recommendations
• 3. How to implement batch mode for custom handlers
• 4. Comparison
  • 4.1 TensorFlow Serving
  • 4.2 Clipper
• Adding Custom Model Artifact
• Customizing BentoHandler
• Deploy yatai server behind NGINX
  • 1. make NGINX proxying HTTP/2 from h2 to h2c
  • 2. config yatai client to use h2
  • 3. (Optional) using self-signed certificates