Advanced Guides

This part of the BentoML documentation is a work in progress. If you have any questions related to this, please join the BentoML Slack community and ask in the bentoml-users channel.

• Offline Batch Serving
• Monitoring with Prometheus
• Request Logging
• Understanding BentoML adaptive micro batching
• 1. The overall architecture of BentoML’s micro-batching server
  • 1.1 Why micro batching matters
  • 1.2 Architecture & Data Flow
  • 1.3 Parameters & Concepts of micro batching
  • 1.4 Sequence & How it works
  • 1.5 The main design decisions and tradeoffs
• 2. parameter tuning best practices & recommendations
• 3. How to implement batch mode for custom input adapters
• 4. Comparison
  • 4.1 TensorFlow Serving
  • 4.2 Clipper
• Adding Custom Model Artifact
• Customizing InputAdapter(Former BentoHandler)
• Deploy yatai server behind NGINX
  • 1. make NGINX proxying HTTP/2 from h2 to h2c
  • 2. config yatai client to use h2
  • 3. (Optional) using self-signed certificates