BLIP (Bootstrapping Language Image Pre-training) is a technique to improve the way AI models understand and process the relationship between images and textual descriptions. It has a variety of use cases in the AI field, particularly in applications that require a nuanced understanding of both visual and textual data, such as image captioning, visual question answering (VQA), and image-text matching. This document demonstrates how to build an image captioning application on top of a BLIP model with BentoML.
Define a BentoML Service to customize the serving logic. The example service.py file in the project uses the BLIP model Salesforce/blip-image-captioning-large, which is capable of generating captions for given images, optionally using additional text input for context. You can choose another model based on your need.
The @bentoml.service decorator defines the BlipImageCaptioning class as a BentoML Service, specifying that it requires 4Gi of memory. You can customize the Service configurations if necessary.
The Service loads the BLIP model based on MODEL_ID and moves the model to a GPU if available, otherwise it uses the CPU.
The generate method is exposed as an asynchronous API endpoint. It accepts an image (img) and an optional txt parameter as inputs. If text is provided, the model generates a caption considering both the image and text context; otherwise, it generates a caption based only on the image. The generated tokens are then decoded into a human-readable caption.
Run bentomlserve in your project directory to start the Service.
The server is active at http://localhost:3000. You can interact with it in different ways.
curl-s-XPOST\-Ftxt='unicorn at sunset'\-F'img=@image.jpg'\http://localhost:3000/generate
importbentomlfrompathlibimportPathwithbentoml.SyncHTTPClient("http://localhost:3000")asclient:result=client.generate(img=Path("image.jpg"),txt="unicorn at sunset",)
Visit http://localhost:3000, scroll down to Service APIs, and click Try it out. In the Request body box, select an image, optionally enter your prompt text and click Execute.
After the Service is ready, you can deploy the project to BentoCloud for better management and scalability. Sign up for a BentoCloud account and get $30 in free credits.
First, specify a configuration YAML file (bentofile.yaml) to define the build options for your application. It is used for packaging your application into a Bento. Here is an example file in the project: