Stability AI has released its first Japanese vision-language model, Japanese InstructBLIP Alpha. It can generate textual descriptions for input images as well as generate answers to questions given images.
Japanese InstructBLIP Alpha is a vision-language model that enables conditional text generation given images, built upon the Japanese large language model Japanese StableLM Instruct Alpha 7B that was recently released.
Figure1. Output: “Two persons sitting on a bench looking at Mt.Fuji”
The Japanese InstructBLIP Alpha leverages the InstructBLIP architecture, which has shown remarkable performance in various vision-language datasets. To make a high-performance model with a limited Japanese dataset, we initialized a part of the model with pre-trained InstructBLIP trained on large English datasets. We then fine-tuned this initialized model using the limited Japanese dataset.
Examples of applications of this model include search engine given images, a description/QA of the scene, and a textual description of the image for blind people, etc.
In addition to generating Japanese text, the Japanese InstructBLIP Alpha has the capability to accurately recognize Japan-specific objects (such as Tokyo Skytree or Kinkaku-ji).
Figure2. Output: “Sakura and Tokyo Skytree
Figure3. Output: “Kinkakuji temple in Kyoto”
Furthermore, input can also include text, such as a question. For example, as shown in the example below, it can answer questions about input images.
Figure4. Prompt: “What are the speed limits written on the road?”, Output: “30km/h”
Figure5. Prompt: “Which is the larger one?”, Output: “Left one”
Figure6. Prompt: “What color is the yukata of the person on the right?”, Output: “purple”
The model is available on Hugging Face Hub and can be tested for inference and additional training. For more information, please visit the Hugging Face hub pages linked below:
The Japanese InstructBLIP Alpha was developed for research purposes and is available exclusively for research use. For more details, please refer to the Hugging Face Hub page.
Stability AI is an open-access generative AI company working with partners to deliver next-generation infrastructure globally. Headquartered in London with developers around the world, Stability AI’s open philosophy provides new avenues for cutting-edge research in imaging, language, code, audio, video, 3D content, design, biotechnology, and other scientific research. For more information, visit https://stability.ai/
Jasper Research Lab’s new shadow generation research and model enable brands to create more photorealistic…
We’re announcing new updates to Gemini 2.0 Flash, plus introducing Gemini 2.0 Flash-Lite and Gemini…
Interactive digital agents (IDAs) leverage APIs of stateful digital environments to perform tasks in response…
This post is co-written with Martin Holste from Trellix. Security teams are dealing with an…
As AI continues to unlock new opportunities for business growth and societal benefits, we’re working…
An internal email obtained by WIRED shows that NOAA workers received orders to pause “ALL…