Basic Information

Model Name: LLaVA v1.6 - Mistral 7B

Developer/Creator: Haotian Liu

Release Date: December 2023

Version: 1.6

Model Type: Multimodal Language Model (Text and Image)

Description

Overview

LLaVA v1.6 - Mistral 7B is an open-source, multimodal chatbot that combines a large language model with a pre-trained vision encoder. It excels in understanding and generating text based on both textual and visual inputs, making it ideal for a wide range of multimodal tasks.

Key Features

• Built on the Mistral-7B-Instruct-v0.2 base model
• Supports dynamic high-resolution image input
• Capable of handling diverse multimodal tasks
• Improved commercial licensing and bilingual support
• 7 billion parameters for efficient computation

Intended Use

LLaVA v1.6 - Mistral 7B is designed for:

• Research on large multimodal models and chatbots
• Image captioning and visual question answering
• Open-ended dialogue with visual context
• Building intelligent virtual assistants
• Image-based search applications
• Interactive educational tools

Language Support

The model demonstrates strong multilingual capabilities, with improved bilingual support compared to earlier versions.

Technical Details

Architecture

LLaVA v1.6 - Mistral 7B utilizes:

• An auto-regressive language model based on the transformer architecture
• A pre-trained vision encoder (likely CLIP-L, based on similar models)
• Integration of text and image inputs using the <image> token in prompts

Training Data

The model was trained on a diverse dataset including:

• 558K filtered image-text pairs from LAION/CC/SBU, captioned by BLIP
• 158K GPT-generated multimodal instruction-following data
• 500K academic-task-oriented VQA data mixture
• 50K GPT-4V data mixture
• 40K ShareGPT data

Data Source and Size: The training data comprises over 1.3 million diverse samples, including image-text pairs and instruction-following data.

Knowledge Cutoff: December 2023

Diversity and Bias: The model's training data includes a wide range of sources, potentially reducing bias.

Performance Metrics

LLaVA v1.6 - Mistral 7B demonstrates strong performance across various benchmarks:

Comparison to Other Models

Accuracy: LLaVA v1.6 - Mistral 7B shows competitive performance compared to similar models.

For example, it achieves 35.3 on MMMU and 37.7 on MathVista benchmarks.

Speed: Specific inference speed metrics are not provided, but the 7B parameter size suggests efficient computation.

Robustness: The model demonstrates strong performance across multiple benchmarks and tasks, indicating good generalization capabilities.

Usage

Code Samples

Ethical Guidelines

While specific ethical guidelines are not detailed, users should adhere to responsible AI practices and consider potential biases in model outputs. The model should not be used for generating harmful or misleading content.

Licensing

LLaVA v1.6 - Mistral 7B follows the licensing terms of the Mistral-7B-Instruct-v0.2 base model. Users should refer to the official licensing terms for specific usage rights and restrictions.