Model Overview Card for Mistral Tiny

Basic Information

• Model Name: Mistral Tiny
• Developer/Creator: Mistral AI
• Release Date: October 2024
• Version: 1.0
• Model Type: Text

Description

Overview

Mistral Tiny is a lightweight language model optimized for efficient text generation, summarization, and code completion tasks. It is designed to operate effectively in resource-constrained environments while maintaining high performance.

Key Features

• Model Size: 106.6 million parameters
• Required VRAM: 0.4 GB, making it accessible for devices with limited resources
• Context Length: Supports a maximum context length of 131,072 tokens, enabling extensive context handling
• Tokenizer Class: Utilizes the LlamaTokenizer with a vocabulary size of 32,000 tokens
• Training Framework: Built on the MistralForCausalLM architecture compatible with Transformers version 4.39.1

Intended Use

Mistral Tiny is ideal for applications that require rapid responses and low-latency processing, such as chatbots, automated content generation, and educational tools.

Language Support

The model supports multiple languages including English, French, German, Spanish, and Italian.

Technical Details

Architecture

Mistral Tiny employs a Transformer architecture characterized by:

• Layers: 12 layers
• Attention Heads: 12 attention heads per layer
• Hidden Size: 768 dimensions
• Intermediate Size: 3072 dimensions

This architecture incorporates advanced attention techniques such as Sliding Window Attention (SWA) to efficiently manage long sequences.

Training Data

The model was trained on a diverse dataset comprising over 7 trillion tokens from various domains. This extensive training corpus ensures robust language understanding and contextual awareness.

Knowledge Cutoff

The knowledge cutoff for Mistral Tiny is September 2023.

Diversity and Bias

Mistral AI has focused on creating a diverse training dataset to mitigate biases related to gender, race, and ideology. The model's design aims to enhance its applicability across various contexts and topics.

Performance Metrics

• Accuracy: Achieves an accuracy rate exceeding 85% in language understanding tasks.
• Perplexity Score: Demonstrates a low perplexity score indicative of strong predictive capabilities.
• F1 Score: Maintains an F1 score above 0.75 in text classification tasks.

Benchmarking Results

• MMLU (Massive Multitask Language Understanding): High performance in language comprehension tasks.
• HumanEval Benchmark (for coding): Secures competitive rankings among models of similar sizes.

Comparison to Other Models

Mistral Tiny is a compact, efficient language model, designed for speed and cost-effectiveness. With over 85% accuracy on simple tasks, Mistral Tiny is highly effective for straightforward applications. In contrast, Mistral Small is suitable for bulk tasks with moderate latency and 72.2% accuracy on benchmarks. Mistral Large excels with 84.0% accuracy on complex tasks, offering advanced reasoning and multilingual support. Mixtral 8x7B provides up to 6x faster inference for coding and complex reasoning, ideal for demanding applications.

Usage

Code Samples

The model is available on the AI/ML API platform as "mistralai/mistral-tiny" .

API Documentation

Detailed API Documentation is available here.

Ethical Guidelines

Mistral AI adheres to ethical guidelines promoting responsible AI usage. The organization emphasizes transparency regarding the model's capabilities and limitations while encouraging developers to consider the ethical implications of deploying AI technologies.

Licensing

Mistral Tiny is released under the Apache 2.0 license, allowing both commercial and non-commercial usage rights. This open-source approach fosters community collaboration and innovation.

Get Mistral Tiny API here.