Model Overview Card for Mistral-Nemo

Basic Information

• Model Name: Mistral-Nemo
• Developer/Creator: Mistral AI and NVIDIA
• Release Date: July 18, 2024
• Version: 0.1
• Model Type: Large Language Model (LLM)

Description

Overview:

Mistral-Nemo is a state-of-the-art large language model designed for advanced natural language processing tasks, including text generation, summarization, translation, and sentiment analysis. It features a large context window of up to 128k tokens, making it suitable for handling extensive inputs and complex tasks.

Key Features:

• 12 billion parameters for robust performance.
• Supports a context window of up to 128k tokens.
• Instruction-tuned for improved task performance and adherence to prompts.
• Multilingual capabilities covering over 10 languages including English, French, Spanish, and Chinese.
• Utilizes the Tekken tokenizer for efficient text and code compression.

Intended Use:

Mistral-Nemo is designed for applications requiring high-quality text generation, such as chatbots, content creation tools, document summarization, and multilingual communication solutions.

Language Support:

The model supports multiple languages, making it versatile for global applications.

Technical Details

Architecture:

Mistral-Nemo is built on a Transformer architecture with the following specifications:

• Layers: 40
• Hidden Dimension: 14,436
• Head Dimension: 128
• Number of Heads: 32
• Activation Function: SwiGLU
• Grouped Query Attention and Sliding Window Attention techniques are employed to enhance performance.

Training Data:

The model was trained on a diverse dataset that includes extensive multilingual text and code data. This training set comprises billions of tokens from various domains, ensuring a broad understanding of language nuances.

• Data Source and Size: The training data includes sources from literature, web pages, and programming documentation to cover a wide range of topics and styles.
• Knowledge Cutoff: The model's knowledge is current as of April 2024.
• Diversity and Bias: Mistral AI has implemented strategies to reduce bias in the training data by ensuring a diverse dataset that represents multiple cultures and languages. This approach enhances the model's robustness across different contexts.

Performance Metrics:

Mistral-Nemo has demonstrated strong performance on various benchmarks:

• Achieves high accuracy on tasks like HellaSwag and Winogrande.
• Outperforms similar models in its size category in terms of reasoning and coding accuracy.

Comparison to Other Models

The Mistral NeMo model demonstrates strong performance across a range of tasks compared to models like Gemma 2 9B and Llama 3 8B. With a significantly larger context window of 128k, Mistral NeMo outperforms in several areas, especially in HellaSwag (0-shot) with 83.5% accuracy, Winogrande (0-shot) with 76.8%, and TriviaQA (5-shot) with 73.8%. In contrast, Gemma 2 9B and Llama 3 8B have smaller 8k context windows and achieve slightly lower performance, with Gemma 2 9B scoring 80.1% on HellaSwag and 71.3% on TriviaQA, while Llama 3 8B scores 80.6% and 61.0%, respectively. Mistral NeMo also leads in other tasks like OpenBookQA (0-shot) at 60.6% and CommonSense QA (0-shot) at 70.4%, highlighting its effectiveness in handling a wide range of language-based benchmarks.

Usage

Code Samples:

The model is available on the AI/ML API platform as "mistralai/mistral-nemo" .

API Documentation

Detailed API Documentation is available here.

Ethical Guidelines

Mistral AI emphasizes ethical considerations in AI development. The organization promotes transparency about model capabilities and encourages responsible usage to avoid misuse or unintended consequences.

Licensing

License Type: Mistral-Nemo is released under the Apache 2.0 license, allowing both commercial and non-commercial usage rights. This open licensing fosters innovation and accessibility within the developer community

‍

Get Mistral Nemo API here.