Basic Information

Model Name: Llama 3.2 3B Instruct Turbo

Developer/Creator: Meta

Release Date: September 2024

Version: 3.2

Model Type: Text-to-Text Language Model

Description

Overview

Llama 3.2 3B Instruct Turbo is a large language model (LLM) optimized for instruction-following tasks, striking a balance between computational efficiency and high-quality performance. It excels in multilingual tasks, offering a lightweight solution without compromising on quality.

Key Features

• 3 billion parameters for efficient computation
• Optimized for instruction-following tasks
• Multilingual capabilities
• Lightweight architecture suitable for edge computing and mobile applications
• Customizable and flexible for specific industry needs

Intended Use

Llama 3.2 3B Instruct Turbo is designed for various natural language processing tasks, including:

• Dialogue generation
• Summarization
• Translation
• Entity extraction
• Real-time text analysis
• Edge and mobile AI applications
• Multilingual dialogue agents
• Translation services

Language Support

The model demonstrates strong multilingual capabilities, with benchmark scores available for:

• English
• Spanish
• French
• German
• Italian
• Portuguese
• Thai
• Hindi

Technical Details

Architecture

Llama 3.2 3B Instruct Turbo uses an optimized transformer architecture with auto-regressive language modeling. It incorporates:

• 3 billion parameters (3.21B to be precise)
• Group Query Attention (GQA)
• Shared embeddings
• 128k context length

Training Data

Overview: Llama 3.2 was pretrained on up to 9 trillion tokens of data from publicly available sources. The training process involved:

1. Incorporation of logits from Llama 3.1 8B and 70B models
2. Knowledge distillation after pruning
3. Several rounds of alignment, including:
   • Supervised Fine-Tuning (SFT)
   • Rejection Sampling (RS)
   • Direct Preference Optimization (DPO)

Data Source and Size: A new mix of publicly available online data, with up to 9T tokens used in training.

Knowledge Cutoff: December 2023

Diversity and Bias: The model's strong performance across multiple languages suggests a diverse training dataset.

Performance Metrics

Llama 3.2 3B Instruct Turbo demonstrates impressive performance across various benchmarks:

Multilingual performance (MMLU benchmark):

• Spanish: 55.1%
• French: 54.6%
• German: 53.3%

Comparison to Other Models

Accuracy: Llama 3.2 3B Instruct Turbo shows competitive performance, especially considering its smaller size. For example, it achieves 63.4% on the MMLU benchmark, compared to 69.4% for the larger Llama 3.1 8B model.

Speed: The model demonstrates high performance with an output speed of 131.7 tokens per second, which is faster compared to the average.Robustness: Its strong performance across multiple languages and various tasks indicates good generalization capabilities.

Usage

Code Samples

Ethical Guidelines

While specific ethical guidelines are not detailed in the search results, the usage terms prohibit:

• Creating malicious code or interfering with computer systems
• Circumventing usage restrictions or safety measures
• Engaging in illegal activities
• Activities that risk harm to individuals

Licensing

Llama 3.2 3B Instruct Turbo is released under an open license by Meta. However, users should refer to the official licensing terms for specific usage rights and restrictions.