Model Overview Card

Basic Information

• Model Name: Llama 3 8B Instruct Reference
• Developer/Creator: Meta AI
• Release Date: April 18, 2024
• Version: 1.0
• Model Type: Text
• Quantization FP16

Description

Overview

Llama 3 8B Instruct Reference is a state-of-the-art language model designed for instruction-following tasks, offering exceptional performance in a compact 8 billion parameter package.

Key Features

• Advanced instruction-following capabilities
• Improved reasoning and code generation
• Efficient tokenizer with 128K vocabulary
• Grouped Query Attention (GQA) for enhanced inference efficiency
• 8,192 token context window

Intended Use

The model is optimized for various natural language processing tasks, including:

• Text generation
• Question answering
• Code assistance
• Content creation
• Dialogue systems

Language Support

Primarily focused on English, with limited capabilities in other languages.

Technical Details

Architecture

Llama 3 8B Instruct Reference utilizes a decoder-only transformer architecture, incorporating several key improvements over its predecessor:

• Tokenizer: A new tokenizer with a 128K token vocabulary, resulting in more efficient language encoding and improved model performance.
• Grouped Query Attention (GQA): Implemented to enhance inference efficiency.
• Attention Mechanism: Uses a masked attention approach to ensure self-attention does not cross document boundaries during training.

Training Data

Data Source and Size

The model was trained on over 15 trillion tokens of high-quality, publicly available data. The training process involved:

• Careful pre-processing and curation pipelines for pre-training data
• Rigorous quality assurance and filtering approaches for post-training data
• 95% of the training data was in English, explaining its strong performance in this language

Knowledge Cutoff

The exact knowledge cutoff date is not specified in the available information.

Diversity and Bias

Meta AI claims to have spent considerable effort on filtering input data to achieve the right balance in the training dataset. However, the model's performance in non-English languages suggests a potential bias towards English-language content.

Performance Metrics

Llama 3 8B Instruct Reference has demonstrated exceptional performance across various benchmarks:

• Outperforms many other AI models, including some larger models, in specific tasks
• Exhibits improved reasoning, code generation, and instruction-following capabilities compared to its predecessors
• Achieves better results than competitive models, including GPT-3.5, in human evaluations of real-world usage scenarios

Comparison to Other Models

Accuracy

The model shows competitive performance against larger models like Gemini Pro and Claude Sonnet in certain benchmarks.

Speed

While specific speed metrics are not provided, the implementation of GQA and an efficient tokenizer suggests improved inference speed compared to previous versions.

Robustness

Llama 3 8B demonstrates enhanced capabilities in handling diverse tasks, including reasoning and code generation, indicating improved robustness.

Usage

Code Samples

Ethical Guidelines

Meta AI has implemented several security measures within the model:

• Sensitivity to topics related to biology, chemistry, and cybersecurity
• Assessment of both input and output for safety
• Inclusion of Llama Guard 3, a multilingual safety model, and Prompt Guard, a prompt injection filter, in the reference system

Licensing

The exact licensing terms for Llama 3 8B Instruct Reference are not specified in the provided information. However, Meta AI emphasizes their "open approach" and encourages broad use of the model.