Model Overview Card for Llama 3 70B Instruct Lite

Basic Information

Model Name: Llama 3 70B Instruct Lite
Developer/Creator: Meta
Release Date: April 18, 2024
Version: 1.0
Model Type: Text

Description

Overview:

Llama 3 70B Instruct Lite is a large language model developed by Meta, optimized for dialogue and instruction-tuned to improve helpfulness and safety. It is part of the Llama 3 family, which includes models with 8 billion and 70 billion parameters.

Key Features:

• High Performance: Outperforms many open-source chat models on industry benchmarks.
• Optimized for Safety: Supervised fine-tuning and reinforcement learning with human feedback.
• Versatile Applications: Suitable for both commercial and research use in English.

Intended Use:

Llama 3 70B Instruct Lite is designed for assistant-like chat applications and various natural language generation tasks. It is intended for use in commercial and research contexts.

Language Support:

Currently supports English, but developers may fine-tune it for other languages following the Llama 3 Community License and Acceptable Use Policy.

Technical Details

Architecture:

Llama 3 uses an auto-regressive transformer architecture. The instruction-tuned variants employ supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF) for alignment with human preferences.

Training Data:

Llama 3 was trained on over 15 trillion tokens of publicly available online data. The fine-tuning process involved over 10 million human-annotated examples.

Data Source and Size:

The training data is sourced from a diverse mix of publicly available data, ensuring a robust dataset. The token count for pretraining exceeds 15 trillion.

Knowledge Cutoff:

December 2023 for the 70B model.

Diversity and Bias:

Efforts were made to ensure diversity in training data; however, biases may still exist. Developers are encouraged to evaluate and mitigate these biases in their specific applications.

Performance Metrics

Accuracy:

Llama 3 models show superior performance on standard benchmarks, such as MMLU (79.5% for 70B), CommonSenseQA (83.8% for 70B), and others.

Speed:

The model's inference speed is optimized using Grouped-Query Attention (GQA) for improved scalability.

Robustness:

The model performs well across diverse inputs and generalizes effectively across various topics and languages.

Usage

Code Samples

Ethical Guidelines:

Meta's Responsible Use Guide outlines the steps for ethical AI development and deployment. Developers should follow these guidelines to ensure safe and responsible use of Llama 3.

License Type:

A custom commercial license is available. For more details, visit Llama 3 License.