Llama 3 70B Instruct Lite by Meta: Optimized for dialogue, high performance, diverse applications.
Model Name: Llama 3 70B Instruct Lite
Developer/Creator: Meta
Release Date: April 18, 2024
Version: 1.0
Model Type: Text
Llama 3 70B Instruct Lite is a large language model developed by Meta, optimized for dialogue and instruction-tuned to improve helpfulness and safety. It is part of the Llama 3 family, which includes models with 8 billion and 70 billion parameters.
Llama 3 70B Instruct Lite is designed for assistant-like chat applications and various natural language generation tasks. It is intended for use in commercial and research contexts.
Currently supports English, but developers may fine-tune it for other languages following the Llama 3 Community License and Acceptable Use Policy.
Llama 3 uses an auto-regressive transformer architecture. The instruction-tuned variants employ supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF) for alignment with human preferences.
Llama 3 was trained on over 15 trillion tokens of publicly available online data. The fine-tuning process involved over 10 million human-annotated examples.
The training data is sourced from a diverse mix of publicly available data, ensuring a robust dataset. The token count for pretraining exceeds 15 trillion.
December 2023 for the 70B model.
Efforts were made to ensure diversity in training data; however, biases may still exist. Developers are encouraged to evaluate and mitigate these biases in their specific applications.
Llama 3 models show superior performance on standard benchmarks, such as MMLU (79.5% for 70B), CommonSenseQA (83.8% for 70B), and others.
The model's inference speed is optimized using Grouped-Query Attention (GQA) for improved scalability.
The model performs well across diverse inputs and generalizes effectively across various topics and languages.
Meta's Responsible Use Guide outlines the steps for ethical AI development and deployment. Developers should follow these guidelines to ensure safe and responsible use of Llama 3.
A custom commercial license is available. For more details, visit Llama 3 License.