Qwen 2 1.5B Instruct

Qwen2 1.5B Instruct Description

Basic Information

Model Name: Qwen2 1.5B Instruct

Developer/Creator: Qwen team (Alibaba Group)

Release Date: June 7, 2024

Version: 2.0

Model Type: Text-to-text language model

Overview

Qwen2 1.5B Instruct is a compact, instruction-tuned language model designed for efficient natural language processing tasks. It offers a balance between performance and resource requirements, making it suitable for a wide range of applications.

Key Features

• Instruction-tuned for improved task performance
• Group Query Attention (GQA) for faster inference and reduced memory usage
• Tied embeddings for parameter efficiency
• Context length supports input of up to 128,000 tokens and can produce outputs containing up to 8,000 tokens.
• Multilingual support for 27 languages besides English and Chinese

Intended Use

Qwen2 1.5B Instruct is designed for various natural language processing tasks, including:

• Text generation
• Question answering
• Language understanding
• Code generation
• Mathematical problem-solving

Language Support

The model supports 29 languages, including Chinese, English, French, Spanish, Portuguese, German, Italian, Russian, Japanese, Korean, Vietnamese, Thai, and Arabic.

Technical Details

Architecture

Qwen2 1.5B Instruct is based on the Transformer architecture with the following enhancements:

• SwiGLU activation
• Attention QKV bias
• Group query attention
• Improved tokenizer for multiple natural languages and code

Training Data

• Data Source and Size: The model was trained on an extensive dataset comprising approximately 18 trillion tokens.
• Knowledge Cutoff: The model's knowledge is current up to September 2024
• Diversity and Bias: The training data includes a wide range of languages and domains to reduce bias and improve robustness

Performance Metrics

Performance comparison between Qwen2 1.5B Instruct and its predecessor:

Comparison to Other Models

• Accuracy: Qwen2 1.5B Instruct shows significant improvements over its predecessor across various benchmarks
• Speed: The implementation of Group Query Attention (GQA) results in faster inference compared to previous versions
• Robustness: Enhanced multilingual support and diverse training data contribute to improved generalization across topics and languages
• In our testing (LLama 3 vs Qwen 2 and Qwen 2 72B vs ChatGPT 4o) we took it through some not-so-obvious prompts with cultural context, like translating an idiom, and it performs reasonably well. Moreover, it is a must when working with Asian groups of languages.

Usage

Code Samples

Ethical Guidelines

Users are encouraged to:

• Respect intellectual property rights when using generated content
• Be aware of potential biases in model outputs
• Use the model responsibly and avoid generating harmful or misleading content

Licensing

Qwen2 1.5B Instruct is licensed under the Apache 2.0 License, allowing for both commercial and non-commercial usage.