Model Overview Card for Qwen2.5 Coder 32B Instruct

Basic Information

• Model Name: Qwen2.5 Coder 32B Instruct
• Developer/Creator: Alibaba Cloud's Qwen Team
• Release Date: November 12, 2024
• Version: Qwen2.5
• Model Type: Causal Language Model

Description

Overview

Qwen2.5 Coder 32B Instruct is a state-of-the-art (SOTA) open-source code large language model (CodeLLM) designed for code generation, repair, and reasoning. It is part of the Qwen2.5 series and aims to help developers code more efficiently and learn new skills.

Key Features

• Excels in code generation, code repair, and code reasoning
• Demonstrates strong general and mathematical abilities
• Supports over 40 programming languages
• Offers human preference alignment
• Provides long-context support up to 131,072 tokens

Intended Use

Designed for instruction-based tasks, code completion, error correction, and assisting developers in various coding scenarios. It can be used in code assistants and other applications.

Language Support

Proficient in over 40 programming languages, including both mainstream and niche languages like Haskell and Racket.

Technical Details

Architecture

Employs a dense transformer architecture with Rotary Position Embedding (RoPE), SwiGLU activation functions, RMSNorm normalization, and Attention QKV bias.

• 32.5 billion parameters
• 64 layers
• Context length of up to 131,072 tokens
• Attention Heads: 40 for Q and 8 for KV

Training Data

Trained on 5.5 trillion tokens, including source code, text-code grounding, and synthetic data.

Data Source and Size

The model was trained on a massive dataset of 5.5 trillion tokens, which includes a diverse range of source code, text-code grounding data, and synthetic data [5, 6]. This extensive training data enables the model to perform well in various coding tasks and programming languages.

Knowledge cutoff

The LiveCodeBench questions used for evaluation are from July to November 2024.

Diversity and Bias

The model incorporates meticulous data cleaning and balancing strategies during pre-training to ensure robust and well-rounded capabilities across various programming languages. Human preference alignment is assessed using an internal benchmark called Code Arena.

Performance Metrics

The Qwen2.5 Coder 32B Instruct model achieves top-tier performance, rivaling GPT-4o in coding capabilities.

• HumanEval: 92.7
• MBPP: 90.2
• LiveCodeBench: 31.4
• Aider: 73.7
• Spider: 85.1
• CodeArena: 68.9

Comparison to Other Models

• Accuracy: Achieves state-of-the-art performance among open-source models in code generation, repair, and reasoning. It rivals the coding abilities of proprietary models like GPT-4o.
• Speed: Output speed of 83.2 tokens per second.
• Robustness: Demonstrates exceptional performance across a wide range of programming languages due to meticulous data cleaning and balancing strategies used during pre-training. It is also designed to assist users in swiftly identifying and fixing code errors.

Usage

Code Samples:

The model is available on the AI/ML API platform as "Qwen2.5-Coder-32B-Instruct" .

API Documentation:

Detailed API Documentation is available here.

Ethical Guidelines

The Qwen2.5 Coder 32B Instruct model has been evaluated for human preference alignment using the Code Arena benchmark to ensure that its code solutions align with user expectations.

Licensing

Qwen Plus is available under specific licensing terms provided by Alibaba Cloud. Users are advised to review the licensing information to understand the permissions and restrictions associated with the model's use.

‍

Get Qwen 2.5 Coder 32B Instruct API here.