StarCoder (16B)

Basic Information

• Model Name: Starcoder
• Developer/Creator: BigCode
• Release Date: 2022
• Version: 1.0
• Model Type: Text (Code generation)

Description

Overview

The bigcode/starcoder model is a 15.5 billion parameter language model developed by BigCode, a project focused on the open and responsible development of large language models for code. It is designed to assist developers with a wide range of coding tasks, including code generation, completion, and infilling.

Key Features

• Multi-Query Attention architecture: Allows the model to attend to multiple queries simultaneously, improving its ability to understand and generate code.
• 8192 token context window: Enables the model to consider a large amount of context when generating code, resulting in more coherent and relevant outputs.
• Fill-in-the-Middle objective: The model is trained using a novel objective that involves filling in missing code segments, helping it learn to understand and generate code at a deeper level.
• Supports multiple programming languages: bigcode/starcoder is trained on over 80 programming languages, making it a versatile tool for developers working in various languages.

Intended Use

The bigcode/starcoder model is intended for use in scenarios where developers need assistance with coding tasks, such as:

• Generating code snippets based on natural language descriptions
• Completing partially written code
• Infilling missing code segments
• Assisting with code refactoring and optimization

Language Support

The model supports over 80 programming languages, including popular ones like Python, Java, JavaScript, C++, and Go. It also supports multiple natural languages, with English being the predominant language used in the training data.

Technical Details

Architecture

The bigcode/starcoder model is based on the GPT-2 architecture, with a few key modifications:

• Multi-Query Attention: The model uses a novel attention mechanism that allows it to attend to multiple queries simultaneously, improving its ability to understand and generate code.
• Transformer-based: Like GPT-2, bigcode/starcoder is a transformer-based model, which means it uses a series of transformer blocks to process the input and generate the output.

Training Data

The model was trained on The Stack (v1.2) dataset, which contains source code from GitHub. The dataset includes code from over 80 programming languages and spans a wide range of domains, from web development to machine learning. The total size of the training data is 1 trillion tokens.

Performance Metrics

The bigcode/starcoder model has been evaluated on several benchmarks and has achieved state-of-the-art results in many cases:

• Human Eval: The model outperforms CodeCushman on the Human Eval benchmark in 12 languages, demonstrating its ability to generate high-quality code across multiple languages.
• DS-1000: The model achieves state-of-the-art results on the DS-1000 benchmark for data science workflows, showcasing its ability to generate code for complex tasks.
• Practical code generation: The model performs well on practical code generation tasks that require the use of external libraries and APIs, demonstrating its real-world applicability.

Usage

API Usage Example

Ethical Guidelines

BigCode has focused on raising the bar for data governance and has been transparent about the data used to train the model. An opt-out process was provided for source code developers who did not want their code included in the dataset.

License Type

The bigcode/starcoder model is licensed under the BigCode OpenRAIL-M v1 license agreement, which allows for both commercial and non-commercial use of the model.