Basic Information

• Model Name: DBRX
• Developer/Creator: Databricks
• Release Date: 2023
• Version: 1.0 Instruct
• Model Type: Large Language Model (LLM)

Description

Overview

DBRX Instruct is a powerful, open-source large language model (LLM) developed by Databricks. It utilizes a fine-grained mixture-of-experts (MoE) architecture with 132 billion total parameters, of which 36 billion are active for any given input.

Key Features

• Fine-grained MoE architecture with 16 experts and the ability to select 4, providing 65x more possible expert combinations than other open MoE models
• Trained on 12 trillion tokens of carefully curated text and code data
• Exceptional performance on benchmarks for general knowledge, commonsense reasoning, programming, and mathematical reasoning
• Outperforms leading open models like Mixtral Instruct and Code LLama (70B)

Intended Use

DBRX Instruct is a general-purpose LLM designed for a wide range of natural language processing tasks, including text generation, question answering, code generation, and more. It is particularly well-suited for applications requiring strong programming and mathematical reasoning capabilities.

Language Support

DBRX Instruct is a multilingual model, supporting a wide range of languages.

Technical Details

Architecture

DBRX Instruct is a transformer-based, decoder-only LLM that was trained using next-token prediction. It utilizes a fine-grained MoE architecture, with 16 experts and the ability to select 4 for any given input.

Training Data

The model was pre-trained on 12 trillion tokens of carefully curated text and code data, with a maximum context length of 32,000 tokens. This dataset is estimated to be at least 2x better quality than the data used to train the MPT family of models.

Performance Metrics

On standard benchmarks, DBRX Instruct outperforms leading open models:

• MMLU (Multiple-choice Model-Linguistic Understanding): 73.7% vs. 71.4% for Mixtral Instruct
• HellaSwag 10-shot: 89.0% vs. 87.6% for Mixtral Instruct
• WinoGrande: 81.8% vs. 81.1% for Mixtral Instruct
• Databricks Gauntlet: 66.8% vs. 60.7% for Mixtral Instruct
• HumanEval: 70.1% vs. 54.8% for Mixtral Instruct
• GSM8k: 66.9% vs. 61.1% for Mixtral Instruct

Usage

API Example

License

DBRX Instruct is available for use under the Databricks Open Model License.