Model Overview Card for Dolly v2 (3B)

Basic Information

• Model Name: Dolly v2 (3B)
• Developer/Creator: Databricks, Inc.
• Release Date: April 12, 2023
• Version: Dolly-v2-3b
• Model Type: Instruction-following Large Language Model

Description

Overview:
Dolly v2 (3B) is an instruction-following large language model created by Databricks, designed to follow instructions and perform various language tasks. Based on the Pythia-2.8b model, Dolly v2 (3B) has been fine-tuned on a dataset of approximately 15k instruction/response pairs to enhance its ability to generate high-quality responses to prompts.

Key Features:

• Fine-tuned on ~15k instruction/response pairs
• Capable of performing tasks such as brainstorming, classification, closed QA, generation, information extraction, open QA, and summarization
• Licensed for commercial use
• Available in larger sizes (dolly-v2-7b and dolly-v2-12b)

Intended Use:

Dolly v2 (3B) is designed for various natural language processing tasks including brainstorming, classification, closed and open question answering, generation, information extraction, and summarization. It is suitable for applications requiring high-quality instruction following, though it is not state-of-the-art.

Language Support:

Supports English. Other languages might be supported but with potentially less accuracy due to the training data being primarily in English.

Technical Details

Architecture:

Dolly v2 (3B) is based on the Pythia-2.8b model, a Transformer-based architecture.

Training Data:

The model was trained on a dataset of approximately 15,000 instruction/response pairs generated by Databricks employees. This dataset, named databricks-dolly-15k, covers various domains mentioned in the InstructGPT paper, including brainstorming, classification, QA, and summarization.

Data Source and Size:

• Source: Public internet, including Wikipedia.
• Size: Approximately 15,000 instruction/response pairs.
• Knowledge Cutoff: The model's knowledge is up-to-date until April 2023.
• Diversity and Bias: The dataset includes data that reflects the interests and biases of Databricks employees, potentially limiting diversity. It is also subject to the biases present in the public internet data from which it was derived.

Performance Metrics:

• Comparison to Other Models:
  Dolly v2 (3B) outperforms its foundation model Pythia-2.8b and shows competitive performance with similar parameter models but underperforms compared to state-of-the-art models like GPT-4 and LLaMA-3.
• Accuracy: Demonstrates strong instruction-following behavior, but may struggle with syntactically complex prompts, programming problems, mathematical operations, factual accuracy, and handling dates and times.
• Speed: Optimized for inference on GPUs; performance varies based on hardware.
• Robustness: Handles a wide range of instructions but may produce errors in specific complex or ambiguous tasks.

Usage

Code Samples/SDK

Ethical Considerations

Databricks is committed to developing AI technologies that are helpful, honest, and harmless. The model has limitations and may produce biased or harmful outputs, reflecting the biases present in the training data.

Licensing

Dolly v2 (3B) is released under a permissive license (CC-BY-SA), allowing for commercial use.