Guanaco (13B)

Basic Information

• Model Name: Guanaco
• Developer/Creator: Tim Dettmers
• Release Date: May 2023
• Version: 13B
• Model Type: Text-based LLM

Description

Overview

The Guanaco-13B is an open-source, fine-tuned language model developed for creating high-quality, multilingual chatbots. It is based on the LLaMA base model and offers competitive performance with commercial systems like ChatGPT and BARD.

Key Features

• Open-source for research purposes
• Efficient 4-bit QLoRA fine-tuning on the OASST1 dataset
• Available in multiple sizes: 7B, 13B, 33B, and 65B parameters
• Multilingual support, though primarily in high-resource languages

Intended Use

Guanaco-13B is intended for researchers and developers looking to experiment with and create multilingual chatbots. It allows for cheap and local experimentation with high-quality chatbot systems.

Language Support

Guanaco-13B supports multiple languages, with a focus on high-resource languages. The model's multilingual capabilities are achieved through fine-tuning on the OASST1 dataset.

Technical Details

Architecture

Guanaco-13B is built upon the LLaMA base model, with LoRA (Low-Rank Adaptation) adapters added to all layers. The rank of the LoRA adapters is set to 64.

Training Data

The model is fine-tuned on the OASST1 (Open Assistant Supervised Task 1) dataset, which is multilingual but heavily weighted towards high-resource languages. The dataset consists of human-written conversations, allowing the model to learn natural dialogue patterns.

Data Source and Size

The OASST1 dataset is sourced from the Open Assistant project and contains a large number of human-written conversations. The exact size of the dataset used for fine-tuning Guanaco-13B is not publicly disclosed.

Knowledge Cutoff

The knowledge cutoff date for Guanaco-13B is not explicitly stated in the available documentation.

Diversity and Bias

As the OASST1 dataset is multilingual, Guanaco-13B has the potential to be less biased towards specific languages or regions compared to models trained on monolingual datasets. However, the dataset's focus on high-resource languages may introduce some biases.

Performance Metrics

Guanaco-13B has demonstrated competitive performance with commercial systems like ChatGPT and BARD on the Vicuna and OpenAssistant benchmarks.

Usage

API Usage Example

Ethical Guidelines

The developers of Guanaco-13B have not publicly released any specific ethical guidelines for the model's use. However, as an open-source model, it is expected that users will adhere to general ethical principles when using the model.

License Type

Guanaco-13B is released under an open-source license, allowing for research and non-commercial use. The exact license type is not specified in the available documentation.