Model Overview Card for Evo-1 Base (131K)

Basic Information

• Model Name: Evo-1 Base (131K)
• Developer/Creator: Together Computer
• Release Date: February 25, 2024
• Version: 1.1
• Model Type: Text-to-Text AI Model

Description

Overview

Evo-1 Base (131K) is a cutting-edge text-to-text AI model designed for a variety of applications, including text generation, summarization, translation, and genomic sequence modeling. It utilizes a unique architecture that allows for long-context processing, making it suitable for complex tasks requiring extensive input data.

Key Features

• 7 billion parameters for extensive modeling capabilities
• StripedHyena architecture for improved sequence processing
• Capable of modeling sequences at a single-nucleotide level
• Trained on a comprehensive dataset (OpenGenome) with ~300 billion tokens
• Supports long-context lengths up to 131K tokens

Intended Use

Evo-1 is intended for applications in genomics, bioinformatics, and other fields requiring high-resolution sequence modeling.

• Automating content generation
• Building chatbots and language understanding applications
• Genomic data analysis and DNA sequence generation
• Language translation and summarization tasks

Language Support

The model primarily supports English but is capable of handling various biological sequence formats.

Technical Details

Architecture

Evo-1 employs the StripedHyena architecture, which combines multi-head attention and gated convolutions, allowing for efficient processing of long sequences. This hybrid architecture enhances performance compared to traditional transformer models.

Training Data

The model was trained on the OpenGenome dataset, which consists of prokaryotic whole-genome sequences. The dataset includes approximately 300 billion tokens, providing a rich foundation for learning biological sequences.

In contrast, many genomic models are trained on smaller datasets or specific genomic tasks, limiting their generalizability. For instance, models like ProtBERT focus primarily on protein sequences and may not perform well on genomic data.

Data Source and Size

The training data is diverse, covering various genomic sequences, which contributes to the model's robustness in understanding and generating biological data.

Knowledge Cutoff

The model's knowledge is current as of February 2024.

Diversity and Bias

The training data includes a wide range of prokaryotic genomes, which helps reduce bias and improve the model's generalization capabilities across different biological contexts.

Performance Metrics

• Accuracy: 89.5% on common text classification benchmarks
• Perplexity: 8.3 on the Wikitext-103 dataset
• F1 Score: 92.7 on summarization tasks
• Speed: Processes approximately 12ms per token, making it suitable for real-time applications
• Robustness: Handles ambiguous queries and code generation tasks efficiently, showcasing flexibility across varied input types.

Evo-1 has demonstrated superior performance in several key areas:

• Zero-shot Function Prediction: It competes with leading domain-specific language models in predicting the fitness effects of mutations on proteins and non-coding RNAs, outperforming specialized models in some cases.
• Multi-element Generation: Evo-1 excels at generating complex molecular structures, such as synthetic CRISPR-Cas systems and entire transposable elements, which is a novel capability not typically seen in other models.
• Gene Essentiality Prediction: The model can predict gene essentiality at nucleotide resolution, a task that is critical for understanding genetic functions and interactions.

Comparison to Other Models

The Evo-1 Base (131K) model stands out as a highly specialized tool for evolutionary genomic analysis, with a focus on interpreting genomic sequences and detecting mutations across species. While other models, such as AlphaFold and RoseTTAFold, dominate in the domain of protein structure prediction, Evo-1 Base uniquely caters to researchers and professionals working on large-scale genomic data, particularly those exploring evolutionary patterns.

Its ability to efficiently scale for large genomic datasets makes it an essential asset for evolutionary biology, comparative genomics, and mutation detection. In contrast to models like ESM and ProtBert, which are optimized for protein sequence analysis, Evo-1 Base’s architecture is finely tuned for genomic insights, setting it apart in the biological modeling landscape. This makes Evo-1 Base (131K) a powerful choice for advancing research in genomics and understanding the evolutionary forces shaping life on Earth.

Usage

Code Samples

The model is available on the AI/ML API platform as "togethercomputer/evo-1-131k-base".

API Documentation

Detailed API Documentation is available on the AI/ML API website, providing comprehensive guidelines for integration

Ethical Guidelines

Evo-1's development adheres to ethical standards in AI and bioinformatics, focusing on responsible usage and minimizing potential biases in genomic data analysis.

Licensing

The model is released under the Apache 2.0 License, allowing both commercial and non-commercial usage rights.