Model Overview Card for MiniMax-Text-01

Basic Information

• Model Name: MiniMax-Text-01
• Developer/Creator: MiniMax AI
• Release Date: January 15, 2025
• Version: 1.0
• Model Type: Large Language Model (LLM)

Description

Overview:

MiniMax-Text-01 is a powerful language model developed by MiniMax AI, designed to excel in tasks requiring extensive context processing and reasoning capabilities. With a total of 456 billion parameters, of which 45.9 billion are activated per token, this model utilizes a hybrid architecture that combines various attention mechanisms to optimize performance across a wide array of applications.

Key Features:

• Hybrid Architecture: Integrates Lightning Attention, Softmax Attention, and Mixture-of-Experts (MoE) for enhanced context handling.
• Extended Context Length: Capable of processing up to 1 million tokens during training and managing inference contexts of up to 4 million tokens.
• Top Performance on Benchmarks: Achieves competitive scores on academic benchmarks, including MMLU and various reasoning tests.
• Efficient Parameter Usage: Activates only the necessary parameters for each token processed, optimizing computational efficiency.
• Open Source Availability: Released under an MIT license, allowing for broad usage and modification.

Intended Use:

MiniMax-Text-01 is intended for software developers, researchers, and data scientists who require advanced natural language processing capabilities. It is particularly useful for applications involving deep reasoning, long-context processing, and efficient handling of large datasets.

Language Support:

The model primarily supports English but can accommodate multiple languages depending on user requirements.

Technical Details

Architecture:

MiniMax-Text-01 employs a sophisticated architecture featuring:

• Total Parameters: 456 billion
• Activated Parameters per Token: 45.9 billion
• Number of Layers: 80
• Attention Mechanisms:
  • Hybrid approach with Softmax attention following every 7 Lightning attention layers.
  • Number of attention heads: 64
  • Attention head dimension: 128
  • Mixture of Experts (MoE) with 32 experts and a top-2 routing strategy.

The model uses Rotary Position Embedding (RoPE) for positional encoding with a base frequency of 10,000.

Training Data:

The model was trained on a diverse dataset that includes a variety of programming languages and general knowledge sources.

• Data Source and Size: The training dataset encompasses approximately 14.8 trillion tokens from publicly available texts and code repositories.
• Diversity and Bias: The training data was curated to minimize biases while maximizing diversity in topics and styles, enhancing the model's effectiveness in generating varied outputs.

Performance Metrics:

Usage

Code Samples:

The model is available on the AI/ML API platform as "MiniMax-Text-01" .

API Documentation:

Detailed API Documentation is available here.

Ethical Guidelines

MiniMax AI emphasizes ethical considerations in AI development by promoting transparency regarding the model's capabilities and limitations. The organization encourages responsible usage to prevent misuse or harmful applications of generated content.

Licensing

MiniMax-Text-01 is available under an open-source MIT license that allows both research and commercial usage rights while ensuring compliance with ethical standards regarding creator rights.

‍

Get MiniMax-Text-01 API here.