AMD Instella-MoE: A Fully Open 16B Mixture-of-Experts Model Trained on AMD Instinct GPUs

• Gated Multi-head Latent Attention (Gated MLA): An augmentation of standard Multi-head Latent Attention (MLA). It adds a lightweight learned output gate that introduces a data-driven non-linearity, selectively attenuating low-utility attention responses to increase expressivity at a modest computational cost .
• FarSkip-Collective: A modification to MoE connectivity that overlaps communication with computation during expert-parallel training, reducing communication bubbles. During pre-training, this technique accelerated training by 12.7% .

The model also uses a Multi-Token Prediction (MTP) objective during its pre-training and mid-training phases .

Context Window

After a dedicated long-context extension training stage, Instella-MoE supports a 64K-token context .

Six-Stage Training Pipeline

AMD published the complete training pipeline, which consists of six sequential stages :

1. Pre-training
2. Mid-training
3. Long-Context Extension (extending the context to 64K tokens)
4. Supervised Fine-Tuning (SFT) — including a feedback-driven data curation stage using fine-grained error analysis and similarity-based retrieval
5. Direct Preference Optimization (DPO) — with router bias updates and the auxiliary load-balancing loss disabled to maintain training stability
6. Reinforcement Learning (RL) — using the Miles RL framework with ROCm support on AMD Instinct GPUs, incorporating IF-specialized RL training and Multi-Teacher On-Policy Distillation (MOPD)

Benchmark Performance

The Instella-MoE-16B-A3B base model achieved an average score of 76.7 on standard benchmarks, positioning it among the top fully open models at its scale . It reportedly outperformed SmolLM3-3B and OLMo-3-7B, and competed with larger models despite activating only 2.8 billion parameters per token .

Full Transparency and Licensing

In line with its commitment to open AI, AMD released all training artifacts openly :

• Model weights from every training stage
• Training configurations and data mixtures
• Intermediate checkpoints
• Inference code

All artifacts are available on Hugging Face under the amd/Instella-MoE-16B-A3B namespace, licensed under a Research RAIL (Responsible AI License) for academic and research purposes .

Strategic Significance

Instella-MoE is more than a model release — it is a direct proof point that AMD's ROCm software stack and Instinct GPU hardware can support frontier-scale AI development, from pre-training through RL, using fully open infrastructure. By delivering competitive benchmarks against leading open-source models trained on Nvidia hardware, AMD positions its ecosystem as a viable alternative to Nvidia's CUDA for large-scale AI research and development .