AI · 1h ago
VIDRAFT Open-Sources 6.6B MoE LLM with Five Attention Types
Korean AI startup VIDRAFT released Aether-7B-5Attn, a 6.59B-parameter Mixture-of-Experts model under Apache-2.0. It uses five heterogeneous attention mechanisms across 49 layers, with only 2.98B active parameters per token. The release includes full training code, data recipes, and intermediate checkpoints for reproducibility.
Meridian48 take
The model's bilingual focus (Korean/English) and novel attention mixing are noteworthy, but benchmark comparisons against similarly sized open models are needed to gauge real-world performance.
Read the full reporting
VIDRAFT Releases Aether-7B-5Attn: A Fully Open-Source MoE LLM with Five Heterogeneous Attention Mechanisms →
DEV Community
open-source-llmmixture-of-experts