Dev Tools · 14h ago
26B MoE model runs on cheapest AWS Inferentia2 instance, 8.6x cost cut
A developer ported a 26B-parameter mixture-of-experts model to AWS's $0.76/hr inf2.xlarge, down from $6.49/hr. The trick: int8 quantization of experts and aggressive memory management. The model runs on a single 2-core Inferentia2 with 32 GB HBM and 16 GB host RAM.
Meridian48 take
This is a practical optimization win, but the claim of 'cheapest' depends on spot pricing and assumes single-user throughput; scaling to production loads may still require larger instances.
Read the full reporting
Squeezing a 26B MoE onto the *cheapest* Inferentia box — from a $6.49/hr 24xlarge to a $0.76/hr inf2.xlarge →
DEV Community
aws-inferentiamodel-optimization