Dev Tools · 2h ago
Porting Gemma-4 models to AWS Inferentia2: a technical deep dive
A developer ported all five Gemma-4 models (E2B, E4B, 12B, 31B, 26B-A4B MoE) to run on AWS Inferentia2 hardware. Each model achieves token-for-token identical output to CPU fp32 reference, with speeds ranging from 34 to 44 tok/s for smaller models and prefill times under 115 ms for larger ones. The ports required hand-rolled solutions as vendor tools lacked support for Gemma-4 architectures.
Meridian48 take
This demonstrates that custom inference optimization can unlock new hardware for cutting-edge models, but the lack of vendor support highlights ongoing gaps in AI infrastructure tooling.
Read the full reporting
Five Gemma-4 models, one accelerator: what porting E2B 31B to AWS Inferentia2 taught me →
DEV Community
aws-inferentiagemma-4