ResearchOCT 29TH, 2026Prime Mega MoE: Fast Fused MoE Kernels for Training

Prime Mega MoE extends DeepSeek's Mega MoE kernel, which previously had only a forward implementation, with a fast overlapping backward pass so it can be used for training. It reports up to 3.11× faster MoE kernels than the default PyTorch-based implementation and up to 1.34× faster end-to-end training than the Prime-RL baseline (34% higher throughput). Supported data types are bfloat16, MXFP8 (Blackwell only) and blockwise FP8 (Hopper only).