Radar: a unified view of Attention-MoE and FFN-MoE
A September 24, 2025 publication describes treating pre-mixed tokens and (V·Wo) as an FFN expert, unifying Attention-MoE and FFN-MoE with shared experts.
On September 24, 2025, a publication presented a paper proposing that pre-mixed tokens and (V·Wo) be treated as an FFN expert. The formulation unifies Attention-MoE and FFN-MoE with shared experts; according to the publication, it reduces perplexity with similar compute.
The approach may help engineers compare sparse attention and FFN designs. To check the scope and findings, consult the original paper and compare its formulation and metrics with the publication’s summary; the supplied material gives no experimental values or conditions.