Skip to content

Radar ·

FP4 training of MoE models on Hopper GPUs

A practical discussion of FP4 training for large-scale Mixture-of-Experts models on Hopper GPUs identifies activation memory and expert-parallel communication as bottlenecks.

The material covers practical FP4 training of large-scale Mixture-of-Experts (MoE) models on Hopper GPUs. It identifies two bottlenecks: memory used by activations and communication among experts in the expert-parallel scheme.

These issues matter to engineers studying memory use and communication in large-scale MoE training. If you use AI to study or apply the material, do not submit internal data, credentials, or personal information; use fictional or anonymized examples instead.

Original source: Treinamento FP4 de modelos MoE de grande escala em Hopper

Curation · AgentLog

Get the essentials

Privacy, AI engineering and security in your inbox.

At most one digest per day, when there is news. Cancel at any time. We use your account's verified email.

Privacy

Looking for something else?DocumentationContact