statementSaid on 2026-10-25
In Energy-Based Transformers, several optimization steps are run on the energy during training and the model is trained through that process itself, which is more expensive.
A statement by post author, said on 2026-10-25. Statements are as reported by the cited publishers — not verified by DecisionGateway.
בEnergy-Based Transformers מריצים כמה צעדי אופטימיזציה על האנרגיה בזמן האימון ומאמנים דרך התהליך הזה עצמו, וזה יקר יותר.
Discussion
Related claims
The problem with EBMs is training: it is not enough to teach the model that real examples have low energy; it must also be taught that everything else has high energy, and that 'everything else' is enormous, which is why EBMs were stuck for years with unstable training and expensive sampling.post author · statementDiffusion is almost an EBM: the direction diffusion learns is exactly the gradient-descent direction of some energy (in the transition from noisy data to clean), meaning diffusion directly learns the slope of the energy surface and skips the energy itself — the very trick that bypassed the difficulty of training EBMs.post author · statementIn transformer-based language models and architectures like Mamba, every token prediction requires activating billions of parameters in parallel through the attention mechanism.author · statementAlexi Gladstone's tweet is a good opportunity to clarify what an Energy-Based Model is, what a diffusion model is, and why the difference between them is smaller and more interesting than the hype suggests.post author · statementThe researchers show that training leads the model to learn an efficient solution path, even though it receives no extra reward for choosing that path.החוקרים · investigationWhen decomposing the standard MTP gradient (trained with standard Cross-Entropy), two opposing forces are hidden inside: a useful self-distillation signal and a residual noise component.post author · statement