---
title: "In Energy-Based Transformers, several optimization steps are run on the energy during training and… — post author — DecisionGateway"
description: "statement by post author, recorded 2026-10-25 — verbatim quote and source citation."
canonical_url: "https://dgw.tmarket.io/claims/01a1e1f3-ce79-7bf5-95ba-b94da6ad7af2"
meta:
  "og:description": "statement by post author, recorded 2026-10-25 — verbatim quote and source citation."
  "og:image": "https://dgw.tmarket.io/og.jpg"
  "og:image:alt": "White network of connected nodes crossed by a timeline on black"
  "og:image:height": "1024"
  "og:image:width": "1024"
  "og:title": "In Energy-Based Transformers, several optimization steps are run on the energy during training and… — post author — DecisionGateway"
  "twitter:image": "https://dgw.tmarket.io/og.jpg"
---

> Index of every page on this site: [llms.txt](https://dgw.tmarket.io/llms.txt)

statementSaid on 2026-10-25

# **In Energy-Based Transformers, several optimization steps are run on the energy during training and the model is trained through that process itself, which is more expensive.**

A statement by post author, said on 2026-10-25. Statements are as reported by the cited publishers — not verified by DecisionGateway.

> *בEnergy-Based Transformers מריצים כמה צעדי אופטימיזציה על האנרגיה בזמן האימון ומאמנים דרך התהליך הזה עצמו, וזה יקר יותר.*

mathyaiwithmikePublished 2026-10-25 [ Read source](https://t.me/mathyaiwithmike/1295)

## **Discussion**

## **Related claims**

[**The problem with EBMs is training: it is not enough to teach the model that real examples have low energy; it must also be taught that everything else has high energy, and that 'everything else' is enormous, which is why EBMs were stuck for years with unstable training and expensive sampling.**post author · statement](https://dgw.tmarket.io/claims/01a1e1f3-ce4b-7960-83c7-fe561635767d) [**Diffusion is almost an EBM: the direction diffusion learns is exactly the gradient-descent direction of some energy (in the transition from noisy data to clean), meaning diffusion directly learns the slope of the energy surface and skips the energy itself — the very trick that bypassed the difficulty of training EBMs.**post author · statement](https://dgw.tmarket.io/claims/01a1e1f3-ce6a-7994-8b63-3d2f141cabfb) [**In transformer-based language models and architectures like Mamba, every token prediction requires activating billions of parameters in parallel through the attention mechanism.**author · statement](https://dgw.tmarket.io/claims/01a1e1f1-92c2-775b-b01e-62ef447243fc) [**Alexi Gladstone's tweet is a good opportunity to clarify what an Energy-Based Model is, what a diffusion model is, and why the difference between them is smaller and more interesting than the hype suggests.**post author · statement](https://dgw.tmarket.io/claims/01a1e1f3-ce33-7357-88a1-e035201915b2) [**The researchers show that training leads the model to learn an efficient solution path, even though it receives no extra reward for choosing that path.**החוקרים · investigation](https://dgw.tmarket.io/claims/01a1e1f1-8d4e-7407-9371-5176e7fb9053) [**When decomposing the standard MTP gradient (trained with standard Cross-Entropy), two opposing forces are hidden inside: a useful self-distillation signal and a residual noise component.**post author · statement](https://dgw.tmarket.io/claims/01a1e1ee-c9b5-7ec8-958e-20cbda9fe423)

[All claims](https://dgw.tmarket.io/claims)

```json
{"@context":"https://schema.org","@graph":[{"@id":"https://dgw.tmarket.io/#website","@type":"WebSite","description":"Decisions, events and falsifiable forecasts from 4,500+ Telegram channels — every claim cited. Read free.","name":"DecisionGateway","url":"https://dgw.tmarket.io/","publisher":{"@id":"https://dgw.tmarket.io/#identity"}},{"@id":"https://dgw.tmarket.io/claims/01a1e1f3-ce79-7bf5-95ba-b94da6ad7af2#webpage","@type":"WebPage","description":"statement by post author, recorded 2026-10-25 — verbatim quote and source citation.","name":"In Energy-Based Transformers, several optimization steps are run on the energy during training and… — post author — DecisionGateway","url":"https://dgw.tmarket.io/claims/01a1e1f3-ce79-7bf5-95ba-b94da6ad7af2","breadcrumb":{"@id":"https://dgw.tmarket.io/claims/01a1e1f3-ce79-7bf5-95ba-b94da6ad7af2#breadcrumb"},"isPartOf":{"@id":"https://dgw.tmarket.io/#website"},"potentialAction":[{"@type":"ReadAction","target":["https://dgw.tmarket.io/claims/01a1e1f3-ce79-7bf5-95ba-b94da6ad7af2"]}]},{"@id":"https://dgw.tmarket.io/#identity","@type":"Organization","name":"DecisionGateway","url":"https://dgw.tmarket.io"},{"@id":"https://dgw.tmarket.io/claims/01a1e1f3-ce79-7bf5-95ba-b94da6ad7af2#breadcrumb","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","name":"Home","item":"https://dgw.tmarket.io","position":1},{"@type":"ListItem","name":"Claims","item":"https://dgw.tmarket.io/claims","position":2},{"@type":"ListItem","name":"In Energy-Based Transformers, several optimization steps…","position":3}]},{"@id":"https://dgw.tmarket.io/claims/01a1e1f3-ce79-7bf5-95ba-b94da6ad7af2#/schema/quotation/1","@type":"Quotation","dateCreated":"2026-10-25","isBasedOn":"https://t.me/mathyaiwithmike/1295","text":"בEnergy-Based Transformers מריצים כמה צעדי אופטימיזציה על האנרגיה בזמן האימון ומאמנים דרך התהליך הזה עצמו, וזה יקר יותר.","url":"/claims/01a1e1f3-ce79-7bf5-95ba-b94da6ad7af2"}]}
```

## Sitemap

See the full [sitemap](https://dgw.tmarket.io/sitemap.md) for all pages.
