statementSaid on 2026-11-05
When decomposing the standard MTP gradient (trained with standard Cross-Entropy), two opposing forces are hidden inside: a useful self-distillation signal and a residual noise component.
A statement by post author, said on 2026-11-05. Statements are as reported by the cited publishers — not verified by DecisionGateway.
כשפירקו את נוסחת הגראדיאנט הרגיל של ה-MTP (שמאומן עם פונקציית Cross-Entropy או CE סטנדרטית), הם גילן שמתחבאים בתוכו שני כוחות מנוגדים:
Discussion
Related claims
If you train models with MTP architecture, stop injecting noisy CE gradients and instead add the Soft-MTP line of code.מייק · commitmentSoft-MTP obtains the same gradient for free by a simple subtraction between gradients already computed, instead of computing KL from scratch (which requires a lot of memory).post author · statementDuring the backward pass, the MTP residual noise and the NTP residual noise meet with opposite signs and cancel each other out, leaving only the clean self-distillation signal; the result is 0 added memory, 0 impact on token throughput, and full compatibility with Fused kernels.post author · statementLarge AI labs train models with MTP, but 80% of the gradient they backpropagate to the model is static noise that interferes with learning.post author · allegationAdding the Soft-MTP line of code isolates the Self-Distillation signal and yields a smarter, faster model in speculative decoding without paying an additional penny in compute or memory.מייק · statementWhen the same gradient keeps appearing across several checkpoints, the writers interpret it as "demand" that SGD fails to supply.the writers · statement