statementSaid on 2026-11-05
Soft-MTP obtains the same gradient for free by a simple subtraction between gradients already computed, instead of computing KL from scratch (which requires a lot of memory).
A statement by post author, said on 2026-11-05. Statements are as reported by the cited publishers — not verified by DecisionGateway.
כאן נכנס הקסם ההנדסי של Soft-MTP: במקום לחשב KL מאפס (הרבה זכרון), החוקרים גילו שאפשר לקבל את אותו גראדיאנט בחינם על ידי חיסור פשוט בין הגראדיאנטים שממילא חישבנו.
Discussion
Related claims
If you train models with MTP architecture, stop injecting noisy CE gradients and instead add the Soft-MTP line of code.מייק · commitmentWhen decomposing the standard MTP gradient (trained with standard Cross-Entropy), two opposing forces are hidden inside: a useful self-distillation signal and a residual noise component.post author · statementAdding the Soft-MTP line of code isolates the Self-Distillation signal and yields a smarter, faster model in speculative decoding without paying an additional penny in compute or memory.מייק · statementTo know where to implant, the agent computes the loss gradient for each token according to its representation in each layer.post author · statementLarge AI labs train models with MTP, but 80% of the gradient they backpropagate to the model is static noise that interferes with learning.post author · allegationWhen the same gradient keeps appearing across several checkpoints, the writers interpret it as "demand" that SGD fails to supply.the writers · statement