statementSaid on 2026-11-05
During the backward pass, the MTP residual noise and the NTP residual noise meet with opposite signs and cancel each other out, leaving only the clean self-distillation signal; the result is 0 added memory, 0 impact on token throughput, and full compatibility with Fused kernels.
A statement by post author, said on 2026-11-05. Statements are as reported by the cited publishers — not verified by DecisionGateway.
בזמן ה-Backward pass(בקפרופ) קורה איזה ״נס״ מתמטי: רעש השארית של ה-MTP ורעש השארית של ה-NTP נפגשים עם סימנים הפוכים ופשוט מאפסים זה את זה!
Discussion
Related claims
When decomposing the standard MTP gradient (trained with standard Cross-Entropy), two opposing forces are hidden inside: a useful self-distillation signal and a residual noise component.post author · statementThe residual component is 4 times noisier than the first term, and is responsible for 80% of the update strength flowing back to the shared trunk; as the model grows it only intensifies and pulls the weights in a direction that interferes with next-token prediction.post author · statementAdding the Soft-MTP line of code isolates the Self-Distillation signal and yields a smarter, faster model in speculative decoding without paying an additional penny in compute or memory.מייק · statementIf you train models with MTP architecture, stop injecting noisy CE gradients and instead add the Soft-MTP line of code.מייק · commitmentLarge AI labs train models with MTP, but 80% of the gradient they backpropagate to the model is static noise that interferes with learning.post author · allegationIn a regular network, if the model errs, the backprop algorithm stops everything, goes back from end to start and updates all the weights; in the brain there is no such central manager, and instead each synapse learns on its own from its local environment using STDP.author · statement