allegationSaid on 2026-11-05
Large AI labs train models with MTP, but 80% of the gradient they backpropagate to the model is static noise that interferes with learning.
A allegation by post author, said on 2026-11-05. Statements are as reported by the cited publishers — not verified by DecisionGateway.
מעבדות ה-AI הגדולות מאמנות מודלים עם MTP, אבל מסתבר ש-80% מהגראדיאנט שהן מפעפעו אחורה (בקפרופ) למודלל הוא בכלל רעש סטטי שמפריע למודל ללמוד.
Discussion
Related claims
If you train models with MTP architecture, stop injecting noisy CE gradients and instead add the Soft-MTP line of code.מייק · commitmentWhen decomposing the standard MTP gradient (trained with standard Cross-Entropy), two opposing forces are hidden inside: a useful self-distillation signal and a residual noise component.post author · statementMIT researchers discovered that despite everyone using MTP, no one noticed a built-in mathematical bug in the training process.post author · investigationIn a regular network, if the model errs, the backprop algorithm stops everything, goes back from end to start and updates all the weights; in the brain there is no such central manager, and instead each synapse learns on its own from its local environment using STDP.author · statementOn algorithmic tasks (CLRS-Text), ten such layers raised an 86M-parameter model from 55% to 84%, after regular training got stuck at 55% and even training twice as long did not move it.post author · statementWhen the same gradient keeps appearing across several checkpoints, the writers interpret it as "demand" that SGD fails to supply.the writers · statement