statementSaid on 2026-11-05
In the Effective Rank metric, regular MTP training collapsed the output matrix from 496 to 131, while Soft-MTP kept it high and stable at 437.
A statement by post author, said on 2026-11-05. Statements are as reported by the cited publishers — not verified by DecisionGateway.
במדד דרגה אפקטיבית (Effective Rank) שבודק כמה ממדים במטריצה באמת פעילים, אימון MTP רגיל ריסק את המטריצה מ-496 ל-131 (קריסת ממדים מפוארת), בעוד ש-Soft-MTP שמר אותה גבוהה ויציבה על 437.
Discussion
Related claims
If you train models with MTP architecture, stop injecting noisy CE gradients and instead add the Soft-MTP line of code.מייק · commitmentWhen decomposing the standard MTP gradient (trained with standard Cross-Entropy), two opposing forces are hidden inside: a useful self-distillation signal and a residual noise component.post author · statementMTP not only improves performance in math and code, but also provides built-in draft heads for inference acceleration via Speculative Decoding.post author · statementMIT researchers discovered that despite everyone using MTP, no one noticed a built-in mathematical bug in the training process.post author · investigationRaising reasoning effort improves average benchmark scores but produces painful performance drops at the individual task level; tasks solved easily and cheaply at Low fail at High or xHigh.החוקרים · investigationLarge AI labs train models with MTP, but 80% of the gradient they backpropagate to the model is static noise that interferes with learning.post author · allegation