investigationSaid on 2026-11-05
MIT researchers discovered that despite everyone using MTP, no one noticed a built-in mathematical bug in the training process.
A investigation by post author, said on 2026-11-05. Statements are as reported by the cited publishers — not verified by DecisionGateway.
אז זהו, שחוקרי MIT גילו שלמרות שכולם משתמשים בזה, אף אחד לא שם לב לבאג מתמטי מובנה בתהליך האימון.
Discussion
Related claims
MTP not only improves performance in math and code, but also provides built-in draft heads for inference acceleration via Speculative Decoding.post author · statementLarge AI labs train models with MTP, but 80% of the gradient they backpropagate to the model is static noise that interferes with learning.post author · allegationThe researchers show that training leads the model to learn an efficient solution path, even though it receives no extra reward for choosing that path.החוקרים · investigationWhen decomposing the standard MTP gradient (trained with standard Cross-Entropy), two opposing forces are hidden inside: a useful self-distillation signal and a residual noise component.post author · statementIn the Effective Rank metric, regular MTP training collapsed the output matrix from 496 to 131, while Soft-MTP kept it high and stable at 437.post author · statementIf you train models with MTP architecture, stop injecting noisy CE gradients and instead add the Soft-MTP line of code.מייק · commitment