statementSaid on 2026-11-08
On algorithmic tasks (CLRS-Text), ten such layers raised an 86M-parameter model from 55% to 84%, after regular training got stuck at 55% and even training twice as long did not move it.
A statement by post author, said on 2026-11-08. Statements are as reported by the cited publishers — not verified by DecisionGateway.
במשימות אלגוריתמיות (CLRS-Text), עשר שכבות כאלה העלו מודל של 86M פרמטרים מ-55% ל-84%, אחרי שאימון רגיל נתקע על 55% וגם אימון ארוך פי שניים לא הזיז אותו.
Discussion
Related claims
The researcher claims superiority at large scale, but the largest model he trained here is 1.2 billion parameters on 31.5 billion tokens.החוקר · statementThe models trained on the easier examples are tested on problems harder than those they saw in post-training and still succeed much more than models trained directly on the test difficulty level, without receiving full solutions or feedback on intermediate steps.המחברים · investigationWhen the authors train the models only on problems at the test difficulty level, the models reach about 50% accuracy, almost like random guessing.המחברים · investigationThe researchers confirm their claims in experiments: they post-train models such as Qwen and Llama on a task of solving systems of equations, where the final answer is 0 or 1 so about half the cases can be solved by guessing.החוקרים · investigationIn transformer-based language models and architectures like Mamba, every token prediction requires activating billions of parameters in parallel through the attention mechanism.author · statementResearchers discovered long ago that about 75% of attention heads in a trained model can be deleted and it will keep working almost the same, and the book shows the structural reason: at the end of each layer the model merges all units' insights into one blend, so the next layer receives a 'fruit shake' and cannot know which unit discovered what.the post author · statement