statement
The researcher claims superiority at large scale, but the largest model he trained here is 1.2 billion parameters on 31.5 billion tokens.
A statement by החוקר. Statements are as reported by the cited publishers — not verified by DecisionGateway.
החוקר מצהיר על עליונות בסקייל גדול, אולם המודל הכי גדול שהוא אימן פה הוא 1.2 מיליארד פרמטרים (1.2B) על 31.5 מיליארד טוקנים.
Discussion
Related claims
On algorithmic tasks (CLRS-Text), ten such layers raised an 86M-parameter model from 55% to 84%, after regular training got stuck at 55% and even training twice as long did not move it.post author · statementIn transformer-based language models and architectures like Mamba, every token prediction requires activating billions of parameters in parallel through the attention mechanism.author · statementLarge AI labs train models with MTP, but 80% of the gradient they backpropagate to the model is static noise that interferes with learning.post author · allegationIn every attention head in a model like BERT there are over 8,000 parameters that mathematically cancel each other out behind the scenes, a phenomenon called 'calibration freedom'.the post author · statementThe researchers confirm their claims in experiments: they post-train models such as Qwen and Llama on a task of solving systems of equations, where the final answer is 0 or 1 so about half the cases can be solved by guessing.החוקרים · investigationThe models trained on the easier examples are tested on problems harder than those they saw in post-training and still succeed much more than models trained directly on the test difficulty level, without receiving full solutions or feedback on intermediate steps.המחברים · investigation