investigation
When the authors train the models only on problems at the test difficulty level, the models reach about 50% accuracy, almost like random guessing.
A investigation by המחברים. Statements are as reported by the cited publishers — not verified by DecisionGateway.
כאשר המחברים מאמנים את המודלים רק על בעיות ברמת הקושי של ה-test, המודלים מגיעים לכ-50% דיוק, כמעט כמו ניחוש אקראי.
Discussion
Related claims
The models trained on the easier examples are tested on problems harder than those they saw in post-training and still succeed much more than models trained directly on the test difficulty level, without receiving full solutions or feedback on intermediate steps.המחברים · investigationThe researchers confirm their claims in experiments: they post-train models such as Qwen and Llama on a task of solving systems of equations, where the final answer is 0 or 1 so about half the cases can be solved by guessing.החוקרים · investigationWhen the test-difficulty examples are replaced with a mixture of easier examples, Qwen jumps to 91.6% and Llama to 98.6%, while the test remains at exactly the same difficulty level.המחברים · investigationThe paper proves that when training reasoning models with RL, it is not always true that training data should match the test-time distribution; sometimes data different from what the model will see at test time yields much better test-time results.המאמר · investigationOn algorithmic tasks (CLRS-Text), ten such layers raised an 86M-parameter model from 55% to 84%, after regular training got stuck at 55% and even training twice as long did not move it.post author · statementRaising reasoning effort improves average benchmark scores but produces painful performance drops at the individual task level; tasks solved easily and cheaply at Low fail at High or xHigh.החוקרים · investigation