investigation
The paper proves that when training reasoning models with RL, it is not always true that training data should match the test-time distribution; sometimes data different from what the model will see at test time yields much better test-time results.
A investigation by המאמר. Statements are as reported by the cited publishers — not verified by DecisionGateway.
אבל מאמר שנסקור היום מוכיח שבאימון מודלי ריזונינג עם RL זה לא תמיד נכון: לפעמים, אם בוחרים דאטה ששונה ממה שהמודל יראה ב-test time, מקבלים תוצאות הרבה יותר טובות(ב- test-time).
Discussion
Related claims
The post's author concludes that in supervised learning a distribution shift between train and test is usually a problem, while for RL training it is part of the solution.כותב הפוסט · statementWhen the authors train the models only on problems at the test difficulty level, the models reach about 50% accuracy, almost like random guessing.המחברים · investigationThe models trained on the easier examples are tested on problems harder than those they saw in post-training and still succeed much more than models trained directly on the test difficulty level, without receiving full solutions or feedback on intermediate steps.המחברים · investigationThe researchers prove that in their synthetic framework, the training data must include enough simple examples (requiring few reasoning steps); with enough such examples the model learns to solve problems step by step, and without them it does not learn.החוקרים · investigationThe researchers show that training leads the model to learn an efficient solution path, even though it receives no extra reward for choosing that path.החוקרים · investigationThe researchers confirm their claims in experiments: they post-train models such as Qwen and Llama on a task of solving systems of equations, where the final answer is 0 or 1 so about half the cases can be solved by guessing.החוקרים · investigation