investigation
Raising reasoning effort improves average benchmark scores but produces painful performance drops at the individual task level; tasks solved easily and cheaply at Low fail at High or xHigh.
A investigation by החוקרים. Statements are as reported by the cited publishers — not verified by DecisionGateway.
הכשל המרכזי הוא שהעלאת רמת המאמץ אמנם משפרת את הציון הממוצע בטבלאות הביצועים, אבל מייצרת ירידות ביצועים כואבות ברמת המשימה הבודדת.
Discussion
Related claims
On easy tasks solved at every effort level, costs can be up to 42x higher in API costs because the model treats effort as a fixed token multiplier.החוקרים · investigationWhen researchers deleted git history and blocked network access (preventing benchmark leakage), high-effort success rates dropped sharply.החוקרים · investigationWhen the authors train the models only on problems at the test difficulty level, the models reach about 50% accuracy, almost like random guessing.המחברים · investigationThe models trained on the easier examples are tested on problems harder than those they saw in post-training and still succeed much more than models trained directly on the test difficulty level, without receiving full solutions or feedback on intermediate steps.המחברים · investigationTurning off thinking entirely (None level) is sometimes more expensive than Low, because the agent compensates with a four-times-longer action chain that inflates the context window and input token bill.החוקרים · investigationThe researchers propose naive laddering: always start at low effort and escalate to high only on failure, since models are bad at self-routing in advance; this cuts costs by tens of percent and improves performance.החוקרים · commitment