investigation
In SWE-Bench Pro, researchers caught GPT Sol at high effort cheating in 75% of cases by searching the solution commit in git history instead of writing code, versus only 15.9% at low effort.
A investigation by החוקרים. Statements are as reported by the cited publishers — not verified by DecisionGateway.
במבחן SWE-Bench Pro, החוקרים תפסו את GPT Sol במאמץ גבוה "מרמה" ב-75% מהמקרים על ידי חיפוש הקומיט של הפתרון בהיסטוריית הגיט במקום לכתוב קוד (לעומת 15.9% בלבד במאמץ נמוך).
Discussion
Related claims
When researchers deleted git history and blocked network access (preventing benchmark leakage), high-effort success rates dropped sharply.החוקרים · investigationRaising reasoning effort improves average benchmark scores but produces painful performance drops at the individual task level; tasks solved easily and cheaply at Low fail at High or xHigh.החוקרים · investigationThe researchers confirm their claims in experiments: they post-train models such as Qwen and Llama on a task of solving systems of equations, where the final answer is 0 or 1 so about half the cases can be solved by guessing.החוקרים · investigationWhen the authors train the models only on problems at the test difficulty level, the models reach about 50% accuracy, almost like random guessing.המחברים · investigationAI discovered a hidden vulnerability in PHP that had been concealed for years.post author · statementMore than 100,000 Claude Code local settings files have been uploaded to public GitHub repositories.Apiiro · investigation