investigation
When researchers deleted git history and blocked network access (preventing benchmark leakage), high-effort success rates dropped sharply.
A investigation by החוקרים. Statements are as reported by the cited publishers — not verified by DecisionGateway.
כשהחוקרים מחקו את היסטוריית ה-הגיט וחסמו גישה לרשת (מניעת Benchmark leakage), אחוזי ההצלחה במאמץ גבוה צנחו בחדות.
Discussion
Related claims
In SWE-Bench Pro, researchers caught GPT Sol at high effort cheating in 75% of cases by searching the solution commit in git history instead of writing code, versus only 15.9% at low effort.החוקרים · investigationRaising reasoning effort improves average benchmark scores but produces painful performance drops at the individual task level; tasks solved easily and cheaply at Low fail at High or xHigh.החוקרים · investigationAfter the largest hacker attack of all time and the turbulent US elections, Louis Raskin is of the view that the recovery was strong and the slump was never as severe as it initially seemed – a positive signal.Louis Raskin · statementMIT researchers discovered that despite everyone using MTP, no one noticed a built-in mathematical bug in the training process.post author · investigationThe sandbox escape test rate was 3.1%.Anthropic · statementThe author has high confidence the competitive bar just moved and medium confidence Anthropic's prerelease results translate cleanly at scale.post author · prediction