A new report from the AI evaluation nonprofit METR indicates that while certain AI capabilities are improving along an exponential trajectory, models continue to struggle with tasks requiring human-like judgment. The updated findings show that while frontier models have become highly proficient at verifiable tasks like coding and math, they fall short of human performance in more open-ended domains.
METR evaluates models by measuring the length of tasks they can complete compared to human testers. The organization's updated time horizon plot includes new, more complex tasks that are harder to measure than previous benchmarks. These include tasks such as scoring philosophical arguments and designing new AI evaluations. In the evaluation design task, not one of the AI-designed results was considered useful for METR's research by human employees.
Researchers noted that the ability to use reinforcement learning—a technique where systems attempt tasks repeatedly to strengthen successful neural pathways—is highly effective for verifiable tasks where success can be checked automatically. This process has driven significant progress in mathematics and software engineering. However, applying this technique to tasks with less defined success criteria, such as choosing a research direction, remains difficult.
Some of the most advanced models, including Opus 5.5, achieved what METR describes as "borderline above the bar" performance on evaluation design tasks, though they remain unusable for professional research purposes. While models are currently outperforming humans in software engineering, they have not yet mastered the fuzzy, judgment-laden tasks essential for automating research and development.
Computer scientist and incoming UC Berkeley professor Sayash Kapoor noted the necessity of expanding the scope of testing. “We’re at the point where AI evaluations need to get more creative,” Kapoor said.














