AI models today have a 50% chance of successfully completing a task that would take an expert human one hour. Seven months ago, that number was roughly 30 minutes — and seven months before that, 15 minutes.These are substantial, multi-step tasks requiring sustained focus: building web applications, conducting machine learning research, or solving complex programming challenges.Today’s guest, Beth Barnes, is CEO of METR (Model Evaluation & Threat Research) — the leading organisation measuring these capabilities.These highlights are from episode #217 of The 80,000 Hours Podcast: Beth Barnes on the most important graph in AI right now — and the 7-month rule that governs its progress, and include:Can we see AI scheming in the chain of thought?

We have to test model honesty even before they're used inside AI companies

It's essential to thoroughly test relevant real-world tasks

Recursively self-improving AI might even be here in two years — which is alarming

Do we need external auditors doing AI safety tests, not just the companies themselves?

A case against safety-focused people working at frontier AI companies

Open-weighting models is often good, and Beth has changed her attitude about it

Rob's intro

Can we see AI scheming in the chain of thought?

We have to test model honesty even before they're used inside AI companies

It's essential to thoroughly test relevant real-world tasks

Recursively self-improving AI might even be here in two years — which is alarming

Do we need external auditors doing AI safety tests, not just the companies themselves?

A case against safety-focused people working at frontier AI companies

Open-weighting models is often good, and Beth has changed her attitude about it