Episode 137I spoke with Davidad Dalrymple about:* His perspectives on AI risk* ARIA (the UK’s Advanced Research and Invention Agency) and its Safeguarded AI ProgrammeEnjoy—and let me know what you think!Davidad is a Programme Director at ARIA. He was most recently a Research Fellow in technical AI safety at Oxford. He co-invented the top-40 cryptocurrency Filecoin, led an international neuroscience collaboration, and was a senior software engineer at Twitter and multiple startups.Find me on Twitter for updates on new episodes, and reach me at [email protected] for feedback, ideas, guest suggestions. Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on TwitterOutline:*

Intro*

Calibration and optimism about breakthroughs*

Calibration and AGI timelines, effects of AGI on humanity*

Davidad’s thoughts on the Orthogonality Thesis*

Understanding how our current direction relates to AGI and breakthroughs*

What Davidad thinks is needed for AGI*

Extracting knowledge*

Cyber-physical systems and modeling frameworks*

Continuities between Davidad’s earlier work and ARIA*

Path dependence in technology, race dynamics*

More on Davidad’s perspective on what might go wrong with AGI*

Vulnerable world, interconnectedness of computers and control*

Formal verification and world modeling, Open Agency Architecture*

The Semantic Sufficiency Hypothesis*

Challenges for modeling*

The Deontic Sufficiency Hypothesis and mathematical formalization*

Oversimplification and quantitative knowledge*

Collective deliberation in expressing values for AI*

ARIA’s Safeguarded AI Programme*

Anthropic’s ASL levels*

Guaranteed Safe AI — *

AI risk and (in)accurate world models*

Levels of safety specifications for world models and verifiers — steps to achieve high safety*

Davidad’s portfolio research approach and funding at ARIA*

Earlier concerns about ARIA — Davidad’s perspective*

Where to find more information on ARIA and the Safeguarded AI Programme*