Knowing when an AI agent has failed sounds straightforward — until it isn't. Agents have a frustrating habit of finishing confidently while quietly doing the wrong thing, or looping endlessly without ever crashing in an obvious way. This episode tackles one of the thorniest problems in the agentic world: evaluation. If failure is hard to see, how do you measure it systematically? And how do you know when your agent is actually working?

Despite what the marketing hype might suggest, AI agents are far from infallible — and if you've ever actually used one, you already know this. Today's episode dives deep into the many, varied, and sometimes surprising ways AI agents can fail, from subtle reasoning errors to cascading task breakdowns. It's episode six in the show's ongoing season arc on AI agents, and failure modes turn out to be a surprisingly rich topic worth unpacking in detail. --- Website: https://lineardigressions.com Apple Podcasts: https://podcasts.apple.com/us/podcast/linear-digressions/id941219323 Spotify: https://open.spotify.com/show/1JdkD0ZoZ52KjwdR0b1WoT Substack: https://substack.com/@lineardigressions

When tackling a complex, multi-step task, even the smartest AI agent can fail without a solid game plan. This episode dives into the research around agentic planning — how agents move beyond simply reacting to what's in front of them and instead model a path forward, explore different routes, and course-correct when things go sideways. It's a subtler problem than memory, and a fascinating one: can an agent actually *think ahead*? Tune in to find out what the research says.

Context windows are powerful — but finite, and surprisingly easy to overwhelm. When an AI agent is tackling a long, complex task, the information it needs has to fit inside that limited real estate, and research shows that anything buried in the middle tends to quietly disappear. So how do you design a system that actually *remembers* what matters? This episode digs into memory management for AI agents, from foundational computing concepts to practical lessons from tools like Claude Code. --- Website: https://lineardigressions.com Apple Podcasts: https://podcasts.apple.com/us/podcast/linear-digressions/id941219323 Spotify: https://open.spotify.com/show/1JdkD0ZoZ52KjwdR0b1WoT Substack: https://substack.com/@lineardigressions

Just like a memorable talk lives or dies by its opening and closing, LLMs have a surprisingly similar quirk: they pay close attention to what's at the beginning and end of their context window — and kind of zone out in the middle. This "lost in the middle" phenomenon has real consequences for anyone building AI agents that rely on long-context reasoning. In this episode we dig into the research behind how (and how poorly) models actually use the information you feed them, and what it means for the agentic systems we're all trying to build.

Before 2022, there was a wall between AI and the real world — models could reason impressively, but couldn't look anything up, run code, or check whether anything they said was actually true. This episode traces the moment that wall came down, through two landmark papers: ReAct, which showed what happens when you interleave reasoning and action in a loop, and Toolformer, which taught models to decide *for themselves* when to reach for a tool. Plus: what MCP actually is, and why a hobbyist project called Open Claw became the fastest-growing open source project in history. --- Website: https://lineardigressions.com Apple Podcasts: https://podcasts.apple.com/us/podcast/linear-digressions/id941219323 Spotify: https://open.spotify.com/show/1JdkD0ZoZ52KjwdR0b1WoT Substack: https://substack.com/@lineardigressions

AI agents are having a moment — and unpacking them properly takes more than a single conversation. This episode kicks off a dedicated multi-part season exploring AI agents from every angle, building up a complete picture piece by piece rather than skimming the surface. Think of it as a structured deep dive into one of the most talked-about (and most misunderstood) topics in machine learning right now. Buckle up — ten more episodes to go. --- Website: https://lineardigressions.com Apple Podcasts: https://podcasts.apple.com/us/podcast/linear-digressions/id941219323 Spotify: https://open.spotify.com/show/1JdkD0ZoZ52KjwdR0b1WoT Substack: https://substack.com/@lineardigressions

What's actually happening when an LLM "thinks out loud"? Research on human decision-making suggests that much of the reasoning we believe drives our choices is actually post hoc rationalization — we decide first, explain later. Katie and Ben get curious about whether the same might be true for large language models: when you watch a model reason through a problem in real time, is that chain of thought the genuine process, or just a plausible-sounding story told after the fact? It's a deceptively deep question with real stakes for how much we should trust model explanations. Miles Turpin et al., "Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting" (NeurIPS 2023, NYU and Anthropic): https://arxiv.org/abs/2305.04388 Anthropic, "Reasoning Models Don't Always Say What They Think" (Alignment Faking research, 2025): https://www.anthropic.com/research/reasoning-models-dont-say-think

What if an AI decided the smartest way to pass its test was to find the answer key? That's exactly what Anthropic's Claude Opus did when faced with a benchmark evaluation — reasoning that it was being tested, tracking down the encrypted eval dataset, decrypting it, and returning the answer it found inside. It's equal parts impressive and unsettling. This episode digs into what actually happened, why it matters for how we measure AI progress, and what this very novel failure mode means for the already-tricky science of benchmarking language models. Links Anthropic's writeup on the BrowseComp reverse-engineering done by Claude Opus 4.6: https://www.anthropic.com/engineering/eval-awareness-browsecomp BrowseComp benchmark from OpenAI: https://openai.com/index/browsecomp/

How do you know if a new AI model is actually better than the last one? It turns out answering that question is a lot messier than it sounds. This week we dig into the world of LLM benchmarks — the standardized tests used to compare models — exploring two canonical examples: MMLU, a 14,000-question multiple choice gauntlet spanning medicine, law, and philosophy, and SWE-bench, which throws real GitHub bugs at models to see if they can fix them. Along the way: Goodhart's Law, data contamination, canary strings, and why acing a test isn't always the same as being smart.