In this episode 600 of the TestGuild Automation Podcast, Joe Colantonio talks with Jason Arbon, founder of Testers.ai, Jank.AI and IcebergQA and author of the new book Testing AI: Engineering Confidence in Non-Deterministic Systems. Take Our 2027 Survey Now: https://testgld.link/27data Jason makes a case most testers have not heard yet. Coding is being absorbed by AI. Specification work is thinning out. Product, development, and test roles are converging into one. And when the music stops, the only seat left belongs to the person who can look at what the machine produced and make an evidence backed call on whether it ships. He calls that confidence engineering, and he argues it is not a rebrand of QA. It is what QA was always supposed to be. Along the way, Joe and Jason get into the containment problem and why alignment, not lockdown, is now the real safety goal. They dig into why testing cost scales quadratically, meaning ten times more generated code creates roughly a hundred times more testing demand. Jason also pushes back hard on skeptics of agentic testing, pointing out that almost nobody has run the obvious experiment of testing a site themselves for a week and comparing their results against what AI finds. You will also hear Jason's most practical piece of advice in the whole conversation. If you are not running the same suite five times against the same build and looking at the actual results, not just flake, you are not testing seriously in an AI world. Plus a detour into grokking, the Chinese Room, and Geoffrey Hinton, because it would not be a Jason Arbon episode without one. Listen up!

Your developers just got supercharged by AI coding agents. Your test coverage did not. So how do you keep quality high when product code is shipping faster than any test team can follow? In this episode Andrew Knight, the Automation Panda and Senior Director of Product and Engineering at Cycle Labs, shares how his six person team is running the biggest release quarter in company history using AI coding agents, spec driven development, and Playwright. You will discover: How Playwright turned itself into an AI automation platform with the MCP server, the planner, generator, and healer agents, and the new CLI and skills approach that cuts your token usage way down. Why Andy uses Spec Kit to codify his testing strategy once, in markdown, so quality standards get baked into every single pull request instead of being caught in review. How to decide which AI generated tests are actually worth running when compute time and budget are finite. What AI slop looks like from a manager's seat, and how to build a team culture that catches it before it ships. Why Andy believes AI coding tools are the new compiler and markdown is the new programming language, plus my pushback on what that means for everything testers were trained to care about. What Andy really thinks about token costs, subscription tiers, and what happens when the AI subsidies run out. Plus the one piece of advice he gives to any tester still sitting on the sidelines of the AI shift. Whether you are an automation engineer, a QA lead, or an engineering manager trying to figure out where testing fits in an AI first workflow, this episode gives you a practical playbook you can start using this week.

Keith Klain has spent 25+ years leading enterprise quality programs in financial services and is one of the testing industry's most respected voices. In this episode, he joins Joe to discuss his BrowserStack talk, "The Death of Test Engineering," why testing isn't dying—but outdated testing roles are—and how AI is reshaping the future of software quality. In this episode: Why "The Death of Test Engineering" isn't what you think The difference between checking and testing Will AI replace software testers? Why confidence engineering matters How to communicate testing in business terms Where automation adds real value Why agentic AI testing is overhyped Notable Quotes "If you look at AI as a competitor, you're screwed. If you look at it as just another tool... it's not a threat." "Uncertainty in testing is a feature, not a bug." "If an organization doesn't value testing, you're doing it wrong." "You can be incredibly confident about something and incredibly wrong." One Thing to Try This Week Ask yourself after every critical defect: "What would the business impact have been if this had gone live?" It's a simple way to shift conversations from test cases to business risk. Resources Mentioned Rethinking Expertise — Harry Collins How Complex Systems Fail — Richard Cook Measuring and Managing Performance in Organizations — Robert Austin Emily Bender Alex Hanna Timnit Gebru Dagmar Monett Olivia Gambelin Ed Zitron Joe's Recommendation: Prophecy

Nandini Srinivasan has spent 25 years in the quality industry and leads a global QA organization of over 150 engineers across the US, Canada, India, and Pakistan. In this episode, she breaks down exactly how she built a QA AI acceleration charter, ran a train-the-trainer model, and used a phased proof-of-concept approach to separate real AI from what she calls "powerful automation dressed up as AI.  We get into: How she frames quality metrics for executives Using language around revenue protection Risk mitigation Feature velocity instead of test coverage percentages. She talks about the four pillars she uses to present her team's value: quality, scalability, performance, and availability. Nandini also shares her take on the future of QA hiring, why the judgment layer will always require a human, and what skills testers need to stay relevant as AI agents take over the more mechanical parts of automation. She is also writing a five-part LinkedIn series called "The Voice of QA in the AI Era" if you want to follow along.

Matt Wynne, co-creator of Cucumber and BDD practitioner, joins Joe for the first time in over a decade to talk about what two years inside a Silicon Valley AI startup taught him about the future of software testing. Matt spent time at Mechanical Orchard working alongside experienced XP practitioners to modernize legacy COBOL mainframes using LLMs, and then spent a week with the team that coined the term "software factory," where the rule was simple: humans never write the code, never read the code. In this episode, Matt breaks down what harness engineering actually means, why shared understanding is still the real bottleneck even in an agentic world, and how testers can use multiple LLMs to review AI-generated pull requests without reading every line. He also gets honest about the grief that comes with realizing you can encode years of hard-won expertise into a Markdown file, and why that does not mean your skills are worthless. If you are working in a brownfield codebase, wondering how to handle the flood of agentic PRs, or trying to figure out where testers fit in a world where agents write the code, this conversation is worth your time. Find Matt at: mattwynne.net leansoftware.ai Also check out his course: Build a Software Factory: Hands-off agentic coding for experienced engineers https://testgld.link/mattcourse

Amit Rawat is an agentic engineer who spent two decades in QA before shifting fully into building AI agents. He's the creator of PromptWright, a desktop tool that turns natural language prompts into automated Playwright browser tests, complete with screen recording, Gherkin scenario generation, and self-healing locators. In this episode, Amit and Joe get into what it actually takes to work with AI agents at a high level, starting with why the planning phase matters more than the prompt itself. Amit breaks down his own workflow, brainstorming with AI, building a detailed plan in HTML before ever executing, and why curiosity and technical depth still matter even as AI gets more capable. They also cover why Amit believes QA professionals, more than developers or DevOps engineers, are best positioned to thrive in the agentic era, how he tracks the ROI on his $200-a-month Claude subscription, the "Chief of Staff," "Chief Health Officer," and "Chief Financial Officer" AI agents he's built to help manage different aspects of his personal life, and how he uses a memory layer so those agents understand his preferences and become more useful over time. If you're a tester, automation engineer, or QA leader trying to figure out where AI agents fit into your workflow and your career, this conversation is a practical look at what's already working today.

Most API testing stops at the happy path. The problem is that the bugs that actually hurt you in production are sitting in everything many testers skip, like the boundary values, the oversized payloads, the missing tokens, the security headers, the inputs that make no sense at all. In this episode, Joe sits down with Liudas Jankauskas, who has spent almost twenty years breaking software and testing APIs since 2008. Liudas demonstrates Rentgen, his free and open-source API testing tool, live on screen. You'll watch him take a single request from a real app, map it in seconds, and generate dozens of tests covering security, boundaries, performance, and load—all from one click. You'll learn: How to discover APIs hiding under the hood of any application, even when there is zero documentation Why happy path testing leaves you exposed How to run a fast hygiene check before your real automation ever starts Liudas also explains why Rentgen runs completely locally with no server and no data leaving your machine, making it safe for banking, healthcare, and other regulated environments. Plus, he demonstrates the killer Copy Bug Report feature that drops a standards-based ticket straight into Jira or Trello. In This Episode You'll Discover How to find and test undocumented internal APIs using the browser DevTools Network tab Why happy path-only testing misses the bugs that matter most How Rentgen turns one request into security, boundary, performance, and load tests automatically Where Rentgen fits in your workflow as a pre-automation hygiene layer—not a Postman replacement How to use it for regression by comparing results across environments The one piece of advice Liudas gives every tester to level up their API testing Try Rentgen, free and open source, at Rentgen.io. Connect with Liudas Jankauskas on LinkedIn: https://www.linkedin.com/in/liudas-jankauskas/

Your AI code review tools read the diff. They stare at your code. But they never actually run it. So the bugs that only show up at runtime, the broken user flows, the bad query plan, the duplicate submission, sail right past review and land in front of your customers. In this episode, Joe Colantonio sits down with Evan Marshall, founder of Ito and a fifteen year engineer who spent five years in applied cryptography securing hundreds of millions of dollars for millions of people. Evan is taking that ship fast without breaking things discipline and pointing it straight at testing. Ito is an agentic QA platform that builds and runs your actual app on every pull request, navigates it like a real user, exercises the frontend and backend as one system, and brings back real runtime evidence: video replays, logs, the exact lines responsible, and steps to reproduce, posted right in your PR. You will learn: Why static code review misses the bugs that cause real production incidents How Ito spins up ephemeral environments and tests across UI, API, and database Why QA is not disappearing, it is leveling up into a manager and quality strategist role How to keep your test layer separate from your code generation so your signal stays honest The skills testers and engineers need as AI writes more of the code If you are shipping AI generated code at high velocity and your QA cannot keep up, this one is for you. Try Ito on your own code. Your first ten pull requests are reviewed free, no credit card required. Check it out at https://testgld.link/itoai now. And as Joe always says, seeing is believing.

What happens to QA when AI is writing ten times more code than your team can test? That is the exact problem Ivan Barajas Vargas set out to solve with Amikoo, a purpose-built AI QA agent designed to help testers, SDETs, and even developers move faster without sacrificing coverage or quality. Ivan is no stranger to AI in testing. Before generative AI became mainstream, he co-founded MuukTest, a test automation platform built on symbolic reasoning and expert systems. After six years and thousands of customer conversations, he went back to first principles to build Amikoo from scratch, this time with a harness of 12 specialized agents and 43 tools trained specifically for testing workflows. In this episode, Ivan and Joe dig into the real-world gap between AI code generation and AI-powered testing, why the QA role is being elevated rather than replaced, how Amikoo uses Playwright and page object model patterns under the hood, and where human judgment still has to stay in the loop. Ivan also shares practical advice on what skills QA engineers should be building right now and which test scenarios should never be fully delegated to an agent. If you are trying to figure out where testing fits in an agentic development world, this episode gives you a clear picture of what is possible today and what is coming next. Visit https://testgld.link/amikoo to try the freemium account, and mention you heard this on TestGuild to unlock double the free usage.

Everyone is talking about AI replacing testers, writing tests, and transforming software quality. But what if we're asking the wrong question? In this solo episode, Joe Colantonio shares a growing concern he's seen while traveling across the country for TestGuild IRL events: a decline in testing fundamentals at the exact moment AI hype is reaching a fever pitch. Drawing insights from Carissa Véliz's book Prophecy: Prediction, Power, and the Fight for the Future, Wayne Roseberry's work on AI and meaning, and Tariq King's concept of Human Experience Testing, Joe explores why AI systems may be far less intelligent than many believe, and why human testers remain more important than ever. You'll discover: ✅ Why large language models generate plausible answers without understanding truth ✅ The difference between prediction, correlation, and genuine understanding ✅ Why AI can test software but cannot experience software ✅ What "Everything is tested, but nothing is experienced" really means ✅ How AI hype may be distracting teams from critical testing fundamentals ✅ Why empathy, context, and human judgment are becoming competitive advantages for testers Whether you're excited about AI, skeptical of it, or somewhere in between, this episode will challenge you to think more deeply about the future of testing and your role in it. Resources Mentioned 📖 Prophecy: Prediction, Power, and the Fight for the Future by Carissa Véliz 📖 Work and presentations by Wayne Roseberry 🎓 Free course: thebullshitmachines.com 🎤 Learn more about TestGuild IRL events: TestGuild.com/irl If you enjoy this episode, be sure to subscribe, leave a review, and share it with a fellow tester who's trying to navigate the AI era without losing sight of the fundamentals. #SoftwareTesting #AI #QualityEngineering #TestAutomation #SoftwareQuality #HumanExperienceTesting #ArtificialIntelligence #TestGuild #QA #TechPodcast