Show notes
📌 Copy the AI system I use to run a one-person $1M+ business: https://www.behindthecraft.com🤝 Subscribe to my practical AI newsletter read by 150K+ professionals: https://creatoreconomy.soShreya and Hamel teach the industry standard course for AI evaluations taken by 4,500+ students from OpenAI, Google, and more. In our interview, I asked them to give live feedback on the AI evals that I built for my skills and demo how anyone can run evals with ChatGPT/Claude and their free evals skill. Don’t miss this up to date primer on evals from the real experts. 🙂P.S., I asked Shreya and Hamel to give my audience 25% off the next cohort of their AI evals course. It has 4.7 stars with 900 reviews which is almost unheard of on Maven. Sign up via the link here: https://maven.com/parlance-labs/evals?utm_campaign=peteryang&utm_medium=affiliate&utm_source=maven&promoCode=PETERYANGWe talked about:(00:00) Why AI evals are completely different now with the latest models(02:30) Demo: Live audit of the evals that I built for my AI skills(05:14) Top down vs. bottom up evals and why AI sucks at the latter(08:16) Spinning up AI agents to grade each eval criteria(14:32) Demo: Using Shreyas AI skill to run evals in Claude(23:46) How to review output labels to identify failures(34:37) How to turn failures into reusable evals(43:19) Where human judgement is still neededResources mentioned:Free eval skills from Shreya: https://github.com/ai-evals-course/evals-skillsDo automated evals work?: https://parlance-labs.com/blog/posts/auto-evals/Thanks to our sponsors:Wispr Flow — 4x faster than typing with your voice — https://ref.wisprflow.ai/peteryang📌 Get the takeaways: https://creatoreconomy.so/p/how-to-build-better-ai-evals-with-claude-code-shreya-hamelWhere to find Shreya and Hamel:Shreya: https://www.sh-reya.com/Hamel: https://hamel.dev/Subscribe to this channel - more interviews coming soon!