We talk with Kai Wern Choong about his experience with tracing performance problems in production. We talk about what “tracing” means in a BEAM system, available tools like recon, great resources for learning how the tools work and general approaches for troubleshooting live production Elixir systems. Kai also shares his Livebook notebook setup for demonstrating tracing techniques in an interactive way.Show Notes online - http://podcast.thinkingelixir.com/66Elixir Community Newshttps://twitter.com/hexpm/status/1439268469296570368 – New version of Hex released that fixes issues with version downgrades and slow version resolutions. Run mix local.hex to upgrade.https://hexdocs.pm/credo/1.6.0-rc.0/changelog.html#1-6-0 – Credo 1.6.0 was released with new "First Run Mode"https://twitter.com/elixirphoenix/status/1438222537301843970 – Removing Phoenix compiler config change can speed up compilation times when on up-to-date versionshttps://github.com/Qqwy/elixir-type_check – New library called TypeCheckhttps://blog.oestrich.org/2021/09/introducing-aino/ – Eric Osterich is experimenting with an alternative HTTP framework for Elixirhttps://github.com/oestrich/aino – The Aino projecthttps://techcrunch.com/2021/09/14/logistics-startup-stord-raises-90m-in-kleiner-perkins-led-round-becomes-a-unicorn-and-acquires-another-company/ – Stord, an Elixir company, raised $90M and became a "unicorn"https://spawnfest.org/ – SpawnFest 2021 happenedhttps://github.com/spawnfest/eflambe – SpawnFest submission - rapid generation of flamegraphshttps://github.com/spawnfest/Sketch – SpawnFest submission - library for creating generative art with Elixirhttps://github.com/spawnfest/lotus – SpawnFest submission - Lotus is a Surface UI wrapper for UIKithttps://github.com/spawnfest/Discovery – SpawnFest submission - Platform for hosting realtime, stateful servers with zero downtime deployment and horizontal scaling on Kuberneteshttps://github.com/spawnfest/crash – SpawnFest submission - Crash is a simple (distributed) docker-on-docker Continuous Integration system written in ElixirDo you have some Elixir news to share? Tell us at @ThinkingElixir or email at [email protected]Discussion Resourceshttps://kaiwern.com/posts/2021/06/27/debugging-with-tracing-in-elixir-with-recon_trace/https://www.naluri.life/https://ferd.github.io/recon/recon.htmlhttps://hex.pm/packages/reconhttps://twitter.com/kw7oe/status/1409126300636254219https://github.com/kw7oe/livebook-notebooks/blob/main/debugging-with-tracing-in-elixir-with-recon_trace.livemd – Livebook notebook showing tracing exampleshttps://www.youtube.com/watch?v=sR9h3DZAA74 – Debugging Live Systems on the BEAM talk by Jeffery Utterhttps://www.youtube.com/watch?v=OR2Gc6_Le2U – Operable Erlang and Elixir talk by Fred Hebert (skip to https://kaiwern.com/posts/2020/11/02/debugging-with-tracing-in-elixir/ – Tracing with dbg moduleshttps://en.wikipedia.org/wiki/Tracing_(software)https://erlang.org/doc/man/dbg.html – dbg module in Erlang:dbg.fun2mshttps://kubernetes.io/blog/2017/12/using-ebpf-in-kubernetes/https://github.com/cilium/ciliumhttps://opentelemetry.io/docs/erlang/https://www.brendangregg.com/blog/2019-01-01/learn-ebpf-tracing.htmlhttps://www.erlang-solutions.com/capabilities/wombatoam/https://hex.pm/packages/recon_exhttps://erlang-in-anger.com/ – Erlang in Anger (Chapter 9)https://www.thegreatcodeadventure.com/testing-genservers-with-erlang-trace/ – Testing GenServers with Erlang Tracehttp://blog.plataformatec.com.br/2016/04/how-to-trace-elixir-nodes-with-erlyberly/ – How to trace Elixir nodes with Erlyberly - Plataformatec Bloghttp://erlang.org/doc/man/seq_trace.html#sequential-tracing – Seq_tracehttp://erlang.org/doc/man/erlang.html#trace-3 – Erlang:tracehttps://github.com/andytill/erlyberly – Erlang tracing for the masseshttps://github.com/tatsuya6502/recon_ex – Elixir wrapper for Recon, tools to diagnose Erlang VM safely in productionGuest Informationhttps://twitter.com/kw7oe – on Twitterhttps://github.com/kw7oe/ – on Githubhttps://kaiwern.com/ – BlogFind us onlineMessage the show - @ThinkingElixirEmail the show - [email protected]Mark Ericksen - @brainlidDavid Bernheisel - @bernheiselCade Ward - @cadebward