The GeekNarrator
The GeekNarrator
Kaivalya Apte
Diving into Kafka Internals with David Jacot
1 hour 10 minutes Posted Aug 19, 2023 at 6:47 am.
Kafka Internals with David Jacot
Defining Kafka
Kafka Architecture(s)
Write Path Producer sending data
How does replication work?
How do we track replication progress?
Failure Modes: Leader fails
Consumers: Push vs Pull
Consumers: How does fetch works?
Consuming number of bytes vs records
Optimising consumption
Offset management and choosing partitions
Ending notes
0:00
1:10:41
Download MP3
Show notes
In this video I talk to David Jacot who works as a Staff Software Engineer at  @Confluent  and has been a long time Kafka user, committer and PMC member. We covered how Kafka works internally in great depth.
We use Kafka for various use cases and it works great, but going one level below the abstraction and truly understanding the protocols, techniques and algorithms used is a fun ride.
Chapters:
I hope you like this episode and more importantly you learnt some amazing techniques Kafka uses to ensure durability, low latency, simplicity and scalability in its architecture.
Do give this episode a like and share it with your network. Also please subscribe to the channel for content like this.
Other playlists:
Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-
Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17
Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d
Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN
Other episodes:
KsqlDB: https://youtu.be/2yE86P6uD_0
Exactly once semantics: https://youtu.be/twgbAL_EaQw
David's Linkedin: https://www.linkedin.com/in/davidjacot/
our website: www.geeknarrator.com
Cheers,
The GeekNarrator