The GeekNarrator
The GeekNarrator
Kaivalya Apte
Scaling Derived Data for Planet-Scale Applications at Linkedin
1 hour 12 minutes Posted Jun 5, 2024 at 5:03 am.
Introduction
The Evolution of LinkedIn's Databases
Challenges with Voldemort and the Birth of VeniceDB
Understanding Derived Data
Planet-Scale Applications and Multi-Region Support
Writing Data into VeniceDB
Merging Data in VeniceDB
Understanding the Architecture
Components of the Write Path
Leader and Follower Architecture
Partitioning and DaVinci Client
Read Patterns and Client Options
Fault Tolerance and Recommender Systems
Kafka Integration and Deployment
Roadmap and Future Improvements
0:00
1:12:42
Download MP3
Show notes
In this video I speak with Felix GV, who is a Principal Staff Engineer at Linkedin, and has done major contributions to the data infrastructure and Linkedin, including VeniceDB.
This episode will give you a good understanding of why we need a new database for storing "Derived Data" in a low latency, high performance manner, which is very important for Machine Learning workloads.
Chapters:
Important links:
VeniceDB blog: https://www.linkedin.com/blog/engineering/open-source/open-sourcing-venice-linkedin-s-derived-data-platform
VeniceDB docs: https://venicedb.org/
Qcon: https://youtu.be/pJeg4V3JgYo?si=vblGUxp5fNdKPHoC
Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarrator
If you like this episode, please hit the like button and share it with your network.
Also please subscribe if you haven't yet.
Database internals series: https://youtu.be/yV_Zp0Mi3xs
Popular playlists:
Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-
Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17
Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d
Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN
Stay Curios! Keep Learning!
#kafka #linkedin #venicedb #Rocksdb