The GeekNarrator
The GeekNarrator
Kaivalya Apte
How would you design a database on Object Storage?
1 hour 8 minutes Posted Dec 2, 2024 at 8:02 am.
Introduction
Simon's Background and Journey to TurboBuffer
Challenges in Database Scalability
Experimenting with Vector Databases
Cost Implications of Vector Databases
Architectural Considerations for Search Workloads
Building a Database on Object Storage
Designing a Simple Database on Object Storage
Handling Multiple Writers and Consistency
Trade-offs in Write Operations
Optimizing MySQL Write Performance
Batching Writes in Object Storage
Time-Based vs Size-Based Batching
Understanding Amplification in Databases
Challenges with Cold Queries
Building and Persisting B-Trees
Separating Workloads in Databases
Multi-Tenancy Challenges
Choosing Storage Formats
Key Innovations in Object Storage Databases
with me.
0:00
1:08:26
Download MP3
Show notes
Join Kaivalya Apte and Simon Hørup Eskildsen from Turbopuffer as they talk about the complexities of building a database on top of object storage. Discover the key challenges, the nuances of various storage formats, and the critical trade-offs involved.
Learn from Simon's rich experience, from his time at Shopify to creating Turbopuffer. This episode covers everything—from approaches to write-ahead logs to multi-tenancy and object storage advancements. Perfect for database enthusiasts and those keen on first-principles thinking!
Important links:
- https://github.com/sirupsen/napkin-math (numbers)
- https://turbopuffer.com/
- https://turbopuffer.com/architecture
- https://sirupsen.com/napkin/problem-10-mysql-transactions-per-second
- https://sirupsen.com (my blog, napkin math)
- https://sirupsen.com/subscribe (napkin math newsletter)
- https://github.com/rkyv/rkyv rkyv rust
Become a member of The GeekNarrator to get access to member only videos, notes and monthly
Like building stuff? Try out CodeCrafters and build amazing real world systems like Redis, Kafka, Sqlite. Use the link below to signup and get 40% off on paid subscription.
https://app.codecrafters.io/join?via=geeknarrator
If you like this episode, please hit the like button and share it with your network.
Also please subscribe if you haven't yet.
Database internals series: https://youtu.be/yV_Zp0Mi3xs
Popular playlists:
Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA-
Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17
Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d
Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN
Stay Curios! Keep Learning!