In this video I speak with Felix GV, who is a Principal Staff Engineer at Linkedin, and has done major contributions to the data infrastructure and Linkedin, including VeniceDB. This episode will give you a good understanding of why we need a new database for storing "Derived Data" in a low latency, high performance manner, which is very important for Machine Learning workloads. Chapters: 00:00 Introduction 01:42 The Evolution of LinkedIn's Databases 03:15 Challenges with Voldemort and the Birth of VeniceDB 08:42 Understanding Derived Data 13:33 Planet-Scale Applications and Multi-Region Support 17:40 Writing Data into VeniceDB 22:53 Merging Data in VeniceDB 40:31 Understanding the Architecture 40:47 Components of the Write Path 41:56 Leader and Follower Architecture 43:58 Partitioning and DaVinci Client 47:57 Read Patterns and Client Options 54:25 Fault Tolerance and Recommender Systems 01:01:19 Kafka Integration and Deployment 01:06:56 Roadmap and Future Improvements Important links: VeniceDB blog: https://www.linkedin.com/blog/engineering/open-source/open-sourcing-venice-linkedin-s-derived-data-platform VeniceDB docs: https://venicedb.org/ Qcon: https://youtu.be/pJeg4V3JgYo?si=vblGUxp5fNdKPHoC Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarrator If you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet. Database internals series: https://youtu.be/yV_Zp0Mi3xs Popular playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA- Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17 Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN Stay Curios! Keep Learning! #kafka #linkedin #venicedb #Rocksdb

In this video I speak with Philippe Noël, about ParadeDB, which is an Elasticsearch alternative built on Postgres, modernizing the features of Elasticsearch's product suite, starting with real-time search and analytics. I hope you will enjoy and learn about the product. Chapters: 00:00 Introduction 01:12 Challenges with Elasticsearch and the Need for ParadeDB 02:29 Why Postgres? 06:30 Technical Details of ParadeDB's Search Functionality 18:25 Analytics Capabilities of ParadeDB 24:00 Understanding ParadeDB Queries and Transactions 24:22 Application Logic and Data Workflows 25:14 Using PG Cron for Data Migration 30:05 Scaling Reads and Writes in Postgres 31:53 High Availability and Distributed Systems 34:31 Isolation of Workloads 39:38 Database Upgrades and Migrations 41:21 Using ParadeDB Extensions and Distributions 43:02 Observability and Monitoring 44:42 Upcoming Features and Roadmap 46:34 Final Thoughts Important links: Links: GitHub: https://github.com/paradedb/paradedb Website: https://paradedb.com Docs: https://docs.paradedb.com/ Blog: https://blog.paradedb.com Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarrator If you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet. Database internals series: https://youtu.be/yV_Zp0Mi3xs Popular playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA- Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17 Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN Stay Curios! Keep Learning! #postgresql #datafusion #parquet #sql #OLAP #apachearrow #database #systemdesign #elasticsearch

In this video I speak with Andrew Lamb, Staff Software Engineer @Influxdb. We discuss FDAP (Flight, DataFusion, Arrow, Parquet) stack for modern OLAP database system design. Andrew shared some insights into why the FDAP stack is so powerful in designing and implementing a modern OLAP database. Chapters: 00:00 Introduction 01:48 Understanding Analytics: Transactional vs Analytical Databases 04:41 The Genesis and Goals of the FDAP Stack 09:31 Decoding FDAP: Flight, Data Fusion, Arrow, and Parquet 12:40 Apache Parquet: Revolutionizing Columnar Storage 17:18 Apache Arrow: The In-Memory Game Changer 23:51 Interoperability and Migration with Apache Arrow 27:10 Comparing Apache Parquet and Arrow 28:26 Exploring Data Mutability in Analytic Systems 29:19 Handling Data Updates and Deletions 29:24 The Role of Immutable Storage in Analytics 30:42 Optimizing Data Storage and Mutation Strategies 34:20 Introducing Flight: Simplifying Data Transfer 35:02 Deep Dive into Flight's Benefits and SQL Support 39:20 Unpacking Data Fusion's SQL Support and Extensibility 46:12 The Interplay of FDAP Components in Analytics 51:49 Future Directions and Innovations in Data Analytics 56:04 Concluding Thoughts on FDAP and Its Impact FDAP Stack: https://www.influxdata.com/glossary/fdap-stack/ FDAP Blog: https://www.influxdata.com/blog/flight-datafusion-arrow-parquet-fdap-architecture-influxdb/ InfluxDB: https://www.influxdata.com/ Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarrator If you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet. Database internals series: https://youtu.be/yV_Zp0Mi3xs Popular playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA- Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17 Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN Stay Curios! Keep Learning! #datafusion #parquet #sql #OLAP #apachearrow #database #systemdesign

In this video I and Pratim Bhosale, Developer Advocate at SurrealDB, talk about SurrealDB, a multi-model database which aims to make Developer’s life easier by letting them focus mainly on the business logic and not on the Database choice. Following chapters will help you understand what is a multi-model database and how SurrealDB shines. Chapters: 00:00 Introduction 01:48 The Genesis of SurrealDB 03:59 SurrealDB's Mission and Use Cases 07:34 Understanding Multi-Model Databases 10:30 Deep Dive into SurrealDB's Architecture 33:09 Deployment and Getting Started with SurrealDB 34:31 Future Developments and Use Case Considerations 43:51 Final Thoughts and How to Get Started Important links: Install SurrealDB https://sdb.li/4bqwn38 SurrealDB Docs: https://sdb.li/3wxjoxx SurrealDB Website: https://sdb.li/3JMK7JI Surrealist: https://sdb.li/4b7wcdh SurrealDB GitHub: https://sdb.li/3JRPNlE Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarrator If you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet. Database internals series: https://youtu.be/yV_Zp0Mi3xs Popular playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA- Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17 Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN Stay Curios! Keep Learning! #surrealdb #elasticsearch #search #vectorsearch #acid #databases #sql #joins #indexes #graphdatabase

In this video, I talk to Dhruba, CTO @Rockset about search and realtime analytics. We discussed deep internals of Rockset, its architecture and why is it a great fit for search and realtime analytics use cases. Chapters: 00:00 Introduction 02:45 The Evolution of Data Systems: From Hadoop to Rockset 07:30 Understanding Rockset: Real-Time Analytics and Search Defined 12:01 The Technical Edge: Rockset vs. Elasticsearch 18:16 Deep Dive into Rockset's Architecture and Internals 28:21 Partitioning, Hashing, and Data Distribution in Rockset 36:56 Exploring Hot Storage and Cache Layers 37:40 Why Hot Storage is Essential for Low Latency 39:05 Optimizing Data Storage with Compression and Delta Encoding 39:49 Balancing Cost and Performance in Data Storage 41:50 The Power of Converged Indexing in Rockset 45:50 Efficient Query Execution and Index Management 54:51 Leveraging Mutability for Real-Time Analytics 59:24 Deep Dive into Query Processing and Optimization 01:04:21 Understanding Joins and Reporting Queries in Rockset 01:12:23 Future Directions and Vector Search Innovations Index Conference: https://rockset.com/index-conf/ Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarrator If you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet. Database internals series: https://youtu.be/yV_Zp0Mi3xs Popular playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA- Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17 Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN Stay Curios! Keep Learning! #rockset #elasticsearch #search #vectorsearch #realtime #databases #sql #joins #indexes

In this episode we explore how to Rapidly Simulate Production Traffic with Michael Drogalis, using his creation ShadowTraffic. I am sure you will be able to relate to all the different problems mentioned in this episode and like how ShadowTraffic aims to solve those problems. I hope you like this conversation. Chapters: 00:00 Welcome to The Geek Narrator Podcast: Exploring Deep Tech 00:18 The Challenge of Simulating Production Traffic 00:59 Introducing Shadow Traffic: A Solution to Data Simulation 02:34 Understanding the Problem Space of Data Simulation 06:03 How Shadow Traffic Works: A Deep Dive 08:17 The Power of Declarative Data Generation with Shadow Traffic 10:40 Shadow Traffic's Architecture and Deployment 13:02 Configuring Load Testing and Throttling with Shadow Traffic 15:47 Testing and Validation in Shadow Traffic 20:42 Mimicking Production Data Distribution with Shadow Traffic 26:48 Innovative Features for Stream Processing Testing 28:47 Shadow Traffic: Adding Faults to Data for Robust Testing 29:04 Antithesis and Shadow Traffic: A Synergistic Approach 32:46 The Challenge of Generating Realistic Test Data 40:04 Enhancing Observability in Data Generation 41:50 Customer-Driven Roadmap and Future Vision 45:27 Closing Thoughts ShadowTraffic: https://shadowtraffic.io/ Contact Michael: https://shadowtraffic.io/contact.html Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarrator If you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet. Database internals series: https://youtu.be/yV_Zp0Mi3xs Popular playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA- Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17 Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN Stay Curios! Keep Learning! #kafka #s3 #postgres #testing #streamprocessing #loadtesting #chaostesting #demo

If you're involved in the Java space, chances are you've come across #GraalVM. And for those active in the tech community, you might have heard about the recent 1BRC challenge initiated by Gunnar Morling. GraalVM truly showcased its capabilities in this challenge, sparking my curiosity. That's why I reached out to Alina to delve deeper into GraalVM, exploring its features and uncovering how it excels in such endeavors. And here we are talking about GraalVM Chapters: 00:00 Introduction 01:47 GraalVM's Impact on the 1BRC Challenge and Its Features 04:34 Exploring GraalVM's Core Features and Benefits 08:34 Real-World Success Stories: GraalVM in Action 16:18 Understanding Native Image Compilation with GraalVM 20:34 Framework Compatibility and GraalVM Integration 25:04 Testing and Integration with GraalVM 25:26 Exploring Testing and Development with GraalVM 25:58 Best Practices for Developing with GraalVM 28:11 Migrating to GraalVM: Strategies and Considerations 31:25 Performance Optimization in GraalVM 35:15 Building and Resource Considerations for GraalVM 38:45 Expanding Horizons: Polyglot Programming with GraalVM 43:15 Future Directions and Limitations of GraalVM 47:40 Engaging the Java Community: GraalVM's Impact 50:21 Getting Started with GraalVM: Resources and Recommendations References and Links: - The GraalVM website with docs, downloads, guides: https://www.graalvm.org/ - Nicolai Parlog's "Modern Java in Action" demo: https://github.com/nipafx/modern-java-demo - My native version of Nicolai's demo: https://github.com/alina-yur/native-modern-java-demo - For news, follow GraalVM: https://twitter.com/graalvm Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarrator If you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet. Database internals series: https://youtu.be/yV_Zp0Mi3xs Popular playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA- Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17 Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN Stay Curios! Keep Learning! #Java #jvm #graalvm #highperformance #JITcompiler #AOT #nativeimage #security #rust #c++

In this episode I am talking to Javier Ramirez from QuestDB, about everything QuestDB. This episode is a great resource to understand how QuestDB works, its architecture, what is it optimised for and whats upcoming as per the roadmap. If you have timeseries data and need a simple yet highly scalable solution, #QuestDB is a great option. Chapters: 00:00 Introduction 03:04 Understanding QuestDB: Origins and Use Cases 09:21 Deep Dive into QuestDB's Architecture and Data Ingestion 19:07 Optimizing Data Reads and Writes in QuestDB 28:40 Exploring Data Granularity and Partitioning in QuestDB 29:29 Optimizing Query Performance with Partition Strategies 30:26 Handling Data Ingestion and Query Efficiency 32:58 In-depth Look at Data Duplication and Ingestion Performance 34:55 Understanding Compression and Its Impact on Performance 38:51 Replication and Data Distribution Strategies 47:10 Observability and Metrics in QuestDB 50:57 Future Developments and Enhancements in QuestDB 58:45 Closing Remarks Links: QuestDB: https://questdb.io/ Github: https://github.com/questdb/questdb =============================================================================== For discount on the below courses: Appsync: https://appsyncmasterclass.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Testing serverless: https://testserverlessapps.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Production-Ready Serverless: https://productionreadyserverless.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Use the button, Add Discount and enter "geeknarrator" discount code to get 20% discount. =============================================================================== Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarrator If you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet. Database internals series: https://youtu.be/yV_Zp0Mi3xs Popular playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA- Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17 Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN Stay Curios! Keep Learning! #questdb #sql #timeseries #timeseriesanalysis #databases #highscale #scaleup #performance #parquet #S3 #replication #writeaheadlog #wal #durability #columnstore

In this episode I talk to Andras Gerlits, who founded omniledger.io. Andras has a very interesting view on how Distributed Consistency should work that can get rid of several bottlenecks when it comes to maintaining Distributed consistency. He argues how getting rid of a global wall clock and using causality to approach Distributed consistency helps you build resilient, simple and performant systems. We have gone deeper into how that can be achieved and how the product works. Chapters: 00:00 Introduction 00:52 Andras's Journey into Distributed Consistency 03:04 The Evolution of Data Consistency in Banking and Beyond 08:04 Introducing Client-Centric Consistency 10:36 Exploring the Standard Model of Distributed Consistency 16:01 Redefining Strong Consistency with a Relativistic Approach 34:25 Practical Implications of Client-Centric Consistency in Banking 36:20 Mitigating Latencies and Partitions in Distributed Systems 41:08 Exploring System Reliability and Availability 41:52 Tuning System Properties for Specific Use Cases 43:07 Comparing Standard and New Models for Data Management 45:08 Understanding Local Progress and Mutex-Free Updates 47:23 Deep Dive into Token-Based Ordering and Global Calibration 58:30 Introducing OmniLedger: A New Approach to Distributed Consistency 01:02:41 Performance Optimizations and Tunable Consistency 01:08:20 Ideal Use Cases and Potential Limitations of OmniLedger 01:14:30 Future Directions and Closing Thoughts Links: Our website: https://omniledger.io A long-form essay on the thinking behind our model: https://medium.com/p/5e397cb12e63 A demo of transactionality https://www.youtube.com/watch?v=XJSSjY4szZE I think my blog in general might be interesting to some https://medium.com/@andrasgerlits The science-paper with all its mathematical rigour: https://www.researchgate.net/publication/359578461_Continuous_Integration_of_Data_Histories_into_Consistent_Namespaces =============================================================================== For discount on the below courses: Appsync: https://appsyncmasterclass.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Testing serverless: https://testserverlessapps.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Production-Ready Serverless: https://productionreadyserverless.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Use the button, Add Discount and enter "geeknarrator" discount code to get 20% discount. =============================================================================== Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarrator If you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet. Database internals series: https://youtu.be/yV_Zp0Mi3xs Popular playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA- Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17 Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN Stay Curios! Keep Learning! #databases #sql #consistency #distributedsystems

In this video I talk to Semih Salihoglu about KuzuDB : A highly scalable, extremely fast, easy to use embeddable Graph Database. Chapters: 00:00 Introduction 00:40 The Genesis of KuzuDB: From Academic Research to Startup 06:40 Graph Databases 101: Understanding the Basics and Beyond 10:24 When to Opt for a Graph Database: Use Cases and Advantages 19:16 KuzuDB vs. Traditional Databases: A Comparative Analysis 24:39 Inside KuzuDB: Optimizations and Data Ingestion Explained 31:08 Exploring Query Optimizations in Graph Databases 31:34 The Relational Nature of Graph Databases 33:33 Factorization: A Key Optimization Technique 38:50 Integrating New Data Sources and Handling Joins 43:39 Optimizing Write Operations and Index Management 50:23 Comparing Kuzu with Other Graph Databases 58:50 Future Developments and Vision for Kuzu Important links: - History of DBMSs and the IDS, which is the first database in history, which had a graph-based model: https://dl.acm.org/doi/abs/10.1145/1147376.1147382 is a good paper by CS historian on this history and a must read for everyone interested in the birth of databases as a field. - https://blog.kuzudb.com/post/what-every-gdbms-should-do-and-vision/ blog on the what every GDBMS should do and vision of Kùzu. - The user survey paper that got Semih into GDBMSs. https://arxiv.org/pdf/1709.03188.pdf - Blog on factorization https://blog.kuzudb.com/post/factorization/ - Kùzu's RDFGraphs feature https://docs.kuzudb.com/rdf-graphs/ =============================================================================== For discount on the below courses: Appsync: https://appsyncmasterclass.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Testing serverless: https://testserverlessapps.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Production-Ready Serverless: https://productionreadyserverless.com/?affiliateId=41c07a65-24c8-4499-af3c-b853a3495003 Use the button, Add Discount and enter "geeknarrator" discount code to get 20% discount. =============================================================================== Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarrator If you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet. Database internals series: https://youtu.be/yV_Zp0Mi3xs Popular playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA- Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17 Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN Stay Curios! Keep Learning!