AWS re:Invent 2016
AWS re:Invent 2016
AWS
DAT315: Streaming ETL for RDS and DynamoDB
41 minutes Posted Dec 25, 2016 at 12:00 am.
0:00
41:00
Download MP3
Show notes
During this session Greg Brandt and Liyin Tang, Data Infrastructure engineers from Airbnb, will discuss the design and architecture of Airbnb's streaming ETL infrastructure, which exports data from RDS for MySQL and DynamoDB into Airbnb's data warehouse, using a system called SpinalTap. We will also discuss how we leverage Spark Streaming to compute derived data from tracking topics and/or database tables, and HBase to provide immediate data access and generate cleanly time-partitioned Hive tables.