– Amit lead a team of ~10 R&D engineers responsible for Data Science / ML

– Roughly 1,000 cars a day are being traded with 8-10k auctions / bids per day

– Motivation for using Flask and Python

– Scikit-Learn and TensorFlow for machine / deep learning

– Did things start off with multiple microservices or was it a monolith early on?

– There’s about 80,000 to 120,000 lines of code across 200-300+ Python files

– The huge refactor to microservices was done without automated tests initially

– After the refactor now there’s 86% test coverage which is enough to be confident

– Flask-Restplus is the main library used to build their RESTful APIs

– Other notable libraries were gunicorn and boto3 (AWS SDK for Python)

– Locust is an open source load / performance testing tool

– With machine learning, speed is important but accuracy is even more important

– gunicorn is very compact, performant and easy to configure

– Most caches were in memory and they used Amazon DynamoDB

– The primary database is MySQL running on Amazon RDS

– SQLAlchemy is used on the Python side as an ORM

– Docker is sort of being used in development

– The platform runs on AWS with Lambda, API Gateway and AWS Fargate with ECS

– What is AWS Fargate and what does it allow you to do?

– Scaling with Fargate while using auto scaling policies and configuration

– Taking advantage of the cloud and setting up load balancing with configuration

– How do you deal with secrets when using Fargate / ECS?

– What about logging and metrics? Are you exclusively using all of AWS’ services?

– What about error reporting, such as getting notified if an error happens

– The deploy process from development to production (includes CI / CD with Jenkins)

– A Walk through of how the different AWS services come together

– Terraform is being used to manage the infrastructure as code (valuable tool)

– Database backups were performed by the DevOps team

– Best tips? Start slow and expect failures, also don’t chase perfection

– You can find Amit on Twitter at @ml_amit and on LinkedIn