– The current stack is a legacy Zope system combined with Flask

– ScholarPack has been running for 12+ years and Zope was very popular then

– 10% of the schools in the UK are using ScholarPack, it peaks at 65k reqs / second

– Their traffic patterns are predictable based on a school’s working hours

– Feature development during school sessions / architecture upgrades during holidays

– Zope vs Flask and the main reason they wanted to move to Flask

– Since Flask is so flexible, you need to be on the ball with setting standards

– 17-18 folks deal with the infrastructure and development of the project

– Gareth has a fetish for microservices but it really does fit well for their app

– Microservices let you split out your responsibilities and independently scale

– At their scale, downtime can have a serious impact on the kids at school

– A well maintained skeleton app works wonders for working with microservices

– A developer’s workflow for starting and working with a microservice

– Mocking responses for unrelated services helps with the development process

– Dealing with multi-tenancy across 1,500+ databases using SQLAlchemy binds

– Splitting the data up with a foreign key and 1 database would be too risky

– A school’s database gets picked from a sub-domain / unique URL

– What it’s like running database migrations on 1,500+ databases with PostgreSQL

– Point in time database backups make running so many migrations less scary

– Point in time backups are why they are on AWS instead of GCP

– Most services render Jinja on the server with sprinkles of JavaScript

– Supporting browsers like IE8 limits what you can do on the front-end

– IE8 is getting a little crusty, but it’s necessary to support it

– Redis and CloudFront are the 2 only other services being used in their stack

– Using signed cookies vs Redis for storing session state

– What about Celery and background workers? Most things are synchronous

– Celery could still be used in the future since it has benefits

– Schools do pay to use this service, but not with a credit card

– Using checks has an advantage of not needing a billing back-end

– Cost and scaling requirements of their old platform lead them to AWS Fargate

– GCP was looked into initially but the lack of point in time backups killed that idea

– The added complexity of going multi-cloud wasn’t worth it and RDS won

– Managed Kubernetes is not that great on AWS (especially not in 2017)

– ECS was also not going to work out due to their scaling requirements

– Fargate allows them to focus on scaling containers, not compute resources

– The TL;DR on what AWS Fargate allows you to do and not have to worry about

– Their microservices set up fits well with the Fargate style of scaling

– You still need to allocate memory and CPU constraints on your containers

– Everything runs in the AWS UK region across its multiple availability zones

– AWS initially limits you to 50 Fargate containers but you can easily raise that cap

– Setting a cap on the number of containers Fargate will ever spawn

– Pre-warming things to prepare for the massive traffic spike at 9am

– It’s fun to watch the traffic spikes on various dashboards

– Number of requests per host is their primary way to measure scaling requirements

– DataDog plays a big role in monitoring and reporting

– But CloudWatch is used too and DataDog alerts get sent to Slack

– Jira is used for error logging and ticket management

– 100s of errors occur a day in the legacy Zope system, but they are not serious

– It’s very rare to have a system level error where things are crashing

– The longest down time in the last 3.5 years has been 35 minutes

– All of the metrics to help detect errors have a strong purpose

– Walking through a deployment from development to production

– The Zope deployment experience has been a dream come true

– The Flask deployment has more steps but it’s still very automated

– Dealing with the challenges of doing a rolling restart

– Complex database changes are done after hours with a bit of down time

– That’s a great time to do a Friday evening deploy!

– Most new additions are behind a feature toggle at the Flask level

– Feature flags can be tricky but a good mindset helps get everyone on board

– A company policy combined with experience dictates best practices

– Switching from Flask-RESTPlus to Connexion

– What is Connexion and how does it compare to other API libraries?

– It only took a few days to get a real API service running with Connexion

– Everything is in git, it’s all deterministic and they use Pipenv with lock files

– The Zope structure is in a RAID file system and has daily backups

– Extensive user auditing is done at the database level (everything is logged)

– The audit tables get a huge amount of writes

– (10) t3.2xlarge (8 CPU cores / 32 GB of memory) instances power the RDS database

– How much does it all cost on AWS? Too much!

– The cloud is nice but you need to really keep tabs on your bills

– Gareth spends 2 days a month reviewing the AWS bills

– RDS will automatically restart stopped instances after 7 days

– Best tips? Look at what you have, what you want to do and how to get there

– A microservice should be broken up by its scope / domain, not lines of code

– There is no “wrong”, there is only the thing that works

– One mistake they did early on was try to be too perfect which delayed shipping

– Gareth is on Twitter @thestub