– Weather data can come from a number of different sources

– 750+ GB of data is cycled every 2 days and it needs to be queryable in ~200ms

– This is Nick’s side project and he’s the sole developer which has its challenges

– The app is open source on GitHub

– Motivation for using Python and Flask

– Picking gunicorn over uwsgi, mainly for ease of deployment

– The web front-end is a monolith but the data processing happens in a different service

– Looking at Go for more performance on the math heavy calculations

– Ingesting data is done through Kubernetes cron jobs and it’s backed by PostgreSQL

– PostgreSQL is really amazing and works great as a queue back-end

– PostgreSQL was originally used for the 750 GB of data but then it was moved to S3

– The files on S3 are 10+ GB in size (each)

– The files are a huge series of floats in a binary format

– PostgreSQL ends up storing a little bit of meta-data that’s used to get data from S3

– It was all moved out of PostgreSQL because…

– Before the S3 change there were over 2.5 billion rows being saved to PostgreSQL

– Seriously, PostgreSQL is really good

– Visualizing weather in a radar view requires a massive amount of computing power

– Predicting the weather in the short term future based on clouds and wind

– Docker is being used in development and it’s making things a lot easier

– Ubuntu is being used as a base OS for the Docker images

– Traefik sits in front of gunicorn to do TLS termination and act as a reverse proxy

– Traefik is rock solid but it’s not a complete drop in replacement for nginx

– (2) servers are sitting in a colo for now, mainly for cost and performance wins

– The TL;DR on what a colo (colocation) is and what it’s like owning your own hardware

– Server 1 has 8 CPU cores, 96 GB of RAM, 6 TB of HDD space in RAID-5 and a 2 TB SSD

– Server 2 is a Threadripper with 16 CPU cores, 64 GB of RAM and (2) 2 TB NVME SSDs

– How much storage could you fit? 10+ SSDs plus the PCI slots on the motherboard

– Getting similar capacity and performance with AWS would be a lot more money

– Nick runs a Debian based distro called Proxmox

– A good GUI is one of the selling points for Proxmox, even though it’s on a server

– Kubernetes is self managed but it was configured with Rancher’s RKE

– Prior to finding RKE, he almost went with Swarm due to Kubernetes’ complexity

– Currently there’s no error reporting set up, but Sentry.io is on the horizon

– Deploying code from development to production

– The weather app is on the Docker Hub at kallsyms/wx_explore

– CI isn’t being used now but a self hosted Gitlab instance might happen in the future

– As long as the kubeconfig stays out of CI, then it’s all good

– How are database migrations dealt with? By avoiding them since they’re not needed

– What about backing up the database? It’s also not needed since it’s all recreatable

– The weather tool is made for both regular people and meteorologists

– StatusCake’s free tier is being used to do uptime monitoring

– Best tips? Learning to self manage your time and break down problems

– Learning best practices and tips on how to optimize Docker performance

– Keeping yourself accountable for hitting self imposed deadlines

-20 – You can find Nick on Twitter @kallsyms and on GitHub too