– 4 developers are actively working on the project

– It’s been running in production since September 2019

– Motivation for using Phoenix and Elixir

– Henry started using Elixir back in late 2014

– Ditching Umbrella apps for dedicated services

– 6DOS is built on 6 independent git repos with a service oriented architecture

– A break down of what those 6 repos are and what they do

– Each service has its own independent database (Postgres, Neo4j, Elasticsearch)

– Neo4j is a graph database which is a great fit for their main data model

– How is Elixir support for Neo4j?

– Each service talks to each other through RabbitMQ events / notifications

– Walking through the request / response cycle when a visitor hits the site

– How did you arrive at this service oriented architecture?

– It’s easy to get Domain-driven Design (DDD) wrong initially

– Are Phoenix contexts being used? Nope

– Monoliths vs micro-services vs something in between and industry trends

– “Instantaneous complexity”

– Using an app skeleton project to help spin up new services quickly

– Using VueJS on the front-end with Webpack, but not through Phoenix

– Currently 6DOS doesn’t need websockets but that could change later

– Quite a lot of work happens in the background

– RabbitMQ handles queueing up all of the jobs

– Docker is being used in production, but not in development (yet)

– The work flow for starting everything up locally in development

– Everything is hosted on a self managed Kubernetes cluster on GCP

– (3) 2 core master nodes, (3) 2 core worker nodes and extra servers for the databases

– The self managed Kubernetes cluster was terrifying to set up initially

– Kubernetes is not a magic button you press to scale your application

– Auto-recovering from a CrashLoopBackOff error with Kubernetes

– Those 2 CPU core servers have 8 GB of RAM but the app isn’t using all of that

– Handling an interesting auto-scaling problem with Kubernetes

– Performing rolling restarts so there’s no down time for each deploy

– Dealing with restarting containers while an important action is happening

– Walking through the deploy process from development to production

– It starts with a self hosted Gitlab instance with automated CI

– On the other side, Keel takes over to automate deploying any services

– Helm is being used for a few things, but not everything

– Humans needing to accept the release happens within Keel’s UI

– Secrets are stored directly in the self hosted config repo with strict access rights

– Balancing your time between low level infrastructure vs app features

– Handling SSL certificates on the cluster with cert-manager

– Everything is behind a Cloudflare proxy too

– Dealing with database migrations when you have automated deployments

– Migrations get run as part of the app boot up process

– Design your software like a space ship

– Diagnosing errors with custom tasks and 3rd party tools

– No one can agree on how to format API JSON errors

– Elixir best practices are still being discovered, the future is bright

– An example of one of Henry’s open source Elixir tools (Pigeon) taking off

– All of the databases get backed up hourly

– Kubernetes really needs to be configured

– Rate limiting is currently being added to all of the services

– What about alerts if something goes down? It’s a digital notification bomb

– Using UptimeRobot as a second sanity check to make sure things are up

– Hung over at 6am out in the middle of the woods and your server goes down

– Using an external tool like UptimeRobot is worth it

– Timber.io is being used for logging but that will change soon

– Kubernetes’ Stern package helps with tailing logs across pods

– Henry isn’t a fan of Kubernetes’ web UI tools to manage the cluster

– Weave Scope was interesting but it used too much resources to run it

– Best tips? Don’t be afraid to break your code up into multiple repos

– If you’re not at that point yet, at least look into using contexts and DDD

– Developers as a whole are getting better over time

– Codedge is Henry’s consulting company and they are on GitHub / Twitter too