– How online streaming video works with HLS and where Mux fits into the picture

– Mux lets you post a video to their API and they give you an HLS playback URL

– Mux has been up and running since January 2016 and went through YCombinator

– Mux Data is another service they offer, it’s like New Relic but for video data

– They process billions of video views per month through Mux Data

– You could use Mux as a lower level alternative to Vimeo or Wistia

– Sometimes embedding iframes can be problematic and Mux can help in this area

– About 45 people work at Mux and half are involved with engineering

– Motivation for using Phoenix and Elixir, even when they were very new tools

– Their main public API is an out of the box Phoenix app

– They have a real-time dashboard that is powered by websockets and channels

– Some of Mux’s customers have millions of concurrent video views through that

– Will you switch to using Live View? Probably not since they are so API driven

– A dozen or so Go microservices and Kafka handle processing the videos

– Go is a great fit for super CPU intensive tasks such as video encoding

– The video processing infrastructure was very well thought out early on

– The public API is RESTful and there’s ~40-50 endpoints with a few private endpoints

– Cookie based auth is done in a browser but there’s tokens for API access

– The exq library is used for processing jobs asynchronously in Elixir land

– exq runs within a supervisor of your app, not a dedicated OS level service

– Prometheus is used for metrics but it’s not hooked into Elixir Telemetry (yet)

– Kubernetes and Docker drive their production infrastructure

– Buildkite is used for their CI / CD pipeline

– Deployments are very automated, a human only needs to merge to a specific branch

– The video processing microservices are in 1 mono repo, but there’s 2 other repos

– There’s PR approvals in place but all developers can merge to the production branch

– Code reviews are really important and you need to trust your developers

– The Elixir app has a PostgreSQL billing DB and also uses ClickHouse (SQL based)

– ClickHouse lets them store billions of rows and access everything quickly

– You do write SQL queries with ClickHouse but it doesn’t work with Ecto out of the box

– The Elixir API runs on AWS with an AWS load balancer sitting in front of it all

– The video infrastructure runs on Google Cloud

– How many servers do you run in total? Hard to tell really, but it’s a lot of compute

– Despite being on AWS, they are not using Amazon’s managed Kubernetes (EKS)

– All payments go through Stripe, including the metered billing which they hand rolled

– Instead of billing based on bandwidth, Mux bills by minutes watched

– SendGrid is used for transactional emails, Sentry for errors and Opsgenie for paging

– All sorts of CI / CD related information gets sent over to a Slack channel

– Developers are broken out into 4 cross functional teams

– There’s 2 flavors of SDKs that Mux has (REST API wrappers and video players)

– They currently have 22 different video players to account for across many platforms

– Efficiently creating so many different SDKs by having a core library for each language

– It’s sort of like having a core payment library and supporting Stripe, PayPal, etc.

– The SDK team needs to be aware of many different languages and players

– Another key metric to track is the video upscale and downscale percentages

– As of today Mux is focused on supplying service quality metrics

– There’s a lot of data stored but it all gets rolled over after 90 days

– The API is deployed all the time, but there’s zero down time deploys

– There’s been one day in the past there they had to put the API in read-only mode

– The data is backed up, but Dylan isn’t sure how often (but it happens, he swears!)

– Video thumbnails can be picked out from any timestamp, even animated GIFs too

– For now you need to supply your own closed captions to Mux

– Captions are downloaded, cached locally until processed and then backed up too

– Smoke tests and various alarms help detect issues in production (they use Flink)

– Uptime is important, Mux has high profile clients where downtime is not an option

– Rate limiting is done at the Elixir level for API calls with the ex_rated library

– It’s a reasonable idea to always assume users are out to get you

– For video rate limiting, it’s up to the CDN and they use a few different CDNs

– You could build a live streaming service like Twitch with Mux’s API

– The Elixir API doesn’t get billions of calls a month but it’s a still a lot

– Best tips? Video is hard and it keeps getting more and more complicated

– Fortunately the video player SDK’s churn isn’t too high due to the HTML5 spec

– You can email Dylan or contact him on Twitter, also Mux is hiring too!