DaskWhat is it?Parallelism for analyticsWhat is parallelism?Doing a lot at once by splitting tasks into smaller subtasks which can be processed in parallel (at the same time)Distributed work across multiple machines and then combining the resultsHelpful for CPU bound - doing a bunch of calculations on the CPU. The rate at which process progresses is limited by the speed of the CPUConcurrency?Similar but a but things don’t have to happen at the same time, they can happen asynchronously. They can overlap.Shared stateHelpful to I/O bound - networking, reading from disk, etc. The rate at which a process progresses is limited by the speed of the I/O subsystem.Multi-core vs distributedMulti-core is a single processor with 2 or more cores that can cooperate through threads - multithreadingDistributed is across multiple nodes communicating via HTTP or RPC Why is this hard?Python has it challenges due to GIL, other languages don't have this problemShared state can lead to potential race conditions, deadlocks, etcCoordination work across the machinesFor analytics?Calculating some statistics on a large dataset can be tricky if it can’t fit in memory// Show NotesCoiled Cloud: https://cloud.coiled.io/Coiled Launch Announcement: https://medium.com/coiled-hq/coiled-dask-for-everyone-everywhere-376f5de0eff4OSS article: https://www.forbes.com/sites/glennsolomon/2020/09/15/monetizing-open-source-business-models-that-generate-billions/#2862e47234fdAmish barn raising: https://www.youtube.com/watch?v=y1CPO4R8o5MMessagePassingInterface: https://en.wikipedia.org/wiki/Message_Passing_Interface----------- Connect With Us ✌️-------------Join our Slack community: https://go.mlops.community/slackFollow us on Twitter: @mlopscommunitySign up for the next meetup: https://go.mlops.community/registerConnect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/Connect with David on LinkedIn: https://www.linkedin.com/in/aponteanalytics/Connect with Matthew on LinkedIn: https://www.linkedin.com/in/matthew-rocklin-461b4323/Timestamps: