How to Avoid Suffering in Mlops/Data Engineering Role // Igor Lushchyk // MLOps Meetup #55 from Agentic Conversations (formally mlops.community)

How to Avoid Suffering in Mlops/Data Engineering Role // Igor Lushchyk // MLOps Meetup #55

57 minutes Posted Mar 12, 2021 at 3:58 pm.

] Introduction to Igor Lushchyk

] Igor's background in tech

] Tips you can pass on

] How these tools work and how they play together and what is underneath?

] Dedicated MLOps team

] Central Data Infrastructure Section

] Transfer over to open-source

] If you don't plan for production from the beginning, then it's going to be painful trying to go from POC to production.

] Ho do you handle data lineage?

] You chose that back in the day but you're regretting it.

] "Try to use tools which solve 80% of your use cases and maybe 20% you'll have the suffering but at least it's not 100% suffering."

] Friction points

] Interaction with Data Scientists

] "We have alignment sessions. We have different levels of representations. We share our progress."

] Build verse by decisions

] When to build or grab an open-source tool

] Build your own or buy open-source?

] Certain maturity and a certain number of engineers

] Startup to go with open-source

] Correct transition process

] "There are no other ways but to communicate with data scientists. Your team needs to have a close loop for future priorities, what to take with you and what to leave behind."

] What to use in monitoring piece

] Prometheus and Grafana

] Do you automatic retriggering monitoring of Models set up?

] Hardware for on Prim model training

] "Machine Learning model prediction is a spear bomb."

] War or horror stories

] "Guys, don't do context switching!"

] "I won't say that Adyen is a company that allows you to make mistakes but you can make mistakes."

0:00

57:55

Download MP3

Share at current time

Show notes

MLOps community meetup #55! Last Wednesday we talked to Igor Lushchyk, Data Engineer, Adyen.

// Abstract:

Building Data Science and Machine Learning platforms at a scale-up. Having the main difficulty in finding correct processes and basically being a toddler who learns how to walk on a steep staircase. The transition from homegrown platform to open source solutions, supporting old solutions and maturing them with making data scientists happy.

// Bio:

Igor is a software engineer with more than 10 years of experience. With a background in bioinformatics, he even started PhD but didn't finish it.

As a data engineer, Igor has been working for the last 6 or 7 years, or maybe more - because he was doing almost the same data engineering stuff but his position was named differently.

Igor has been doing a lot of MLOps in 4-5 years now. He doesn't know what he was doing more then - Data Engineering or MLOps. And that’s how this topic came about.

----------- Connect With Us ✌️-------------

Join our Slack community: https://go.mlops.community/slack

Follow us on Twitter: @mlopscommunity

Sign up for the next meetup: https://go.mlops.community/register

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Igor on LinkedIn: https://www.linkedin.com/in/igor-lushchyk/

Timestamps:

[

← See all 563 episodes of Agentic Conversations (formally mlops.community)