hi everyone Welcome to our event this event is brought to you by data dos club which is a community of people who lovedata and we have weekly events and today one is one of such events and I guess weare also a community of people who like to wake up early if you're from the states right Christopher or maybe not somuch because this is the time we usually have uh uh our events uh for our guestsand presenters from the states we usually do it in the evening of Berlin time but yes unfortunately it kind ofslipped my mind but anyways we have a lot of events you can check them in thedescription like there's a link um I don't think there are a lot of them right now on that link but we will beadding more and more I think we have like five or six uh interviews scheduled so um keep an eye on that do not forgetto subscribe to our YouTube channel this way you will get notified about all our future streams that will be as awesomeas the one today and of course very important do not forget to join our community where you can hang out withother data enthusiasts during today's interview you can ask any question there's a pin Link in live chat so clickon that link ask your question and we will be covering these questions during the interview now I will stop sharing myscreen and uh there is there's a a message in uh and Christopher is fromyou so we actually have this on YouTube but so they have not seen what you wrotebut there is a message from to anyone who's watching this right now from Christopher saying hello everyone can Icall you Chris or you okay I should go I should uh I should look on YouTube then okay yeah but anyways I'll you don'tneed like you we'll need to focus on answering questions and I'll keep an eyeI'll be keeping an eye on all the question questions so umyeah if you're ready we can start I'm ready yeah and you prefer Christophernot Chris right Chris is fine Chris is fine it's a bit shorter umokay so this week we'll talk about data Ops again maybe it's a tradition that we talk about data Ops every like once peryear but we actually skipped one year so because we did not have we haven't hadChris for some time so today we have a very special guest Christopher Christopher is the co-founder CEO andhead chef or hat cook at data kitchen with 25 years of experience maybe thisis outdated uh cuz probably now you have more and maybe you stopped counting Idon't know but like with tons of years of experience in analytics and software engineering Christopher is known as theco-author of the data Ops cookbook and data Ops Manifesto and it's not thefirst time we have Christopher here on the podcast we interviewed him two years ago also about data Ops and this onewill be about data hops so we'll catch up and see what actually changed in inthese two years and yeah so welcome to the interview well thank you for havingme I'm I'm happy to be here and talking all things related to data Ops and whywhy why bother with data Ops and happy to talk about the company or or what's changedexcited yeah so let's dive in so the questions for today's interview are prepared by Johanna berer as alwaysthanks Johanna for your help so before we start with our main topic for todaydata Ops uh let's start with your ground can you tell us about your career Journey so far and also for those whohave not heard have not listened to the previous podcast maybe you can um talkabout yourself and also for those who did listen to the previous you can also maybe give a summary of what has changedin the last two years so we'll do yeah so um my name is Chris so I guess I'ma sort of an engineer so I spent about the first 15 years of my career insoftware sort of working and building some AI systems some non- AI systems uhat uh Us's NASA and MIT linol lab and then some startups and then umMicrosoft and then about 2005 I got I got the data bug uh I think you know mykids were small and I thought oh this data thing was easy and I'd be able to go home uh for dinner at 5 and lifewould be fine um because I was a big you started your own company right and uh it didn't work out that wayand um and what was interesting is is for me it the problem wasn't doing thedata like I we had smart people who did data science and data engineering the act of creating things it was like thesystems around the data that were hard um things it was really hard to not haveerrors in production and I would sort of driving to work and I had a Blackberry at the time and I would not look at myBlackberry all all morning I had this long drive to work and I'd sit in the parking lot and take a deep breath andlook at my Blackberry and go uh oh is there going to be any problems today and I'd be and if there wasn't I'd walk andvery happy um and if there was I'd have to like rce myself um and you know andthen the second problem is the team I worked for we just couldn't go fast enough the customers were superdemanding they didn't care they all they always thought things should be faster and we are always behind and so um howdo you you know how do you live in that world where things are breaking left and right you're terrified of making errorsum and then second you just can't go fast enough um and it's preh Hadoop eraright it's like before all this big data Tech yeah before this was we were usinguh SQL Server um and we actually you know we had smart people so we we webuilt an engine in SQL Server that made SQL Server a column ordatabase so we built a column or database inside of SQL Server um so uhin order to make certain things fast and and uh yeah it was it was really uh it's notbad I mean the principles are the same right before Hadoop it's it's still a database there's still indexes there'sstill queries um things like that we we uh at the time uh you would use olapengines we didn't use those but you those reports you know are for models it's it's not that different um you knowwe had a rack of servers instead of the cloud um so yeah and I think so what what Itook from that was uh it's just hard to run a team of people to do do data and analytics and it's notreally I I took it from a manager perspective I started to read Deming andthink about the work that we do as a factory you know and in a factory that produces insight and not automobiles umand so how do you run that factory so it produces things that are good of goodquality and then second since I had come from software I've been very influencedby by the devops movement how you automate deployment how you run in an agile way how youproduce um how you how you change things quickly and how you innovate and sothose two things of like running you know running a really good solid production line that has very low errorsum and then second changing that production line at at very very often they're kind of opposite right um and sohow do you how do you as a manager how do you technically approach that andthen um 10 years ago when we started data kitchen um we've always been a profitable company and so we started offuh with some customers we started building some software and realized that we couldn't work any other way and thatthe way we work wasn't understood by a lot of people so we had to write a book and a Manifesto to kind of share our ourmethods and then so yeah we've been in so we've been in business now about a little over 10years oh that's cool and uh like whatuh so let's talk about dat offs and you mentioned devops and how you were inspired by that and by the way like doyou remember roughly when devops as I think started to appear like when did people start calling these principlesand like tools around them as de yeah so agile Manifesto well first of all the Imean I had a boss in 1990 at Nasa who had this idea build alittle test a little learn a lot right that was his Mantra and then which mademade a lot of sense um and so and then the sort of agile software Manifestocame out which is very similar in 2001 and then um the sort of first realdevops was a guy at Twitter started to do automat automated deployment you knowpush a button and that was like 200 Nish and so the first I think devopsMeetup was around then so it's it's it's been 15 years I guess 6 like I wastrying to so I started my career in 2010 so I my first job was a Javadeveloper and like I remember for some things like we would just uh SFTP to themachine and then put the jar archive there and then like keep our fingers crossed that it doesn't break uh uh likeit was not really the I wouldn't call it this way right you were deploying youhad a Dey process I put it yeahright was that so that was documented too it was like put the jar on production cross yourfingers I think there was uh like a page on uh some internal Viki uh yeah thatdescribes like with passwords and don't like what you should do yeah that was and and I think what's interesting iswhy that changed right and and we laugh at it now but that was why didn't youinvest in automating deployment or a whole bunch of automated regressiontests right that would run because I think in software now that would be rarethat people wouldn't use C CD they wouldn't have some automated tests you know functionalregression tests that would be the exception whereas that the norm at the beginning of your career and so that'swhat's interesting and I think you know if we if we talk about what's changed in the last two three years I I think it isgetting more standard there are um there's a lot more companies who aretalking data Ops or data observability um there's a lot more tools that are a lot more people areusing get in data and analytics than ever before I think thanks to DBT um andthere's a lot of tools that are I think getting more code Centric right thatthey're not treating their configuration like a black box there there's severalbi tools that tout the fact that they that they're uh you know they're they're git Centric you know and and so and thatthey're testable and that they have apis so things like that I think people maybe let's take a step back and just do aquick summary of what data Ops data Ops is and then we can talk about like what changed in the last two years sure so Iguess it starts with a problem and that it's it sort ofadmits some dark things about data and analytics and that we're not really successful and we're not really happy umand if you look at the statistics on sort of projects and problems and eventhe psychology like I think about a year or two we did a survey ofdata Engineers 700 data engineers and 78% of them wanted their job to come with a therapist and 50% were thinkingof leaving the career altogether and so why why is everyone sort of unhappy well I I I think what happens isteams either fall into two buckets they're sort of heroic teams whoare doing their they're working night and day they're trying really hard for their customer um and then they getburnt out and then they quit honestly and then the second team have wrappedtheir projects up in so much process and proceduralism and steps that doinganything is sort of so slow and boring that they again leave in frustration umor or live in cynicism and and that like the only outcome is quit andstart uh woodworking yeah the only outcome really is quit and start workingand um as a as a manager I always hated that right because when when your teamis either full of heroes or proceduralism you always have people who have the whole system in their headthey're certainly key people and then when they leave they take all that knowledge with them and then thatcreates a bottleneck and so both of which are aren aren't and I think themain idea of data Ops is there's a balance between fear and heroisthat you can live you don't you know you don't have to be fearful 95% of the time maybe one or two% it's good to befearful and you don't have to be a hero again maybe one or two per it's good to be a hero but there's a balance um andand in that balance you actually are much more prod