Welcome back to the Klaviyo Data Science podcast! This episode, we dive into… Using NLP to communicate at scale Last episode, we discussed the history and practice of natural language processing, or NLP. This month, we’re here to discuss an exciting and cutting-edge application: using NLP to help businesses converse with their customers at scale. See the power of NLP in action as we talk with NLP experts on the Conversation AI team at Klaviyo about: How NLP enables a qualitative shift in how businesses communicate What intent classification is and why it matters Tips on tailoring NLP to a highly specific use case “There’s a lot of ways to think about the term ‘intent’. One way is what is the customer saying, and you can assign some sort of value to that. But the real intent that we’re interested in is what response are they hoping to get.” - David Lustig, Data Scientist See the full show notes on Medium!

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into… What’s the deal with natural language? Natural language processing, or NLP, is one of the dominant forces in modern data science, and it’s produced a host of data science-powered products many people take for granted as a basic fact of life. It hasn’t always been so powerful or pervasive, though — NLP has a long and interesting history, and some of the advances powering today’s technology would have seemed like science fiction only decades ago. This month, we dive into the history and foundations of NLP, examining: Why natural languages are so difficult to work with in the first place Early attempts by mathematicians and data scientists to use natural languages, and why they failed What distinguishes today’s cutting edge models and allows them to succeed “Language is the natural medium for humans to communicate in. So if you want to build a really immersive, interesting product, for especially non-data scientists to interact with, it almost has to involve NLP.” - Robert Huselid, Data Scientist See the full show notes, including resources to learn more, on Medium.

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into… Thinking big-picture with A/B testing We’ve discussed A/B testing multiple times on this podcast, for good reason. But there’s an important angle we have yet to cover: in the life of a researcher or marketer, there’s no such thing as an A/B test. There’s an entire system of A/B tests run for specific purposes over time. What is the best way to construct a system of A/B tests to help you learn, improve, and grow over time? How does that translate into tenets to hold while building software to help people run A/B tests? We’ve brought on three members of the data science team at Klaviyo, and you’ll hear about A/B tests in a variety of ways, including: Real data-driven trends observed by successful A/B testers on Klaviyo Why up-front thinking and vision translate into long-term success Why dad jokes might be far more powerful than you think “The more experimental you can be, the more creative you can be, the more you can learn about your customers to really deliver authentic experiences and see return on your investment.” - Woody Austin, Senior Machine Learning Engineer Check out the full show notes on Medium for more information!

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into… Using data science to help people write Using machine learning models to generate text, images, and other creative objects is, as they say, a bit of a hot topic right now. There are examples of models like this in action all across the internet and across different fields and disciplines. Today, we discuss one of those fields in more depth: marketing. In particular, the Klaviyo data science team recently released the Subject Line Assistant tool, which helps marketers craft better subject lines. We take a close look at that tool, how it works, and the thinking behind it to examine what it looks like to use AI to help a human write. We’ve brought on four experts from Klaviyo, and you’ll hear about subject lines from a variety of angles, including: What a subject line is, and why it’s arguably the most important part of an email What holds people back from writing great subject lines and how the team went about solving those problems How a specialized human-in-the-loop model for a highly specific context can look “Subject lines are a very unique type of text generation problem. We’re not asking for a short story where there’s a lot of leeway to really hit a home run — you have a limited amount of space to communicate a brand message, communicate what the email is communicating, make a connection with your audience, and encourage them to interact.” - Josh Villarreal, Data Scientist Head over to the full show notes to see all the information about this episode!

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into… Writing code for computers and people No matter what sort of data science work you do, it’s fairly inevitable that you’ll have to write code to accomplish your goals. For substantial projects, it’s also fairly inevitable that you’ll have to work with other people to see them to completion. As anyone who’s dived into a legacy code base can tell you, writing code that other people (and yourself in the future) can understand is both an essential skill to have and a difficult practice to master. This episode, we talk specifics about improving your coding skills. We’ve brought on four software engineering experts from Klaviyo, and you’ll hear about writing good code from a variety of angles, including: What exactly is good code? The biggest misconceptions that come with writing code How to prepare for your first code review Our panel’s top tips for improving your coding skills, tailored to your level of experience “You don’t have to make a perfect work of art. It doesn’t have to be bug-free. But it should absolutely be an act of polite and intelligible communication for the next person who will interpret what you create.” - Zac Bentley, Lead Site Reliability Engineer

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into… What are data privacy and security? Data privacy and security are huge and hugely important topics — in all likelihood, you already know a little about them if you’re reading this intro. But they are both crucial to any good data science work, and this month we explore the fundamentals of both topics: why data privacy and security are necessary to deliver the value you promise your customers, who they matter the most to, and how to build privacy and security into your own data science work. The panel includes some of the foremost experts on the topics at Klaviyo from data science, engineering, and security and risk governance, so you’ll get to hear about these topics from a variety of angles, including: How approaches to data privacy that seem intuitive can fail, and fail spectacularly The consequences of not taking privacy and security carefully enough How to make people actively want to work within the security environment you set up “The worst case is that you violate your customers’ trust. And if you think about personal relationships you have where someone has violated your trust, it’s really hard to build that back.” - Dom Lombardi, Security Risk and Compliance Manager Learn More Privacy and security failures mentioned in the episode  — The SWIFT hack of the Bank of Bangladesh  — The CafePress data breach Differential privacy —Overview:  A non-technical primer from Nissim et al.   — Example: Apple’s DP Sketch algorithm  — Example: Google’s RAPPOR Data Privacy  — The Harvard Business Review’s New Rules of Data Privacy For the full show notes, see the writeup on Medium.

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into… Customer-focused research This month, we focus on research — but specifically research that’s aimed at your customers, delivering the sort of insight they would try to glean by running experiments and analysis using their own data. In particular, we dive into two different case studies drawn from the recent topics explored by the Klaviyo data science team. You’ll hear about: Why customer-focused research can be some of your highest-impact work Whether or not to use emojis when you’re sending out an email How to react when you encounter surprising results in your research “It was startling. It was the type of number that when you see it, you think: oh, what did I do wrong?” - Mike Galli See the full writeup, including links to the blog posts we mention, in the show notes on Medium.

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into… 2021 Year in Review Once again, as the new year starts, we begin by recapping the old. Instead of diving deep into a specific topic, I asked 7 members of the Klaviyo data science team to give their personal highlight for 2021 as a year in data science. You’ll hear about fascinating data science topics, including: How companies used domain knowledge to hyper-charge their ranking algorithms Powerful estimating methods that account for covariance How 2021 provided new opportunities — and pitfalls— for state-of-the-art experimental analysis techniques Be sure to check out the show notes in Medium to learn more about the topics we discuss in this episode!

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into… Customer research: your secret weapon You can study as much mathematical theory, invent as sophisticated a machine learning model, or write as clean production-ready code as you want — if you don’t make sure you’re solving the right problems to begin with, all that effort could be for nothing. It’s not a topic you learn about in most data science coursework, but understanding your end customer is a crucial part of being an effective data scientist. We spend this whole episode describing why and how to do great customer research. Topics include: Why customer research is such a big deal in the first place How talking with customers can drastically change your thinking How to run the perfect customer call Be sure to check out the show notes in Medium to learn more about the topics we discuss in this episode! If you have any questions, comments, or concerns, please contact me on Twitter.

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into… Fuel for the Creative Fire It’s no secret: being creative is hard. Creativity requires time and energy, at the bare minimum, and lacking creativity can spiral into writer’s block and other such conditions. That may be okay if you’re just sending out a tweet here or there — but what if your core user base consists of people who need to be creative, day in and day out? The Creative team at Klaviyo recently tackled the problem of helping users get inspired to create content, and I sat down to discuss the thinking that went into the resulting feature, Showcase. You’ll hear about the development process for Showcase, but also about the underlying problems that Showcase is trying to solve and the process of coming up with a solution like Showcase. Specific topics include: Using data science to answer questions that seem simple… even when they aren’t Ensuring data privacy in solutions that have to scale Controversial sandwiches, and why they make great marketing tools “There are actually a lot of sites where you can subscribe to literally every single email that a company sends out… but you have no sense of: did these emails do well? What about them was good? Is this something I should copy? It’s just throwing out a bunch of data with no context or insight whatsoever.”  — Charlie Natoli, Senior Data Scientist See the full episode writeup, including links and who's who, on Medium.