Klaviyo Data Science Podcast

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

When Things Break

Welcome to the November episode of the Klaviyo Data Science Podcast for this year! November is a unique month for ecommerce, which makes it a unique month for any software solution built for ecommerce; it’s a tradition on this podcast to take the opportunity to celebrate some of those unique challenges.

In an ideal world, software and data science products would never break. We do not live in an ideal world, though, so an important question to answer is: what should you do when things do break? This month, we discuss incidents, incident response, and getting things back on track as quickly and effectively as possible to continue delivering value to your customers.

Listen along to learn more about:

• Why not all ways of recognizing something has gone wrong are created equal
• How to cut through disagreements when the stakes are at their highest
• What sorts of unique challenges data science breakages and incidents pose

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

Off the Happy Path

In most discussions about data science and data science features on this podcast, we make a basic, foundational assumption: the users whose data we are thinking about and customer experience we are trying to improve are, generally speaking, trying to use the platform in a way we recognize and approve of. Not all users of an application have this intention, and the data science behind detecting users who misuse a platform— and even abuse it — constitutes a complex and vast field of study.

Listen along to learn more about:

• Different types of human behaviors motivating platform misuse, and how that translates to different types of data
• What makes many-to-one problems so challenging
• Why keywords alone are not enough

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

Presenting your work for fun and profit

Presenting technical work is not something you automatically learn how to do — just like the technical skills themselves, it has to be learned and practiced, and opportunities to practice it can be hard to find. This episode, we discuss one opportunity that Klaviyo put together for its R&D teams this summer: the Klaviyo R&D Science Fair. Listen along to hear about:

• How, much like software development, explaining technical work is an iterative process
• The best ways to engage a crowd and get them interested in what you have to say
• The unique and powerful allure of scissors and glue guns

“We put together a little game: try to find all of the accessibility problems in this form, without using the tool that we built…. And then when they react, ‘oh my God, like that one was impossible, I don’t know how you expected me to find that,’ that’s when we can say: exactly! That’s why we needed this feature!”— Maya Nigrin, Senior Software Engineer

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

An introduction to production

What comes after you finish building a data science model? If you’re working on a software project, the answer likely involves that model serving customers in production. Understanding production is crucial for any data scientist or software engineer, so we spend this episode learning about best practices from three experienced Klaviyo engineers.

Listen along to learn more about:

• How to make sure your code is “battle-ready,” whether you’re working on a data science project or not
• Why error messages you think are safe to ignore may not actually be safe to ignore
• One key lesson for safely deploying your code, no matter what environment you work in

“That’s stuck with me through the years: there are these knock-on effects between things. Even if it’s not your code, you should still try to understand how it’s working and whether it can have a ripple effect that comes back and affects your code.”— Chris Conlon, Lead Software Engineer

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

Research is a core part of data science. But data science is far from alone in that respect — other fields rely on research just as heavily, and they have their own set of hypotheses, methods, complications, and concerns. This month, we talk to three Klaviyos about research they did before joining the team — both data science research and other kinds — to see what we can learn about conducting effective data science research.

Listen along to learn more about:

• What tiny iron meteorites teach us about the importance of using your results to tell a compelling story
• What data science research into commerce and policy teaches us about iterating on your research questions
• What rubber beams teach us about the importance of getting feedback early

“Everybody has a unique perspective could be the one that opens up a brand new door. You’re looking at doing specific algorithms, you’re looking at doing the research a specific way, but there could be an alternative path.” 

- Mike Galli, Data Scientist

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

Few parts of your product, application, or webpage are more crucial than the very initial experience. In a web application like Klaviyo, that means the home page. Everyone sees it every time they log on to do anything, and interactions with that page set the tone for everything that follows. Meaning: if you’re going to change the home page, you need to really know what you’re doing.

This month, we talk with the Klaviyo engineering team that did just that. We discuss many aspects of that redesign, including:

• How to get buy-in from teams you depend on without taking away your own independence
• The unique difficulties that come with large front-end engineering projects and smart data visualization
• How to filter through the noise when evaluating the success of a feature

“There are very few features ever been released in Klaviyo that have seen that sort of change… At the end of the day, if we can help our users complete tasks faster and more effectively, that’s our highest priority.”
- Griffin Drigotas, Senior Product Designer

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

The question is slightly tongue-in-cheek, but only slightly. Data science is a new field — while many people today are graduating with degrees in data science, the same was not true a decade ago. Many of the people who work (and will work) as data scientists were not classically trained as a data scientist, but as something else. This month, we examine that process: the process of working in a field that’s distinct from data science and becoming a data scientist.

We discuss several parts of that journey, including:

• What attracts someone to data science in the first place
• How to approach gaining the technical skills you need to get a data science job
• How similar some parts of the data scientist job are to washing dishes

Where do data scientists come from?“You really need to practice using these tools. I did my best to come up with excuses to use data science techniques in all my projects… maybe instead of trying to automate a workflow in Excel VBA, I’d try to automate it in python instead.”
- Steven Her, Data Scientist

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

Back by popular demand: data science is a broad, deep field with an extraordinary amount to learn, and we’re here to help you learn it. We asked four members of the Data Science team at Klaviyo what one of their favorite data science books was, and we got four different answers. Listen on if you’ve wanted to know more ways to learn about:

• How to think about and employ the Bayesian framework (and corgis)
• Learning intro-to-intermediate coding skills necessary for data science work
• The theory that drives natural language processing
• The mindset of a data scientist in general

“it gives you a different lens to apply to different problems. And sometimes taking that different lens, suddenly a problem that was really hard to formulate using traditional frequentist statistics or machine learning techniques, suddenly it can be really easy to frame in this other way” - Tommy Blanchard, Senior Data Science Manager

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

When you work in data science, it’s inevitable that the world will change for you. Sometimes it’s due to global events, macroeconomic trends, or sudden shifts in consumer behavior. Other times it’s due to new features added by a commonly-used piece of software. When your lifeblood is data, all of these can be equally shocking and disruptive.

This month, we discuss one of the latter cases: the changes to the world of email marketing data brought about by the iOS 15 privacy updates. We bring on a panel of product managers, data scientists, and software engineers to discuss:

• How one software update can drastically alter your data landscape
• How to do research while the world is changing, and how to test your conclusions while the ground truth is still in flux
• How using different sources of data can help you adapt your product to a new reality

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

As the new year starts, we take a look back at 2022. We spoke to 8 data scientist and people who work closely with data scientists, and we asked them all the same question: what is the coolest data science thing you learned about in 2022? You’ll hear about fascinating data science topics, including:

• Advances in AI, NLP, and data science in general in 2022
• How understanding data science and ML operations makes a recruiter’s job easier
• New ways to classify models, visualize data, and play video games

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

We talk a lot on this podcast about the results of data science and software engineering work. We even talk about the process of doing data science and software engineering work. But one thing we haven’t shed much light on, until this month, is: what specific tools help a Data Science team — or any developer or data scientist similarly engaged in building a scalable and intelligent system — actually do their work? We asked several data scientists, machine learning engineers, software engineers, designers, and product managers the same question: what is your favorite tool that helps you do your job? You’ll hear all their answers in this episode, including:

• Why some well-known tools fully deserve the hype
• Specialized packages for specialized purposes
• How to slow down and really force yourself to think about the problem
• How to avoid analysis paralysis

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

It’s our third November on the Klaviyo Data Science Podcast, and if you work in ecommerce, you know that November means one thing: Black Friday and (usually) Cyber Monday, i.e. the month of the year where everything changes. Traditionally, we’ve talked about things that help prepare builders of software for when the world is about to change, such as infrastructure, readiness, scale-out testing, and other things along those lines. This year, we’re approaching it from another angle: ecommerce stores go through the exact same struggle every year. How can a platform like Klaviyo help prepare them for the unexpected? One answer: by automatically figuring out when unexpected things are happening, i.e., by detecting anomalous behavior. You’ll hear all about anomaly detection on this episode, including:

• How to pivot your research when your idea is valuable but your results aren’t providing that value
• How to label large swaths of data efficiently
• How to design algorithms for an extraordinarily diverse base of end users

I’ll let you in on a secret: this podcast does not cover everything. We cover a wide array of projects, go into detail on a variety of aspects of them, and speak to a diverse panel of data scientists and people related to the data science world, but we still can’t cover everything. This month, to give you a taste of what we haven’t been able to showcase on this podcast, we’re asking six Klaviyos who work on or with the Data Science team one simple question: what is your favorite data science project you’ve worked on? You’ll hear about all of the following and more:

• How data science and product management can work together to maximize their strengths
• How two different viewpoints on the same project can illuminate different, equally fascinating parts of it
• An unexpectedly powerful way to use data about first names

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

Natural language processing, or NLP, is one of the dominant forces in modern data science, and it’s produced a host of data science-powered products many people take for granted as a basic fact of life. It hasn’t always been so powerful or pervasive, though — NLP has a long and interesting history, and some of the advances powering today’s technology would have seemed like science fiction only decades ago. This month, we dive into the history and foundations of NLP, examining:

• Why natural languages are so difficult to work with in the first place
• Early attempts by mathematicians and data scientists to use natural languages, and why they failed
• What distinguishes today’s cutting edge models and allows them to succeed

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

We’ve discussed A/B testing multiple times on this podcast, for good reason. But there’s an important angle we have yet to cover: in the life of a researcher or marketer, there’s no such thing as an A/B test. There’s an entire system of A/B tests run for specific purposes over time. What is the best way to construct a system of A/B tests to help you learn, improve, and grow over time? How does that translate into tenets to hold while building software to help people run A/B tests? We’ve brought on three members of the data science team at Klaviyo, and you’ll hear about A/B tests in a variety of ways, including:

• Real data-driven trends observed by successful A/B testers on Klaviyo
• Why up-front thinking and vision translate into long-term success
• Why dad jokes might be far more powerful than you think

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

Using machine learning models to generate text, images, and other creative objects is, as they say, a bit of a hot topic right now. There are examples of models like this in action all across the internet and across different fields and disciplines. Today, we discuss one of those fields in more depth: marketing. In particular, the Klaviyo data science team recently released the Subject Line Assistant tool, which helps marketers craft better subject lines. We take a close look at that tool, how it works, and the thinking behind it to examine what it looks like to use AI to help a human write. We’ve brought on four experts from Klaviyo, and you’ll hear about subject lines from a variety of angles, including:

• What a subject line is, and why it’s arguably the most important part of an email
• What holds people back from writing great subject lines and how the team went about solving those problems
• How a specialized human-in-the-loop model for a highly specific context can look

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

Data privacy and security are huge and hugely important topics — in all likelihood, you already know a little about them if you’re reading this intro. But they are both crucial to any good data science work, and this month we explore the fundamentals of both topics: why data privacy and security are necessary to deliver the value you promise your customers, who they matter the most to, and how to build privacy and security into your own data science work. The panel includes some of the foremost experts on the topics at Klaviyo from data science, engineering, and security and risk governance, so you’ll get to hear about these topics from a variety of angles, including:

• How approaches to data privacy that seem intuitive can fail, and fail spectacularly
• The consequences of not taking privacy and security carefully enough
• How to make people actively want to work within the security environment you set up

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

This month, we focus on research — but specifically research that’s aimed at your customers, delivering the sort of insight they would try to glean by running experiments and analysis using their own data. In particular, we dive into two different case studies drawn from the recent topics explored by the Klaviyo data science team. You’ll hear about:

• Why customer-focused research can be some of your highest-impact work
• Whether or not to use emojis when you’re sending out an email
• How to react when you encounter surprising results in your research

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

Once again, as the new year starts, we begin by recapping the old. Instead of diving deep into a specific topic, I asked 7 members of the Klaviyo data science team to give their personal highlight for 2021 as a year in data science. You’ll hear about fascinating data science topics, including:

• How companies used domain knowledge to hyper-charge their ranking algorithms
• Powerful estimating methods that account for covariance
• How 2021 provided new opportunities — and pitfalls— for state-of-the-art experimental analysis techniques

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

You can study as much mathematical theory, invent as sophisticated a machine learning model, or write as clean production-ready code as you want — if you don’t make sure you’re solving the right problems to begin with, all that effort could be for nothing. It’s not a topic you learn about in most data science coursework, but understanding your end customer is a crucial part of being an effective data scientist. We spend this whole episode describing why and how to do great customer research. Topics include:

• Why customer research is such a big deal in the first place
• How talking with customers can drastically change your thinking
• How to run the perfect customer call

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

We’ve devoted quite a bit of time on this podcast to robust, carefully tuned, and vetted-in-a-thousand-ways solutions. This episode, we venture beyond the land of neatly trimmed hedges and into the unknown, where scrappy solutions may be the only ones that are feasible — or even possible. And we’ll hear about settings where a quick calculation on a napkin can be the difference between success and failure — including the biggest weekend of the ecommerce year. You’ll hear about all that and more, including:

• How to solve Fermi problems (and possibly get put on a watch list)
• When quick calculations can save hours of painstaking work
• How even the simplest math can help you prepare for the most complex engineering challenges of the year

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

This month, we talk with Shane Suazo, the founder of Plytrix Analytics, about using data science to drive efficient business growth. Shane and Plytrix work with Vital Proteins, and we dive deep into their story and highlight the places where using specific — and powerful — data science techniques helped accelerate a growth opportunity into a growth story. You’ll hear about all that and more, including:

• Establishing a single source of truth as a foundation for advanced analyses
• Preventing churn with minimal cost
• The most important advice for translating general data science techniques to the reality of a specific business

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

A question we frequently get asked is: what books should I read to be a better data scientist/machine learning engineer? This may not surprise you, but there isn’t just one answer — in fact, we spent an entire episode talking about three ways to level up your data science knowledge and skills. This month, we’re back with three more:

• One of the foremost foundational texts for understanding machine learning models in a statistical way
• A survey course for a broad variety of machine learning models, with the opportunity to go in depth on topics like deep learning
• A foundational text in designing and analyzing experiments — both in ideal scenarios and in cases where the standard assumptions aren’t met

We discuss the following books and courses in this episode:

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

This week, we talk with Ben Knox from Super Coffee and Gina Perrelli from Lunar Solar Group about using data science to motivate the growth of a business. No hypothetical business cases this week — Super Coffee is a real business with a real growth story, and we’re here to showcase the ways that they have partnered with Lunar Solar Group and used inquisitive problem-solving methods to answer questions core to Super Coffee’s business needs. You’ll hear about all that and more, including:

• How expert insight translates into valuable questions
• Dealing with findings that stumped even the experts
• The data science feature that has helped Super Coffee the most

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

We’ve talked about quite a few aspects of data science on this podcast, but one that’s perhaps conspicuously absent so far is running experiments on your product. It’s no secret that experiments provide extraordinarily high-quality data to help you make decisions, but it’s also no secret that you only get good experimental results if you run good experiments. You’ll hear about running a good experiment and more, including:

• How experimentation fits into the design cycle
• What sorts of changes can drive unexpectedly large growth
• How to understand and adapt to counterintuitive results

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

Most of us reading this writeup have probably had at least one interaction with a recruiter. Most of us reading this writeup probably don’t have a deep knowledge of recruiting — what recruiters do, how they help teams scale, and what the other 90% of the iceberg you don’t see as a candidate consists of. Recruiters are on the front lines of attracting talent and making sure that a team grows the right way, and this episode we talk about how to make sure that happens. You’ll hear about all that and more, including:

• Common misconceptions about recruiting
• The most difficult aspects of scaling a team
• Why some recruiters hate the one-page résumé

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

A question we frequently get asked is: what books should I read to be a better data scientist/machine learning engineer? This may not surprise you, but there isn’t just one answer — depending on the skills you have, your knowledge base, the point of your career that you’re in, and many other factors, there are many books you could read that will help you learn more. This month, we cover several ways to improve the skills you need to contribute to a data science team. You’ll hear about all that and more, including:

• Object-oriented programming, how to think about it practically, and how it can help anyone on a data science team
• The ethics of machine learning and AI, and why understanding AI ethics is one of your most powerful tools
• How Pac-Man delivers some of the most powerful data science insights of our time

Some more reading or viewing that we mention in this episode:

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

This is a math-heavier episode than usual — we’re going to dive into probabilistic distributions and talk about systems of estimators. Even if that’s not your background, though, you should still find this episode useful. That discussion is all based in something crucial to real-life businesses around the world: customer lifetime value, or CLV. What exactly does CLV tell you, how exactly is it calculated and predicted, and why exactly does it matter to your business? You’ll hear about all that and more, including:

• Statistical approaches to modeling customer behavior
• Difficulties that arise when customers don’t act exactly like they’re modeled
• How the humble Tungsten cube can teach us about the entire customer journey

Contact me

You’ve probably heard of benchmarks. You’ve probably even used them. But what exactly are benchmarks, how are they useful, and how can you go about building a system to make benchmarks in your own industry? You’ll hear about all that and more, including:

• How to use benchmarks to make informed decisions about improving your business
• Why the humble stoplight served as a key insight for making complex math understandable
• How to assess your personal levels of spice intake

Some more reading or viewing that we mention in this episode:

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

We have a bit of a different episode this month. Instead of diving deep into a specific topic, I asked 14 members of the Klaviyo data science team to give their personal highlight for 2020 as a year in data science. You’ll hear about a bunch of fascinating data science topics, including:

• Using machine learning to take a quantum leap in drug discovery
• Discovering methods from much older years that are still relevant for the problems of today
• How 2020 provided some new opportunities — and some sobering real-world stress tests — for data science

Full Episode Notes

See https://medium.com/klaviyo-data-science/klaviyo-data-science-podcast-ep-8-2020-a-data-science-year-in-review-88be9b534183.

Contact Me

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

All data science work at scale rests on a solid foundation of engineering. We discuss how to establish that foundation — from what goes into software engineering to begin with to the specifics of how to prepare for big seasonal events like Black Friday and Cyber Monday. You’ll hear from software engineers on the team about:

• Why building software is a lot like running a bar
• How to make things be — or at least seem — real-time
• How a tiny leak can crash a whole system

Welcome back to the Klaviyo Data Science podcast! This episode, we dive into…

As the calendar changes, so do the right steps to take for your e-commerce business. We wade into the waters of seasonal changes in behavior, data, and logistics, and we take a deeper look at how to navigate them. You’ll hear from data scientists and product analytics about:

• The fact that gardening is sometimes more powerful than the biggest holiday of the year
• Why you should stop worrying and love the error bars
• How to immortalize yourself as a data science meme

Full show notes are available at https://medium.com/@michael-lawson-96765/klaviyo-data-science-podcast-ep-6-navigating-seasonality-in-e-commerce-1bac11b8bf13.

In this episode, we take a deep dive into a recent feature the team built, signup form A/B testing, to give you a taste of what it’s like to build software for data science. You’ll hear from data scientists, product designers, and software engineers. We discuss:

• The reason multi-arm bandits are called multi-arm bandits
• How to distill a two-minute explanation into a single word
• The way people think about randomness, and what that means when you’re designing a feature that involves randomness

In this episode, we discuss how our careers in data science began, lessons we’ve learned along the way, and mistakes we’ve made and learned from. You can expect to hear:

• When we stopped wanting to be astronauts and started wanting to be data scientists
• Advice we’d give to anyone just starting in the field
• Advice we’d give to anyone currently at the helm of a dinosaur-based movie franchise

We mention a few books and other resources in the course of this episode. Check them out here: