Shreya Shankar — Operationalizing Machine Learning - Gradient Dissent  ## Metadata - Author: **Gradient Dissent** - Full Title: Shreya Shankar — Operationalizing Machine Learning - Category: #podcasts - URL: https://share.snipd.com/episode/d63530e9-7cd6-4ae1-9f41-3310ca9d1766 ## Highlights - The Four Stages of a Successful ML Deployment Key takeaways: (* Practitioners interviewed for the podcast identified four stages in their workflow around experimentation, evaluation, deployment monitoring, and response., * The velocity of a practitioner's experimentation and deployment process is important, as is the ability to validate early on in the process.) Transcript: Speaker 1 And so we interviewed around 20 practitioners and the criteria was that they have worked on or are working on a model that's being used in production. So basically it's serving some predictions or some output that customers are using. And somebody will get an alert if the system breaks, like that's kind of our definition of production. And we interviewed people across company sizes and across applications like self-driving cars, banking, whatsoever. And we found, we looked for common patterns across people's interviews. We found four kind of high level stages of their workflow around experimentation. Like the evaluation and deployment monitoring and response and then data collection, which wasn't often performed by the ML engineers that we interviewed, but it was like a critical part of the production ML pipeline. So we identified these four components or these four stages and then we also identified kind of what are the kind of variables that govern how successful their deployments will be. Like what are the things to think about whenever evaluating tools to use in each of these stages? How do I know if I'm on the right track to a successful deployment? And we found that the velocity really matters a lot, the ability to validate as early as possible matters a lot. You don't want to like push a bad model to production. You don't want to wait until your final stage of like A.B. Testing in order to find out that something is not going to work well. ([Time 0:09:50](https://share.snipd.com/snip/5a99c388-bd63-498f-9cd3-cc3dd292740e))