The AI Playbook | A Conversation with Eric Siegel

Monica Manney Welcome back to UVA data points. I'm your host, Monica Manney, in his new book The AI playbook mastering the rare art of machine learning deployment, Eric Siegel offers a detailed strategy for how business professionals can launch machine learning projects providing both success stories where private industry got it right, as well as cautionary tales others can learn from. Siegel laid out the key findings of his book during a wide ranging conversation with Mark ruggiano, director of the University of Virginia's Collaboratory for Applied data science, a partnership between the school of data science and the Darden School of Business. And Michael Albert, an Assistant Professor of Business Administration at the Darden School. The discussion featuring three experts in business analytics takes an in depth look at the intersection of artificial intelligence, machine learning, business and leadership. So with that, here's Mark Ruggiano, Michael Albert and Eric Siegel. Mark Ruggiano I'm Mark Ruggiano. I am the head of a collaboration here at the University of Virginia between the Darden business school and school of data science. And our collaboration is focused on topics at the intersection of data science, and business, which is where I spent about two and a half decades in the corporate world, working in various capacities that utilize data analytics and technology, primarily in areas that many people would call marketing. I also teach at the Darden School, and here at the UVA School of data science, and so really excited to be talking about machine learning with these wonderful colleagues, Michael, I'm Michael Albert Michael Albert, I am an Assistant Professor of Business Administration at the Darden School of Business. And I specialize in teaching topics around data science, I teach data science, both to our Masters of Business Administration, students, and our Masters in Science and Business Analytics. My research focuses on the application of machine learning and artificial intelligence to economic problems, using things like machine learning to design new market structures, for example. So I'm excited to be here as well, Eric, Okay, Eric Siegel my turn. I'm Eric Siegel, I am a longtime independent consultant in machine learning for 20 years now. And I more recently spent a year as the inaugural bodily Bicentennial professor and analytics at UVA Darden, during which, and this was a this is I was the inaugural person in that position. So one year position, during which I had the pleasure of working with both mark and Michael. And a lot of the work that I did during that year, was the genesis of what's now become a book called The AI playbook, which is just shipping, virtually as we speak. And, and the topic of the book has to do with getting machine learning projects deployed, which is what I'm hoping will be a general theme of our conversation today. Mark Ruggiano That's awesome. And so, Eric, you you know, you mentioned machine learning projects, and with all the, you know, the headlines and hype around AI. I think the definitions are starting to get clouded in certainly in popular conversations, if not elsewhere. So what what do what do you mean by machine learning and and how can we, how can we use a set that as a baseline for our conversation here today? Eric Siegel Sure, I think M, ml, machine learning is well defined value proposition is well defined. In contrast to AI. Machine learning is learning from data to predict that sort of an actionable definition of it or prac, practical, applied definition. You know, it's the application of machine learning methods like decision trees, log linear regression, neural networks, ensembles, you learn from data to predict in order to target operations such as which individual customer to target for marketing, or inspect for fraud, or which word to generate. Next, in generative AI, which is built on machine learning how to change a pixel in the next iteration during the generation of a new image with generative AI. So whether it's at that level of detail or another it's in a fairly microscopic level, even in the business applications and and those are the more established ones. I suggest we we distinguish from generative AI, by calling it predictive AI or predictive analytics is one way to distinguish it. Those are the established use cases, determining whether each individual is going to click by lie or die committed an act of fraud or what have you, in order to improve large scale operations because those pro predictions directly inform the action, the treatment taken with that very individual whether to contact audit whether to approve for a credit card application, whether to drill for oil in this spot, because not only only human individuals, individual cases or situations, whether to check out the satellite might be at risk of running out of battery. So it's all those large scale operations that are made up of many micro decisions. That's, that's sort of the long standing established, proven track record of, of deployed enterprise machine learning, standing for decades. You know, that's still where most of the money is, and where most of the proven returns are generative AI, it's kind of like apples and oranges. That's a very exciting area. I fear, it's overhyped, but I'm also extremely excited about it. And that's where it's, you know, generative isn't a technical term. It's simply a reference to how we're using machine learning to generate new content items like writing and images. Mark Ruggiano Michael, what does that, Does that does that description of machine learning resonate with you? Michael Albert I think that's a, that's roughly in line with how I view the definition of machine learning and AI. I think, traditionally, AI has had the term AI has had a bigger focus on automated decision making, which I think is something that has gotten a little bit muddled, and and the kind of the popular usage, Eric pointed out, traditional machine learning has been more about prediction, though, I think even that's a little bit of a gray area, when you bring in things like reinforcement learning, which are certainly a subclass of machine learning. But traditional AI has often been concerned with, how do we go about doing kind of optimal search within a space? Or how do we go about making decisions based on uncertain data? Which is certainly related to machine learning, but but somewhat distinct. In the common, you know, discussion, I think that AI has primarily come to mean, generative AI, which maybe is a bit muddled for our conversation. So I appreciate the distinction that Eric made with generative AI versus machine learning. Mark Ruggiano Awesome. So, you know, my, my take on it is that, for the purposes of our conversation, and you know, exploring why machine learning projects aren't always as successful, or even failed to deploy, you know, the, the definition would exclude, you know, some of the things that we're reading in the headlines about artificial intelligence these days. What, what are what do you guys think about, you know, machine learning? It, you know, it's been a thing for quite a while, and as you pointed out, Michael, and Eric, but in my, you know, in my personal experience, there are many other things that have been around for a while and have gone through the cycles where, you know, it's a great new technology, it is expected and maybe even invested in to deliver, you know, great benefit. It doesn't do that, or doesn't live up to expectations at least, and people get disillusioned. And then we take a step back collectively and figure out okay, how do we actually create value? You know, with some of these tools and technologies, my personal, the deepest experience I have, and that is in a thing called CRM, which was, you know, the next best thing since sliced bread for, you know, half a decade or a decade, gave rise to all sorts of, you know, predictions about how amazing technology and data and data science, we're going to, you know, change the way we interacted with companies and customers, and then fell into, you know, what, what some call the trough of disillusionment. So, my two cents is that we've seen these things before we've seen this cycle. And we we ought to be able to do things differently, as you know, in education, as scholars, as business leaders, etc. What do you guys think? My Eric Siegel mission is to save the world from that disillusionment? And, and I'm here to do that right now. So I think that there's I think there's two layers of fog that are, are are preventing us from from capturing and realizing actual value in the successful deployment of machine learning. Right, which stands to improve, essentially, virtually all our existing large scale operational processes across industry sectors. And the two layers of fog are first, the whole AI hype thing. And yes, AI is largely just a reference to generative AI and many uses. But it means a lot of different things. I think that the misleading narrative and public these days is that we're actively headed towards Artificial General Intelligence computers capable of anything a person can do, including running a fortune 500 company, you can or onboard it, like a human employee and let it rip. Let's just call it what it is. It's an artificial human, I think that there's a prevalence of belief that we're actually headed in that direction, even within decades. And I think that's a misbelief. And I think it's misleading. And, for example, it means Hey, even if we only have 5% of it right now, whatever it is, that would be extremely valuable. So I think it's really important to quash that the second layer of fog though, even if you dispense with the sort of general mainstream hype, and you focus on so the an antidote to hype, is to focus on concrete value proposition I'm going to predict which cancels, excuse me, which customers are going to cancel in order to target a retention offer that it provides a discount or incentive to keep customers around one way or another. These predictive use cases are about triage prioritization, the allocation of limited resources or time and expenditure in order to run things more effectively numbers a business game, and this is the way to tip the odds bit more in your favor. Now, here's the thing, even if you've sort of cut to the chase, and you have a pretty specific value proposition, predict x in order to do Y predict customer churn to target offers predict fraud in order to target the use of an audit team or decide which transactions to automatically hold instantly. Whatever it is that that pair defines it, what's predicted and what's done about it. Yeah, actually need to get a lot more specific. So those projects even as concrete as potentially and, and potent as they are, they failed to deploy routinely, the majority of new machine learning initiatives do fail to achieve deployment and therefore don't realize value. And I believe that the problem is a lack of proper planning. And that has to be collaborative. From the get go. Between both the tech and the business, the the quants, the data scientists and the business stakeholders, the person in charge of large scale operation that's meant to be improved with the deployment of a model. Until now, there's been no established, standardized practice framework playbook paradigm that's well known to business professionals. But you need a very particular specialized business practice to run machine learning projects successfully through to deployment. Most business professionals don't even realize that let alone know the name of one. So what I've done with the book is I coined a five letter buzzword, and I'm here to evangelize it. And it's biz ml, bi Z ml, the business practice for running machine learning processes, because there needs to be a brand around this concept, a buzzword, something where it's going to gain traction, and therefore visibility and common understanding, we need a specialized customized practice for running these projects. And I think that, then we stand to actually realize on the promise of machine learning, whereas although there's lots of successful projects, now, the majority of enterprises are behind big tech and a handful of leaders, because there's this lack of universal understanding around the other half of each of these projects, the business part. That's what we need to get in place so that we can get it right now there's decades of track record of success, and also even more failures. But we can really, we're sort of only tap the tip of the iceberg, and we can really tap a lot more value. It is very practical and real. Machine learning is ready. It's not about technological improvements. It's about organizational ones. Michael Albert One of the challenges that that machine learning faces in a lot of organizations is that the skill set to traditionally the skill set to build machine learning models has been very kind of computer science tech heavy, a very statistics heavy, and the mindset to evaluate them, frankly, has been very economics heavy. And those two areas often don't really interact. Well. I, I would push back and I will come back to this later in the conversation. I'd push back a little bit on this idea that we don't have a framework. You know, I think that there's room to improve frameworks. But But I think there have been various iterations of frameworks and place, I just genuinely think it's really complicated. You know, even for example, in, you know, I think Eric mentioned the example of customer retention, trying to predict churn so that you can do some kind of intervention to avoid a customer leaving leaving your platform your product or whatever. The one of the problems with that is that you can predict which ones are likely to leave. And, and that's a well defined problem that, you know, we understand how to evaluate it, we can say strong statistical things about the quality of our solution. But once we add in the intervention, it becomes much more difficult to say, whether or not a model is provided value, for example, we probably already as a company, if the hypothetical company probably already has some kind of customer retention strategy in place, right? I'm thinking of, you know, Legacy cable providers, they already know that you have to call the table. Yeah, they already know you have to call them to, to cancel, and they're already going to offer various interventions when you call, right. If they built a model to predict churn, it's unclear that that would actually provide a lot of additional value, because they have this, this this existing layer, right? So you have to really view the quality of your solution, primarily as how does it change? The decision making process? Which is really a subtle question a lot of times, right. And I don't I'm not providing an answer here. I just genuinely don't know, how we, you know, really get machine learning teams, data science teams, and and the managers and other business units to really focus in on kind of this broad decision question. That's, that's what these models are feeding into. Mark Ruggiano Yeah, I think the the challenge is to continue the example that you that you brought up Michael, of the cable industry, which, you know, is perennially at the bottom of the list of, you know, industries as far as customer satisfaction, customer experience, you know, and other things, which is not news to anyone in this modern day and age. But to continue with that example, right. I think one of the challenges as a business person involved in that effort that you described, the hypothetical effort is to understand, while we could build models that, you know, very accurately predict who is likely to turn who's likely to leave, we can't accurately value all of the possibilities that we might do to keep them at least, you know, we could, but it would take an extraordinary amount of time, and money and other things that, you know, in a limited resource environment are likely better applied against other business opportunities. So, you know, that business person is put in a position of having to make a judgment, even with an extraordinarily well built and tested and, and, you know, proven model at their disposal. And I think that, you know, getting the data science team and function, and practice and discipline, more integrated with the business practitioners across the board, whether it's marketing operations, etc, is one way to bring that together, we can't make everybody an expert at everything. But we can't also put the full decision and all of its complexity on the shoulders of someone who is only an expert in one area. So how do we strike that right balance? I think that's one of the areas where education and management practice can improve. Eric Siegel Yeah, I mean, that's a great way of framing sort of what the issue is, how do you balance it? Not everyone can know everything? My assertion is that we need to get the business side stakeholders to ramp up on this very certain semi technical understanding, which consists of what's predicted how well what's done about it, as I mentioned, what's predicted and what's done about it sort of the way that those are the ingredients that define a particular use case, how well the second of those three is the metrics, and you have to get involved in a certain amount of detail. It's not the rocket science part. It's sort of understanding how to get value of it. So I'll circle back to my assertion that there needs to be an A well adopt a broadly adopted broadly understood business paradigm. I totally agree with Michael that there has been plenty of attempts at frameworks one in particular, that's well known called. Okay, well known amongst many senior data scientists called crisp DM. But my assertion a moment ago, just to clarify it, and I'll say it more emphatically, there's been no established standardized practice. That's well known to business professionals. And this is a business practices, the business professionals need to know it. And that way crisp DM, the by far the most successful one in the past, from 30 years ago, has the word data mining and its name has never been updated, failed, because it's not well known. And again, I think that even business stakeholders don't even don't even know that you need a very specialized particular framework, or practice in the first place, let alone the name of any particular one. So what I advocate for is, however, you break down the lifecycle and I break it down to six steps, from inception to deployment with the biz ml thing, but however you break it down, the more important thing, the bigger theme, is to get that deep collaboration and have the people on the business side of that collaboration, having had them ramp up on the semi technical understanding, and but I say, what's predicted how well what's done about it, I mean, to really concrete amount of detail, for example, not just who's going to buy or who's going to cancel, but which customers who've been around for, for four months, are going to decrease their spending by 80% in the next three months, and not increase their spending another channel, because that wouldn't count as a defection, etcetera, etcetera, all the caveats and qualifiers that are business relevant, maybe three times as many as those I just spat out to define that prediction goal, otherwise known as the dependent variable. And to get concrete so the examples you gave where there's this trickiness, a how do you know whether marketing worked? Right? Well, you should have a control set. But listen, what I'm advocating for it isn't necessarily sufficient. But I would argue strongly that it's necessary. We got to get them on the page and that concrete level of detail so they understand, okay, with targeting response model, you're predicting customer purchase in light of contact with churn modeling, typically, you're predicting customer defection, in light of no contact. Right. And. And, Michael, you know, you and I wrote a article together on that. Let's see, remind myself the title of it, I have it right here in analytics magazines run by informs to avoid wasting money on AI, business leaders need more AI acumen. And then the example we get into is the difference between targeting, marketing versus targeting sales, where with sales, you're not going to necessarily be controlling the treatment of customers in that data collection in the same way. So you have to be very specific for for targeting sales, and you might be predicting will the individual buy, given how the sales team has previously interacted with them. For as for marketing, you're predicting will the individual buy, if contacted with this particular marketing treatment? So those are the kinds of things we break down. And the problem we're identifying in that article that we co authored, is that lack of business understanding of the particulars there? Again, I'm not I don't know how to solve all the world's problems as much, as much as I facetiously said so upfront, but I am arguing that these are necessary, even if not completely sufficient conditions. Michael Albert Well, so I appreciate you bringing up the article. Because I think that in that article, kind of one of the things that we pinpoint, and this is a bit of a technical term is the distinction between kind of predictive machine learning and causal machine learning, you know, this, this idea that, that things might be correlated, and therefore you can predict, you know, that something based on whether or not something is correlated, but it's not, that's often at least orthogonal to what a business leader really cares about, right? I think a lot of times Eric Siegel you want to make people buy? Yeah. Ideally, you want to make you want to make people not do fraud. But that's not usually the objective, though. Those projects are just about how should we just how do we balance false positive and false negative rates, as far as blocking, you know, credit card payment transactions, so you don't have to worry about causality as much with those projects. Michael Albert And I think that we have seen huge successes in machine learning on projects where we don't need to worry about causality. Right. I think like, I think fraud is actually one of the best examples of machine learning applied to industry, right? Fraud detection for credit cards. It's probably it's maybe the biggest success story for the, you know, traditional machine learning. I think the problem is that there are actually relatively few questions that are as as nice and simple and businesses as like, you know what well defined as predict fraud and just decline the transaction. Right. And even that, I think, you know, there's subtleties involved, where you could imagine not just declining, but doing some other kinds of treatments that where you might start to get into, you know, concern about causality. But, but I think, you know, going back to this example of sales versus marketing, I worked with a company that had, they had they had data on their salespeople, and their salespeople, you know, they didn't have data on conversions for leads that were reached out to. And, and they wanted to build a predictive model to evaluate how likely a particular potential customer would be to convert. The problem was, is that the the model that they built, had encoded the behavior of the salespeople. And so there was no real way for them to, to alter business practices, without making their model completely invalid. They couldn't, they couldn't, you know, predict the counterfactual right. And I think that that's where most of the time, what we really care about, is we care about understanding what if I had done something different? And that's a really subtle question that I think people struggle with, I think, I think you might I teach a class on Data Science here at Darden. And I spend a lot of time we go into a lot of technical detail, we have extremely sharp students, by the end of the class, I still think most of the students struggle with this distinction. It's just, it's just challenging. I don't know if you guys have thoughts on ways to, to, you know, raise awareness of of kind of the limitations of traditional ml. And there are ways to get around it, including costly ways experimentation, things like that. But that's a that's a harder sell than, Oh, I can run this Python code and come out with a deployable model, right? Well, Eric Siegel I'll see this, I'll just make one quick comment, the CEO of her, Harrah's Casino is famous for saying he'll fire anyone who steals from the company or fails to use a control set. Mark Ruggiano Well done. It's, it's, uh, you know, whereas in Caesars are great examples of, you know, of a lot of this sort of enterprise transformation that, you know, perhaps underlies the more effective, you know, development and deployment of machine learning. And so I'm glad, Eric, that, you know, that you bring that you bring that one up. But, you know, to take a half step back, you know, when we talk about, you know, the difference between sales and marketing as one, you know, one example of how, you know, business practitioners thinking needs to, you know, advance in terms of their understanding of how best, you know, to integrate to incorporate machine learning. And, you know, on the opposite end of that spectrum, how a data scientists, even a senior one, you know, their understanding of sales and marketing needs to advance it, that the contemporary practice of, you know, thinking about sales and marketing together, is to not think about them as distinct fields, distinct activities, distinct motions, in, in a corporation, right, a sales outreach is just a different channel. And it can be broken down into all of the same constituent parts that any other marketing interaction could be broken down into. And so, you know, what, what is different is the fact that a human being delivers the sales interaction, right, and we can't quantify every aspect of that behavior, and every impact that that potential customization may have on the outcome of the interaction, whereas a digital, it's Eric Siegel called a, it's called a control group. And it's much easier to control machines that do marketing than humans to do sales. Mark Ruggiano And that's a great example right to a business person. You know, having a control group or a holdout on your, from your prospect list is throwing money down throwing opportunity, perhaps down the drain, right, why would I not call on, you know, 10 or 20% of my prospect list? Because you need a control group? Right, Mr. or Ms. Data Scientist? Yeah. And that's, that's what's Beto come up with another alternative, come up with another way we can do Eric Siegel this. Yeah. Wha wha wha, exactly, Michael Albert I mean, there are the you know, I think in the realm of kind of academic research, there are alternatives right, there are ways in which with careful measurement and some kind of understanding the situation. We can draw causal diagrams and use, you know, kind of advanced tools to estimate these causal effects without control groups, assuming isn't a variation among some dimension. But I would say that, you know, there's still a ton to get wrong, right. And it requires a really a really subtle look at the situation. And control groups make that all easy. It's not strictly necessary. But the article that Eric referenced, I went back to the company after seeing what their data look like. And I suggested they, they, you know, they collect some more data around the behavior of their sales team, in order to do this in order to try to tease out these causal effects without having without having to run these costly experiments. But data collection is itself costly, and they were not ever they never were able to actually collect the data needed. So, you know, I think, you know, it is you have to have buy in from the beginning of the project, right, you have to have people who, who say, I'm really going to I understand enough about the issues. To understand why you need this data, I believe in the final outcome, even though I don't know what the data is going to say. Right. So like, whatever the final outcome might be that it turns out this problem is unpredictable. You know, we we can't we build our best model, and, and we've got really no statistical power here. But you got to make all that that commitment investment upfront before, you know, people start just building predictive models that really aren't are well suited to the, to the decision. Problem. Eric Siegel I sounds like we all agree that it's harder with sales and marketing. But if it's marketing, and you have a control group, isn't it reasonably straightforward to at least establish? how well your marketing campaign did? Michael Albert Oh, yeah, I agree with that. But again, the cost is that you're, you know, you're, you're burning some some subset of potential customers, right. And that with our control group, that's a cell you've got to make to, you know, who's ever deciding whether it's okay. Right. And Mark Ruggiano I would add, Michael, I think, for that person who, you know, is deciding whether it's okay. Right, that person, that individual needs a deeper understanding of, you know, what was done to demonstrate, you know, the, the effect that either they're looking for, and not finding, or the effect that they're trying to agree just happened, right, they're trying to confirm? And I think one of the things we're we're saying here is that, you know, there are steps that, you know, leaders and education leaders in the corporate world, etc, can take, you know, to improve their ability to do so. And, you know, I think one of those is, is certainly, you know, for the, for the business leader in question, you know, who's, who's being asked the question, you just post, they need to understand the methodology, the technique, and the interpretation of those, in order to make a well informed business decision. And most business leaders, in my experience are not yet fully equipped to do that. Let Michael Albert me let me argue something slightly from the other side, right. I mean, we could expect, you know, the C suites of modern companies to become, you know, amateur, you know, machine learning engineers. My experience, that's, that's a pretty heavy lift. The other side of it, and I think that is something that, you know, the certainly, I'm working on, with the students that I that I teach, and I think the school of data science is certainly very much engaged in, is training our data scientists to be better at communicating the value, to be better at thinking about the big picture to be and to be more winsome, and how they ultimately deliver the results to the stakeholders. This is, you know, this puts the burden on on the data scientist, you know, where they have to be really clear about, you know, when they, when they're presenting the outcomes of their, you know, prototype model to their, to their, you know, CTO or whoever else, they've got to do a lot of education, they've got to help them understand what the limitations of modeling are, what the challenges around you know, making good decisions with a model or, and so I guess there's a there's a question, you know, do we think that it's, it's likely to be more successful kind of going kind of top down and building up this, this bridge or going bottom up and building up this bridge? Eric Siegel Can you spell that out for me? What are the what What's the metaphor? Yeah, so Michael Albert I think we've been, we've been talking a lot about how how, you know, decision makers in business need to, you know, tech up, right, they need to be come more aware of what machine learning models are doing, what they're not doing. My I agree with that, I just also know from experience, that that's not an easy task, right? Somebody who's well established in their career, like, it is actually quite challenging to you know, that, well, some of the concepts to us all seem very simple, you know, to lots of people, these are very difficult concepts to cross, right. And so, the, the alternative would be to put the onus of, of, you know, in, of communicate of understanding and communication on on the data scientist, right, that they, they have to come to, you know, the decision maker with a compelling demonstration of value, right, that they have a easy to understand view of the trade offs and challenges associated with implementing these models. And so the question is, do we go bottom up by training the data scientists to be a more effective communicator to the decision makers? Would we go top down and try to train the decision makers to be more effective data scientists? Eric Siegel Yeah, I mean, I did you see the Indiana Jones movie, I mean, people on both sides of the chasm have to do something to get across that, like both sides have to participate. I think that the I think the whoever selling the project in that sometime is the data scientists, these are really does have a responsibility to take a more business side role. I mean, everything you just said, I totally agree with. And I think it's a little bit of an uphill battle, often to get data scientists to recognize the need for them to take a more business oriented role, expand outside their cubicle and take responsibility on that level for the communication and for guiding the project in that way, in a meaningful way. Especially if you're selling in the first place. One of the failure stories I tell in the book is my own. Early in my consulting practice, I had a big successful online dating site. And I said, Hey, churn modeling, you've got these premium paying customers. Look at how many this is high churn, so many coming in and leaving every month. You know, if you just can predict which ones are defecting, you can target retention and potentially have a very effective, you know, financially effective method that really increases your lifetime values and all those kinds. So I did the that. And they were like, cool. That sounds great, right? I mean, they were so flushed with cash. And all of a sudden, as as a brand new independent consultant, I was getting three times the hourly than I sort of even imagined. And I was like, This is great. I'm doing churn modeling, I'm actually using this stuff in the real world. For real, we're real problem. And then of course, the power gets stuck in PowerPoint, right? That's the traditional thing. So I present it to them. And I'm, like, done, didn't die. And I show a clip of the decision tree, and here's all the, you know, the projected returns if you target a retention campaign, of course, when you project the returns, you're a lot more speculative for targeting churn modeling, than for targeting response modeling. Namely, for the reason I mentioned earlier, you're predicting what would happen in in lieu of contact rather than in light of contact? And that makes a big difference, because then you have to assume Okay, well, even if you contact the right person, someone who was destined to leave, what are the chances you'll change their mind? And if you do, how much longer are they going to stick her out? Right? So there's all these assumptions without a control set? Right? And well, so I presented it to them, and they're like, cool, you want us to do something about it, you want us to start a new initiative and new marketing campaign, right? They were too busy. I hadn't sold the project properly. I'd sold the number crunching intrinsically. They didn't get there's an operational deployment part of it. Michael Albert So Eric, I'd like to just ask a quick question, because I thought you're going a different way, the story. It seems like churn is the goal of a dating site. Right? Like, you know, but you should all be the hope. Eric Siegel No, this was this was actually let's just be completely transparent. You can read about in the book, this was gay.com, which at the time was a male hookup site. So it wasn't about finding it wasn't about true love forever. It was just it was an online service. And besides, even if even if your more traditional online dating site, my father was a psychiatrist, and we would always joke about you want to cure your patients, but not entirely, right. You're gonna lose their business. Yeah, Michael Albert well, I mean, there's one. I had a student who worked with a, in his in his job, he worked with a data science consultancy. They were big mining company. And this data science consultant came back and said, you could improve your operational efficiency by making sure that you don't run the mining trucks empty half the time. And when they told him this, everybody laughed at them, because they're a mining company, right? You've got a big pit here that you're mining stuff out of, you drive the empty trucks down the pit, and then you drive the full trucks out. And so there's a, you know, it was it was an example that stuck out to me as a particularly egregious one of, you know, of not not seeing full the full context. But I agree that the, in your particular example with the dating website, yeah, it's churn modeling is a perfectly valid thing to do there. Mark Ruggiano Yeah. And how would you and Eric, how would you, you know, to generalize that, right, how would someone in the position you were in at the beginning of that story? Know, the business well enough to be able, even if you were so inclined, you know, to modify your proposal, you know, to account for the fact that this is a site where people don't stick around by design? And, you know, to me that that reinforces the need, whether it's driven top up, top down, or bottom up? We haven't we haven't dug into that yet. But either way, it reinforces the need for, you know, that sort of common understanding to grow. Right, the data scientists knowing more about business, and vice versa. Eric, Michael Albert I would love to, for you to run down kind of why you think that you know, Bismil is likely to be a, you know, a widespread playbook for AI, right, like what is what has been the metal have that but the crisp DM that you referenced earlier? Like, how does it how is it likely to succeed without as failed? Well, Eric Siegel first of all, because ML is specific for machine learning crisp DM is sort of data mining in general, which is subjective same as data science, basically, number crunching, using number using data in a meaningful way. Whereas for machine learning, in particular, where you have to define the dependent variable, that is to say exactly what's being predicted. And then in the deployment, you're actually going to integrate those predictions. So they change existing operations. That means driving business processes with probabilities, right, that's really what it comes down to. Those are very particular things and those are in are weaved through the six steps. I mean, the the six step breakdown is pretty straightforward. And I'll outline it, but let me just say, what really needs to happen right now is the packaging. So it's done in a way that is understandable, relevant and interesting to business stakeholders, which includes, amongst other things, a really nice five letter buzzword biz ml. And, and, and then therefore, we can put on the radar of the broader public that are touched by or even involved in any way with machine learning projects, which is only growing that this population, the need for particular business practice, just the need for it, the knowledge that there is one that's being generally adopted. And maybe even more importantly, that you need to get business side stakeholders involved in that level of detail, not the sparkplug under the hood, but how to drive a car momentum friction rules of the road and mutual expectations of drivers. That's a lot of expertise to drive a car, same level of semi technical understanding, for business side stakeholders to run machine learning project. So the understanding that they need to ramp up on what's predicted how well what's done about it to get into the details, if they don't get their hands dirty, their feet will get cold, right, get into those details in the process from end to end of the project. That's the important thing right now, the break down into six steps, some do five, some to eight, I didn't include monitoring or maintaining. Refreshing afterwards. I mean, it's included in the book, but in terms of the six steps, culminating with let's at least get the thing deployed and make it plausible that it could be deployed. The six steps break down in a way that to most data scientists, when you think it through is clearly quite obvious and is along the lines of what people already sort of formalize it as the main thing is to now convey that break down to business stakeholders. And anyway, the first three correspond to those three semi technicals facets, what's predicted how well what's done about it, but not in that order. So predict a deployment goal, which is what's predicted was done about it. I'm sorry, the first step is to establish the deployment goal, which is what's predicted and what's done about it. That pair that defines the use case the value proposition. The second is to get more specific about the what's predicted part, the flesh out the full definition, semi technical details of the dependent variable, the thing that you're predicting, and then the third is the metrics, which have to not only be technical but business metrics. acts like a way to forecast the actual returns or profit of the operation or the operational improvement. So those would sort of pre production. And then the other three steps are what everybody knows, if you're a data scientist, which is prep the data, train the model and deploy it. So what this is really about, it's not so much those six steps, but hey, let's, let's bring the other side of the company into those six steps. Unknown Speaker I think, you know, Eric, I think the six steps and biz ml, I think, are a great proposal. And I, I am hopeful that the adoption, you know, is there in the practitioner and academic communities. And to connect it back to, you know, what, what Michael asked us a few minutes ago to wrestle with, you know, how do we get all parties in, you know, an enterprise on the same page, and where that page, you know, contains more insight and understanding about the business and about machine learning. These six steps biz ml, is a process are a process that ensures that those stakeholders come together, and realize when each one or the other don't know, enough, you know, to proceed, and they can be educated by, you know, the person proposing the project or, you know, the the vendor or the team internally that, you know, is bringing it forward. But by having a set of steps like this, like this ml, we're able to have a built in sequencing of interactions among all the stakeholders on a team on a project in an enterprise that need to come together. And those interactions will point out where the gaps are in knowledge and understanding if people enter into them, you know, in an open and collaborative, right. Eric Siegel Yes, hallelujah. Right. That's what I'm saying. That's what I mean, by deep collaboration on end to end on each of the steps. Exactly. And I Mark Ruggiano think that to me, that, you know, that bridges, the bridges, the the gap, crosses the chasm, you know, solves solves the problem in the Indiana Jones movie of, you know, top down versus bottom up how to bring the two sides together, they are together, because they are all bought into this set of activities that you know that you've that you've christened busy now, I'm Michael Albert curious, a number that's often thrown around like 90% of machine learning models are never deployed, or, you know, 90%, machine learning projects or whatever, never deployed. I'm curious. And your guys view is that 90% driven by it's certainly some of both, but is that 90% primarily driven by data scientists making useless models? This will be you know, similar to the, to the sales example that we talked about where this company had built a model. That was useless. And they didn't realize it was useless, but it was useless. Or is it they the stakeholders, being unable to evaluate whether or not a model is likely to be valuable? Eric Siegel I think it's more the latter. I mean, I think that I mean, but they kind of go hand in hand, right? If the business stakeholders aren't guiding, then the then the data scientist isn't making, they haven't defined the dependent variable, specifically enough aligned with the exact deployment plan, but the deployment plans, concept is moot because it wasn't in itself wasn't fleshed out enough to have business stakeholders really understand, hey, I'm gonna actually change my largest scale operations in a substantive way with probabilities. So by the way, though, that 90%, I kind of hear 80% thrown around more, but that was actually one of the three main prongs of my work, during my analytics professorship at Darden. Was Was industry research. And I, I hooked up with RXR analytics that does a data science, a big data scientist survey, kind of convinced them to add questions about deployment success rates, and the results were that only 22% of data scientists say, new analytics initiatives meant to drive new capabilities usually deploy in the sense that 80% or more deploy, so it's not like how many of your models deploy, it's how many data scientists say at least 80% Deploy, it breaks down like that, you know, we've written it up, it's on KT nuggets, across all initiatives, not just sort of new capabilities, the way we defined it. They're only four it was more but only 43%. That said that 80% or more. I'm sorry. So let me correct that. 43% actually say 80%, or more fail. So and by the way, this failure rates that we're talking about if you try to estimate that into what it is, as far as the number of models that could be deployed that aren't, are actually sim similar with just digital transformation initiatives in general, like there's a cross a lot of other kinds of technology. There's there's a lot of reports with similar difficulties. But my assertion is for machine learning, in particular, you need to get into those kinds of details about what does it mean to drive? I mean, ultimately, what does it mean to drive operations with probabilities? What do you need to predict that to that? and probability of what of what is the dependent variable? What data do you need? Right? So those details and getting the business stakeholders in there, I think that it's a lack of deployment, sort of like, the question is, is the business unwilling to or can't because they need more ml ops infrastructure or more data pipeline infrastructure? Either way, it's a lack of planning. So I see it as a business organizational problem that can be addressed by better planning. Mark Ruggiano Yeah, in my in my last corporate role before joining UVA, I was responsible for a team, a team of modelers and other folks. And I would say that we didn't, I didn't actually, you know, Tally this up. But I would say that less than 10% of the modeling projects that the team had undertaken over, you know, my tenure, there were an active deployment. And, you know, I think, you know, the survey type responses that, you know, that we hear, including the ones you mentioned, Eric, are our points in time. And obviously, if we obviously, if we aggregate those, you know, over some period of time, we can probably come up with any percentage of failure that we want, depending on how we, you know, how we combine those numbers. But my team, you know, they didn't do that our projects did not deploy for a couple of reasons. One of those, I don't think we've spent a lot of time talking about here today. That is, they weren't designed for broad deployment, right? We, we have a team, many organizations have teams that do a lot of exploratory work, right to understand and probe, you know, where the opportunity exists for the business or to demonstrate feasibility, you know, to narrow down the approaches that should be invested in, you know, to address a problem. And, you know, when Yeah, Eric Siegel just to clarify, we did we did in the question posed to data scientists, of the models you developed with the intention to to, to deploy Mark Ruggiano Yeah, exactly that that qualification is a very important one, as is the work that, you know, a data science team might be doing on things that are not intended to deploy, right? Those, those are oftentimes very, very valuable as well. But the second reason, I think, you you touched on this, that my team, you know, had projects that didn't deploy, is the failure to, you know, for the organization to recognize that this wasn't just a model, this was a transformation of how we did something major, right? If it's going to have major impact, it has to influence something very significant that the company is doing. And, you know, to think that a group of people are going to sit, you know, in a lab like environment for some number of months and come up with something that completely changes the fortunes of the company. Right, without having to get everyone else involved, is a fundamental misunderstanding. Right? And so the the main reason what's a common one? Oh, yeah, absolutely. And, you know, it was the reason why a number of our project, despite our best efforts, didn't go forward because people loved the possibility. And were turned off by the work that would be required to realize it. Eric Siegel Well, this is the best technology. And I only mean that half tongue in cheek, right. It's the why probably you guys too, I'd say most techies myself, I got involved in this because the core technology, the idea of learning from limited number of examples to draw general generalizations that hold that pan out in new, unseen situations. And then in that sense, the computer is automatically learning is the best kind of science and technology is the coolest, it's the most fascinating and potent and interesting. And the Harvard Business Review calls it the most important general purpose technology of the of the century or something like that. So we're using the best technology of course, it's valuable and that misconception I call it the ML fallacy in the book, that misconception that it's intrinsically valuable rather than the value, in fact only coming when you act on it, when you deploy it, when you use it, that's sort of like we're, we're because we're fetishizing the technology, we're in love with the technology, it's like being more excited about the rocket science than the launch of the rocket. Mark Ruggiano Interesting. So what what do you guys think we, you know, we'll see, you know, if, if everyone listens to this podcast, as I know, they will do, and takes the lessons that we're sharing and reads the book and takes those lessons home to, you know, their educational, you know, institution to their cooperation, etc. Machine learning and the way we use it and experience it today, regardless of what side you're on is going to change. Right? What do you think? What do you guys think the future is for machine learning, and for the people who are engaged in it as users, as developers, or anywhere in between? Eric Siegel I mean, I think Michael made a great point that in on a certain level, we can't hope for any sort of clear cut Silver Bullet where the really intractable challenge is getting, especially when control sets are needed, and what it means to define them. Experimental design, where that's, that's so hard, you know, what I'm advocating for, of having a common language understanding brand on the need for the practice calling it biz ml. Again, honestly, I mean, honestly, I see what I'm advocating for as necessary. And yet, I have the humility to say, not necessarily sufficient, per Michael's concerns, a big part of those concerns, by the way, kind of come from the fact that with a lot, a majority of these projects, were using found data write sort of like longitudinal studies, rather than controlled experiments, or any experimental design. And that's sort of what the all the excitement about the big data movement was as you you're collecting these, this data, residually, it's like a side effect of conducting business, all the transactions are getting logged, it's experienced from which to learn. Let's leverage it. So the big and big data, you know, there's we're going to run out of adjectives, big, bigger, biggest, because data's grow so quickly, but with the real, what that word big is about the excitement, the potential value of found data, data that's already been collected anyway. But when you do that, you don't have a control set, you deal with a lack of causality. And then when you go to certain applications, like marketing, it can be really tricky, really tricky, especially if there's a refusal to have a control set on the deployment. So that is, that is an unsolved problem. You know, I worked hard to scope out a book that's, you know, not much longer than 260 pages, right. So that if you see it in the airport bookstore, it's not going to look too thick for your next flight. Which means that although I covered a lot of, of ramp up for business readers, on what I'm saying is semi technical understanding. I don't get into control sets in that book at all I do in my first book. Michael Albert I mean, one of the things that I'm that I think we're seeing, and I have mixed opinions about this is the democratization of data science. You know, historically, the, the technical skills required to use these tools have been relatively relatively substantial. Now, frankly, I can with a chart, GBT Plus subscription, I can go armed with a CSV file, and as chatty with you to write me Python code, and then run the Python code and show me graphs and give me all sorts of, you know, fairly good from a technical perspective models, based on this data. It's very easy, frankly, I think what we're going to see, as we're gonna see more and more people engaged in the process of building predictive models. It's, which is good on many, in many ways, but I think that, you know, the, the flip side of having a significant technical barrier to entry to building these kind of models, is that you also had generally developed statistical skills and, and some sense of experience and perspective that can help evaluate what you what you've produced. And I you know, and I think that this, this newfound access to these very easy to use kind of modeling tools and you know, not not including things like auto ml, though, that's another discussion that can build these really accurate predictive models with almost no effort. It's, I suspect, we're gonna see a lot of bad modeling out there. Right. I think that you know, I think we're gonna see a lot of people, you know, run with with these machine learning models, because they are these want this wonderful technology without understanding the context in which they're going to be deployed. Ai I think that's Eric Siegel your say, when you say you have mixed feelings about democratization of data science, do you really mean that? Or do you mean that you're mostly have a name? Because I would say I mostly negative, I'm more like, Oh, gee, you know, you do need some expertise. And just to be clear, when I say semi technical ramp up and collaboration, I'm saying that's a necessity on the business side. But you still need the data scientist, human? No, I Michael Albert actually say I have an ambiguous feelings about mixed feelings about it. I think that there's a lot of opportunities for people to ask questions that, you know, the typical data scientists just would never think about, right, because that's not their functional unit. There's a lot of opportunity within an organization for people to experiment. I think the, you know, if there's a if there's a gate, if there's a gate before deployment, unfortunately, I'm not sure that especially in smaller organizations, I'm not sure that there will necessarily be this gate before deployment. But it doesn't get before deployment, I think it could, it could lead to kind of a creative flourishing within the kind of machine learning space and a lot of organizations. Whether or not that happens isn't is an open question. But I mean, that's why I'm, I have mixed feelings. I think democratization of tools, the, you know, is is generally a good thing. It's just that there's, you know, these, these powerful tools need to be wielded carefully. Eric Siegel Yeah, indeed. And the ability and abilities chat TPT to make the Python code doesn't really introduce a new element to that maybe increases the prevalence, but there's already been long standing paid commercial, sort of super user friendly tools, I always call them PhD tools, push here, dummy. Mark Ruggiano Yeah, on, on that top on this topic, you know, I'm probably a little bit more extreme than, than either of you, I think that the, the arc that we have seen in many technologies, is the same one that, you know, will happen that is happening here. And, you know, if you think that not needing to be able to rebuild the engine on your car, you know, is a good thing, then, you know, not needing to start from a blank screen and write your first line of code, or not needing to start from, you know, a statistical theory and figure out how to implement it, those are also good things, right. But to the same, you know, on the on the same, by the same token, right, we It doesn't mean that the car engine doesn't need maintenance, it doesn't mean that someone shouldn't have the expertise to make sure that the theory is translated into a tool with high fidelity, right, or that the output is meaningful and used appropriately. Right, but I think we're going to see many of those safeguards, also built into the same sets of tools, right, in the same way that some of the commercial tools today warn you about problems with your data set, or issues that would make one technique more appropriate than another, right. And people who might not be able to discern those things themselves, you know, can see the warning, and either figure out what to do themselves, or they can bring in an expert, I think those are the types of things that are on the horizon, right, where the expertise that is resonant in a data scientist who was trained, a trained professional, can be applied at the top of the data scientists license so to speak, right and not have that data scientist be doing the kinds of things that will become are becoming everyday things. And that again, that that push and pull to me, is something that's embodied in and will be, will happen. If both sides, all sides, all stakeholders are following a process, a set of steps, you know, that that are most appropriate for these projects. And I think, you know, biz ml offers the opportunity for that. Michael Albert So I think the example of a car is interesting, because we do have driver's test before we let you drive your car, right? You do have to understand something about a car, you have to understand how it operated, you have to understand what makes it safe. What makes it dangerous, right? You have to you have to demonstrate to somebody that you can use it with some with some care. Back to this article that, that Eric and I wrote, As organizations working with, I went to them and I explained to them that their model, you know, I was the gate right? That their model did not learn what they thought it had learned that predictive power did not translate into value here. And they need to do additional data collection. And the response that I got was that if we don't build this model, marketing is going to build it because we're very focused on ml and We want to control it. And so we're building it. And as far as I know, that model is deployed right now. And, and so there's, I think, Mark Ruggiano sounds like a bad client. Michael Albert You know, I did, I did my part, I explained to them very clearly that this was not the this model should not be deployed, I went, I went, you know, to the boss of the person who I was directly working with, at this organization to explain this, and it still got pushed through. And, and so I think there is, there is a danger here, right, that, that these things, none of the tools will ever, you know, data is agnostic to causality, right. And, and you can't understand the context purely from any any given data set, right. So it's going to be impossible for any of these automated tools, any of these auto ml tools or charge up to you or anybody else, without the business context to ever provide a safeguard, right? Because it's just not it's not even theoretically possible. And so, you know, I, that's where my, my concern comes in. I agree that, that needing needing to know how to write Python code is unnecessary. Just like needing to know how to rebuild your engine is not necessary. For everyone, everyone, for everyone, for everyone. Right. But I would say that the, that some element of these building, and particularly deploying these models is actually more akin to driving the car, where it is important that we evaluate somebody's capability of doing that. Mark Ruggiano I would agree. And so, you know, like I said, I think I'm a bit more, you know, out there on that issue than, you know, the neither of the two of you are, but I would certainly agree that where we are today, in 2024, is not a place where, you know, all the risks have been addressed. And so there are absolutely challenges and risks. And, you know, I think in our environment at at UVA, and, you know, in, in other contexts, you know, where people are reading your book, Eric, you know, there are a lot of business risks that are, you know, growing because these tools, and these techniques are more accessible. But I'm an optimist. And I think that through education, training literacy efforts, by better by putting in better processes that bring together data science experts and domain experts in business and other fields, in, you know, an organized way, like we've talked about the six steps we've talked about here today, I think, you know, I'm an optimist, that those steps will ameliorate the risks and begin to, you know, unlock some of the power of machine learning and AI more broadly. Eric Siegel I agree, I'm optimistic, I think that Michaels story should be, should be qualified, because that means there's always going to be organizations, especially smaller ones, in this case, right? Where they act on superstition, right, they're like, they also hire a clairvoyant, and they avoid stepping on cracks in the sidewalk, and they think that's going to help their business. Right, you can't, there's always going to be situations where they just don't listen to the expert. But I think that we're moving more towards data driven culture and the understanding of, of that expertise. I think that we need to make sure that the technical side of the projects are, are conducted in a more sound manner. But that the first more fundamental, generally missing pieces a lack of a business, an utter lack of a business side process. Mark Ruggiano Interesting. So my, my final comment is, is, you know, see if you guys see if you guys agree, but we've mentioned a few things, you know, throughout this conversation, that, you know, even in our, our programs here at UVA, and in other, you know, training and development initiatives, even in the companies that, you know, that I've been a part of, are just not front and center, we've mentioned the difference are the level of difficulty difference in collecting data that is specifically and most relevantly applicable to a particular challenge versus using data that's been collected over time for some other reason or for no reason at all. We mentioned, you know, increasing the literacy in experimentation, and causation. We, we've talked about, you know, the importance of ethics and safeguard ethical use and application of, you know, data science and machine learning and these tools. And we've talked about the driver's license concept where, you know, we, we have a hard time today, in business and elsewhere. are agreeing on, you know, who is a data scientist? Who is a machine learning engineer? And What qualifications do you have to have in order to be called one? Right? And without that you can't you can't have a driver's license. There's no such thing if there isn't a driving test. And so, you know, we have those those challenges, and I throw those out there to you, for you to guys, because I think those are challenges that, you know, that we need to be at the forefront of tackling whether it's through books, or education or in other means. Eric Siegel Yeah, I'm, I feel like that's a burning question that I'm to this day, agnostic about, you know, how should it be regulated or form? Formal? I mean, there's actuarial standards for an actuary. Right, but not for data science. And I see pros and cons. I don't know who's going to do that. How, you know, and how it's executed, obviously, would be important. Michael Albert You've got me I mean, more people should sign up for advanced degrees at UVA. That's probably the the answer to it. Mark Ruggiano Is there anything else that we that we need to talk about here? Or have we solved all of the problems that that we can in one discussion of of the AI playbook? Well, I Eric Siegel think we've definitely asked a lot of the right questions, which is is often the hardest part. Why don't I mentioned that I'm going to be back on the grounds it may 10, providing a keynote at the knowledge continuum, which is hosted by UVA CMIT, the Center for the management of it. Awesome. Mark Ruggiano Well, good to chat with you guys. It's It's always fascinating to hear your thinking on some of these, some of these complex and very timely topics. So thanks for the opportunity. Eric Siegel Yeah, thank you both, I think both of you that I think that's a great discussion a lot, a lot of great food for thought. Monica Manney Thanks for checking out this week's episode. If you'd like to learn more about this topic, we recommend checking out Eric's new book, The AI playbook, which is on shelves starting February 6 2024. If you're enjoying UVA data points, be sure to give us a rating and review wherever you listen to podcasts. We'll be back soon with another conversation about the world of data science. We'll see you next time.