I’ve seen some “advanced analytics transformations” succeed and many others fail. It’s well known that most of these transformations end in failure, wasting huge amounts of time and money. And what is worse, these failures typically sour the organization on data science and machine learning for the future. So failures in advanced analytics transformations cause longer-term damage than is typically appreciated.
I was recently challenged by a colleague of mine to write down what I thought our business would look like in five years if our own analytics transformation was successful. A few items I wrote were typical: we’d have the ability to deploy machine learning models into production quickly; we’d have data science embedded across the org chart, and so on. But what I quickly realized was that although all of those goals are good ones, the real difference I’d like to see in five years is not technical, but cultural. We should be aiming first and foremost at cultural change. I now believe that this is the right way to think about how to add data science and machine learning to your business.
Scientific culture
Think about how typical business decisions are made. In a well-run business, experts are consulted and possible solutions to business problems are debated in an open and collaborative way. In those discussions, there is a lot of conventional wisdom and many assumptions about potential risks and rewards. Decision makers adopt a “measure twice, cut once” approach that is risk-averse and intended to commit to a reasonable course of action over the long run.
In contrast, think about how science is done. I happen to have worked alongside some excellent researchers in my academic career at universities and in the national laboratory system. An effective scientist, in my experience, prioritizes generating and testing hypotheses over commitment to a solution. The most brilliant researchers don’t adopt a “measure twice, cut once” mentality. Instead, they generate many hypotheses and they focus on testing them rapidly. If this sounds like a startup mentality, that’s because it is. The best methodologies in the tech startup world, in my opinion, are successful because they’re scientific.
Let’s consider an example. Suppose your business wants to do a better job reaching high-value customers. A common, but unscientific, approach is to bring in a team of consultants who will develop “personas” of your customers over the course of weeks or months. Some of those personas will be of more desirable customers. The assumptions built into the “desirable” personas are used to develop marketing campaigns. It is understood that the company is committing over the long term to using those personas to segment its customers and measure performance.
If this story seems natural, and you can easily imagine it happening in your business, then you don’t have a scientific culture. No scientist worth their salt would be willing to commit to a theory in such a manner. A scientific approach would be to come up with lots of hypotheses and a wide variety of potential personas. To a good scientist, those hypotheses could come from anywhere. They could be conventional wisdom, or they could have occurred to someone in a fever dream. They might be the result of some exploratory analysis, or not. But they are all provisional and tentative. All the hypotheses would be parked somewhere until they’ve been tested. Discussion quickly turns from generating hypotheses to the more important question of how to test them.
Cultural change motivates other changes
Of course, this is a simplistic view of what a scientific culture looks like. But the main elements are there, the most important of which is a focus on generating and testing hypotheses.
When my colleague asked me to describe what our organization would look like after our analytics transformation, this sort of scientific culture is what I described. I will judge our transformation as successful if the default way to make decisions is scientific. And most importantly, this is not limited to the technical workers. We’ll only be successful if this mentality is adopted across the entire org chart.
All the technical and organizational goals of an analytics transformation fall out of this overarching goal. For example, it’s unfair to ask your marketing team to generate and test hypotheses if they don’t have access to data and the ability to rapidly roll out marketing campaigns. You can’t ask your designers to be scientific if you don’t provide the ability to A-B test their designs.
Adopting a scientific culture entails all the technical and organizational changes that we’re used to hearing about, and it places those changes into context, making them understandable. For example, it’s commonly suggested that an organization should integrate its data science team with the other teams across the business. This is perfectly true, but why should we do this?
A common but inadequate answer is that the data scientists need to understand the business context of their work, and the other people need to be able to take advantage of data science expertise. This is true, so far as it goes. But it misses the real point. You won’t get any benefits if the team doesn’t adopt a scientific mindset. If they don’t change their methods to emphasize the quick generation and testing of hypotheses, the business will not enjoy the benefits that data science has to offer. With this in mind, there’s a new reason for integrating data science into other teams: to infect teams with the data-driven, scientific mindset that you (hopefully) have within the data science team.
When we think about the goal of creating a scientific culture, we can see some of the more subtle changes that have to occur. The most important of these changes, in my opinion, is that incentives have to be realigned. Returning to our previous example, if we’re developing customer personas, we have to reward people for testing their ideas and gathering data. They’ll be doing a good job if they incrementally improve these personas over time based on measurements. Contrast that with the usual way of doing things: People would be rewarded for rolling out marketing campaigns based on polished and detailed descriptions of customer personas — a distinctly unscientific approach.
The bottom line
Most analytics transformations fail because a new data science team is simply bolted onto the organization without being given the support it needs. In other cases, the data science team gets the right level of technical and non-technical support, but the effort still fails because of a lack of understanding of the business. But the transformation can also fail even when it seems like the business has made all the changes necessary for success.
In my experience, this latter kind of failure is especially mysterious and frustrating for everyone. The failure seems inexplicable because it feels like the business did everything right. But thinking about your analytics transformation in terms of effecting a cultural change helps put into focus the rest of the changes that the business is undergoing. All the rest, including organizational, technological, and other changes should be understood as means toward the end of creating a more scientific culture.
Zac Ernst is Head of Data Science at insurance tech startup Clearcover.
Credit: Google News
