The team matters even more than ever
The team plays a crucial role in the life of each product and having a good team behind it is essential for success. I was lucky to have a brilliant team of senior developers with years of experience in software engineering. Given that most team members were new to AI, we learned together.
AI and machine learning are marathons rather than sprints. As with most business initiatives, it’s helpful if you have people who can stick around for a while to see the initiative through. In my case, at least half of the projects that we worked on were part of the third horizon of innovation. You might spend months on something before seeing concrete A/B testing result numbers. Working in this environment requires a lot of motivation, energy, focus, and discipline.
I wouldn’t get into details about who should be on the AI team, but make sure that in addition to a full-stack product team definition, your engineering team members cover AI software engineering, data science, and data engineering roles. This allows the team to be more hands-on with AI. As a side note, depending on how much data you have and how good it is, it’s very likely that data engineering would become a big part of the team’s work, and should be well-staffed accordingly.
Integrate into company planning and goal setting processes
Before joining the AI initiative, I used to think that our AI team was working on the most pressing and vital tasks for the business, and there was no reason that anyone should usurp their time or disturb them from their core work. But as I learned later, our AI team was eager to expand their work beyond the proverbial laboratory walls and become a part of the overall company strategy. They just didn’t know how to plug-in.
To fix this, I started meeting with every initiative to review their OKRs, yearly plans, KPIs, and what challenges they have. I began capturing insights and ideas where AI could be applied to drive the business forward. We quickly found good business problems to approach, but after doing this for a few quarters, it became clear that this should be the foundation of the AI strategy moving forward. This process has become our AI transformation goal — a tool that stipulates how we build a bridge between team OKRs without AI and those with AI. I can’t share our proprietary business methods here, but we came up with a process that guides the selection of the most promising AI opportunities. It requires creation of a business design document with financial information, resourcing, and proposed execution plans for testing hypotheses via MVP and series of A/B experiments. For your organization, this process might look different. But I would strongly recommend introducing a structured way for AI opportunities discovery rather than bouncing around. It creates a higher likelihood of engaging a more extensive range of stakeholders and company leadership to secure interest in AI and make your continuous progress visible at every level. Ultimately, it’s all about communication.
Build mutual commitment and shared goals
Say you have a promising model to test drive but merely no space in the A/B testing calendar to give it a try. Or perhaps your AI project is blocked because the people responsible for providing you data are busy with something else. So many things could get out of control and go wrong.
Regardless of how your product organization is structured, whether you work in silos or everything is cross-integrated, AI often is prone to disruptions, challenges, and higher risks — unless you move slowly. I like to think that we (the AI team) operate as Growth Hacking team, which runs from area to area with a hunger for big-swing opportunities and moon shots. Unless you are a small startup or your company is optimized for innovations through disruption, you will need to invent your own way to integrate your AI team’s work into the product roadmap, testing calendar, access to domain experts, data ETL and real customers for every project you do. For us, formal project agreements and OKRs are the way we build consensus, split responsibilities, and pre-commit for AI projects. Make sure that both teams have the same desired business outcome and commit to the same mutual goal. I would also recommend that all teams involved explicitly declare how much team effort they are going to invest in this project. This should help you to better plan the roadmap and timing of the project.
An accurate model is as important as a good experiment design
When I first joined the AI team, they were already working on a model that classified incoming customer requests into two buckets to handle them differently. But the problem was that despite the decent model accuracy, there was a significant level of Type 1 errors (false-positive predictions), which made this new flow too expensive.
Interestingly, while we iterated on the model itself, we discovered that 80% of the false positives get eliminated after a few minutes as the request ages and changes its state. By applying this knowledge, we designed a different experiment where we used the same old model, but we changed the timing of the model application. This lead to the results we were seeking. The point of this story is that a good model is, of course, essential, but it is nevertheless important how and when it gets applied. By managing the context in creative ways, we smoothly navigated between constraints and better-controlled outcomes. Use data to guide your decisions.
Balance exploration and exploitation
You want the AI team to be successful, but based on my experience, two major non-technical issues could prevent AI from growing: either your team is very academic, or all your projects are long-term. Both issues prevent the AI team from demonstrating success and business potential. Don’t get me wrong; you should be scientific, and you should work on some hard problems that might take you a few quarters or even years. But by not delivering business results for a long time, you can harm the team’s reputation and faith in AI. Therefore, try to balance work on good theories and deliver testable product features by making them a reality. At first, not everything will work, and that’s okay. The goal should be to build, measure, learn, and iterate. This will create a meaningful feedback loop for the team and company leaders. And most importantly, it will strengthen the team’s AI muscles.
Along with strategic projects, you should be seeking easy, quick wins, low hanging fruit, or at least some forms of Proofs-of-Concept and MVPs to test out your hypothesis before spending significant resources on something that might not fly. Always choose wisely. Andrew Ng recommends the first project ideally to fit in 6–12 months. I found a sweet spot by allocating 50% of the team’s time on quick wins, 30% on the on-going hard but narrow problems to solve, and 20% on analytical research with deep dives where it’s necessary to examine data for a new business case. Your ratio might be different but try to have it explicitly defined so you know how much return you could promise today and how much you can invest in the future.
Good luck on your AI journey.
Many thanks to John Coleman, Andrey Boreyko, Pavel Surmenok, and Leah Shough for your help on this article.