Lyft open sources Flyte tool for managing machine learning workflows

Ride-sharing company Lyft Inc. has open sourced a new debugging tool for artificial intelligence data that its pricing, locations, estimate time of arrivals, mapping, and self-driving developer teams have been using in-house for the last three years.

The tool is called Flyte, and is described by Lyft as a “structured and distributed platform for concurrent, scalable and maintainable machine learning workflows.”

“With data now being a primary asset for companies, executing large-scale compute jobs is critical to the business, but problematic from an operational standpoint,” Lyft engineers Allyson Gale and Ketan Umare wrote in a blog post on Medium. “Scaling, monitoring, and managing compute clusters becomes a burden on each product team, slowing down iteration and subsequently product innovation. Moreover, these workflows often have complex data dependencies. Flyte’s mission is to increase development velocity for machine learning and data processing by abstracting this overhead.”

Gale and Umare explained that Flyte is a “multi-tenant” service, which means teams can work on separate data repositories and employ them without affecting the rest of the platform. Software code is versioned and containerized with its dependencies to ensure all executions are still reproducible. Developers are able to invoke different parameters, or variables internal to AI models that define its skill relating to a particular problem. Flyte can also use cached outputs from previous model executions to save time and memory.

Flyte is also capable of linking tasks together and passing data between them via a Python-based domain-specific programming language. Further, because every entity in Flyte is immutable, with each change captured as a new version, it’s possible to iterate or rollback workflows quickly, and share versioned tasks across workflows.

“Flyte is built to power and accelerate machine learning and data orchestration at the scale required by modern products, companies, and applications,” the company said. “Together, Lyft and Flyte have grown to see the massive advantage a modern processing platform provides, and we hope that in open sourcing Flyte you too can reap the benefits.”

Image: Lyft