Observability is among the buzzwords of our times. Unlike many buzzwords, it’s there for a reason: To describe something new, or the evolution of an existing notion. The term observability was introduced by none other than Twitter, to describe the evolution of what was previously called monitoring.
The four pillars of Twitter’s Observability Engineering team charter were monitoring, alerting/visualization, distributed systems tracing infrastructure, and log aggregation/analytics. So, more than just monitoring, indeed. It’s about setting up the infrastructure to collect, analyze and digest granular data on applications and infrastructure.
Perhaps one of the most telling examples of the shift towards observability comes by taking note of the evolution of New Relic. New Relic, one of the key players in what was once called APM (Application Performance Monitoring), is trying to evolve to keep with the times. Its messaging is evolving, too, and it’s now all about observability.
Open source observability? Enter Grafana
An incumbent like New Relic is actually a good yardstick to use to measure up the observability landscape. When covering New Relic’s new platform, we also wondered about the feasibility of using open source solutions for observability. It’s a natural question to ask: Since open source has disrupted IT everywhere else, why not observability?
The answers given by many people on Quora were along the lines that yes, you could do that, but it would end up costing you dearly. These are somewhat predictable answers, and not just because many of the people who provided them are affiliated with (non-open-source) vendors in this space.
It’s a well-known property of open-source software: It can be free as in speech, but not necessarily as in beer. Refining, integrating, customizing, and battle-testing open-source software is hard work. Of course, there are good reasons why people want to use open-source, so somebody has to do the hard work, which is why open-source vendors proliferate, and enterprise vendors go open source, too.
Observability is data problem, too. Image: Grafana Labs
Grafana Labs wants to be the vendor to build an open-source-based observability platform. Grafana Labs have just closed $24 million in Series A Funding to double down on open-source strategy and build what it dubs the world’s first open and composable observability platform. ZDNet discussed this interesting development with Grafana Labs co-founder and CEO, Raj Dutt.
Dutt said Grafana Labs has come a long way since it was founded in 2014. He emphasized that it built a sustainable business, and while it didn’t need to fundraise, it took the opportunity to inject cash in the organization to accelerate its investments in the global open-source community, and the Grafana Labs platform.
Grafana Labs is built around Grafana, an open-source analytics platform that allows users to query, visualize, alert on, and understand metrics no matter where they are stored. It also enables users to create, explore, and share dashboards with their team. Grafana has nearly 1K contributors and many adopters, including the likes of NetApp and Wix.
Grafana Labs offers Grafana Enterprise, a commercial version of Grafana, as well as Grafana Cloud, a managed cloud version of the software. Grafana Labs contributes heavily to Grafana and shapes its direction. But probably the most interesting part of Grafana and the one that showcases how Grafana Labs people think and operate is the Grafana Labs ecosystem.
Composability is Grafana’s difference
Grafana, much like other observability platforms, relies on integrating data from various data sources. The difference is in how Grafana does it, said Dutt: “Every other observability vendor is essentially a database. We are not.”
Dutt went on to explain that while all other observability platforms ingest data in their own underlying back end store, Grafana does not. What it does is connects to data sources, get the data, and display them or uses them for alerts and other purposes. It may be a subtle difference, but it’s an important one:
“Most vendors are trying to sell you an ‘everything in my closed database’ approach. By unifying your existing data, wherever it lives, we deliver unprecedented insights, choice, and flexibility. Grafana Labs offers the only composable and open observability platform to solve the complex mess of vendors and silos,” said Dutt.
Indeed, there is an entire ecosystem created around Grafana under the auspices of Grafana Labs. Grafana Labs has integrated solutions like Prometheus, Loki, Cortex, Graphite, and Metrictank, enabling Grafana to connect to and manage data from sources like Kubernetes, log files, and more. These solutions are open source, and Grafana Labs contributes to them, too.
What Grafana Labs means by composability is the ability to integrate data from a variety of data sources. Image: Grafana Labs
But there’s more. Grafana Labs also offers official data sources for AWS Stackdriver, GCP Cloudwatch, and Microsoft Azure, so users can graph custom metrics alongside the data available from your cloud provider. Other integrations include sources like Apache Kafka and Cassandra, MySQL, Oracle, Postgres, and Splunk.
Also, AppDynamics, Datadog, Dynatrace, Elastic, and New Relic. Confused? We were a bit. But this is what Dutt referred to as “composability” — being able to use data coming from any other source, including competitors like New Relic. Not all of these integrations are open source though, and this is part of the Grafana Labs strategy.
Enterprise data sources are only available in Grafana Enterprise and Grafana Cloud. And those are different than open-source Grafana, said Dutt: “We are not pretending these are open source — they are not. While Grafana is available under an Apache 2.0 license, those are closed source, different branches of our software, although common ground does exist.”
This helps keep Grafana Labs away from open-source licensing woes, and it also enables them to heap praises on the competition. Open source observability has made strides in the last five years, and the best observability platforms today are open source, said Dutt, citing for example, Elastic and Prometheus. Not coincidentally, both platforms Grafana Labs connects to.
The best observability platforms today are open source?
Of course, this meta approach sounds great, but there are also the flip sides. Not storing data means your platform is more lightweight and your storage bill may be reduced, but it also means data needs to be fetched on the fly, which may impact performance.
But how does Grafana Labs compare to a solution like New Relic? At this point, the New Relics of the world seem to have more integrations, and more advanced features like AIops, predictive capabilities based on machine learning. Can Grafana Labs close the gap?
As far as integrations go, Dutt emphasized the open-source community aspect. What typically happens, he said, is that when a previously unsupported data source is in demand, someone in the community will start working on it. Then Grafana Labs will pick up on this and contribute, sometimes bringing original creators on board, to the point where it becomes production-ready.
Grafana is rising. As to whether Grafana Labs will rise as an observability platform to question the status quo, too, check back again in a few months. Image: Grafana Labs
As far as advanced features go, Dutt said their initial journey in 2020 will be about assembling the building blocks of metrics, logs, and traces. During the second half of 2020, they will be looking to deepen the level of insight, and will explore adding predictive capabilities. This will likely be done through a variety of techniques including statistical analysis and machine learning.
So. to be fair, to say that the best observability platforms today are open source is probably a stretch, at least as far as features go. That said, however, open-source, a composable architecture, and a healthy dose of funding can go a long way pretty fast.
The first outcomes of Grafana Labs $24 million Series A funding, led by Lightspeed Venture Partners and with participation from Lead Edge, are already visible in terms of recruiting. Grafana Labs have recruited new executives with experience from the likes of Zendesk and Dynatrace.
When commenting that these hires seem to be spot on for the growth that Grafana Labs wants to achieve, Dutt agreed, going on to add, in a playful tone, we should ask again in a few months. While always true for new hires, perhaps more importantly, we should check again in a few months to see how much of Grafana Labs plans and projected traction will have materialized.
Credit: Zdnet
