IT Teams Are At the Start of a Long-Term Strategy to Deploy Cloud-Native Observability

Written by Gregg Ostrowski, executive CTO, Cisco AppDynamics

The surge in cloud computing initiatives over the past two years has resulted in soaring complexity and an increasingly fragmented and dynamic IT estate. In particular, rapid adoption of cloud-native technologies, such as microservices and Kubernetes, over the last two years, are presenting a major challenge for technologists, who are all too aware of the need to deliver brilliant digital experiences to customers and employees at all times.

Against this backdrop, technologists now recognise the need for visibility across their cloud environments and consequently, the importance for a new generation of solutions to monitor IT availability and performance in a software-defined, cloud environment where everything is changing in real-time.

Scale becomes critical — these highly distributed systems rely on thousands of containers and spawn a massive volume of metrics, events, logs, and traces (MELT) telemetry every second. And as it stands, most technologists simply don’t have a way to cut through this vast data volume and noise when troubleshooting application availability and performance problems caused by infrastructure-related issues that span across hybrid environments.

This is why, more and more, technologists are looking to move past domain monitoring and to achieve full-stack observability, where they can quickly transform availability and performance data, from up and down their IT stack, into unified and actionable insight based on business context.

In fact, a recent AppDynamics report, The Journey to Observability, reveals that 55% of businesses in the United Arab Emirates (UAE) have now started the transition to full-stack observability, and a further 38% plan to do so during 2022.

The importance of cloud-native observability
The key driver for this soaring appetite for full-stack observability is the fact that technologists are now acutely aware that most traditional monitoring solutions aren’t designed to handle dynamic and highly volatile cloud-native environments. Traditional approaches to availability and performance were often based on legacy physical or virtualised infrastructure. Flashback 10 years, and IT departments operated a fixed number of servers and network wires — they were dealing with constants and fixed dashboards for each layer of the IT stack. The introduction of cloud computing added a new level of complexity and as a consequence, organisations found themselves continually scaling up and down their use of IT, based on real-time business needs.

With traditional APM solutions, it can be difficult for technologists to generate real-time visibility into applications and underlying infrastructure for large, managed Kubernetes environments running on public clouds. What IT teams now need is a new generation, cloud-native observability solution, to provide visibility into highly dynamic and complex cloud native applications and technology stack.

In order for technologists to be able to properly understand how their applications are behaving, they need visibility across the application level, into the supporting digital services (such as Kubernetes) and into the underlying infrastructure-as-code (IaC) services (such as compute, server, database and network) they leverage from their cloud providers.

Importantly, technologists should be looking to implement a purpose-built solution which can observe distributed and dynamic cloud-native applications. Traditional monitoring solutions continue to play a vital role — and will do so for years to come — but it becomes problematic when cloud functionality is bolted onto existing monitoring and APM solutions.

Too often, when new use cases are added to existing solutions, data remains disconnected and siloed, forcing users to jump from view to view to try to identify the root causes of performance issues. Very few of these solutions provide complete visibility, for example insight into business metrics or security performance, and many are naturally biased towards a particular layer of the IT stack depending on their legacy, whether that is the application or core infrastructure.

The transition to full-stack observability is a marathon, not a sprint
However, despite a clear pivot towards cloud observability, it’s worth remembering that the shift to cloud computing is a multi-year strategy for a vast majority of companies. Every organisation is charting its own unique digital transformation journey, at its own speed and on its own scale.

We all know how challenging digital transformation is, particularly within large enterprises where it can take time to implement change, whether that is a complete rip-and-replace of legacy technology, expanding their technology stack through cloud services or bringing in new thinking and ways of working. While some organisations are progressing rapidly along their transformation journey, others are still taking their first tentative steps.

The reality is that most enterprise businesses are still running the majority of their core application portfolio within traditional environments. And whilst they may have intentions to make the shift to cloud-native environments over the next few years, they’re not there yet. Within many organisations, traditional monitoring and APM solutions are still performing a vital function, enabling technologists to identify issues and take appropriate action within specific domains. And for many, this will continue to be the case for some time to come.

This is why it’s important for technologists to be thinking about both their current needs and their future needs. They need to work with a partner that can meet them where they’re currently at and then help them to transition to a cloud-native solution in a gradual and seamless way, when the time is right for them. That could be in two years or 15 years.

There is no doubt that full-stack observability has become a critical part of every major digital transformation initiative. It is key to ensure that organisations are able to deliver on their objectives and maximise return on investment. It is now incumbent on technologists to develop an observability strategy that can flex to support their organisation as it progresses on its transformation journey, and to work with a partner that can deliver the right observability solutions and the right support and guidance along the way.