I had the opportunity this past week to spend time with Ram Chakravarti, the CTO of BMC. In our conversation, some of which will air in an upcoming video, we had a wide-ranging conversation around how the world of observability and operations is evolving in the era of AI. We discussed how, in today’s digital era, where technology landscapes are continuously shifting and expanding, the principles and applications of observability have emerged as pivotal elements in ensuring the smooth functioning and governance of IT infrastructures.
This conversation came hot on the heels of a recent announcement by Broadcom of its WatchTower solution, which I covered here. Broadcom’s WatchTower platform, announced at the SHARE event, is set to revolutionize mainframe observability by enhancing business performance and operational efficiency through improved incident resolution and visibility across IT domains. This platform and the industry’s move toward open telemetry signify a shift in the directions of more transparent, interoperable IT operations and a proactive approach to mainframe management. These advancements promise a new era of mainframe observability aligned with modern digital transformation strategies. These two moments crystallized the trend I am seeing in steady-state operations.
This significance is magnified as IT operations teams extend their purview beyond the initial phases of application deployment, venturing into comprehensive, steady-state operations that demand constant vigilance and adaptability, especially when the systems in question hold the crown jewels of the company’s data. The evolution of observability from its origins as a relatively straightforward monitoring utility into a complex, multifaceted discipline marks a significant transformation in how IT ecosystems are managed and optimized. Observability now serves not merely as an adjunct tool but as the very spine of daily IT operational activities, offering insights and foresights that were previously unattainable.
This deeper exploration into the world of observability will traverse the intricate journey of its evolution, shedding light on how it has become an indispensable asset within IT operations. The narrative of observability is one of growth and sophistication, evolving to meet the demands of increasingly complex IT environments. It encompasses a rich tapestry of techniques and methodologies, from monitoring and logging to tracing and anomaly detection, each contributing to a comprehensive understanding of system health and performance. This evolution reflects a broader trend toward more proactive and intelligent IT management strategies, leveraging data to anticipate issues and optimize system performance proactively.
Moreover, the integration of observability into IT Service Management (ITSM) frameworks has catalyzed a paradigm shift in how IT services are delivered, managed, and improved. This synergy enhances the ability of IT teams to respond with agility to incidents and issues, supported by deep, data-driven insights into system behavior and performance. By embedding observability into ITSM processes, organizations can achieve a more responsive, efficient, and resilient IT service delivery model, one that is capable of adapting to the dynamic demands of the business and its customers.
The advent of AI has further revolutionized the field of observability, particularly in the domain of log monitoring. The application of AI technologies has transformed log analysis from a manual, time-consuming process into an automated, insightful practice capable of parsing vast quantities of data to identify trends, anomalies, and predictive indicators. This innovation has paved the way for the emergence of AI for IT operations (AIOps), a groundbreaking approach that melds AI and machine learning with IT operations, heralding a new era of efficiency, accuracy, and predictive capability in IT management.
In this transformative landscape, several companies stand at the forefront, shaping the future of steady-state operations through their pioneering contributions. Splunk, with its robust platform for searching, monitoring, analyzing, and visualizing machine-generated data, has become synonymous with operational intelligence and agility. ServiceNow, through its comprehensive ITSM and IT operations management (ITOM) solutions, has redefined how organizations manage and deliver IT services, integrating observability into the fabric of ITSM. BMC, known for its innovative IT solutions that bridge the gap between traditional IT management and modern AIOps, empowers businesses to navigate the complexities of digital transformation and a company that arguably covers the most diverse technology set from mainframe to the cloud. Lastly, PagerDuty, with its focus on digital operations management, leverages AI to optimize incident response processes, enhancing the speed and effectiveness with which IT teams can address and resolve issues. While I call out these companies, numerous others with proven technologies and innovative roadmaps also occupy this pace.
As we delve into the nuances of observability’s evolution and its pivotal role in shaping the future of IT operations, it becomes clear that observability is not just a component of the IT management toolkit; it is a fundamental paradigm that underpins the agility, resilience, and efficiency of modern IT infrastructures. The contributions of leading companies in this space not only illuminate the path forward but also underscore the critical importance of observability in steering steady-state operations toward a future characterized by unprecedented operational excellence and innovation.
Deeper Dive into the Evolution of Observability
In its infancy, observability was synonymous with basic monitoring, primarily focused on tracking the uptime and performance metrics. This initial approach was somewhat effective for simpler IT infrastructures but quickly became obsolete as technological landscapes grew in complexity and scale. The advent of cloud computing, distributed systems, and microservices demanded a more sophisticated approach to observability—one that could provide a comprehensive view of IT systems.
Modern observability transcends traditional boundaries, offering a holistic view of the system’s health by amalgamating metrics, logs, traces, and telemetry data. This rich dataset provides IT operations teams with the insights needed to not only troubleshoot issues but to also understand the intricate behaviors and interactions within their systems. It’s a fundamental shift toward a more proactive and predictive management philosophy, ensuring operational resilience and efficiency in today’s dynamic IT environments.
The Synergy between Observability and ITSM
The fusion of observability with ITSM frameworks marks a significant advancement in how IT services are delivered and managed. This integration enables a seamless flow of data and insights from observability tools into ITSM processes, enhancing the ability to manage incidents, changes, and service requests with unprecedented agility and accuracy. Automated incident reporting, powered by observability data, accelerates the identification and resolution of issues, minimizing their impact on service delivery. Moreover, the data-driven insights gained from observability tools enrich ITSM decision-making processes, fostering a culture of continuous improvement and innovation.
Revolutionizing Log Monitoring with Artificial Intelligence
The application of AI to log monitoring is revolutionizing the field, shifting the paradigm from manual, rule-based analysis to dynamic, AI-driven insights. Traditional log monitoring methods are increasingly inadequate for navigating the complexities of modern IT systems. AI and machine learning algorithms can process and analyze vast volumes of log data in real-time, identifying anomalies and predicting issues before they escalate. This predictive capability not only enhances operational efficiency but also opens new vistas for proactive system maintenance and optimization, heralding a new age of intelligence-driven IT operations.
Embracing the Future with AIOps
AIOps stands at the forefront of this new era, embodying the integration of AI technologies into IT operations. It signifies a move away from reactive management toward a model where operations are predictive, proactive, and automated. AIOps leverages big data, machine learning, and other AI technologies to streamline the detection, diagnosis, and resolution of IT issues. This shift not only reduces the operational overhead for IT teams but also aligns IT operations more closely with business objectives, ensuring that IT infrastructures are not just stable and efficient but are also dynamically aligned with the evolving needs of the business.
What’s Next?
Companies like Splunk, ServiceNow, BMC, and PagerDuty, among numerous others, are at the vanguard of the transition toward advanced observability and AIOps. Splunk’s data-to-everything platform exemplifies how deep insights into system data can drive operational intelligence, offering a comprehensive view of the IT landscape. ServiceNow’s integration of observability into its ITSM and ITOM solutions is transforming how services are managed, enabling more agile and efficient workflows. BMC’s solutions blend traditional ITSM excellence with the cutting-edge capabilities of AIOps, automating complex operations and ensuring peak system performance and the breadth of its supported platforms from mainframe to cloud. The work Broadcom is doing to bring Open Telemetry to the mainframe to revolutionize how mainframes interact with the rest of the hybrid cloud and PagerDuty’s focus on digital operations management with AI at its core streamlines incident response, setting new standards for operational speed and efficiency – all standout to me as indicators of where the industry is heading.
Looking Ahead: The Future Is Observability-Driven
The transformation of observability from its initial status as a peripheral monitoring tool to its current position as an indispensable element of IT operations is emblematic of a significant paradigm shift within the IT industry. This evolution reflects a move toward more sophisticated, intelligent, automated, and predictive approaches to IT management. Observability’s journey is intertwined with the broader narrative of IT operations’ evolution, which has expanded to cover not just the deployment but the entire lifecycle of applications and services. This comprehensive approach necessitates a seamless integration of observability, ITSM, and AI, transitioning from a nice-to-have to an absolute necessity for modern IT operations.
In this nuanced landscape, observability emerges as more than a tool; it becomes the very lens through which IT operations are viewed and managed, providing a clear, continuous, and comprehensive view of the IT ecosystem. It enables IT teams to not only react to incidents and issues but to anticipate them, leveraging vast amounts of data to predict potential system failures or bottlenecks before they occur. This predictive capability is crucial in a world where downtime or performance issues can have immediate and significant impacts on business operations and customer satisfaction.
Furthermore, the integration of AI with observability tools has catalyzed a leap forward in operational capabilities, automating routine tasks, and uncovering insights that would be impossible for human operators to detect amidst the noise of everyday data. AI’s role in this context is transformative, offering the potential to automate the analysis of data from various sources, identify patterns, and even recommend or initiate corrective actions without human intervention. This automation extends beyond mere efficiency, touching on the ability of IT operations to remain resilient and agile in the face of evolving challenges and workloads.
Leading companies in the observability and IT management space are not just participants in this evolution; they are its vanguards, driving innovation and redefining what is possible. These organizations are crafting the tools and platforms that underpin the future of IT operations, pushing the boundaries of technology to offer solutions that are increasingly sophisticated, yet simpler to manage. Their contributions are pivotal in shaping how businesses understand and leverage their IT infrastructures, transforming complex, dynamic systems into comprehensible, manageable, and scalable assets.
As the role of IT operations continues to expand and evolve, the centrality of observability to its success cannot be overstated. It is the foundation upon which the future of IT operations is being built—a future characterized by resilience, efficiency, and agility. Observability empowers organizations to navigate the complexities of modern IT environments, ensuring that they can not only withstand the challenges of today but also adapt and thrive in the face of those tomorrow. In this context, observability is not just a component of IT operations; it is the very bedrock upon which the next generation of IT management practices is being established, heralding a new era of operational excellence and strategic insight.
Disclosure: The Futurum Group is a research and advisory firm that engages or has engaged in research, analysis, and advisory services with many technology companies, including those mentioned in this article. The author does not hold any equity positions with any company mentioned in this article.
Analysis and opinions expressed herein are specific to the analyst individually and data and other information that might have been provided for validation, not those of The Futurum Group as a whole.
Other Insights from The Futurum Group:
Changing Observability in the Mainframe Ecosystem with OpenTelemetry
Key Takeaways from Splunk’s Q4 and FY 2024 Earnings Report
Author Information
Regarded as a luminary at the intersection of technology and business transformation, Steven Dickens is the Vice President and Practice Leader for Hybrid Cloud, Infrastructure, and Operations at The Futurum Group. With a distinguished track record as a Forbes contributor and a ranking among the Top 10 Analysts by ARInsights, Steven's unique vantage point enables him to chart the nexus between emergent technologies and disruptive innovation, offering unparalleled insights for global enterprises.
Steven's expertise spans a broad spectrum of technologies that drive modern enterprises. Notable among these are open source, hybrid cloud, mission-critical infrastructure, cryptocurrencies, blockchain, and FinTech innovation. His work is foundational in aligning the strategic imperatives of C-suite executives with the practical needs of end users and technology practitioners, serving as a catalyst for optimizing the return on technology investments.
Over the years, Steven has been an integral part of industry behemoths including Broadcom, Hewlett Packard Enterprise (HPE), and IBM. His exceptional ability to pioneer multi-hundred-million-dollar products and to lead global sales teams with revenues in the same echelon has consistently demonstrated his capability for high-impact leadership.
Steven serves as a thought leader in various technology consortiums. He was a founding board member and former Chairperson of the Open Mainframe Project, under the aegis of the Linux Foundation. His role as a Board Advisor continues to shape the advocacy for open source implementations of mainframe technologies.
