Original source: Outshift by Cisco
This video from Outshift by Cisco covered a lot of ground. 3 segments stood out as worth your time. Everything below links directly to the timestamp in the original video.
As AI systems become more complex and interconnected, understanding their behavior is critical. This initiative could set the industry standard for ensuring AI transparency and reliability.
Cisco Outshift Drives AI Agent Observability Standards with OpenTelemetry and Agency Collective
Cisco Outshift is collaborating with OpenTelemetry to establish a unified schema for observing agentic AI systems, aiming to address the current lack of a standardized model for understanding interactions between AI agents. The initiative involves open-sourcing foundational components to provide end-to-end visibility and comprehensive evaluation capabilities, moving AI applications from experimental stages to production readiness.
This effort is part of the broader Agency open-source collective, launched over a year ago by Cisco Outshift with partners like Google, Dell, Red Hat, and Oracle. The consortium has grown to over 80 partners, focusing on building an "infra layer" with basic foundational elements such as protocols, discovery, identity, observability, and evaluation, designed to accelerate the widespread adoption and industrial emergence of agentic AI applications.
"We have decided to collaborate with the other stakeholders in OpenTelemetry to accelerate the process of defining a rich unified schema."
Cisco Outshift Advances AI Observability with Semantic Analysis and Temporal Knowledge Graphs
Cisco Outshift is developing advanced AI observability solutions that move beyond basic data collection to integrate with enterprise platforms like Splunk. The strategy involves structuring raw data using temporal knowledge graphs, which model the time-dependent behavior of AI applications, enabling deeper analytical insights. This progression aims to characterize AI behavior beyond individual session evaluations by incorporating semantic and cognitive layers.
This approach emphasizes the critical importance of semantic understanding in agentic AI systems, especially those interacting with users through natural language. By delving into the meaning behind each step in an AI workflow, the goal is to provide greater transparency and explainability, facilitating root cause analysis and fostering trust in complex AI deployments within enterprise environments.
"We have to really dig deeper into what are the semantics of each step within the agentic workflow and aim and what is called in AI explainability."
Cisco Outshift Emphasizes Explainability for AI Systems After Identical Sessions Yield Divergent Results
Cisco Outshift is highlighting the critical need for explainability in AI systems, moving beyond simple observability and evaluation. The company demonstrated this necessity with an example where two identical AI sessions produced significantly different outcomes, a discrepancy that could not be resolved by standard metrics or activity graphs. Deeper semantic analysis was required to uncover environmental factors, such as a VPN connection causing a language change in a data source, as the root cause.
This issue underscores a significant challenge for AI reliability and adoption, as external variables can profoundly influence agent behavior in unpredictable ways. The capability to reconstruct root causes through causal analysis and provide remediation suggestions is crucial for building trust and ensuring the predictable performance of AI applications in diverse operational environments.
"You have no way just looking at either the agentic graph or the value of the metrics to say why they are not the same."
Also mentioned in this video
- A presentation on observability and evaluation in agentic AI systems,… (0:16)
- The current struggle with interoperable observability due to varied… (2:31)
- The limitations of current observability and evaluation solutions, noting their… (6:16)
- The speaker details the components of their observe and eval solution,… (12:22)
- The Open Agentic Schema Framework (OSF) for characterizing agents and agentic… (15:43)
- The need for evaluation beyond traditional monitoring metrics, emphasizing the… (17:48)
- The presentation showcases a demo of their open-source observability bricks,… (22:20)
- The presentation details the creation of a Multi-Agent System (MAS) ontology… (29:07)
- Observing single sessions is insufficient for understanding normal behavior,… (32:37)
- Anomaly detection based on normal behavior characterization, and describes… (36:20)
- A demo showcases the impact assessment feature, displaying the relative… (41:59)
- A multi-layered approach to analysis, moving from syntactic and semantic layers… (49:35)
- Ongoing work on embedding an operational 'spatial agent' within agentic… (51:32)
Summarised from Outshift by Cisco · 54:27. All credit belongs to the original creators. Streamed.News summarises publicly available video content.
