Original source: Outshift by Cisco
This video from Outshift by Cisco covered a lot of ground. 20 segments stood out as worth your time. Everything below links directly to the timestamp in the original video.
Curious how AI is changing coding? This live demo shows how GitHub Copilot can instantly streamline complex data preparation, making dashboard creation much faster.
GitHub Copilot Generates Grafana-Compliant API Endpoint Live
During a live demonstration, an engineer successfully used GitHub Copilot to automatically generate a FastAPI endpoint designed to display network interface data in a format compatible with Grafana dashboards. By providing the AI with the expected output from a Genie Parser, the tool created an API endpoint that extracted specific details like interface name, IP address, status, and protocol from raw command-line interface (CLI) output, presenting a clean, simplified list in the browser.
This demonstration highlights the growing efficiency of AI-assisted development, particularly for tasks involving data transformation and API generation. Automating the creation of data structures optimized for visualization platforms like Grafana significantly reduces manual coding efforts, enabling developers to build monitoring dashboards more quickly and with fewer errors.
"Please create an API endpoint for interfaces brief. The expected output from Genie Parser is the following, and I take that expected output to help my LLM a little bit."
GitHub Copilot Streamlines Grafana Dashboard Data Preparation
To simplify the creation of Grafana dashboards, an engineer has integrated a custom skill into GitHub Copilot that automatically generates FastAPI endpoints. This skill ensures that data, initially in complex Python dictionary structures, is transformed into a specific, simplified list format that Grafana can consume directly, eliminating the need for extensive manual data processing or additional token expenditure on large language models (LLMs) for filtering.
This approach addresses a common challenge in data visualization: adapting raw data into a usable format without tedious manual intervention. By pre-configuring GitHub Copilot to understand Grafana's data requirements, developers can significantly accelerate the dashboard creation process and reduce the time spent on data manipulation, fostering more efficient development workflows.
"GitHub Copilot is going to do it for me, and that's it."
Containerized Architecture Enables Real-time and Historical Network Data Visualization
A new containerized architecture leverages Docker Compose to integrate FastAPI, InfluxDB, and Grafana for comprehensive network data monitoring. The FastAPI middleware provides API endpoints to query services, such as device interfaces or VLANs. InfluxDB, a time-series database, stores historical configuration data and metrics, while Grafana visualizes both current operational data and its evolution over time using Flux queries.
This modular and containerized setup allows for seamless data collection, processing, storage, and visualization, offering a robust solution for tracking network device configurations and performance. By separating concerns into distinct containers, the architecture enhances scalability, maintainability, and the ability to analyze trends and anomalies over extended periods, crucial for modern network management.
"Everything is containerized. It's going to be very nice and pretty."
Genie Library Transforms Raw CLI Output into Usable Python Dictionaries
A demonstration showcased the 'parse' API endpoint, which uses the Genie library to convert raw command-line interface (CLI) output from network devices into a clean, nested Python dictionary. This transformation, exemplified with a 'show IP interface brief' command, eliminates the need for manual parsing, such as regular expressions or TextFSM, making the data immediately usable for programmatic automation.
This capability significantly streamlines data extraction from network devices, which traditionally involves complex and error-prone text processing. By providing structured data, Genie empowers developers to build more robust and efficient automation scripts, accelerating tasks like network configuration, monitoring, and troubleshooting without extensive data filtering overhead.
"This is usable. I can work with this, and I don't need to burn my eyebrows or spend LLM tokens on filtering raw data."
Grafana Dashboard Enhanced with Time-Series Traffic Data from InfluxDB
A third visualization was added to a Grafana dashboard, incorporating time-series data from InfluxDB to display interface traffic measurements. This involved executing a Flux query to retrieve specific data points, including measurements, fields, and device information, from a designated InfluxDB bucket within a defined time range. The addition enriches the dashboard's functionality beyond current configurations to include historical trends.
Integrating time-series data is critical for comprehensive network monitoring, allowing administrators to observe traffic patterns, identify performance bottlenecks, and analyze trends over time. This capability transforms the dashboard from a static configuration viewer into a dynamic analytical tool, providing deeper insights into network behavior and operational health.
"I want something else for evolution in time. I want to use the InfluxDB one."
Customized Grafana Dashboard Offers Interactive Real-time and Historical Data Views
A fully customized Grafana dashboard was demonstrated, showcasing its interactive capabilities, including dynamic updates when switching between different network devices. Users can also view historical data over various time ranges, from hours to a week, and configure refresh rates to obtain minute-by-minute updates of real-time configurations and traffic data. While visual customizations like header mapping were noted, they were not fully explored in the demonstration.
This level of customization and interactivity provides network administrators with a powerful tool for monitoring and troubleshooting. The ability to instantly pivot between devices and timeframes, coupled with configurable refresh rates, ensures that critical network information is always current and easily accessible, supporting agile operational decision-making.
"I can set this dashboard to be refreshing, I don't know, every 15 minutes, every hour, so I can get these accurate views."
Genie Library Streamlines Network CLI Output Parsing
The Genie library, packaged with the Rath Git SDK, offers a robust solution for parsing raw command-line interface (CLI) outputs from network devices. Instead of relying on complex manual methods like regular expressions or TextFSM, Genie provides preconfigured filters that automatically transform diverse CLI outputs into structured, usable Python dictionary formats. These filters are specifically tailored to command types, vendors, and platforms.
This capability significantly simplifies the automation of network tasks by providing developers with immediately usable data. By abstracting the intricacies of text parsing, Genie enables programmers to focus on higher-level logic, accelerating the development of network management and monitoring tools while reducing the potential for parsing errors.
"What Genie allows us to do Genie is a series of preconfigured filters."
New API Endpoint Calculates and Stores Interface Traffic Data in InfluxDB
An engineer demonstrated an 'interfaces traffic' API endpoint designed to calculate the total traffic for a specific network interface at any given moment. This endpoint sums the in-rate and out-rate measurements and then creates a timestamped record, or 'point,' within an InfluxDB bucket. Each record includes the device, interface, and the calculated total traffic rate.
This functionality is crucial for collecting time-series data, enabling comprehensive monitoring and analysis of network performance. By systematically capturing these traffic snapshots, organizations can track trends, identify anomalies, and make informed decisions regarding network capacity planning and troubleshooting.
"I have a bucket, like my RadKit bucket. I'm creating a point called interface traffic."
Grafana Dashboard Integrates AI-Generated Endpoint for Interface Status
A second table visualization was added to a Grafana dashboard, pulling data from the 'interfaces brief' API endpoint previously generated by GitHub Copilot. This integration efficiently displays network interface status information without requiring any manual formatting or data manipulation, directly leveraging the pre-optimized structure created by the AI for Grafana's consumption.
This demonstration highlights the practical benefits of AI-assisted development in creating ready-to-use data structures for visualization platforms. By streamlining the data pipeline from raw command-line output to a Grafana-compliant format, developers can rapidly deploy and update dashboards, improving monitoring capabilities and reducing the time spent on repetitive data preparation tasks.
"I didn't need to move a finger because all these structures are optimized for Grafana, for Grafana to just pick them up."
Containerized Dashboards Offer Flexible Network Monitoring Solutions
The presented dashboards offer versatile applications for monitoring both current and time-series network configurations, showcasing a wide range of potential use cases. The solution emphasizes minimal device interaction, as the underlying Ratkit capabilities handle most data extraction without requiring extensive configurations like SNMP or traps on network devices. This streamlined approach is supported by a flexible, fully containerized architecture that allows for easy integration and customization.
This design philosophy grants users immense freedom to adapt the system to their specific needs, from tracking traffic patterns to observing configuration changes over time. The combination of minimal device tampering, a robust containerized setup, and boundless customization options ensures that the monitoring solution is both powerful and adaptable, making it suitable for diverse network environments and future expansions.
"The tampering is minimal. The next thing is, by keeping this architecture fully containerized, flexible, we can add whatever we want, integrations, whatever."
FastAPI Endpoints Maintain Continuous Connection for Dynamic Updates
The core of a demonstrated network monitoring solution lies in its FastAPI API endpoints, which, upon container startup, authenticate once with the Rath Git service using security certificates. This initial authentication establishes a continuous connection, allowing new API endpoints to be applied and become active immediately without requiring the container to be rebuilt or restarted. This design significantly streamlines development and deployment.
This persistent connection mechanism enhances the agility of the system, enabling developers to rapidly iterate and deploy new monitoring or automation features. By avoiding frequent container rebuilds, the solution minimizes downtime and operational overhead, making it an efficient framework for dynamic network management applications.
"If I add here a new endpoint and I refresh, it's going to be applied immediately."
Understanding InfluxDB: The Four Pillars of Time-Series Data Storage
InfluxDB, a time-series database, structures data around four core concepts: buckets, tags, fields, and points. Buckets serve as collections of records, akin to tables in a SQL database. Tags define the structure and data types of these records, while fields hold the actual content. A 'point' represents a specific record, complete with its timestamp, forming the fundamental unit of data storage.
This structured approach to data organization is essential for efficient time-series analysis, allowing for rapid querying and aggregation of data points over time. Understanding these pillars is crucial for anyone working with InfluxDB to effectively store, manage, and analyze large volumes of time-stamped information, particularly in monitoring and IoT applications.
"At the end of the day, we're going to be creating points with fields and tags, and we're going to put them into buckets."
Grafana Dashboard Integrates Dynamic Device Selection via FastAPI
A new Grafana dashboard was created to include a dynamic 'device' variable, which is populated by querying the '/devices' endpoint of a FastAPI service. This configuration allows users to select a specific network device from an automatically updated, live list, making the dashboard interactive and responsive to changes in the device inventory.
This dynamic device selection capability significantly enhances the usability of monitoring dashboards, especially in environments with frequently changing network infrastructure. By linking directly to a live data source, the dashboard ensures that users always have access to the most current list of active devices, streamlining the process of data exploration and targeted monitoring.
"If I change something in my inventory, it's we're going to have the changes here as well. Now we can pick a device. Cool. Okay."
Grafana Dashboard Displays Device Version Details from FastAPI Endpoint
A new table visualization was added to a Grafana dashboard to provide quick access to device version details. This visualization fetches key information such as chassis, operating system, platform, uptime, and a short version string directly from a FastAPI endpoint. This setup offers an immediate introductory overview of any selected network device within the dashboard interface.
Integrating such device-specific details directly into a monitoring dashboard enhances its utility, allowing network administrators to quickly ascertain the foundational characteristics of a device without navigating to separate tools. This streamlined access to critical information contributes to more efficient troubleshooting and management workflows.
"So, whenever you switch to a device, I have this nice introduction straight off the bat. Hello, my name is this. This is my platform. I've been awake alive for all this time, okay?"
Grafana Visualization Enhanced with Regex for Readable Interface Names
An InfluxDB visualization within Grafana was refined to improve readability by applying a regular expression (regex) transformation. This hack renames interface entries, replacing raw point notations with clear, user-friendly interface names, making the time-series data much easier to interpret. Further visual customizations, such as line styles or unit measurements, were mentioned but deferred for brevity.
This technique underscores the importance of data presentation in making complex monitoring dashboards accessible and actionable. By transforming cryptic data labels into intuitive names, administrators can more quickly understand network behavior and identify specific interfaces affected by performance issues, enhancing the overall utility of the visualization.
"This looks ugly. Let's make it better. More practical."
Cron Jobs Automate Grafana Dashboard Data Refresh via Ephemeral Containers
A repository includes a cron job mechanism that automates the refresh of Grafana dashboard data. A YAML configuration file defines the frequency at which API endpoints are triggered, specifying minute, hour, day, or week intervals. A bash script then creates a cron job that, when triggered, deploys an ephemeral container. This container subsequently calls the desired FastAPI endpoints, ensuring continuous and scheduled updates of dashboard information.
This automation solution provides a reliable method for maintaining up-to-date monitoring dashboards without manual intervention. By using ephemeral containers, the system efficiently manages resources, spinning up only when needed to refresh data, thereby ensuring that real-time configurations and traffic data are consistently current and available for analysis.
"All of this is going to get triggered as frequently as I want, as I need. So, all of this is getting refreshed again and again and again, okay?"
Cron Jobs Automate Network Data Collection for InfluxDB and Grafana
A scheduled data collection mechanism, driven by cron jobs, has been implemented to automatically gather network data and store it in InfluxDB. These cron jobs trigger specific FastAPI API endpoints at predefined intervals, ensuring continuous updates of information without requiring any manual intervention. The collected data is then visualized using Flux queries within a Grafana dashboard.
This automated approach is critical for maintaining up-to-date network monitoring. By reliably collecting data on a schedule, the system provides a consistent flow of information for performance analysis, trend identification, and proactive issue resolution, significantly reducing operational overhead and improving the accuracy of monitoring insights.
"I want all of this to be executed without me checking it, without me pressing the button."
FastAPI Endpoint Retrieves Live Device Inventory from Rath Git Service
A FastAPI endpoint has been created to retrieve a list of active network devices directly from the Rath Git service inventory. When this API call is made, it returns a structured JSON output containing the names of all currently active devices. This demonstration highlighted the immediate and accurate presentation of the device inventory in a web browser.
This functionality provides a crucial building block for dynamic network management and monitoring applications. By accessing a live, up-to-date inventory, other tools and dashboards can ensure they are interacting with the correct and most current set of network devices, enhancing reliability and efficiency in automated workflows.
"These are the active devices. You saw that we had like the toggles, right? These are the active devices in my inventory."
FastAPI Endpoint Executes CLI Commands on Network Devices, Returns Raw Output
An engineer demonstrated a FastAPI endpoint capable of executing specific command-line interface (CLI) commands, such as 'show IP interfaces brief,' on a selected network device. The endpoint takes a device name and a command, executes it via the underlying service, and then returns the raw output, mirroring the experience of interacting with a device through a terminal via SSH.
This capability is fundamental for network automation, allowing programmatic interaction with devices to retrieve real-time operational data or perform configurations. By exposing CLI execution through a standardized API, developers can integrate network device control into broader automation frameworks, improving efficiency and reducing manual operational tasks.
"This is the same output that I would get from a CLI, right? I open my terminal, I connected via SSH, show IP interfaces brief, I get this, right?"
Interfaces Traffic API Endpoint Collects Timestamped Data in InfluxDB
The 'interfaces traffic' API endpoint was triggered, successfully generating multiple timestamped records, or 'points,' for each of the 18 network interfaces. These records, representing traffic measurements, were then stored in InfluxDB. This action confirmed the effective collection of time-series data for network interfaces.
This collection process is vital for building historical datasets that can be used for performance monitoring, anomaly detection, and capacity planning. By capturing traffic data at specific moments in time, network administrators can analyze trends and troubleshoot issues more effectively, providing deeper insights into network health and usage patterns.
"I have 18 interfaces, so I have 18 records in this moment right here right now. In this moment in time, all timestamped, it's there in InfluxDB."
Summarised from Outshift by Cisco · 42:22. All credit belongs to the original creators. Streamed.News summarises publicly available video content.
