Establishing an infrastructure monitoring layer brings a plethora of benefits to IT systems, especially in regard to ensuring these systems’ reliability, performance, and availability. With a variety of open source tools that offer infrastructure monitoring capabilities, determining which solutions fit best—and how to implement them—is crucial.
Anais Dotis-Georgiou, lead developer advocate at InfluxData, joined DBTA’s webinar, Infrastructure Monitoring Basics with Telegraf, Grafana, and InfluxDB, to provide a comprehensive understanding of infrastructure monitoring, as well as how Telegraf, InfluxDB, and Grafana can be used in tandem to meet modern monitoring needs.
Understanding that monitoring and observability represent different processes is fundamental, Dotis-Georgiou explained. Monitoring refers to the process of collecting and analyzing metrics, logs, and events to track system performance. Observability, on the other hand, instruments code and infrastructure to expose relevant data, enabling teams to understand system behavior.
Monitoring spans several different fields, including:
- Network monitoring to ensure efficient data transmission, detect bottlenecks, and the status of devices
- Server monitoring to capture CPU usage, memory consumption, disk space, and active processes
- Application performance monitoring which measures latency, code inefficiencies, and errors
- Cloud infrastructure monitoring which measures uptime, cost, and resource allocation
Dotis-Georgiou examined an “imaginary problem”—which was constructed by ChatGPT—where monitoring was the central issue. The fictitious product, Whisper GPT, was designed with a purpose to employ natural language processing (NLP) and machine learning (ML) techniques to provide users with highly accurate, context-aware, and personalized responses (sound familiar?). The problem was unprecedented growth; the product, due to popularity, introduced several different challenges, including bottlenecks, latency issues, and the need for seamless scalability.
The question, then, was how can the Whisper GPT team monitor and optimize their scaling solution's network, application, and cloud infrastructure to maintain optimal performance, reliability, and user experience?
To solve this problem, we need to build a scaled monitoring solution in a hybrid architecture that accommodates both on-prem and cloud environments, according to Dotis-Georgiou. This architecture is split into three stages: data collection, data storage, and data action.
The data collection stage leverages Telegraf, an open source data collection agent for metrics and events. With over 300 plugins for ingesting and outputting data, Telegraf is one of the most versatile ingest agents for time series data, said Dotis-Georgiou. This solution acts as the collection backbone deployed on all three servers, as well as the cloud infrastructure, collecting data from OTEL, Prometheus, CloudWatch, and raw server-based metrics.
For data storage, Dotis-Georgiou pointed to InfluxDB, a database purpose-built for handling time series data at massive scale for real-time analytics. Developers can ingest, store, and analyze all types of time series data—metrics, events, traces—in a single platform designed to handle high-speed, high-volume, and high-cardinality data. Setup with four repositories that represent each data source, called buckets, InfluxDB allows us to store metrics, logs, and traces in one datastore.
As we enter the data action stage, using Grafana—an open-source data visualization and monitoring platform—allows users to create interactive dashboards for real-time data analysis and tracking of metrics across various data sources. Grafana acts as the observability hub, where the usage of FlightSQL and Jaeger data sources allow us to query data from InfluxDB 3.0.
For the full, in-depth discussion of infrastructure monitoring with Telegraf, InfluxDB, and Grafana, which features examples, demos, and more, you can view an archived version of the webinar here.