Data observability is a relatively new discipline in the fields of data engineering and data management. While many are familiar with the longstanding concepts of observability and monitoring in enterprise IT networks and infrastructure, data observability has only really come into the spotlight in the last two years. However, it has managed to turn a lot of heads in that short time.
The uptake of traditional observability and monitoring solutions has been driven by the proliferation of cloud applications and infrastructure as organizations find it increasingly difficult to keep an eye on the growing number of systems they have deployed across various locations. Much in the same way, we’re now seeing this happen with data.
Having healthy data goes beyond simply understanding its quality. Nowadays, real-time is king, and that requires solutions that can alert organizations to issues with their data before those issues are propagated into downstream applications and processes. This is where data observability comes in.
Small and mid-size businesses (SMB) require the extra support observability provides, as they have fewer human resources to deliver projects to support business decisions and operations, and therefore have less time to resolve issues with their data and pipelines. It is clear that data observability solutions can bring tremendous benefits and time savings by enabling these individuals to be more proactive in their data management and engineering activities.
What is Data Observability?
Many IT and data professionals are familiar with the concepts of monitoring and alerting, either in data or other IT systems. The simplest form of this is perhaps data quality profiling, where analyses run on a daily basis, and users are alerted when certain rules are violated or thresholds are crossed. Nowadays, these kinds of processes can be embedded into data pipelines so that monitoring and alerting occurs in real-time, giving organizations the capabilities to handle data in motion. Data observability, however, goes beyond simple monitoring and alerting and helps users to answer the why questions like, “Why has my pipeline execution failed?” or, “Why is there no data on my dashboard?”
Think of any enterprise data or analytics system as a car:
Monitoring is equivalent to the various sensors capturing important metrics about the health of the vehicle, perhaps the engine temperature, fuel level, tire pressures, etc., and making those metrics available to the driver via the gauges on the dashboard in real-time.
Alerting is represented by the lights on the same dashboard. These are illuminated to warn the driver of an issue where the health of the vehicle is below optimum, e.g., low fuel level or low tire pressure. These indicate to the driver that some light maintenance is required before things go very wrong and the vehicle is no longer able to function correctly.
When the warnings are more serious and complex—such as an engine warning light or service advisory—drivers don’t have the ability to diagnose issues like this on their own and must rely on a mechanic with specialist equipment to read the error messages and translate them into a solution to fix the vehicle.
This is what data observability brings to the table for organizations. Like a mechanic, it can take the alerts from the monitoring outputs and understand what has caused the issue, perhaps by interpreting the error messages or by knowing how the vehicles systems are interconnected and how to put things right again. Taking the analogy one step further, data observability can also play the role of some car manufacturers, who can even collect data from vehicle sensors in real-time and use it to predict and prevent failures.
Data observability makes it possible to alert users of issues, inform them of the causes and predict upcoming failures. These alerts are sent in real time when, for example, data quality rules are broken. They can show the data lineage, point to where things went wrong and perhaps even flag if there are anomalous data volumes being ingested compared to previous runs. Having this level of insight into data operations is a game-changer for SMBs.
Why is it so important for SMBs?
Data observability enables a more proactive approach to data engineering and data management and ultimately stops data issues from causing problems for business decision-making and operations. Because SMBs are often resource-constrained, anything that takes strain off users and enables them to spend more of their time doing tasks that add value is beneficial for budgets.
But where are the time-savings and additional value coming from when applying data observability practices and solutions in the typical tasks of data users in SMB organizations? Here are some examples:
Data Quality – Identifying and fixing data quality issues in large data repositories such as lakes and warehouses can be extremely time-consuming, especially after years of legacy data has been ingested. Data observability is a proactive, quicker way to flag data quality issues during ingestion and move records into quarantine as needed.
Data Lineage – When data quality issues are identified, it’s essential to find the cause of the issues and fix them at the source to prevent future challenges. Data observability gives users an edge with data lineage, using the visualization of data processes and pipelines to quickly identify the root cause of their problems.
Schema Changes – Changes (knowingly or unknowingly) made to schemas in databases or API payloads can sometimes bring down analytical or operational systems, causing headaches for the data engineers who must make the business-critical systems functional again. Even worse, engineers often only find out about the issue after downstream users begin to complain that they can’t do their jobs – and it’s already had an impact on the business. With data observability, schema change notifications can notify data engineers in advance of any consumers so they can implement a fix with minimal disruption.
Anomaly Detection – The sheer volumes of data in ingestion pipelines and scattered across hundreds or thousands of data fields makes it next-to-impossible for consumers to manually detect problematic data, so issues might only be flagged at the point of visualization or reporting (if they’re even flagged at all). Data observability enables anomaly detection across several different data and metadata types to notify users of issues in advance so they can put a fix in place before having adverse effects on downstream use. For example, data engineers can be notified of fluctuating data ingestion volumes in case a data source has connectivity issues (and is missing from an important analysis), or when a text field containing mainly surnames suddenly has email addresses appearing in it, potentially exposing sensitive information and breaking regulatory compliance.
Data Freshness – Nobody wants to make decisions using out-of-date data, but the truth is that many data consumers are probably unaware of when the data they are using was last updated. Data observability gives consumers the ability to see how fresh their data is, understand why the data may be out of date and how to update it through tracing lineage and the associated pipelines and processes for their datasets.
Data Discovery and Accessibility – Knowing what data is available and where to find it has been one of the biggest challenges for many data consumers in modern organizations. It is a problem that is ubiquitous across all organizations, regardless of size. With data observability, it's also important to keep track of what data is available and what it might contain, so data discovery and cataloging is an important piece of the jigsaw. Users can be notified when new data sets become available or when datasets are removed, and they can even be notified about the content and possible categorization of the dataset. This is crucial for democratizing access to data and driving innovation in organizations where multiple personas need rapid access to new datasets.