It is not unusual for operational systems to implement change. And similarly, it is not usual that, as those operational changes roll out into production, the business intelligence area is left uninformed. This situation remains true even when those operational changes are so very dramatic that they will cause the business intelligence’s data extractions to fail.
One could point to this status as an argument suggesting data warehouses and business intelligence be categorized according to the view of the old comedian Rodney Dangerfield and say that they both “get no respect.” But the truth is far simpler. Changes and enhancement to solutions are hard, even under the best of circumstances; yet harder still is documenting and communicating those changes to the people and areas that may need to know about them at any point-in-time.
An orderly mind might wish for wonderful and proactive change management processes, imagining that changes are published to the “right” receivers in advance and activities are coordinated across an organization’s solutions so that when a change rolls out, all of the impacted downstream considerations are also changed at just the right time. Having an environment that functions in such a well-oiled manner is extremely rare. Maturing an organization into such a stance is a daunting challenge of its own, often so daunting that it doesn’t arise.
While a lack of documentation and communication is common, sometimes change controls exist in a form that does indeed require documentation. Just any old existing document of changes does not mean that the issue of understanding change impact is resolved. The generated artifacts might not be coherent or simply may entomb too much noise around the vital details, meaning that recognizing the changes important to any one downstream system can be nearly impossible. It may force downstream business intelligence areas to commit excessive resources to reviewing these materials in hopes of finding those important changes in time to respond. Similarly, the business intelligence area may need to dedicate staff to involvement with each source system’s project activity meetings. In this way, a figurative ear-to-the-ground is in place that may allow for the business intelligence group to be aware of changes as they are defined within the sources.
The data warehouse must establish a philosophical stake-in-the-ground as it relates to source changes. The approach extremes require that the data warehouse functions like the U.S. Post Office, “Neither rain, nor sheet, nor snow …” versus assuming a stance of, “If something has changed, everything may be wrong now.” In a relational world, many changes can remain unnoticed. When extractions are well written, columns added to tables will not impact queries that do not address them. Alternately, should one use a staging area, processing can be composed to compare source and staging table structures prior to the extraction cycle. If changes are found, notifications can be sent out, and the later cycle can be stopped before it begins.
When it comes to change, each organization has unique aspects and restrictions. There is no one-size-fits-all answer. What can be made to work is up to the circumstances and creativity within the design and development team. Challenges to managing change are more awkward still when acquiring data from external organizations. However, organizations need to remember that when acquiring external data they are often “the customer” and should expect some service from their vendor – they need not always be a victim of change. Whatever the conditions or circumstances surrounding each organization’s data warehousing initiatives, budgets and resources are always limited. Only so much can be implemented and watched over, and that “so much” may be less than ideal. The challenge is getting enough of a manual or automated data check process in place to watch-dog source changes in order to be minimally sufficient.