In the data warehousing arena, development databases often get short shrift. Frequently, this situation arises because development databases are considered too much work to be done properly. So, instead of embracing the problem and following through on the necessary work, it is ignored or done poorly. One could almost say that ignoring the development database has become the standard practice.
There is no end to the number of times one hears the comment, “Since this is only reading the data, why not just read directly from production and be done with it?” Or, alternatively, “Since this is all new data without the use of anything previous existing, just work directly in the production database.”
Often, the development area might be an out-of-date copy of production that is only updated under the direst of circumstances. Establishing a development database environment using a sub-set of carefully chosen production data representing a valued selection of test cases is clearly a time consuming task. Such detailed efforts potentially tie-up resources that also may be needed elsewhere.
A reasonable query might be asked, concerning whether development is only reading data or efforts involve only new and isolated data, then what harm is caused by allowing development inside the production database.
The harm in these approaches comes from two places; one is obvious and the other is a subtle, sneaky, creeping deception that can act like a cancer. The obvious one is that when production is being used for development then the organization must allow the production environment to be open to development staff. Therefore, locking down the data for the true users – business – is only partially achieved. IT development staff are actually stealing bandwidth from the user community. Certainly, if there is excessive bandwidth available in production then that added development burden itself is not harmful, until it is.
The other harm is a stickier, fuzzier, human issue. When one cuts a corner, it is only human nature to try to keep cutting corners. Short cuts become the first thought. Short-cutting arises without even consciously deciding to cut a corner, and pretty soon things are being done in a sloppy fashion. Processes are no longer tightly adhering to designs.
Close becomes good enough. Then, as time passes, small problems are exposed, and maybe some are even significant, but by then these close-enough problems are costly problems. These are problems that only existed because shortcuts were used in place of initially intended solutions, and now have become problems which must be ignored or worked around. And so our cycle of less-then-proper solutions expands and repeats.
As an industry, both data warehousing and business intelligence have a very lackadaisical approach to supporting an isolated development database area. The proof is easily seen since one need look no further than at all of the efforts done directly inside the production databases for an organization. This is one area where we have clearly and greatly lowered standards as to what is an acceptable development practice.
In lowering our standards, have we lowered the bar on the rest of the work that we do? Choosing “good enough” over “best practices” is a slippery slope that opens the door to just trying to get by on more and more things.
This descent into lowered standards happens without conscious thought while we are focused on other things. It creates a circumstance where the developers do not even recognize what has been missed. This is often because, instead, they may be celebrating their cost-cutting tactics. Maybe that estimated excessive cost for maintaining a separate and supported development area isn’t really so very excessive as it seems. Quality issues arising post-implementation may prove equally costly.