As organizations grow their business intelligence function, friction generally arises between the operational IT staff and the business intelligence IT staff. This friction is not driven by animosity or concerns over political fiefdoms, it is actually a very natural occurrence. There are fundamental differences in priorities and approaches in building operational solutions and in constructing analytical supporting solutions. However, these differences can produce a gap in understanding that occasionally results in minor battles and skirmishes between these groups along the path of getting work done.
A critical area of conflict is access to production data. When building an operationally focused solution such as a website, the application is supreme. It is the application, via controlled and predetermined mechanisms, that creates and changes data. And the application’s components must be tested and verified before they are ever promoted into a production environment. The original coding and unit testing is done in a separate and distinct development environment, so that the valuable production data cannot be harmed by bad code. In fact, developers often have their own personal development space to produce solutions before they promote their objects into a shared development environment for integration testing.
Given the lengths to which these developers have gone to ensure the safety of the production environment and data, it is only natural for staff entrenched inside this controlling culture to be scandalized by business intelligence and analytics personnel who blithely want direct access to production sources as a starting point for their projects. Then the data scientists and even report builders push the limits further by needing to create and populate additional data structures to support the tasks necessary to go from start to finish, because not every report is humble enough to be built from a single, straightforward SQL SELECT statement. Complex tasks, temporary tables, permanent tables reading, inserting, updating—operational staff consider this development work that should be hidden away from the production world. Needing to operate against production data, doing this on an ad hoc basis, exploring, looking around, keeping some elements and discarding others is enough to cause great consternation in the minds and souls of operational staff. Often, this conflict has resulted in operational and analytical worlds being split. And, in general, such splits are an easy fix to conflicts.
However, nothing is as easy as it should be, so separating operational versus analytical environments while potentially a positive, isn’t always so good on the procedural side. The downside is that while there is work that is very ad hoc and dynamic, such as reading production data and blending datasets and creating on-the-fly data structures, there is also the work of building and populating data warehouses and data marts. And the ETL related to building those structures is more like the operational work. Having a controlled production with separate development environments is indeed good. Therefore, in some ways, this ETL work very much needs to mimic and piggyback off of the more rigid production-controlled world.
This combination of dynamics easily confuses staff members that are more familiar with a pure operational environment. First, they believe they have made the situation better by splitting worlds; then they find out parts should be controlled, ideally, following one managed corporate change control practice. Furthermore, if an organization has allowed the reporting and analytics area to embrace a free-range approach before trying to institute an architected data warehousing practice, even the analytics staff will be confused and not understand a need for control at all—which is yet another organizational hurdle to overcome.
Overall, the key is getting operational and analytical staff members to work together, so those processes that need to be controlled are controlled well, ideally via a single process, regardless of whether it is for operational solutions or analytical ones. Additionally, it is important to make sure the work that needs freedom to flex is allowed its liberty.