Now more than ever, data has evolved into an asset more strategic and valuable than any raw material or capital construction project. Companies are scrambling to "compete on analytics," recognizing that the one to most effectively leverage information coming out of their systems gets the greatest competitive advantage.
Naturally, this commitment to the power of information lands squarely into the laps of data professionals, who are being called upon to provide services and value far beyond the administrative tasks - such as applying security patches or de-duping data–that dominated their jobs up until just a few years ago. Data professionals are being called upon to perform higher-level tasks and interact with the business, taking on roles that more appropriately should be called "data stewards."
To assume these new, more business-savvy roles, data professionals need to be able to increase their skills, without taking their eyes off the ball of the maintenance-type database tasks they still are responsible for. Those more routine "maintenance" concerns aren't going away anytime soon, either–driven by business, driven by increased digitization, and driven by compliance mandates, databases are growing to enormous sizes. In addition, data is expected to be available on a 24x7 basis to a growing base of end-users.
In a new survey, data professionals are saying it loud and clear: They need additional training, new tools, improved automation of routine tasks and career advancement. The survey, conducted by Unisphere Research among members of the International DB2 Users Group (IDUG), finds that many data managers and professionals are expanding their range of expertise to provide higher-level services to enterprises. However, many feel that data professionals are not receiving the training and support needed to improve their skills base. The survey, which covered 853 enterprises, was conducted in cooperation with CA.
Survey respondents represented a cross-section of database platforms, job functions, and geographies. Job titles of respondents included that of database administrator (54%), IT consultant (8%), team leader (6%), and systems architect (5%). Respondents also represented a wide range of sizes of organizations from many industries.
The roles of data managers and professionals are evolving quickly, the survey found. Until recently, databases were stand-alone environments that were only accessible to a limited number of users. These databases had caretakers, database administrators, who saw to the day-to-day operation of these systems and regulated the data that flowed into and out of them. Today's data professionals, however, engage in a wide range of duties, from traditional database administration, database and application design, quality assurance, integrating applications, to managing performance and data security.
The survey looked at the training requirements of building an effective database team that can serve today's organization, and focused on four key areas shaping the job of the data professional: business consulting, application development and integration, database administration, and data security. The survey found respondents are actively engaged in all of these areas.
Improving database performance is driving many sites, and a majority of respondents see a strong connection between improving database performance and overall corporate performance (increasing revenue/reducing costs). Data managers are concerned about managing the explosion of data, and are looking for ways to manage the process. Top business challenges include managing data growth (40%); compliance management (27%); enterprise information integration (25%); business intelligence and data warehousing (21%); and master data management (21%). Top operational (or technical) challenges include database performance (56%); data recovery and availability (55%); and data security (47%).
The survey also found that performance management skills are the most sought-after skill among data professionals - cited by 57 percent of respondents. Also ranking high on the list are application development and integration skills (44%), reflecting the growing convergence between database administration and development in day-to-day tasks. In fact, 37 percent reported their time engaged in application development and integration has grown over the past year. Most efforts focus on SQL statement development and data modeling and design.
The increasing diversity of database sites - even among predominantly IBM systems sites - is creating more requirements for transferable skills as well. About 44 percent of respondents also cited a need for more cross-training as a top priority, to build skill levels on other databases beyond DB2. To address their need for new skills, most cited ad hoc on-the-job training such as in-person, off-site training in the form of industry conference and local user groups.
Another shift taking place within data professionals' jobs is toward more engagement with the business. Forty-four percent reported their time engaged in business consulting activities has grown over the past year. Most efforts are focused on database design/modeling and resolving storage or archiving issues. By working closely with the business to achieve better data output and performance, data managers and professionals are dramatically changing their roles within organizations, with new tools as well as resourcefulness. For example, one respondent noted how within his organization, "a specific partitioned data warehouse table was closely reaching its maximum size and this table is used by operational systems also. I increased the partition numbers from 20 to 100 to better distribute the data and slow partition growth rate. An outage on this table would result in a Severity 1-level outage of many critical operational systems."
Respondents were in agreement that improving both database and business process performance is one of the most essential parts of their jobs. For example, one respondent discussed how discrete changes to SQL statements were frequently credited with dramatic performance gains and substantial cost and time savings. Another indicated that SQL statements are regularly reviewed, explained, and tuned. Efforts in this area have had dramatic results, enabling hours-long running programs to complete in seconds. Still, another explained, "I was able to improve performance of an application by analyzing indexes and make proper change to optimize data retrieval."
Skills Challenges Similar Across All Database Types
Readers of Database Trends and Applications, which represent a wide range of database brand shops, also participated in the same skills survey administered to IDUG members. A total of 343 DBTA readers participated in the survey, representing not only users of DB2 (16%), but also Microsoft SQL Server (52%), Oracle (29%), MySQL (16%), and Sybase (8%).
As with the IDUG group, DBTA readers see improving database performance as a key driver at their sites. A majority of respondents see a strong connection between improving database performance and overall corporate performance (increasing revenue/reducing costs). Data managers are concerned about managing the explosion of data, and are looking for ways to manage the process.
Top business challenges include managing data growth (34%), enterprise information integration (27%), and business intelligence initiatives (25%). While the DB2 users in the IDUG survey also ranked data growth as the top priority, compliance was the second top-ranked priority. DBTA readers ranked compliance in sixth place.
In terms of skill requirements to meet the organization of the future, both IDUG members and DBTA readers placed performance management and application development and integration as the first- and second-ranked most important skill areas. However, while DB2 site managers see a driving need for database administration on an unfamiliar platform for cross-training purposes, this is less pressing at more diverse DBTA sites. IDUG members ranked this third, compared to eighth among the DBTA group.
DBTA readers are also slightly more inclined to be engaging the business as data consultants along with the operational aspects of their jobs. Forty-nine percent reported their time engaged in business consulting activities has grown over the past year, compared to 44 percent of DB2 professionals in the IDUG survey. Most efforts are focused on database design/modeling and data integration issues. DB2 professionals also ranked database design/modeling as their top activity. However, while DB2 respondents ranked resolving storage or archiving issues in second place, this drops to ninth place at the DBTA reader sites.
Top Business-Related Data Management Challenges
(Percent assigning challenge "top priority" status)
IDUG* (Rank) DBTA* (Rank)
Managing data growth
40% (1) 34% (1)
Compliance initiatives (e.g., Sarbanes-Oxley) 27% (2) 21% (6)
Enterprise information integration 25% (3) 27% (2)
Business intelligence initiatives 21% (4) 25% (3)
Data warehouse/data mart initiatives 21% (5) 22% (5)
Master data management 21% (6) 24% (4)
Service-oriented architecture initiatives (SOA) 20% (7) 12% (7)
Avoiding or postponing CPU upgrade 19% (8) 8% (9)
Database tool vendor management 13% (9) 7% (10)
Other 14% (10) 9% (8)
* IDUG respondents primarily represent IBM DB2 installations. DBTA respondents represent mixed database environments.
Most Important Skills Needed at This Time
IDUG* (Rank)
DBTA* (Rank)
Performance management
57% (1)
50% (1)
Application development and integration
44% (2)
44% (2)
Database administration on an unfamiliar
platform - cross-training purposes
44% (3)
24% (8)
Data modeling and design
38% (4)
32% (5)
Data security
28% (5)
37% (3)
Database administration
32% (6)
31% (6)
Change and capacity management
25% (7)
34% (4)
Business management/consulting
17% (8)
26% (7)
Other
5% (9)
2% (9)
* IDUG respondents primarily represent IBM DB2 installations. DBTA respondents represent mixed database environments.