Organizational focus has been placed on the emergence of "big data" - large-scale data sets that businesses and governments use to create new value with today's computing and communications power. Big data poses many opportunities, but managing the rapid growth adds challenges, including complexity and cost. Leaders must address the implications of big data, increasing volume and detail of information captured by enterprises, the rise of multimedia, social media, and the internet.
For example in the last 5 years, Gwinnett County Georgia government data increased from 38TB to 257TB (675%), and in 2011, nearly doubled from 130TB to 257TB. This growth is attributable, in part, to the introduction of video over internet protocol camera systems to monitor facilities and traffic intersections; videos of law enforcement-related traffic stops; license plate recognition systems, and red- light camera systems.
Gwinnett's data growth also has increased significantly due to the expansion of their data backup systems which are used for disaster recovery and continuity of operations purposes. Additionally, open records requirements play a role in data retention strategy.
Analysts suggest that worldwide information volume is growing annually at a minimum rate of 60%, and although this in and of itself poses a significant challenge in managing big data, business and IT leaders must focus on data types and how quickly data streams must be processed to meet demand. IT leaders should educate their business counterparts on these challenges while ensuring some degree of control and coordination so that big data is leveraged as an opportunity, rather than looked at only as a problem that raises compliance risks, increases costs and/or creates more organizational silos.
As noted in The Petabyte Challenge: 2011 IOUG Database Growth Survey: "The growing demands of the business have led to significant increases in the size and complexity of big data." As the economy has led to tighter budgets, each facet of the Gwinnett County government is "doing more with less," and looking to leverage technology to help create efficiencies. Increasingly, Gwinnett's business units are relying on data warehouse and business intelligence applications, which create a greater reliance on data.
According to the IOUG survey, 1/10 of respondents report their data stores exceed 1 petabyte. Gwinnett County will likely surpass that in 2012. And, according to the survey, seven out of 10 respondents cited that regulatory mandates and requirements have compelled them to keep data for longer periods of time, which increases storage costs. Gwinnett County is one of those seven.
Better use of data helps government agencies operate more efficiently and create more transparency. For example, Gwinnett County developed an application called "Where your property taxes go" which leverages tax related data in order to educate citizens and businesses on the value of what their property taxes fund. Tools like this can help create "smart cities/counties" and create expansive possibilities for the use of data.
Another challenge for most organizations is data governance. It is imperative that public and private sector organizations create an organizational data policy that helps facilitate data innovation while overcoming obstacles to adoption. Data privacy is important, so data security and identity management should be viewed at an enterprise level as opportunities versus impediments.
As data expands, so does the need for education. As a CIO, I need to ensure that my organization understands trends such as big data. Through best practices and lessons learned, publications and research, and networking opportunities, my team stays active in the Independent Oracle Users Group (IOUG). Each year, IOUG jointly leads the annual user-driven Oracle education and networking event - COLLABORATE. This year, COLLABORATE 12 - the IOUG Forum will be held in Las Vegas from April 22 to 26, and I am looking forward to attending a number of sessions including:
- Big Data Analytics, "R" You Ready?
- Top Exadata Features You Need to Know
- Unlocking the Value of Big Data with Oracle
- Challenges of Big Databases with MySQL
- Architecting and Implementing Backup and Recovery Solutions
At COLLABORATE 12, IOUG will hold its Big Data Bootcamp organized by Oracle ACEs to provide attendees with a suggested course curriculum for a comprehensive learning experience. All bootcamps are included in the price of conference registration.
So I guess the phrase, "go big or go home" resonates since big data is not just a trend, it's a reality that IT leaders need to effectively manage. Get educated and stay ahead.