The proliferation of data sources, types, and stores is increasing the challenge of combining data into meaningful, valuable information.
The need for faster and smarter data integration capabilities is growing. At the same time, to deliver actual value, people need information they can trust—now more than ever during this COVID-19 pandemic—balancing data governance is absolutely essential.
DBTA recently held a webinar with Quinn Lewis, consulting director, Denodo; Keith Lambert, VP, marketing and business development, Kore Technologies; and Ginger Gatling, senior director, SAP, who discussed key technologies and best practices for overcoming data integration and governance challenges.
According to Lewis, current challenges in data integration and governance include faster and more complex demands for decision making, reduce cost of data infrastructure, and regulations, enterprise-wide governance and data security.
Companies can balance these challenges with a logical data warehouse, Lewis said. The logical data warehouse (LDW) is a next-gen DW that uses a multi-engine to fulfill conflicting demands. Rather than seeing its components as competing solutions, they are complementary engines.
Key strengths include:
- Faster and more effective data delivery (e.g. real-time)
- Increased flexibility
- Improved decision making
- Maximized ROI
- Responsiveness to big data and IoT
Lambert suggested the top 10 must-haves for overcoming data integration challenges. This includes:
- Best practices- Define your data model and naming standards; Create a data flow diagram; Build a source agnostic integration layer; Adopt a data warehouse architecture standard; and Consider an agile data warehouse methodology.
- Multi-source and database aggregation
- Point-in-time snapshots
- Incremental database updates (CDC)
- Automation with message-based architecture
- Detailed transaction logging
- Ability to analyze and profile data sources
- Template-driven ETL software that is easy to change
- Development environment and tools
- Technology partner with reliable and flexible software
The Kourier Integrator Enterprise Integration and Data Management suite can help organizations roll out these must-haves.
The platform is a purpose built ETL and EAI solution that offers rapid implementation, reliable technology, high speed, integration automation, multi-source aggregation, template driven clicks, flexibility, message-based architecture, and management by exception.
Advanced analytics and machine learning challenge data management to a new degree, Gatling explained.
“Accept that you will have to use multiple tools and processing engines, but be sure to build a strong metadata foundation to ensure data understanding, quality, and standard catalog and glossary,” she said.
SAP Data Intelligence provides intelligent information management. The platform combines data ops with a data catalog along with data fabric and data orchestration to operationalize machine learning.
An archived on-demand replay of this webinar is available here.