June 24, 2014
A look at some of the issues facing companies looking to achieve enterprise data integration.
If you’re reading this blog, you probably have a data integration horror story you could share. The most recent one I’ve heard involved a bank director. He was provided with three different reports all trying to answer the same question. Each one gave a different answer. When the director asked the data analysts to identify the best report, the analysts said the reports were all consistent with the data sources they were based on.
Why are integration projects often doomed to fail? Is integration at cloud scale even possible? Is the ongoing battle between business and IT the biggest barrier of all?
We believe part of the problem has to do with an antiquated and overly ambitious notion of integration. Some enterprises are rethinking their ideas about integration all together. In the process, they’re uncovering techniques that lend themselves to a more practical and less complicated integration style.
Data management teams are no longer mapping everything they can; instead, they’re mapping (or deriving implicit mappings) with lighter, more dynamic approaches, scaling their efforts with simple interfaces and standard templates, all with a shorter time horizon and more feasible objectives in mind. Governance, rather than seeking to impose all manner of controls over how data gets handled or shared, instead assumes the characteristics of guardrails on an expressway.
Over the next few months, PwC’s Technology Forecast research team will be pondering what the emerging, cloud-inspired enterprise integration fabric looks like. We’d like to hear your opinions as we explore the following areas:
Issue overview: Integration fabric
The data lake topic is the first of three topics as part of the integration fabric research covered in this issue of the PwC Technology Forecast. The integration fabric is a central component for PwC’s New IT Platform.
The main areas we’ve explored include these:
Integration fabric layer: Data
Integration challenges: Data silos, data proliferation, rigid schemas, and high data warehousing cost; new and heterogeneous data types
Emerging technology solutions: Hadoop data lakes, late binding, and metadata provenance tools
Enterprises are beginning to place extracts of their data for analytics and business intelligence (BI) purposes into a single, massive repository and structuring only what’s necessary. Instead of imposing schemas beforehand, enterprises are allowing data science groups to derive their own views of the data and structure it only lightly, late in the process.
Integration fabric layer: Applications and services
Integration challenges: Rigid, monolithic systems that are difficult to update in response to business needs
Emerging technology solutions: Microservices
Fine-grained microservices, each associated with a single business function and accessible via an application programming interface (API), can be easily added to the mix or replaced. This method helps developer teams create highly responsive, flexible applications.
Integration fabric layer: Infrastructure
Integration challenges: Multiple clouds and operating systems that lack standardization
Emerging technology solutions: Software containers for resource isolation and abstraction
New software containers such as Docker extend and improve virtualization, making applications portable across clouds. Simplifying application deployment decreases time to value.