March 20, 2014
We can no longer deny the drive to diversify data management technology that began in the mid-90s. The aspiration to achieve one single and simple database management system has died.
I grew up with the advent of commercial relational databases in the late 80s and early 90s. At the time, the promise was clear: you could store everything in a relational database that was carefully modeled and expandable. And in doing so, you acquired the ability to access, govern and securely manage every bit of data in a single technology environment.
Most companies decided on a relational database standard and ported some or all of their applications towards that single database backend. All the principles of good architecture – including cost and skill optimization played out – until they didn’t.
All seemed swimming until one of my clients – a major European railway operator – wanted to geo code every bit of equipment and every centimeter of their railway network. As hard as we tried, we couldn’t meet the client’s demands well with a relational database. The advent of spatial data management systems came to the rescue. Questions like ‘What is the total book value of all assets deployed within 50 meters of switch #1099?’ were so much easier to perform and took a fraction of the computing resources.
Fast forward to 2014. We can forget about keeping data well-organized and secure without redundancy. Gone are the days when we could manage a small set of skills and an even smaller number of vendor relationships.
Today, there are several reasons why we need to rethink the technology standardization approach in information management:
- ‘The Big Data problem’ a.k.a. the acceleration of data volumes, velocity, variety and veracity overwhelms traditional data management architectures
- Information Driven Decision-Making requires a more agile and innovation-driven approach. Therefore, traditional ways of modeling data, creating robust interfaces to systems and data sources, etc. is too slow and too expensive to supply data to an innovation process with an unknown value.
- Commoditization of storage and computing resources make it unnecessary to reduce the amount of data stored or the granularity of data. The quest to avoid redundancy is counterproductive to insight generation
- Open source software and commodity hardware are orders of magnitude below traditional commercial offerings; that allows for controlled redundancy and exposing massive compute power to data scientists
- The ‘Human Touch’ or the individualization of consumer interactions (i.e. ‘I want to be treated as, marketed to and understood as an individual’) presents all companies with scalability issues that previously were only a problem for telecom and internet companies
And, last but not least, the ascent of a large number of well-funded software startups that employ new and innovative technology approaches to data management and analyses are working closely with open source communities and academia.
The dream of data management simplicity has dissipated. A diverse set of technologies will define the future of information management – and that future is bright. New technologies for information management and analytics open up a world of possibilities to push the boundaries of innovation at a breakneck pace.
At the very least, a forward looking information management and analytics landscape will have to have a set of different core data management technologies such as in memory databases, columnar or other NoSQL databases and long term storage and exploration platforms, like Hadoop. On top of that, there will be a diverse set of data exploration, analyses and visualization tools to enable everything from data science-type exploration to robust, multi-device and mobile dashboards, analytic applications, stream processing and self service reporting tools.
To avoid getting lost in the technology complexity, use your business goals as the guiding force of your information management investments and initiatives.