Digital haystacks: Extracting insight from enterprise data

May 23, 2016



How big data innovation helped PwC transform enterprise search to deliver key data and increase employee effectiveness.

Scene on a farmPhoto_R_RGB_AU_D6_JA_01998.jpg

Did you know knowledge workers spend an average of 22 hours a week searching for people and information to do their jobs?

Today’s enterprises operate within a vortex of data. Information is scattered like leaves in the wind, and the meteoric pace of business makes yesterday’s data stale within 24 hours (or less). Extracting insights from mounds of data costs businesses time, productivity and quality of work. What’s more, today’s employees bring consumer expectations to the enterprise, demanding intuitive, real-time, and mobile solutions. Traditional document retrieval search engines no longer cut it. It’s time to reinvent enterprise search.

Innovation in design

My last post on thought experiments reviewed an innovation framework that helped transform enterprise search at PwC. I shared our journey from assessing needs through co-creation, ideating through crowdsourcing, and experimenting through reimagining the team. In this post we will explore how our innovations in development and delivery shifted the paradigm of enterprise search from a document retrieval engine to a cognitive business intelligence platform:


Deconstructing the problem helped diagnose root issues without getting distracted by symptoms or preconceptions, while voice of the customer interviews and surveys further enhanced our “needs analysis” phase. Seeking outside perspectives during “ideation” allowed us to visualize a new way to deliver prioritized information in real time, and most importantly, collaborating outside the usual suspects surfaced new paradigms and possibilities.

Innovation in technology development

Custom enterprise software development can become a sinuous path.  To stay focused while “experimenting,” we adhered to discrete stage-gates for failing fast and pivoting as needed. We engaged internal and external experts to design the working prototype and we asked 500 eager users to test it daily over three months. Crowdsourced ideas and user feedback fueled the development team to release new iterations every four weeks. Collaboration among experts, actual users and software developers was key to realizing our vision.

To scale the prototype to an enterprise-grade solution, we assembled a vanguard team of software developers, and combined best-of-breed consumer technologies. Today, 40,000+ employees in the U.S. can access real-time business intelligence, cognitive search, and a data concierge, all from one platform.

Here is the secret sauce:


    1.  A graph store models data and creates structure in an ontology.
    2.  Natural language processing understands conversational questions, connects questions with answers in the data model, or triggers computations.
    3.  A “Smart” User Interface offers tailored responses:
      • Reports: Dynamically generated summaries, charts and dashboards.
      • Answers: Facts returned directly from a source or computed in real-time.
      • Shortcuts: Keyword-driven navigation to commonly used tools and websites across the enterprise.
      • Lists: Intelligently curated links, filterable by metadata inherited from the source.

The platform uses application program interfaces (APIs) to connect data sources and technologies. The flexible architecture effortlessly integrates new content and capabilities as business needs change or new technologies are introduced.

Innovation in delivery

Leveraging the “ship and iterate” model allowed for continuous improvement with speed to market. Once the Alpha release was developed into an enterprise-grade platform, we piloted a Beta version to 10% of our people, soliciting even more feedback from power users.

In an unprecedented move for our firm, we deployed a work in progress to the entire organization. Fathom continuously improves with monthly releases to address feedback and product backlog. Through collaboration and iteration, we delivered a game-changing platform to our people, and now to our clients.

Fathom in action

Fathom enables us to do the following:

          • Increase efficiency (we cut search time by 50 percent in pilot focus groups)
          • Enhance profitability by uncovering experts and enabling collaboration across the enterprise
          • Improve decision-making by surfacing and integrating siloed data buried in disparate repositories
          • Introduce a new model for accelerated enterprise software development
          • Establish a platform for speciality business intelligence apps

Fathom is the successful result of applying an innovation methodology to deliver a superior technology product in dramatically less time than traditional models—even in a large, complex organization like PwC. Using this same framework, we help our clients address their unique needs and implement agile, flexible and continuously improving solutions that transform productivity, quality and speed.

Can you fathom the possibilities in transforming real-time access to data and insights in your enterprise?




Chris Curran

Principal and Chief Technologist, PwC US Tel: +1 (214) 754 5055 Email

Vicki Huff Eckert

Global New Business & Innovation Leader Tel: +1 (650) 387 4956 Email

Mark McCaffery

US Technology, Media and Telecommunications (TMT) Leader Tel: +1 (408) 817 4199 Email