Say goodbye to your mouse

August 3, 2016



The journey to creating human-computer interfaces for 3D spaces is only beginning, and it promises to be an exciting ride.

What will replace your mouse in the future? Nothing, predicts Meron Gribetz, the CEO of Augmented Reality (AR) smartglasses maker, Meta.

Human-computer interfaces have hit an inflection point. Today’s interfaces were designed for computers. Tomorrow’s interfaces will be designed for humans.  (see figure and my article: How will people interact with augmented reality?)  This means computer interfaces will move from the constricting confines of 2D screens to the unbounded expanse of 3D spaces. Buckle your seatbelts and brace for blast off because the design of human-computer interfaces is about to get interesting.

In his keynote speech at the 2016 AWE conference, Gribetz forecasted the need for a “new paradigm of interface design based on the science of our senses,” interfaces inspired and informed by neuroscience. He is calling it “the natural machine.” He believes we can replace the mouse of the 2D computer interface era with absolutely nothing in the AR/Virtual Reality (VR) world of 3D interfaces. He illustrated his vision with a demo that virtually explored the viscosity of fluids and moved through dust and sculpted objects. The pot of gold at the end of the rainbow is interfaces that are natural, require no device, and no training or learning curve.

Kevin Kelly, senior maverick at Wired Magazine, in his recent article on VR, observed: “The VR industry is waiting for its Doug Engelbart (inventor of the mouse) to invent the equivalent of the mouse.” He feels an intuitive and user-friendly interface method is a key success factor in the adoption of virtual (and augmented) reality.

Both Kelly and Gribetz make profound observations about the importance of human computer interfaces in the emerging area of AR/VR. Both expect significant changes that include easier and more natural interfaces. But what does it mean for interfaces to be natural? How will natural be created?

As I thought about this question, it occurred to me that there is almost a hierarchy of levels to help us understand what is natural to us and what elements are necessary to facilitate a natural experience of interacting with a machine. I am modeling this after the Maslow’s hierarchy of needs, see Figure: Hierarchy of natural interfaces The ideal natural state is at the top of the pyramid and the most fundamental at the bottom. As in the Maslow’s hierarchy, the lower levels are a stepping-stone for higher levels.

  • At the fundamental level is our neurological activity in the brain.  If one can tap into the understanding at this fundamental level, one could create natural interfaces. Gribetz seems to be doing this at Meta by having neuroscientists on the team.
  • At the next level is our senses. Here what is natural will match or interact with our sense of vision, touch, hearing, taste and smell. So advancements in machine vision, speech, haptics and motion sensing are all making interfaces possible that are modeled after our perception systems. Many companies are working at this level.
  • At the next level is our intuition or instinct. We develop intuition or instinct from experience through repetition, from environmental and behavioral conditioning. An example of tapping into our intuitions is the notion of affordances: which are the actions that an object (or interface) suggests to us. Good product or interface designs elicit the right (intuitive) action from the user. Experience and knowledge of product design and industrial design can be quite useful at this level.
  • At the highest level is anticipation of our intent. Systems that can anticipate our actions or intent will be natural, meaning the system will perform actions without us asking for it. Personal assistants on mobile phones are some early examples of agents that are looking across a wide range of contextual and historical information to anticipate what a user might want to do, for example leave for a meeting.

What this means for business is that there is a new lens with which to transform business activity. Good  interfaces have always been essential to the success of physical world objects or digital applications. As the digital and physical world fuse with AR/VR and other technologies, the interfaces from the two worlds also merge, creating a new playing field for deploying technologies in enterprise work. We are still far away from knowing what will replace the mouse of the 2D interface and if it will be one device or a collection of devices or no device at all. However, the journey to creating human-computer interfaces for 3D spaces is only beginning, and it promises to be an exciting ride.



Chris Curran

Principal and Chief Technologist, PwC US Tel: +1 (214) 754 5055 Email

Vicki Huff Eckert

Global New Business & Innovation Leader Tel: +1 (650) 387 4956 Email

Mark McCaffery

US Technology, Media and Telecommunications (TMT) Leader Tel: +1 (408) 817 4199 Email