What is mixed reality?

  • 5 minutes

The first step in any learning process is defining the core concept in simple terms. Here is what mixed reality means at its most basic level:

Mixed reality is a spectrum of immersive experiences, connecting and blending physical and digital worlds together in augmented reality and virtual reality applications.

In visual terms, imagine mixed reality as a creative space that exists between the extremes of the physical and digital worlds. Experiences range from overlaying virtual content on objects in the physical world, like in augmented reality apps, to an entirely immersive experience where the user doesn't have any input from the real world, as in virtual reality.

Diagram of the mixed reality spectrum with the physical world on the left and the digital world on the right.

Because mixed reality covers such a broad range of possible user experiences, it comes with a set of interaction types that are entirely unique. These interaction types include but aren't limited to:

  • Environmental input, like capturing a user's position in the world by mapping surfaces and boundaries in the area.
  • Spatialized sound, which is 3D sound that has position and depth in a virtual space, just like in the real world.
  • Locations, positions, and persistence of objects in real and virtual spaces.

These features are part of a relationship between human and computer input known as human-computer interaction (HCI). Human input covers the more familiar ways of interacting with technology, such as using a keyboard, mouse, touchpad, or your own voice. As sensors and processing power have increased in computers, so has this new area of computer input from the environment. The interaction between computers and environment is called perception.

The intersection of computer processing, human input, and environmental understanding is where the power, creativity, and emerging capabilities of mixed reality come to life. Movement through the physical world can translate to movement in the digital world. Boundaries in the physical world can influence application experiences, such as game play, in the digital world. Without environmental input, experiences can't blend between physical and digital realities.

Venn diagram with circles for human, computer, and environment interaction with mixed reality at the intersection.

As a historical footnote, the term mixed reality comes from a 1994 paper titled A Taxonomy of Mixed Reality Visual Displays, written by Paul Milgram and Fumio Kishino. That paper doesn't describe the version of mixed reality that people have today. The authors explore the concept of a virtuality continuum and the categorization of taxonomy applied to displays. You can find a link to the paper at the end of the module.