What Is Mixed Reality?

Mixed reality, or integrating digital information into the real world, is a concept people have been thinking about for over 100 years. Since at least the 1930s, writers and thinkers have imagined a future where technology could take us outside our real-world experiences. But it wasn’t until 1994 that the phrase mixed reality was coined by Paul Milgram, an American economist who won the Nobel Prize in Economics in 2020, and Fumio Kshino, former director of the Cybermedia Center at Osaka University.

Their paper, A Taxonomy of Mixed Reality Visual Displays, discussed mixed reality visual displays and identified six classes of display environments for mixed reality. In doing so, they also defined the concept of a virtuality continuum, the idea that the real and virtual worlds exist at opposite ends of a spectrum, with mixed reality forming the gradient in between. In the decades since, mixed reality technology advancements now include spatial mapping and environmental understanding, hand- and eye-tracking, awareness of sound in a spatial dimension, and the ability to conduct collaborative work on 3D assets. 

What is mixed reality?

Mixed reality technology allows us to interact with digital content superimposed in the real world. For example, a social media filter that makes you look like a cat works with mixed reality technology to alter our view of the real world. Unlike virtual reality, which removes users from the real world, mixed reality allows us to merge the two naturally and intuitively. 

Mixed reality vs. AR vs. VR

Mixed reality is closely related to augmented and virtual reality. In fact, these concepts exist on a scale called the virtuality continuum or the mixed reality spectrum. Reality exists on one end, the physical world we experience. At the opposite end of the continuum, virtual reality exists as an environment completely removed from reality. To enter VR spaces, you first put on a headset or other device that gives you the sensation of being entirely elsewhere. In between these two extremes, mixed reality and augmented reality blur the lines of what is real and what is virtual, allowing you to interact simultaneously with real and virtual objects. 

Virtual reality

Virtual reality is a fully simulated environment you experience by closing your senses off from reality, such as through a headset. Virtual reality allows you to experience things you would not be able to do otherwise, such as riding a rollercoaster while sitting on your living room couch or walking on the surface of the moon. While you don't see any elements of the real world (your living room) you are limited to the parameters of your actual space when using VR. For example, you couldn’t explore very much of the moon's surface before the physical reality of your environment would force you to interact with it, such as opening a door.

Augmented reality

Augmented reality keeps you in the real-world environment and superimposes digital content you can see and interact with through screens or other devices. For example, you can use augmented reality to project an image of a bookshelf into your living room to see if it fits the space and design of your room before you make a purchase. Augmented reality allows you to add new information or make new connections with the reality in front of you.

Mixed reality

You may see the term mixed reality used as an umbrella term to refer to any blend of the real and virtual worlds, such as in the mixed reality spectrum mentioned above. As the technology that powers mixed reality becomes more advanced, the term is starting to refer to a specific subset of augmented reality where you can interact simultaneously with real and digital objects in real time. This technology goes further than augmented reality because of your increased ability to interact with what you see on the screen or headset in front of you. 

How does mixed reality work?

In simple terms, mixed reality works through sensors that provide a computer with insight into our physical world. From our point of view, it looks like the computer is perceiving the real world. Using different sensors, a computer processor can perceive a lot of different information about the world we live in, such as the positions of objects in the room, levels of ambient light and sound, and our physical location as we move through space.

Other technologies that power mixed reality include computer vision, cloud computing, input systems, graphical processing, and display technologies.

• Computer vision: Computer vision refers to the technology that allows a computer to make sense of visual information like videos or images and take corresponding action. Mixed reality uses computer vision to see and understand the physical world. 

• Cloud computing: Cloud computing is the technology that allows you to access software, systems, or other files through the cloud instead of stored locally on your company’s equipment. An example of cloud computing in mixed reality is Azure Remote Rendering, which offers graphical processing you can access through the cloud on-demand. 

• Input systems: Input systems are the means through which you can interact with a mixed reality environment, such as using a controller. Other examples of input systems could allow you to interact with the digital content through gestures or pointers. 

• Graphical processing: Graphical processing units (GPUs) use graphical processing to render 3D graphics. You can use this mixed reality component to render digital images in a physical space. 

• Display technologies: Display technology outputs visual information, such as a television screen, computer monitor, or the face of a smartphone. Many forms of mixed reality require a display screen to view digital content. 

What is mixed reality used for?

The uses and applications of mixed reality are vast and are already changing the landscape of industries like health care, education, entertainment, gaming, business, and architecture. Let’s take a look at a few of the most common use cases for mixed reality.

• Gaming: The game Pokemon Go, which launched and became popular in 2016, introduced many people to the idea of using augmented and mixed reality for gaming. Today, many other video games, such as Angry Birds FPS, and franchises like Star Wars are taking advantage of the technology. 

• Health care: Mixed reality offers solutions for health care in various ways, from training new health care professionals to improving imaging and patient care. For example, HoloAnatomy is an app developed at Case Western Reserve University using Microsoft’s HoloLen to project advanced anatomy maps into a space where users can interact with them. Healthcare professionals can also use mixed reality in surgery while screening patients for cardiac anomalies, precision in radiology, and more. 

• Education: Mixed reality can be a tool for training and education, both in a classroom setting and to provide on-the-job training. Instead of engaging in complex or dangerous training exercises, instructors and students can complete simulated training exercises instead. Grand Canyon University conducted experiments using mixed reality in the classroom, and it found many benefits to integrating the technology, including the ability to create a lab for experimentation anywhere through virtual means, eliminating some of the costs of running a lab, and providing an engaging learning exercise for students. 

• Engineering: Engineers have already established virtual reality as an important tool for designing complex builds and modeling conceptual structures. Mixed reality offers additional capabilities by bringing those 3D models out of isolated computer screens and into the real-world environment where a team can manipulate them collaboratively. This form of mixed reality uses holographic computing, such as Microsoft’s Hololens.  

Careers in mixed reality 

Mixed reality technology requires many professionals to design, create, test, and support mixed reality software and hardware. Although we’ve outlined the differences between virtual, augmented, and mixed reality above, many job descriptions and titles aren’t as specific. You may find the work you’re looking for in mixed reality under a job title labeled augmented or virtual reality. Let’s take a closer look at three potential careers in the mixed reality or extended reality field: AR/VR developer, AR/VR software engineer, and virtual reality producer. 

AR/VR developer

Average annual US salary (Glassdoor): $115,374 [1]

Job outlook (projected growth from 2022 to 2032): 25 percent [2]

Education requirements: AR/VR developers typically need to earn a bachelor’s degree in computer information science or a related field, such as engineering or math. 

As an augmented or virtual reality developer, you will help create experiences using AR and VR technology. You will use tools like Unity and Unreal Engine to create software directly, and you may also play a role in designing and testing the software and creating the product's general strategy. In this career, you will be a part of a team of other developers and professionals like designers and project managers.

AR/VR software engineer

Average annual US salary (Glassdoor): $126,431 [3]

Job outlook (projected growth from 2022 to 2032): 25 percent [2]

Education requirements: AR/VR software engineers typically need to earn a bachelor’s degree in computer information science or a related field, such as engineering or math.

As an augmented or virtual reality software engineer, you will design, debug, and develop prototypes for products that create VR or AR environments. In this role, you may be working on building VR/AR software solutions, or you may be working on hardware products to support VR/AR environments. 

Virtual reality producer

Average annual US salary (Glassdoor): $81,180 [4]

Job outlook (projected growth from 2022 to 2032): 7 percent [5]

Education requirements: Virtual reality producers typically need a bachelor’s degree in film, broadcasting, communications, or a related field. Alternatively, you could come into the field from a VR development background with a computer science and information technology degree. 

As a virtual reality producer or content creator, you will create the visual component of virtual reality projects. In this role, you may create content in one of two primary ways: entirely digital means based on coding an interactive experience or collecting video and audio from the real world to populate your virtual world. 

Getting started with Coursera

If you’re ready to take the next step and learn more about a career in mixed reality or extended reality, consider Intro to AR/VR/MR/XR: Technologies, Applications & Issues, a beginner-level course offered by the University of Michigan on Coursera. This course takes approximately 28 hours to complete and is part of the greater Extended Reality for Everybody Specialization.