Mixed reality

Mixed reality, or MR, is a combination of augmented reality and augmented “virtuality” in which virtual and real-life objects interact in a physical space. Mixed reality, often referred to as hybrid or extended reality, is considered the next wave in computing after mainframes, PCs, and smartphones. It provides both the semi-digital world of augmented reality and the fully digital world of virtual reality and is going mainstream for both consumers and businesses.

The most current examples include Instagram or Snapchat filters, virtual makeup applications, and virtual furniture fitting and arrangement, among many others. Mixed reality, also called augmented reality 2.0, highlights the essence of the natural world while promoting a digital environment with visuals. A key difference between mixed reality and virtual reality is that MR does not detach users from their physical worlds.

Background

Virtual reality (VR) has centuries-old roots dating to the stereoscope, a hand-held device British scientist Sir Charles Wheatstone invented in the early 1800s. Stereoscopes use mirrors to blend twin photographs into a single image and create the illusion of depth.

American Morton Heilig then patented the Sensorama Simulator in 1962. His immersive sensory environment functioned in the manner of a gigantic View-Master, allowing a user to experience stereoscopic images paired with stereo sound and aromas.

The label virtual reality picked up steam in the 1980s and is usually credited to American computer scientist Jaron Lanier. Lanier is the inventor of the dataglove, an input device worn like a glove that is used to report the position of a user’s hand and fingers to a computer.

On the business side, Boeing researcher Thomas Caudell coined the term augmented reality in 1990 to describe the workings of the head-mounted displays Boeing’s electricians were then wearing. These headsets could project a digital version of wiring maps onto plywood, eliminating the need to physically print those maps. Using augmented reality allowed electricians to modify the wiring maps digitally in real time and saved Boeing money as well.

On the consumer side, among the first commercial applications of augmented reality technology were the yellow first down lines that appeared in televised football games beginning in 1998. Another example was the smartphone app Pokémon Go, which layered digital characters on the phone camera’s field of vision.

The first mention of the term mixed reality came in the 1994 research paper “A Taxonomy of Mixed Reality Visual Displays” by Paul Milgram and Fumio Kishino. The pair viewed MR as a combination of visual displays that elevate real and virtual environments to allow all elements to communicate.

Overview

Mixed reality requires discrimination between virtual reality and augmented reality. Virtual reality puts users in a fully virtual world and completely replaces their experience, while augmented reality inserts virtual objects and information into a user’s real world. The mixed reality combination presents real world and virtual world objects together within a single display.

At least two forms of reality technologies are referred to as mixed reality. They are mixed reality that starts with the real world and mixed reality that starts with the virtual world. In the former, virtual objects can interact with the real world and a user can interact with virtual objects. This is an advanced form of AR. In the latter, the digital environment replaces the real world and a user is fully immersed in the virtual environment with the real world blocked out. It sounds like virtual reality, except the digital objects overlap the real objects.

Mixed reality is quickly integrating into many enterprise digital transformation initiatives. Nearly 90 percent of businesses are reportedly exploring, piloting, or deploying mixed reality, and 69 percent believe mixed reality is critical to achieving strategic goals, according to the Harvard Business Review.

Examples of large companies making notable use of MR include Airbus, which is using the Microsoft Hololens MR headset to enhance its manufacturing and design efforts, and Mercedes Benz, which is using MR solutions for remote technical support. With MR, a specialized technician in an office can help field technicians repair complicated parts, thereby reducing repair time and saving money on travel.

Many other MR applications in production, construction, or logistics also are possible in the corporate environment, including optimization and acceleration of industrial processes; immediate display and validation of planning status; collision control; evaluation of the design of new product components; virtual guidance on real products; simplified target/actual comparison; improved team-internal communication; and improved customer communication.

Specific examples include construction supervisors using MR to inform workers about required tasks. A supervisor’s virtual pin might send the message that a piece of equipment is not working correctly, and workers will use headsets to view schematic diagrams and diagnose the problem. Manufacturing inspectors can use headsets during plant tours and view real-time data for mechanical components. In construction, MR has been especially helpful in site-mapping and visualization.

Mixed reality has been used frequently in marketing, particularly to consumers who prefer to experiment virtually with a product before purchasing it. Using MR, designers also can interact with new products and prototypes before they are built. International corporations can simulate face-to-face meetings with product teams across a wide map, and participants can view a life-size, 3D version of the product under discussion.

MR has also made notable inroads in health care. MR simulations have aided in medical training as well as surgery. Surgeons can use MR to practice inserting medical implants into patient’s bodies, and medical students may soon be using MR imaging instead of cadavers in anatomy classes.

Some applications and possible applications are considered playful and new concepts continue to emerge. For example, a television studio might combine real actors with virtual reality in real time in a virtual studio, or electronic devices may exist only virtually but respond to real touch, enabling artificial sensory enhancements such as X-ray vision and open-world computer games.

Navigation is a notable field in which mixed reality is conceivable in several different applications, including maintaining complex industrial plants; disaster management; automotive cockpit displays that offer the driver graphical navigation and traffic information in live images; and aircraft MR used with head-mounted or head-up displays to increase safety and efficiency in poor visibility and unfavorable weather. One of the first industrial applications of MR took place in military aircraft.

As a navigation partner, Google began offering users live MR navigation in Google Maps. Pedestrians can use live view instead of the normal 2D map view to see directional arrows, street names, and other information.

Stylized, enhanced education/training, also called edutainment, is another area ripe for MR applications. Learners can view material on a computer via webcam and receive informational elements that further explain the topic, with specialized sound sequences part of the package. This approach provides a feeling of tangible proximity and individual and interactive help and can reduce or eliminate learning barriers.

Mixed reality experiences require both cloud computing and artificial intelligence, and current content is typically delivered through headsets, which cost anywhere from $300 to $3,000. Advanced AI sensors, cameras, and graphics processing units (GPUs) power MR devices, such as smart glasses, gloves, body suits, and even smartphones, to process and store data in three dimensions. The better equipped an MR device, the better the mixed reality experience.

MR devices can connect users to a wired or wireless computer, console, or PC to access software that can add, clone, or move virtual objects around users to create new and different immersions. New-age mixed reality headsets, such as HTC Vive or Meta Quest 2, create engaging high-fidelity virtual environments that eliminate the barriers between reality and technology.

Certain MR applications scenarios should be and are easily distinguished from VR scenarios according to a peer-reviewed research team led by Anett Mehler-Bicher. One example is a Living Mirror, in which a camera detects the viewer’s face and places three-dimensional objects on it, with projection typically done via a large screen or a beamer to create a mirror effect. Another “living” example is Living Print, a scenario in which a print medium is recognized and corresponding augmentation is introduced. Distinction is made between different print media, such as greeting cards, brochures, leaflets, or packaging materials. Living game mobile, living architecture, living posters, living presentations, and living meetings are other distinguishable examples of MR applications.

Mobile devices, such as smartphones, are the basis of living game mobile. In living architecture, a viewer can experience a room or a building by determining its representation through movements such as turning the head or walking through a real room and using speech or gestures. A living poster is an advertising message in a public space that is enhanced with manipulative information elements using augmented reality. Living presentations refer to presentations, such as those at trade show booths, made spectacular and more engaging by MR technology.

In a post-pandemic world and due to increasing globalization, an increased number of meetings take place as teleconferences or videoconferences. When augmented reality enhances teleconferences or video conferences, they can be referred to as living meetings.

All augmented reality applications that use mobile systems to enhance real environments or facilities with more information of any kind, such as text, 2D or 3D objects, video, and audio sequences, are called Living Environment. The goal is for a user to acquire extra information in a timely fashion when a camera captures one or more objects.

Once thought of as the stuff of science fiction, mixed reality is now a widespread reality. Some of its most highly touted benefits include the creation of memorable customer experiences, concentration enhancement, extremely personalized media experiences, and fewer accidents and mishaps in dangerous scenarios. Downsides include high upfront costs, a shortage of highly trained specialists and engineers, slower return on investment, and customer apprehension.

Industry experts predict that mixed reality will become a $1.2 billion to $6.9 billion global industry by 2024. Although gaming will continue to play a key role in its development, enterprise MR is expected to dominate future research efforts. Camera-equipped smartphones, tablets, and smart eyeglasses are soon expected to replace headsets, and rumor holds that Apple is developing mixed reality eyeglasses that will connect to the user’s iPhone, much like the Apple Watch.

An up-and-coming form or mixed reality includes natural feature recognition that allows tracking without the use of artificial markers and recognizes unknown environments and thus new application possibilities. Further development of this technology will allow mobile devices to recognize natural environments and enrich them with virtual objects. Face tracking can make possible new forms of communication with viewers. The viewer is recorded as a face, and age- and gender-specific characteristics are recognized, evaluated, and interpreted, resulting in a corresponding individual response. Advertising is just one area in which this would be particularly helpful. While posters currently reflect static information, it is possible to develop interactive posters based on mixed reality that react smartly to their counterparts. Based on age- and gender-specific characteristics, a mixed reality poster would react individually and offer suitable advertising messages.

Technology experts began 2023 with a prediction that it would prove to be the year of mixed reality, highlighting a metaverse that is more real and impactful than cartoonish or escapist. While virtual reality can fascinate and captivate, these experts have said, most people have a limited comfort level with detachment from physical reality. The real metaverse will not separate people from their physical surroundings but seamlessly combine immersive virtual content with the physical world.

A wave of new MR products already has been introduced in 2023, but many more are on the horizon. Developers are expected to continue to push the limits of creativity and artistry in MR, because in many ways, experts say, mixed reality is still only just beginning.

Bibliography

Bayern, Macy. “Your Guide To Mixed Reality Technology.” ZDNET, 2 Oct. 2019, www.zdnet.com/article/your-guide-to-mixed-reality-technology/. Accessed 10 Aug. 2023.

“How Mixed Reality Solutions Can Take Your Business to the Next Level.” 4Experience, 2022, 4experience.co/how-mixed-reality-solutions-make-businesses-grow/. Accessed 16 Aug. 2023.

Mattoo, Shreya. “What Is Mixed Reality? The Better Side of Technology.” G2, 16 Sept. 2022, learn.g2.com/mixed-reality. Accessed 10 Aug. 2023.

Mehler-Bicher, Anett, Lothar Steiger, and Dirk Weitzel. “Mixed Reality Applications in Business Contexts.” Intechopen, 5 Aug. 2022. www.intechopen.com/chapters/82961. Accessed 16 Aug. 2023.

Rosenberg, Louis. “Why 2023 will be ‘The Year of Mixed Reality.’ ” Big Think, 3 Jan. 2023, bigthink.com/the-present/2023-year-mixed-reality/. Accessed 16 Aug. 2023.

Tokareva, Julia. “The Difference Between Virtual Reality, Augmented Reality and Mixed Reality.” Forbes, 2 Feb. 2018, www.forbes.com/sites/quora/2018/02/02/the-difference-between-virtual-reality-augmented-reality-and-mixed-reality/?sh=792d058f2d07. Accessed 10 Aug. 2023.

“What Is Mixed Reality?” Microsoft Learn, 24 Jan. 2023, learn.microsoft.com/en-us/windows/mixed-reality/discover/mixed-reality. Accessed 10 Aug. 2023.

Wigmore, Ivy. “Mixed Reality (Hybrid Reality, Extended Reality).” Tech Target, 2023, www.techtarget.com/whatis/definition/mixed-reality?Offer=abt‗pubpro‗AI-Insider. Accessed 10 Aug. 2023.