Virtual Reality Training Vs. Mixed Reality Training – Can You Feel the Difference?
XR-based training devices, those that use head-mounted displays (HMDs) versus traditional dome and flat-panel displays, are becoming more common as XR technologies advance to support more demanding requirements. XR solutions can be broken down into two primary categories: virtual reality solutions, where the trainee is fully immersed in a computer-generated environment, and mixed-reality solutions, where trainees experience an immersive environment consisting of computer generated content and simultaneous access to the physical world.
What are the use cases and benefits of virtual and mixed reality for pilot training? Each option provides features and benefits with associated pros and cons that can impact which option to choose for training deployment.
In pilot training, the adoption of XR-based solutions is often held back by obstacles, such as insufficient resolution and field of view, excessive latency, and ergonomic issues, but when these are addressed with mature headset technology, deployment options abound.
Enterprise-class HMDs, such as Varjo’s XR-3, VR-3, and Aero, are now supporting pilot training programs. For example, the US Navy is reimagining their flight training curriculum to leverage virtual and mixed reality devices to support basic pilot training. The US Airforce is looking to mixed-reality solutions to train for B-52 aerial refueling. In Europe, civilian helicopter pilots can log flight training hours on VR-based training systems.
So what are the benefits of using these XR-based technologies?
Benefits of mixed and virtual reality training
There are many benefits in adopting virtual and mixed reality training technologies. They include:
- Major cost efficiencies. Training costs can be reduced by up to 99%+ with low-footprint, immersive training systems to supplement or replace traditional training solutions such as dome simulators
- Increase the value of existing part-task simulators enabling new training tasks by adding full field of view and immersive visuals using mixed reality technology
- Enable portability of devices to provide training at the point of need, saving travel costs and time away from home for pilots
- Increase training velocity with greater availability of training tools so trainees achieve more reps and sets, enabling more trainees to become proficient in less time
- Provide engaging training that encourages trainees to repeat tasks until they achieve mastery
- All-around improved scalability
When training organizations successfully implement XR-solutions, they will train more pilots faster, provide more flexibility in the training at lower costs than ever before.
Today’s Virtual and Mixed Reality Training Use Cases
XR-based training solutions are used today to support a wide variety of core commercial and military flight training tasks including the following:
- Aviation Aptitude Assessment
- Cockpit Familiarization
- Air Traffic Pattern Review
- Checklist Practice
- Emergency Procedures
- Communication
- Navigation
- Sensor Deployment
- Formation and Tactical Flight
What Is Virtual Reality Training
In virtual reality training, all visual content seen by the trainee is entirely replaced with computer-generated 3D scenes representing the virtual environment needed to support the training scenario.
The trainee is completely immersed in the virtual training environment and is, for the most part, cut off from the real world. This enables full freedom to create and customize different training scenarios as needed and, depending on the device, support interaction with the virtual content and objects using hands, controllers or eyes.
Virtual reality experiences can vary from a full photogrammetric capture of the real world to a computer-generated scene that is 3D modeled and animated in a gaming engine. Some VR headsets are tethered to a PC which is typically needed for rich, realistic content that requires the most powerful graphics. Alternatively, some VR headsets are untethered, with content being created by on-board graphics and computer processors. Where this allows users to move around freely without cables, the quality and resolution of the content is dramatically limited and training scenarios are limited by battery life.
While VR-based solutions can provide an effective, fully immersive experience, they can suffer limitations in a user’s ability to interact with real-world objects. Collaboration with real people such as co-pilots or instructors is limited since they cannot be seen while fully immersed. Communication and interaction with the physical world can be difficult when compared to traditional training or mixed reality.
While VR headsets have limitations not present in traditional devices or with mixed reality headsets, VR-based training solutions can be very effective for training certain types of pilot training tasks, especially those that do not relay on the development of muscle memory or complex interactions with physical devices. Where these requirements exist, mixed-reality training solutions may be a better fit.
Pros of Virtual Reality Training
- Complete flexibility in building the digital experience and supporting any aircraft platform
- True immersion to support the suspension of disbelief
- Smallest footprint and greatest portability
- Lowest cost immersive solution
Cons of Virtual Reality Training
- Isolated experience
- The user is detached from their real surroundings and colleagues
- Limited ability to develop muscle memory
- Limits ability to train some tasks where physical hardware (such as a physical cockpit) is needed to achieve transfer of training
What Is Mixed Reality Training
Mixed reality is all about digitally merging virtual and real-world content inside a headset – enabling a trainee to see and interact with select real-world people and objects while fully immersed in a virtual environment. Ideally, in a mixed reality solution, it’s impossible to discern which elements are real and which are computer generated. Virtual objects appear as a natural part of the real world and will be occluded behind real objects where appropriate. Physical objects around the user can be seen and touched as needed and the real objects can also influence the shadows and lighting of virtual elements.
With mixed reality, a user is not separated from the real-world environment so they can see their hands and body and interact with colleagues and real-world objects, such as physical control sticks and instrument panels. Mixed reality-based devices enable trainees to immediately grasp the point of each lesson and experience the world in proper context as they would in a real aircraft, and they are able to see cause and effect relationships. Most significant is the ability of a trainee to develop true muscle memory, since they can interact with physical hardware as they do in the real world.
For mixed reality solutions to achieve full immersion and suspend disbelief, they must be convincing – blending real and virtual content to the point that it is difficult or impossible to tell where reality ends and the virtual world begins. Mixed reality typically comes with a higher price tag compared to VR solutions due to hardware and development costs, but added fidelity and support for tactile feedback is gained.
The processing power required to collect and process the video and to display these experiences means that mixed reality headsets must currently be tethered to a computer.
Pros of Mixed Reality Training
- Support for tactile feedback and development of muscle memory
- An immersive environment that matches reality
- Enables direct communication with instructors while fully immersed
- Enables use of real flight hardware and operational flight programs
- Suitable for simulations that need to reflect more realistic operating environments
- Provides complete flexibility of the virtual world with the reliability of the real world
Cons of Mixed Reality Training
- Requires physical hardware which can raise costs and limit portability
- More complex solution development
- Requires tethering to a PC for sufficient graphics quality
- Higher price and weight compared to VR headsets
- More complex to set up and align