Reality to the rescue
We've all been there: just bought a flat-packed piece of furniture and the 2D instructions do not quite match the 3D version of bits and pieces of wood in front of you. In our case we can simply phone up a friend or even the shop from which the furniture was bought to get help. But what if you're an astronaut on the International Space Station or even further away as will be the case in future missions to the Moon or Mars?
Astronauts face operational problems almost on a daily basis and these difficulties make putting together your flat-packed wardrobe look like peanuts. Although they are of course trained extensively for almost any scenario, including the use of virtual reality as an enhancing technique, quite often the decisions they need to make would become much easier if they had some sort of help (especially in cases where contact with Earth is not possible or severely time-delayed).
ESA has been studying Augmented Reality (AR) and Virtual Reality (VR) as a possible aid for astronauts for several years now. A number of studies and projects have already shown the advantages of AR/VR for instance in the field of medical procedures.
In any six-membered crew it can be expected that one member has been trained in medical procedures. But on a mission to Mars or even on the Space Station it may eventually be needed to carry out an emergency medical procedure for which this crew member may not have the necessary knowledge or, worse, what happens if the doctor becomes the patient?
Several ESA astronauts have already been involved in testing AR/VR equipment, such as WEAR (Wearable augmented reality), EdcAR (Engineering Data in Cross-Platform AR) and 3DViT. WEAR includes onboard location and object identification to show astronauts precise information about what they are looking at, as well as step-by-step instructions to guide them through lengthy procedures.
The EdcAR project developed a new AR system that considerably reduces the number of mistakes in the performance of tasks while speeding up and clarifying operations. As such the system could be used in future operations at the international space station, in particular for in-flight Maintenance tasks. The first practical tests carried out at ESA's Astronaut Centre (EAC) were successful.
Andreas Mogensen also used VR techniques during his iriss mission to perform maintenance tasks on the space station; the 3D visual training tool (3DViT) was successfully put through its paces. 3DViT provides efficient training on the spot where needed and can be expanded with a number of routine tasks.
ESA's Advanced Concepts Team (ACT) also researches the use of VR and AR to complement future space missions. In particular, the team is investigating the idea of using a VR headset such as the Oculus Rift to remotely inspect the space station via images streamed from the stereo-vision camera of the SPHERE drones currently on-board the station. However, for complicated operational procedures VR and AR systems have so far not been studied extensively. Especially in emergency situations, where every second counts, AR technology will be indispensable.
A new GSP study called "Augmented reality and Virtual reality techniques to enhance the execution of operational procedures by astronauts and spacecraft operators" has been started to assess whether and in which form AR and VR techniques can aid the performance of astronauts and spacecraft operators.
The main objective is to try and root out human error in stress and emergency situations where often the main cause of problems is the discrepancy between the 2D and textual description of such procedure and the 3D reality.
This article was first published on the ESA Discovery and Preparation pages.
