One of the most important and sensitive challenges in the staff training is the creation of an environment in which the user can learn in a safe and isolated manner from the company’s production domain.
The structure may differ depending on the type of training, but in any case it is necessary to reproduce an ecosystem as similar as possible to the real case in order to not confuse the user during his actual entry into the practical world. Training can require considerable costs and resources depending on the job (practice with the maintenance of a car engine, practice with a surgical operation, etc…) thus representing a demanding challenge that must be overcome in the best possible way. Would it be possible to simulate the training that could lead to a simplification of the infrastructure dedicated to the training? Mixed reality could be a valid solution to create a job simulator in a reliable and credible way.
But what is mixed reality?
Augmented, virtual and mixed reality
Before explaining what mixed reality is, it is good to mention two other conceptually opposite realities that form the mixed reality.
Augmented reality
Maybe this term doesn’t ring a bell to someone, but there is a mobile app that is a very good example of what really is the augmented reality: PokémonGo.
This app allows you to fight and capture different types of Pokémon by simply walking around the streets and cities. A very special feature of PokémonGo is the ability to place a creature on flat surfaces such as tables or bedside tables and this is a synthesis of what augmented reality does.
This technology is mainly based on the real world which is represented, for instance, through the phone’s camera. The real world is then “augmented” via virtual components (like Pokémon). The application, thanks to image analysis, manages to understand if in the video there are surfaces where a Pokémon can be placed to give the feeling that the virtual avatar is actually on the table. There are other techniques that can be added to give more realism and coherence to the avatar, such as adding his shadow on the flat surface. However, it is important to note that there can be no interaction between the real world and the virtual one. All augmented reality does, is add some value to the real world.
Virtual reality
On the other hand there is the virtual reality where instead of using the real world as a foundation, it creates a new whole virtual world. To achieve this, there are special helmets (Oculus Rift, HTC Vive) with the aim of “deceiving” the user’s brain into believing that he is elsewhere. The person is equipped with a helmet that detects the movements of the head and two controllers that are equally detected in order to give the user an interaction with virtual components. The possibilities offered by the applications are infinite and it is possible to obtain a remarkable realism (even knowing that the user is conscious that what is presented is fake) thanks to the accuracy of the movements captured by the device. In virtual reality there is therefore no trace of real elements.
Mixed reality
Mixed reality is the union of the characteristics of the two previously described realities. The real world is the main scene where the user operates but, unlike augmented reality, there is a deeper knowledge of what surrounds the person and an interaction between real and virtual elements can be offered.
Microsoft has been working for years on a helmet called HoloLens that allows you to work with mixed reality.
The Microsoft’s helmet uses semi-transparent lenses to be able to show virtual components on the glass in the form of holograms placed on the real world. In order to place virtual objects in an intelligent and plausible way, the helmet has several sensors placed around the head with the sole purpose of mapping the surrounding area and converting it into a digital format. This process is called “Spatial Mapping” and is a key feature in the world of mixed reality. In other words, “Spatial Mapping” allows applications to be informed about what surrounds the user (walls, tables, chairs, floor, etc…). The better are the obtained information, the better is the quality of each hologram placed around the user because they will be positioned in the world in a more accurate way.
In the first version of the HoloLens, the user had only one possible gesture in order to interact with the digital components, thus limiting the naturalness of the user’s interaction with the virtual world.
With the arrival of the second version of Microsoft HoloLens, interaction with digital components has become richer and more natural thanks to the recognition of finger and hand movements. Because of this new feature, the user is able to manipulate virtual elements instinctively without having to learn how to use the helmet.
Simulation using mixed reality
So how can mixed reality be used to simulate an environment in which a person can be formed?
The first step is to inform the helmet about the area where the training will take place. In case the “morphology” of the training area is not complicated and particular (a normal room with easily recognisable objects), the “Spatial Mapping” process is used. However, in most cases, you have to work on a specific object or machinery that must necessarily be reproduced as precisely as possible in a digital format (for example a car engine). In these cases, you have to rely on another technique.
So let’s take an example: training a mechanic on the maintenance of a particular car engine.
You might have a really car engine where you will need to perform manual operations. The simulator inside the HoloLens can inform the user about these operations step by step by indicating the position of the components to be worked on and also showing the action to be taken. Other demonstration elements can be added such as a screen in which you can play a video with also an explanatory text. In order to do all this, the helmet must know exactly the type of engine, its dimensions and the position of each internal component. In this case, the helmet may not be able to create an accurate and detailed virtual version of this engine, especially due to the fact that some components are inside and invisible from the outside. In this case, you can rely on an accurate representation of the engine in a digital format that can be placed over its real counterpart. With this approach, the helmet interacts with the digital model (invisible and places exactly like the real engine) while the user only sees the interactions with the real engine. Finally, thanks to the tracking of hand movements, the application knows exactly how the user is interacting with the engine and can intervene in case he is doing something wrong or in case he needs help.
Let’s take another example of training in the field of medicine /chemistry: the manipulation of a fluid using two test tubes.
During the various processes necessary for the creation of a drug, it is necessary to carry out manipulations of fluids in a sterile and controlled environment. A special infrastructure allows users to transfer fluids within a vertical air stream that allows to limit the contamination of these fluids.
The operations and movements that take place inside the vertical flow are very limited and must take place in very specific ways in order to not “break” the air flow, thus affecting the quality and purity of the fluid. Working in these conditions requires a heavy training phase to accustom a person to move correctly. The vertical flow is invisible to the human eye and it is therefore more complicated to instruct the person on the consequences of his movements without visual feedback. Using mixed reality, however, it is possible to simulate the physical behaviour of the vertical flow and to visualise its movements in order to facilitate the learning of this physical phenomenon.
The task of the simulator inside the HoloLens helmet would be to detect the movements of the user’s hands within the infrastructure to influence the vertical current. Additionally, the simulator has to inform the user if he is correctly proceeding or if it is necessary to correct his actions. At the end of the procedure, the user can be evaluated based on his behaviour by showing a report of the correct or wrong operations in order to improve.
Final considerations
Mixed reality is still a relatively new and underused technology that is therefore slowly evolving. However, from the results obtained so far, it seems a very promising solution, especially if used in the field of education. Microsoft is spending a lot of resources to advance in this world and with the advancement of the quality of movement tracking (not just arms and hands!) And the computing power, it is more and more possible to approach the virtual world in a natural way thus helping the application ergonomics. We are still a long way from using this technology in everyday life, but little by little we could begin to opt for this solution event for small things, receiving excellent surprises in exchange.
