Transfer of Skills Evaluation for Assembly and Maintenance Training

One of the research topics within the EU-project SKILLS was the training of Industrial Maintenance and Assembly (IMA) tasks. The IMA demonstrator developed comprehends two different training platforms, one based on technologies of Virtual Reality (VR) and the other one on Augmented Reality (AR). To qualify the efficiency of the developed training systems different studies have been conducted, followed by a final by service technicians strial training c Parma. This evaluation included qualitative methods (feedback collection in questionnaires) as well as quantitative methods (experiments with control groups). The results demonstrate that both platforms are useful and suitable training tools for IMA tasks, and that the AR training decreased the number of unsolved errors in the task.


Introduction
Industrial maintenance and assembly (IMA) tasks are complex tasks that require the knowledge of specific procedures and techniques for each machine.Therefore, the training of the service technicians to acquire the necessary skills to perform those tasks efficiently is a challenging point.The main skill involved in this type of tasks is the procedural skill, which is based on getting a good representation of the task organization: what appropriate actions should be done, when to do them (appropriate time) and how to do them (appropriate method).Following the guidelines and recommendations obtained along different experimental activities, two IMA training platforms were developed.
This paper presents the results of the final evaluation of both platforms that was carried out by expert 1 IST FP6 ICT-IP-  technicians in the training facilities of SIDEL in Parma.In addition to analyzing the transfer of skills, this evaluation collected the feedback of the technicians about the easiness of use and effectiveness of both platforms through specific questionnaires.The results show that both platforms offer high potential for exploitation in the field of industrial training.

IMA platforms
The IMA demonstrator comprehends two different platforms, one based on VR technologies (IMA-VR) and the other one on AR technologies (IMA-AR), fully covering the different aspects and constraints of a training process.Although different technologies are used in each platform, both follow the recommendations obtained in previous experiments (e.g.: [1], [2], [3], [4], [5], [6], [7]), mainly: provide all the aids in a controlled way, divide the whole task into a set of logic sub-tasks (set of related steps with a common goal), and provide enriched information about these sub-tasks to facilitate the trainees to develop an appropriate and more accurate mental model of the whole task.

IMA-VR platform
The IMA-VR platform provides enactive training in a controlled multimodal virtual environment for transferring the motor and cognitive skills involved in assembly and maintenance tasks.It supports the aplearning by doing ctive multimodal interaction (visual, audio and haptic) with the virtual scenario, eliminating the constraints of using the physical scenario (such as availability, time, cost, and safety constraints).With this platform, the trainees interact and manipulate the components of the virtual scene, sensing the collisions with the other components and simulating different assembly and disassembly operations (see Figure 1).
The system also provides different type of information along the training session, such as: information about o , technical description of the components and tools, critical information of the operations and detailed description of errors.The critical information can also be sent through audio messages.
The system also logs automatically information about the task execution for further analysis of performance.
This platform has the flexibility to adapt itself to the demands of the task and trainees.For example, it can be used with different types of haptic devices (e.g.desktop or large space devices) and it provides different learning strategies to guide trainees during the training process, including the possibility of remote supervision/training through an on-line interaction between trainer/trainee.During the skill transfer evaluation the trainees were trained with one learning strategy based on subtask level est, this strategy provides information about the current sub-task by means of: 1) visual aids: showing a second display with a copy of all the pieces involved in the current sub-task in their final position, and 2) textual messages: displaying the name and description of the current sub-task/step and a dynamic list with the names of the pieces/tools involved in the current step.If this aid is enabled, but the trainees do not know yet how to continue with the task, they can request an additional aid to obtain direct information about the immediate action within a step, for example: the target piece is highlighted.

IMA-AR platform
Within the AR-training system the trainee is performing assembly operations on the real machines using the real instruments for interaction.Thereby the trainee is guided via a mobile platform supporting AR-Visualisation as well as Tactile Feedback (c.f. Figure 2).The AR-platform gives information on request only that means, Virtual Post-Its are linked to the tracked machine parts.The Virtual Post-It icon indicates that instruction is available that illustrates the handling of this tracked machine part.Only if the trainee needs this instruction she/he selects the Post-If and the linked information is visualized.Thereby the visualized information is separated from the tracking.Beside Virtual Post-Its contextual illustrations are superimposed (on request of the trainee) to illustrate the purpose of the current task within the assembly procedure.

Evaluation Goal and protocol
The skills transfer study was run at the end of the project in order to evaluate the two platforms described above.The goals of this transfer study were: a) to analyse the efficiency of both platforms as training tools for industrial maintenance activities and b) to compare the trainees performance trained with both platforms and traditional training methods.This evaluation involved four experimental groups: Group 1 Control-VR: participants watched twice an instructional video showing the steps of the task.Group 2 VR: participants performed twice the virtual task using the IMA-VR platform with a large see Figure 1) Group 3 Control-AR: participants performed once the physical task while they were watching an instructional video showing the steps of the task.Group 4 AR: participants performed once the physical task using the IMA-AR platform.The experimental task was composed of 25 steps grouped in 6 sub-tasks that are showed in Figure 3.
Forty technicians from Sidel served as participants.They had at least 2 years of experience on field assembly/disassembly operations and a discrete competence in the use of personal computers and hi-tech in general.The evaluation followed a between-participants design, hence each trainee was assigned to only one experi-BIO Web of Conferences 00028-p.2mental group (ten participants in each group).Before starting the evaluation, the participants filled in a demographic questionnaire whose answers were used to distribute them along the four groups in a homogeneous way.The previous capability of participants was tested through a simple assembly task just before starting the test (capability test).
Table 1.Skills transfer study results All the experimental groups followed the same protocol.In the morning, participants were trained in the target task: learning how to assemble an electromechanical actuator.Later, in the afternoon, they had to perform the physical task (the same one as in the training session) by themselves and without any help.But if for any reason they could not continue with the task they could consult a book of photographs of the task and this action was re end, the VR and AR groups were requested to fill in further questionnaires in order to gather their feedback in terms of easiness of use and effectiveness for training purposes of both platforms.The main data analysed were: training time, real task performance (c.f.Table 1) and the subjective evaluation of the platforms.Since the experimental task included only 25 steps, every repetition on it during training could have resulted in better performance.Hence, it was possible to compare only experimental groups with the same number of repetition during the training: group 1 vs. group 2 (2 repetitions) and group 3 vs. group 4 (one repetition).

Evaluation results and analysis
The demographic data of participants as well as their score in the capability test were similar among the groups.Generally speaking, performance in the real task was good for both training platforms, VR and AR, without any aid and with almost no errors during the task as well as without unsolved errors.The difference in performance between the VR and the Control-VR group was not significant. .It should be noted that the VR platform provides a new interaction paradigm that needs time to use it efficiently, so it is expected that future results of training in the VR platform will be even better.For the AR platform, while performance time and the number of solved errors was not significantly different compared to the Control-AR group, the number of unsolved errors was significantly smaller (t(18) = 2.52, p = .02)in the AR group.This result is of great importance since it indicates that wit traditional training, hands-on using only instructional video, there is a danger of performing too many errors (1.3The International Conference SKILLS 2011 00028-p.3prevent these errors by emphasizing the key points in the task.Also true for the AR platform is that when trainees will gain experience using it and several technical issues will be solved, it will be even more effective.

Conclusions and future work
Within an exhaustive evaluation study, the two IMA platforms have been evaluated in comparison to the state-of-the-art training methods (instructional videos with and without on-line performance of the real task during watching).In summary, the results of the skills transfer evaluation performed by the technicians from Sidel demonstrated both IMA platforms are useful and suitable training tools.In both cases, all the technicians performed the real task without any aid and most of them (80%) performed it correctly being the average of unsolved errors very low (0.3 in the IMA-AR and 0.4 in the IMA-VR platform).Performance following the VR training was similar to performance following traditional training, and the AR training significantly decreased the number of unsolved errors.
The result is promising, as the service technicians had very low background in VR/AR technologies, so it is expected that the productivity and the efficiency in the use of the systems will increase significant with the familiarization of AR/VR technologies.This is in particular the case for the VR platform, as here completely new interaction paradigms are used, while in Augmented Reality interaction mostly is performed on the real machinery.As important as the quantitative -of-qualitative feedback collection using questionnaires.Although the developed technologies are highly exceptional for the service technicians, most of them would mostly recommend both platforms, VR and AR, as a training tool.The exploitability of the platforms in particular is supported as flexibility of the platforms has been a clear focus within the development.Thus, the technology can be easily transferred to new hardware platforms (e.g. using different haptic devices in VR or using Smartphone technologies for AR) and also new content with different training protocols can be generated easily.
Finally, it is important to take into account that the -of-ted to proof the potential of the IMA-AR platform (but it was used to be comparable to the IMA-VR platform).Therefore a large area scenario has been realized within the Sidel training centre.For this scenario also developed technologies like, e.g., the Head Mounted Display, has potential (for the desktop sized workspace the HMD has been eliminated as it was not accepted in usability studies).

Acknowledgments
The activities described in this paper were carried out with the financial assistance of the EU which cofunded the project SKILLS.Authors wish to thank especially to all the scientists/technicians who helped in the development of the IMA demonstrator that has been developed in close cooperation with all SKILLS partners in particular with the colleagues of PERCRO, DLR, Aalborg University, OMG, KUKA, Queens University Belfast, University of Montpellier, CEA-List, Haption, OMG and PERCRO.

Figure 1 .
Figure 1.left: manipulation of virtual components with the IMA-VR platform; right: information presentation at the sub-task level.

Figure 2 .
Figure 2. AR-based Training at Sidel Training Center.

Figure 3 .
Figure 3.The actuator assembly task composed of 25 steps grouped in 6 sub-tasks.
in average for this short task), maybe due to overconfidence.As demonstrated, a proper AR training may help to