Learning new skills in Multimodal Enactive Environments

A European consortium of researchers in movement and cognitive sciences, robotics, and interaction design developed multimodal technologies to accelerate and transfer the (re)learning of complex skills from virtual to real environments. The decomposition of skill into functional elements the subskills and the enactment of informational variables used as accelerators are here described. One illustration of accelerator using virtual reality in team rowing is described.


Introduction
Skill is the capacity acquired by learning to reach a specified goal in a specific task with the maximum of success and a minimum of time, energy or both.Technology such as virtual reality devices or multimodal human-machine interfaces is more and more used, obviously for financial or technological reasons, to speed up the learning of new movements or procedures.
Training in technological environments becomes a new era both for trainers and trainees, allowing the elaboration of new training scenarios and training protocols.However, the positive or negative transfer effect of this technologyoriented training (i) often seems to contradict fundamental knowledge about skill learning, (ii) requires the adequate decomposition of (complex) skill into functional elements that make sense from theoretical and pragmatic point of views, and (iii) remains to be proven using careful evaluation and transfer studies.Figure 1 illustrate six technological human-machine interfaces developed within the SKILLs european project, aiming at accelerating and transferring the (re-)learning of complex coordinative skills in virtual and real environments, and for which the three above aspects were carefully addressed.Various domains of application are concerned including sport and entertainment, surgery and rehabilitation, and industrial maintenance.

Skill decomposition
The decomposition of complex skills in functional, (re-)trainable elements is an important step in technological training, both from theoretical and practical standpoints (see [1] for details).Elementary functional units are often described as coordinative structures [2], or motor primitives [3].Coordinative structures are temporary (soft) assemblies of elements constrained to behave as a single functional unit, while motor primitives are force fields generated by muscle units, resulting from the stimulation of hardwire neural circuitry.Stable attractors (fixed points vs. limit cycles) are also envisaged to constitute the building blocks of (continuous vs. discrete) movements and skills [4].Pragmatically, skill decomposition raises the question of which skill elements contributing to the global performance can be temporarily isolated from their neighbours, receive a specific training, before being incorporated again in a more general training protocol.In Table 1, the fifteen skill elements selected by the SKILLS consortium the sub-skills are described.They cross many of the complex skills described in Figure 1, and received a specific attention.The first nine sub-skills (upper section of Table 1) are from the sensorimotor repertoire while the last five sub-skills (lower section of Table 1) are from the cognitive repertoire.In general, sensorimotor sub-skills are skills that relate to the relationship between perceptual components and motor components, and cognitive skills are related to higher-level cognitive activities that orient, formulate, monitor and regulate sensorimotor performances.The distinction is partly arbitrary as the two categories are obviously largely interdependent.It is however a convenient way to help researchers and designers to elaborate technological tools to enact and learn these skill elements.
These 15 sub-skills are general, abstract, well documented, and cross a large number of domains, from rehabilitation to surgery to industrial maintenance to sport and entertainment.For instance, bimanual coordination is a very generic (sensorimotor) sub-skill that exists both in juggling, rowing, or surgery.It includes the concatenation of limit cycle primitives spanning over several muscles and joints.Similarly, procedural sub-skills are (cognitive) components that exist both in complex surgery and industrial maintenance applications.Among a virtually infinite number of sub-skills that compose human activities, the fifteen elements described in Table 1 have been selected because of (i) their key importance for the successful achievement of skilled performance, (ii) their possible yet challenging enactment using multimodal virtual reality technology, (iii) their coverage of complementary perceptual modalities or effectors, (iv) their visible evolution over time and learning, and (v) their anchorage in a solid state-of-the-art in basic human movement and cognitive sciences.Each sub-skill is matter of theoretical and experimental research, is defined by specific variables, can be captured and rendered using specific hardware and software technologies.During the coarse of the SKILLS project, they have been investigated in several evaluation and transfer studies [5].They have been shown to be the main components of efficient and adaptive behaviours in technology-driven learning scenarios.It is throughout the interaction of these (and other) sub-skills, under appropriate practice and training conditions, using multimodal interfaces that skilled behaviours progressively (re-)emerge.
Table 1.Sensori-motor and cognitive skill elements investigated in the SKILLS european project.

Balance / postural control (BPC)
The regulation of posture (segments, muscles, joints, etc ) and balance (static, dynamic, etc ) allowing the distal/manual performance to be successfully achieved.BPC is captured by inter-segmental and inter-muscular coordination, as well as center-of-pressure variables [Nashner & McCollum, 1985].

Bimanual coordination (BC)
The functional synchronization in space and time of the arms/hands/fingers.BC is captured by the relative phase between the coordinated elements and its stability [Kelso, 1995].

Hand eye coordination (HEC)
The synchronization of eye / gaze / effector with reference to the main information perceptually detected.HEC is assesses by gain, relative phase, and in general coupling variables between eye, gaze, and hand [Biguer, Jeannerod, & Prablanc, 1982].

Interpersonal coordination (IPC)
The coupling between two or more persons.It emerges from a nexus of components including sociality, motor principles, and neuroscience constraints.PC is assessed by the relative phase between persons [Schmidt & Turvey, 1994]

Perceptionby-touch (PbT)
The coetaneous component of the haptic modality.Various receptors embedded in the skin provide information about mechanical properties (vibration, compliance and roughness), temperature and pain.PbT is evaluated by psychophysical methods of touch perception [Klatzky & Lederman, 1999].

Prospective control (PC)
The anticipation of future place-of contact and time-to-contact based on spatiotemporal information contained in optic, acoustic, or haptic energy arrays.PC requires the coupling between movement parameters and information contained in various energy arrays, and is measured by time-to-contact and related variables [Lee, 1980]

Proximodistal coupling (PDC)
The spatio-temporal coordination of proximal, gross components with distal manipulatory components.PDC refers to the organization of the body underlying arm movements, or to the synergy between arm postures and hand movements.PDC is assessed by cross-relational variables [Berthier, Rosenstein, & Barto, 2005].

Respiratorymovement coupling (RMC)
The synchronization of breathing and movement (segments, muscles, joints, etc ) that allows efficient performance.RMC is measured by amplitude, phase and frequency synchronization patterns [Bramble & Carrier, 1983].

Fine force control (FFC)
The online regulation of the internal forces applied on the surface to successfully reach the goal (drilling, pasting, navigating, etc ).FFC depends on the properties of the surface in relation to the forces developed by the effectors, and is evaluated by the ratio between the two [Faisal, Selen, & Wolpert, 2008].

Control flexibility and attention management skills (CFAM)
The ability to change response modes and performance strategies, to apply and manage new attention policies in order to cope with task demands and/or pursue new intentions and goals.CFAM is measured by adjustments to changes in task demand and attention allocation [Gopher, Weil, & Siegel, 1989].

Coping strategies and response schemas (CSRS)
A vector of importance or attention weights computed over the many sub-elements of a task, which are associated with the achievement of a specific goal.CSRS is measured by the number and type of strategies to cope with variations in task demand and change in intention [Gopher, 1993].

Memory organization, structure and development of knowledge schemas (MO)
Level of formulated and organized multi hierarchy, task specific memory and knowledge bases that facilitate encoding, retrieval and the conduct of performance.MO is measured by speed and accuracy of encoding, response and decision-making performance, and number, diversity and speed of generating alternative solutions [Baddeley, 1997].

Perceptual Observational (PO)
The ability to detect, sample and extract task relevant information from the environment and perceive static patterns and dynamic regularities.PO is measured by speed, efficiency, amount of conscious supervision, and use of higher-level structures and redundancies [Carlson, 1997].

Procedural skills (PS)
Sequences of ordered activities that need to be carried out in the performance of tasks.Performance of every task can be subdivided into a large number of procedures, the competence in the performance of which is developed with training.PS is evaluated by speed and efficiency of performance, type of supervision (un/conscious) [Anderson & Lebiere, 1998].

Enactive learning and skill acquisition
As mentioned in the previous section, the distinction between sensori-motor and cognitive sub-skills is pragmatically operational but partly arbitrary as the two categories are largely interdependent.This is because of the natural embodiment of cognitive phenomena into sensorimotor dynamics, and the possibility to enact highly cognitive phenomena using contemporary rendering techniques.Although there are several views on embodied cognition and enaction [6], the terms refers to the basic fact that cognition is largely for action, that off-line cognition (cognition decoupled from the environment) is largely dependent on the dynamics of body movements and behaviour, and that the acquisition of knowledge is largely realized by doing and interacting with the environment.Perceptual inputs for instance (e.g., vision) can elicit covert motor activity in the absence of any task demand.One illustration is judgments of whether a screwdriver is screwing or unscrewing, which have been shown to be faster when the orientation of the handle is consistent with the manual dominance of the observer [7].Other illustrations come from brain imaging studies having shown that the perception of objects affords actions towards these objects with identical activity in cortical regions [8].In the same vein, the fact that when individuals observe an action, a brain activation similar to the one arising when actually producing that action is generated in their premotor cortex [9], suggests that embodied cognition plays a role in representing and understanding the behaviour of conspecifics [10], such as in learning by imitation for instance.Hence perception is not solely a visual or auditory or tactile process, preceding symbolic representations of actions to be performed.What we perceive in the world is influenced not only by, for instance, optical and ocular-motor information, but also by our purposes, physiological state, and emotions.Perception and cognition are thus embodied; they relate body and goals to the opportunities and costs of acting in the environment [11].In that respect, theories of embodied cognition and enaction largely rely on the influential work by Gibson in the last fifty years [12] calling for a mutual between information and movement, between perceiving and acting.Virtual reality technology allows the enactment of new perception-action and cognitive components that are of importance for learning.The enactive interfaces develops within the SKILLS project represent a mean to enhance the conditions for carrying out intuitively manipulative procedures or for learning complex perceptuo-motor skills, to study his hands inthere ], to improve subjective (the performatory) fidelity [14], and to evoke new affordances using virtual environments.

Illustration: learning team rowing using VR
The decomposition of a complex perceptuo-motor skill into functional sub-skills allows focusing virtual reality based training on specific elements.An example is given here in team rowing.When rowing in team, the difference in performance between two teams often depends on the ability of the athletes to row together in a highly synchronized way during the race [15].For identical movement patterns, the highest speed of the boat is indeed obtained when continuous relative phase between rowers is null.By using the combination of virtual reality rendering techniques and real-time motion capturing, we [16] recently investigated whether it was possible to learn the specific interpersonal coordination sub-skill with a virtual teammate on an indoor rowing machine, and to transfer the acquired skill to synchronizing with a real teammate.In our learning protocol, the synchronization was either spontaneous (no feedback other than the presence of the avatar), or increased by online visual information embodied on the avatar, giving real-time information about the between-bodies coordination (see Figure 2B&C).The results were straightforward and indicated that all participants improved their ability to synchronize with a real teammate (pre-post test comparison, see Figure 2A) following a training protocol with a virtual partner.However, learning was better for the participants that had the embodied feedback available

Figure 2 .
Figure 2. Up: A VR training protocol for learning team rowing showing real pre-test and post-test (A), training sessions with or without embodied on-line visual information about the between-rowers relative phase (B & C); Bottom: Difference in coordination performance (left) and coordination variability (right) between a control group and feedback group at pre-test, post-test, and retention.Adapted from [16].