Interacting with an intelligent dancing figure : artistic experiments at the crossroads between art and cognitive science
Michel Bret, emeritus university professor
Marie Hélène Tramus, professor in Art and Image Technology at the University of Paris 8
Alain Berthoz, physiologist, professor at the Collège de France (LPPA)
Marie Hélène Tramus, teacher in Art and Image Technology at the University of Paris8
Alain Berthoz, neuro-physiologist, professor at the Collège de France
Digital arts, movement perception, interactivity, connectionism, evolutionary science, artificial life.
Interactivity has introduced a certain type of sensorial awareness in the arts, especially considered from the point of view of the spectator. Our hypothesis is that this sensorial aspect may also be envisaged from the point of view of the work of art itself, by endowing the work with perceptions of its own. This raises one of the most crucial questions in contemporary digital arts: that of the relationships between natural and artificial “perception-movement-action” functions.
We led studies and carried out experiments on these relationships, drawing from the results of research into the fields of connectionism, artificial life and the perception of actions and movements.
One of our aims is to create art installations showing virtual actors who are endowed with artificial perceptions enabling them to react in an autonomous way to the cues given by a spectator, thus opening arts and cognitive science to a whole new range of possibilities for the exploration of virtual life.
1.1 State of the Art
Our purpose is set in the context of interactive arts in relation with artificial life and it follows on from research that we will present briefly and non-exhaustively in order to give a series of landmarks and reference points.
Flavio Sparacino  distinguishes systems that are merely “reactive” ( systems in which sensors transfer data from the audience’s actions to scripts that map pre-defined reactions), from the “behavioural” systems ( systems that apply the results of classical artificial-intelligence research, as, for example “group behaviour” theory, introduced by Reynolds in 1987 ) and finally, from “autonomous systems”, first introduced by Brooks  in the case of robotics, then developed by Maes   and that Karl Sims  first applied to the arts. Blumberg  elaborated a general model for perception and action selection in real-time. He thus made a model of a dog that was capable of interacting with human beings as well as with other virtual actors on the behavioural mode). With the “Neuro-Animator”, Terzopoulos offers a new approach in order to create animations that are realistic in physical terms by exploiting the features of neural networks that are trained off-line in order to imitate the dynamics of moving physical models. CML (“Cognitive Modelling Language”)  outreaches behavioural models by controlling what a virtual actor knows, how he/she acquires the knowledge and how he/she uses it in order to plan his/her actions.
According to Jean-Arcady Meyer  the animat approach postulates that it is possible to study human cognition through a bottom-up approach that proceeds from minimal control architectures and simple environments and then makes them gradually more complex. Evolutionary robotics apply the laws of genetics and natural selection to encode a robot’s phenotype in its genotype, the robot is then submitted to an artificial process of natural selection by using genetic algorithms and genetic programming.
Intensive research has been led on “cyberdance”. Among the most eloquent examples are Merce Cunningham who used Tom Calvert’s Life Forms program. Nadia Magnenat Thalman  who realized performances by putting virtual actors on stage alongside real actors.
We chose to base our research on models drawn from cognitive science and biology, especially connectionism, genetics and the physiology of perception and action, in order to head towards what we suggest calling “second interactivity”, in reference to “second cybernetics” that deals with more complex and fuzzy relations, that are closer to intuitive human behaviour.
Even though the work produces meaning-effects (effets de sens), these impressions are not primarily related to the play of language, concepts and symbols, but to an often unknown and despised form of thought, that Marie-Hélène Tramus  calls “body-thought”. Hence, the work is entirely contained in the series of unique perceptions that the audience may experience, once or more, during the interplay. This work only exists if it is visited, explored, felt. According to Francisco Varela’s phrase, it is fundamentally experience-related (expérientielle). It is, literally, a body art.
The body this art deals with is not only the spectator’s, as the spectator also engages in a dialogue with a counterpart the work calls into existence, and that Alain Berthoz calls the “doppelganger” . The doppelganger is characterized by the fact that it is a mirror-image, while keeping the separate identity of an autonomous being. This ambivalence is at the source of emotions that only art can provide.
According to Varela, autonomy means internal law ( related to self generation, self organisation and the affirmation of identity), it is opposed to allonomy (external law, or command). 
What is at stake in this dialogue with virtual creatures is the issue of their autonomy, the quintessential feature of these virtual objects who have become automata. A feature that allows them to move and act in an independent way and a manner adapted to their environment perceptions ( the environment being the spectator in this case, perceived through the use of sensors).
The connectionist approach offers a possible direction, yet not the only one, for interactive experimental art, as it gives the virtual creature a certain degree of autonomy, thanks to neural networks that generate unpredicted and non-explicitly programmed behaviour. The global approach at work in these artistic experiments draws inspiration from contemporary biology theory, in particular the views of neurophysiologists, such as Alain Berthoz, according to whom the most refined features of human sense and sensibility are dynamic processes, ever changing relations between the brain, the body and the environment. In his view, movements play a fundamental role, as the ability to coordinate actions is indeed at the source of the highest cognitive functions in the brain. Alongside a call for the reintegration of action and movement at the very heart of brain-studies, these installations show a move in favour of a kind of digital art that should also draw inspiration from physical sensations and movement.
We thought neural networks, that have the capacity to self configure, were favourable to the development of experiments on the “body-brain-environment” interactions of a virtual creature. During a first phase, we chose to use networks with supervised learning hidden layers, as they are very easy to implement. Furthermore, they are very widespread, their algorithms are widely published  and, above all, they are very efficient at solving certain problems that have fuzzy constraints and no known resolution algorithms. However, they are very far removed from the neurobiological features of the brain.
This exploration of networks of supervised neurones is connected to a specific stage in our research. But we are carrying out experiments on other kinds of networks such as Kohonen’s  unsupervised networks with competitive learning that are able to recognize regularities. We are also interested by other paradigms, like the dynamic approach in animat research. 
2 “Intelligent” interactive installations.
The exploration of the possibilities of « intelligent gesture related interactivity » between real and virtual actors in the digital arts was enriched by its encounter with other disciplines: in particular the cognitive science research focusing on an understanding of movement, perception and action in relation with emotions and expressiveness, but also the performing arts, such as dance, theatre and the circus, that all found their expressive power on movement.
2.1Description of two installations with an interactive virtual character
This virtual character obeys biomechanical laws and is endowed with reflex behaviour patterns that help it maintain its balance on the ground. Furthermore, neural networks enable it to react to the spectator’s movements in an “intelligent” way.
In the case of the installation called The Virtual Tightrope Walker the spectator is invited to become a tight-rope walker. The image of the virtual tight-rope walker is shown on a screen facing the spectator who is equipped with a movement sensor attached to his or her waist. This sensor sends information about the position and the orientation of the spectator to the computer that interprets it in real time as a set of forces that influence the moving synthetic actor, controlled by neural networks. The tight-rope walker is not a copy of the spectator, but rather an artificial being that is sensitive to the spectator’s movements. If the spectator tries to unbalance the tight-rope walker, she will attempt to regain her balance by developing autonomous strategies in real time. These strategies are the result of a previous training phase. The duet between both “actors” then develops around a game of unbalance and balance (figures 1,2,3,4,5,6,7).
In the installation Dance with Me, the spectator is now invited to interact in real time with a virtual dancer. The spectator interacts through the means of a movement sensor attached to his/her waist. The sensor’s variations in speed are interpreted by the computer as forces influencing a virtual body set in a gravity field and constrained by a floor it cannot fall through. When she faces a moving spectator, the virtual dancer improvises dance steps that are a compromise between previously learnt choreography and balancing strategies, and the spectator’s movements (figures 8 and 9). This artificial being, albeit very elementary and very far removed from the very complex forms of natural adaptation and anticipation, outreaches simple retroactive loops in the way it comprises certain features of live creatures. For example, generalisation, a property of neural networks, endows it with a potentially unlimited array of unlearnt, yet adapted, reactions. Its intelligence appears as a feature emerging from interactions between its elements (artificial neurons), the information it senses in its environment, and its structure ( the simulation of a human body, endowed with certain behaviour patterns).
2.2 Experiments with spectators, acrobats, dancers : interactivity in art installations as interdependence vs. autonomy
Experience taught us, every time we showed the work and observed spectators react, to gradually understand the issues of the relationship between both beings, in particular the delicate balance between autonomy and interdependence.
The response of the previously trained networks is modulated by the spectator’s intervention via the sensor. So the data coming from the neural network and the data issuing from the sensor-related interaction module are mixed. Setting the right proportions between both of these sources is essential as this is what allows the virtual and the real beings to act together in a complex balance of mutual interdependence and autonomy. By moving and observing, the spectator should be able to sense not only how he or she influences the figure and its reactions, but also the figure’s own autonomy. A virtual being with too great an autonomy wouldn’t engage in a relation, and if the spectator’s control were to strong, the relation might lack any sense of surprise. The spectator experiences things through movement, he or she gradually discovers this partner with her unpredictable reactions, adapts, tries things, invents his or her own kind of movements.
We filmed some of the moments of these various displays. The films show careful spectators seizing the balancing-pole, looking at the tight-rope walker and hesitantly starting to move. Some of them copy the acrobat’s movements, in order to become tight-rope walkers themselves, by reproducing the same gestures a split second later. This has produced beautiful scenes showing moments of harmony that surprised the spectators looking for their mirror-image. This configuration differs from usual experiments involving synthetic clones, as the latter copy exactly the gestures of the movement-sensing device they are enslaved to. But the spectators mimetic quest for a duplicate is soon outplayed by his own copycat behaviour which, on the contrary, upsets the virtual figure’s balance and thus disrupts the whole choreographic harmony. The spectator makes new attempts, and the succession of these new attempts produces more movement similarities and more disruptions of the figure’s balance, thus establishing an original interrelation of gestures. Other spectators project themselves in the tightrope walker as if they were dealing with a clone that should follow each and every of their gestures: attempts, failures, more infelicitous attempts that may lead them to abandon any form of relationship with such a disobedient character. Others just set out on an adventure, they make attempts, move, manipulate the balancing-pole, observe the figure’s movements and thus discover and establish quite spontaneously a relation with the virtual creature, through movement and action.
An experienced tightrope walker put her skills in practice to interact with the virtual tightrope walker. It was very moving to see the gestures of equipoise and unbalance of both performers together and to sense the similarity between them. For instance in the ways they enhanced their tread in order to regain their balance, leaning either backward or forward, and in the balancing moves of the bust. But also in the way the figure moved in search of her balance or when she lost it. It was mesmerizing to see the real tight-rope walker laugh and exclaim in front of the virtual figure’s extravagant unbalanced poises. It was fascinating to witness her scrutinizing her, as if she wanted to guess her “intentions”.
We also got the virtual dancer to take part in a dance and music improvisation session, gathering several dancers and a group of musicians. A dancer chose to dance as if he was unaware of the fact that the virtual dancer was there. Another started playing immediately with the image projected on the wall, whereas one dancer set off on a very subtle dialogue of gestures: a series of slight hip movements, and swaying variations on these movements to the sound of music. And yet another started a series of leaps that in turn triggered off the virtual figure’s leaping response, creating an energetic moment of shared intensity.
In the light of the virtual tightrope walker’s and dancer’s experiences, it appears that the work emerges from the network of invisible and unique relationships woven between the real and the virtual being and filling the discrepancy between them, thanks to the interactions of the bodies sharing the space of the installation hall.
2.3 Technical description
These installations, designed by Michel Bret , comprise four modules:
The dynamic module calculates the movements of a body subjected to different forces: gravity, environment reactions, biomechanical constraints, simulations of forces provided by the sensors and the behavioural module.
The behavioural module simulates rebalancing reflexes and forces generating voluntary movements provided by the connectionist module.
The connectionist module builds in real time adaptive strategies thanks to a neural network whose inputs are connected to the interactive module and whose outputs are interpreted as movement projects. The projects are groups of torques, applied to the body’s joints and sent to the behavioural module that produces a movement. The network was instructed by a series of experiences that acted as a learning phase.
The interactive module manages interrelations between the model and the world. Its inputs are fed by external forces sent by the movement sensors. At the same time, the interactive module manages the relations between the model and itself, as some inputs proceed from internal forces, such as the biomechanical constraints of the virtual body.
In this sense, projects produced by the connectionist module are dynamically confronted to the particulars of the interaction that modify them. They can be interrupted at any given moment by another project that is better adapted to the present situation.
We programmed the error back-propagation algorithm  (see appendix), for supervised learning on a layer network. A group of learning pairs is presented to the network whose connection matrix has previously been randomly initialized. For each input, the network calculates an output that is generally different from the required output. The difference between both outputs is used to correct the connection weights in order to minimize that error. Through a series of trial and errors, the network configures itself and learns all the provided examples. If the series of examples is representative enough of the different situations the synthetic actor is to be confronted with, the network will be able to respond correctly, even in the event of unlearnt examples.
In practical terms, we connected the sensors to the network inputs, and the network outputs to the actuators influencing the muscular system (figure 10).
Firstly, the tightrope walker was taught how to keep her balance on the rope, while the dancer was taught dance movements. Secondly, after these virtual characters had been trained, they were put in front of spectators or dancers or even acrobats so that a kind of gesture invention made of interdependence and autonomy could take place.
We also implemented a real-time version of this method by parallelizing the learning process and the interaction phases. From this arose a more elaborate interaction in which the spectator could witness, control and even attempt to modify the effect of his/her actions on the virtual being’s behaviour.
At last, we are planning to use other kinds of non-supervised networks this time, like Kohonen’s . They should allow the virtual being to discover the regular features of its environment as well as the most adapted responses.
This interaction between the spectator and the artificial creature that is endowed with a certain amount of autonomy and a certain capacity to invent gestures, creates an unprecedented kind of artistic event: while remaining close to a real life situation, it remains unpredictable and wishes to inspire improvisation, inventiveness, imagination and wonder.
3 Integration in the autonomous model of elements drawn from the physiology of perception of movements and actions
This section describes the integration of a few principles and natural laws of movement into the virtual body. They influence some of its autonomous behaviour patterns, a bit like the brain defines movement strategies in order to reduce the number of control parameters. If the body is to be rehabilitated in modern neurobiology, it is important to rediscover the rules that govern its movements. These rules have been intuitively sensed by sculptors who managed to reproduce the body’s movements in relation with feelings, as well as by actors in traditional oriental theatre. They teach us that movements are first expressed through posture, that the kinematics of movements conveys meanings and that the trajectory of a finger, a head shift, the demeanour of a swaying body, all respond to laws that are situated on the borders between mechanics and neurology. They also teach us that a natural movement is a source of pleasure.
3.1 About neural networks
The virtual actor’s network of neurones learns through a series of examples based on samples of natural movements recorded by the captors.
By replacing the network controlling the legs by several networks having learned different things, the question of what network to choose arose: In a first phase, we chose the network with the smallest Euclidian distance between one of its learning inputs and the virtual actor. In a second phase, rather than a single most-adapted-network, we chose a combination of responses.
In this way, we developed a multi-network environment allowing combinations that were our very first move towards an action selection model.
Here, movement organisation is based on a synergy directory, each synergy being an action possibility. But not only is it necessary to have a library of easy-to-trigger available movements that are all compatible because they share the same reference frames or the same geometrical principles, it is also necessary to be capable of making choices among them.
The formulation of this problem inaugurates a series of reflections and we want to develop the study of other selection modes, as models for action selection, in further research.
Thanks to input multiplexing, it was possible to control the degree of autonomy of the virtual figure. In the same way, we simulated a “goal-directed action”, using a method close to that of inverted kinematics used in robotics.
3.2 Concerning the role of the head and the gaze
D wij = -n * dQ / dwij
D wij = -n * di * aj avec di = -n * ¶Q / ¶ai * ¶ai / ¶ei
di = 2 * f(ei) * (oi – ai) (1)
OUT = f(IN) = 1 / (1 + e-k*IN), its derivative is :
¶OUT / ¶IN = OUT * (1 – OUT)
di = 2 * ai * (1 – ai) * (oi – ai) (1’)
di = ai * (1 – ai) * dk * wki (2’)
1. F. Sparacino, G. Davenport, A. Pentland, “Media in Performance: Interactive spaces, for dance, theater, circus, and museum exhibits”, IBM Systems Journal Vol. 39 Nos 3&4, 2000
2. Graig W. Reynolds , “Flocks, Herds and Schools: A distributed Behavioral Model”, Computer Graphics Vol. 21, 25-34, (1987)
3. R. A. Brooks, “Intelligence Without Reason”, IJCAI-91 Vol. 1, 569-595 (1991)
4. P. Maes, “Situated Agents Can Have Goals, Designing Autonomous Agents: Theory and Practice from Biology to Engineering and Back”, P. Maes, Editor, MIT Press/Bradford Books, Cambrideg, MA (1990).
5. P. Maes, “Modeling Adaptative Autonomous Agent”, Artificial Life 1, 135-162 (1994).
6. Karl Sims, “Artificial Evolution for Computer Graphics”, Computer GraphicsVol 25 No 4, 319-328 (1991).
7. Karl Sims, “Evolving Virtual Creatures”, Computer Graphics proceedings, 15-22 (1994).
8. B. M. Blumberg, T. Galyean, “Multi-Level Direction of Autonomous Creatures for real-Time Virtual Environments”, Computer Graphics proceedings, 47-54 (1995).
9. B. Blumberg, M. Downie, Y. Ivanov, M. Berlin, M.P. Johnson, B. Tomlinson, « Integrated learning for interactive synthetic characters », Proceedings of the 29th annual conference on Computer graphics and interactive techniques, 2002, San Antonio, Texas
10. R. Grzeszczuk, D. Terzopoulos, G. Hinton, “NeuroAnimator: Fast Neural Network Emulation and Control of Physics-Based Models”, Intel Corporation University of Toronto (1998).
11. J. Funge, X. Tu, D. Terzopoulos, « Cognitive Modeling : Knowledge, reasoning and planning for intelligent characters », J. Funge, Proc. ACM SIGGRAPH 99 Conference, Los Angeles, CA, August, 1999, in Computer Graphics Proceedings, Annual Conference Series, 1999, 29-38. R.
12. A. Guillot, and J.A. Meyer, « La contribution de l'approche animat aux sciences cognitives », Cahiers Romans de Sciences Cognitives. 1 (1), 1-26, (2003).
13. A. Drogoul, J.-A. Meyer, Hermes, Intelligence artificielle située, cerveau, corps, environnement, 1999.
14. J.H. Holland, Adaptation in natural and artificial systems, The University of Michigan Press (1975).
15. D.E. Goldberg, Genetic Algorithms in Search, Optimization, and Machine Learning, Addisson Wesley (1989).
16. J. Koza, Genetic Programming: On the Programming of Computers by the Means of Natural Selection, The MIT Press (1992).
17. T. Calvert, A. Bruderlin, S. Mah, T. Schiphorst, and C. Welman, “The evolution of an interface for choreographers”, In Proc. InterCHI Conference, pages 115-122, April (1993).
18. T. W. Calvert, C. Welman, S. Gaudet, T. Schiphorst, and C. Lee, “Composition of multiple figure sequences for dance and animation”, The Visual Computer, 7(2-3):114-121, (1991).
19. S. Kshirsagar, N. Magnenat-Thalmann « A Multilayer Personality Model », Proceedings of 2nd International Symposium on Smart Graphics, pp 107-115., June, 2002.
20. S. Carion, G. Sannier, N. Magnenat Thalman, « Virtual Humans in CyberDance », MIRALab, C.U.I., University of Genova
21. Michiel Van de Panne, Eugene Fiume, “Sensor-Actuator Networks”, Computer Graphics proceedings, 335-342 (1993).
22. M.-H. Tramus, M. Bret, E. Couchot , « La seconde interactivité », in Arte e vida no século XXI, Organizadora Diana Domongues, UNESP, Brasil (2003).
23. M.-H. Tramus, « La question de la multisensorialité dans les arts numériques interactifs », Actes Isea 2000, http://www.isea2000.com.
24. A. Berthoz, La décision, Ed O. Jacob, Paris, 2003
25. F. J. Varela, Autonomie et Connaissance. Essai sur le vivant, édit. Le Seuil, 1989.
26. A. Berthoz, Le sens du mouvement, Editions Odile Jacob, 1997.
27. H. Abdi, Les réseaux de neurones, Presses Universitaires de Grenoble, 1994.
28. T. Kohonen, Self-Organization and Associative Memory, Springer-Verlag 1984.
29. Expositions de La funambule virtuelle : Virtual World 2000, Paris, 2000 ; Salon d’automne, Paris, 2000 ; Les chemins du numérique, Reims, 2001 ; Média-Village, Paris, France, 2001 ; Le cube s’ouvre à Issy les Moulineaux, exposition d’art numérique, 2001.
Expositions de Danse avec moi : Arts-outsiders, Paris, France, 2001, le Salon d’automne, Paris, 2001.
30. M. Bret, “Virtual Living Beings”, in Lecture Notes in Artificial Intelligence, Virtula Worlds 119-134, Ed. Jean-Claude Heudin, Springer 2000.
31. M. Bret ; « Experimentation dans les arts du connexionnisme », Actes Isea 2000, http://www.isea2000.com
32. P. Viviani, T. Flash, « Minimum-jerk, to-thirds power law, and isochrony : Converging approaches to movement planning”, J Exp Psychol Hum Percept Perform, 21, 32-53 (1995).
Figure 1: Stephanette Vandeville interacting with the virtual tightroper
Figure 2: Stephanette Vandeville interacting with the virtual tightroper
Figure 3 : Stephanette Vandeville interacting with the virtual tightroper
Figure 4 : The Virtual Tightrope Walker
Figure 5 : The Virtual Tightrope Walker
Figure 6 : The Virtual Tightrope Walker
Figure 7 : The Virtual Tightrope Walker
Figure 8 : Dance with me
Figure 9: Dance with me
Figure 10 : sensors connected to the network inputs, and the network outputs connected to the actuators
Figure 11 : phase plane
Figure 12 : The law of the power of one third