MODELING THE MIRROR: GRASP LEARNING AND

ACTION RECOGNITION

 

by

 

Erhan Oztop

 

A Dissertation Presented to the

FACULTY OF THE GRADUATE SCHOOL

UNIVERSITY OF SOUTHERN CALIFORNIA

In Partial Fulfillment of the

Requirements for the Degree

DOCTOR OF PHILOSOPHY

(COMPUTER SCIENCE)

 

 

August 2002

TABLE OF CONTENTS

DEDICATION.. ii

ACKNOWLEDGEMENTS. iii

LIST OF FIGURES. x

1    INTRODUCTION.. 1

2    CHAPTER II: BIOLOGICAL BACKGROUND.. 4

2.1     Abbreviations. 4

2.2     Premotor areas. 4

2.2.1      Area F5. 5

2.2.2      Area F4. 10

2.2.3      Areas F2 and F7 (dorsolateral prefrontal cortex) 11

2.2.4      Area F1 (the primary motor cortex) 12

2.2.5      Areas F3 (SMA proper), F6 (pre-SMA) 12

2.3     The superior temporal sulcus. 12

2.4     Parietal Areas. 13

2.4.1      The anterior intraparietal area (AIP) 13

2.4.2      The caudal intraparietal sulcus (c-IPS) 15

2.4.3      Areas VIP, MIP and LIP.. 17

2.4.4      Areas 7a and 7b (PG and PF) 18

2.5     Connectivity and other brain regions. 19

2.6     Mirror neurons in humans. 23

2.7     Summary.. 25

3    CHAPTER III: MIRROR NEURON SYSTEM MODEL. 26

3.1     The mirror neuron system for grasping and FARS model 26

3.2     The hand-state hypothesis. 29

3.2.1      Virtual fingers. 29

3.2.2      The hand-state hypothesis. 30

3.3     The MNS (mirror neuron system) model 31

3.3.1      Overall function.. 33

3.3.2      Schemas explained.. 33

3.4     Schema implementation.. 36

3.4.1      Grand schema 1: reach and grasp.. 36

3.4.2      Grand schema 2: visual analysis of hand state. 38

3.4.3      Grand Schema 3: core mirror circuit. 42

3.5     Simulation results. 46

3.5.1      Non-explicit affordance coding experiments. 46

3.5.2      Explicit affordance coding experiments. 50

3.5.3      Justifying the visual analysis of hand state schema. 54

3.6     Discussion and predictions. 56

3.6.1      The hand state hypothesis. 56

3.6.2      Neurophysiological predictions. 57

4    CHAPTER IV: MULTILAYER SUPERVISED HEBBIAN LEARNING AND PROBABILITY CODING    60

4.1     Neural coding. 60

4.2     Operation of the proposed network. 61

4.3     Testing the proposed architecture. 64

4.3.1      Deterministic environment. 64

4.3.2      Stochastic environment. 66

4.3.3      Combining multiple layers. 68

4.4     Summary.. 69

5    CHAPTER V: INFANT GRASP LEARNING.. 71

5.1     Motivation.. 71

5.2     Infant reach and grasp.. 71

5.3     Neural maturation versus interactive learning. 74

5.4     Infant Learning to Grasp Model (ILGM) 75

5.4.1      Layers of infant learning to grasp model 76

5.4.2      Functional description of ILGM layers. 77

5.5     Joy of grasping. 78

5.5.1      Mechanical grasp stability.. 78

5.5.2      Implementing the grasp stability.. 79

5.6     Learning approach direction with palm orienting behavior. 80

5.6.1      Simulation results. 80

5.6.2      Conclusions and predictions. 82

5.7     Is infant palm orienting learned or innate? Learning the wrist orientation.......... 82

5.7.1      Simulation results. 83

5.7.2      Conclusions and predictions. 84

5.8     Task constraints shape infant grasping. 84

5.8.1      Simulation results. 85

5.8.2      Conclusions and predictions. 86

5.9     Affordance input matters. 87

5.9.1      Simulation results. 88

5.9.2      Comparison of ILGM with Lockman et al. (1984) 88

5.9.3      ILGM kinematics analysis (five months of age) 88

5.9.4      ILGM kinematics analysis (nine months of age) 89

5.10       Summary and conclusion.. 91

5.11       Discussion.. 92

6    CHAPTER VI: NEUROPHYSIOLOGICAL VIEW OF LEARNING TO GRASP....... 93

6.1     Grasp learning circuit and mirror neurons are complementary networks. 93

6.2     Introduction to primate grasping. 94

6.3     Neural correlates of infant reach and grasp.. 95

6.4     Primate grasp development hypotheses. 97

6.4.1      Hypothesis I: two coexistent grasping circuits. 97

6.4.2      Hypothesis II: single grasping circuit. 98

6.5     Affordance-based learning to grasp model (LGM) 98

6.5.1      Localizing learning to grasp model in primate cortex. 99

6.5.2      What does cerebral cortex know about a grasp?. 101

6.5.3      Simulation level description of LGM layers. 102

6.5.4      Why LGM is relevant: good model versus bad model 102

6.6     Wrist orientation-learning revisited: neural level analysis. 103

6.6.1      Neural level analysis. 103

6.6.2      LGM represents a ‘menu’ of grasps in terms of neural activity.. 104

6.6.3      Predictions and discussion.. 106

6.7     Object axis selectivity: neural level analysis. 106

6.7.1      Neural level analysis. 107

6.7.2      Conclusions and neurophysiological predictions. 108

6.8     Object size selectivity.. 109

6.8.1      Simulation results. 109

6.8.2      Conclusions and neurophysiological predictions. 111

6.9     Generalization: learning to plan based on object location.. 113

6.9.1      Simulation results. 113

6.9.2      Summary and Conclusion.. 115

7    CHAPTER VII: BIOLOGICALLY REALISTIC F5 VISUAL SERVO CIRCUITS FOR GRASPING AND EMERGENCE OF MIRROR NEURONS. 117

7.1     Motivation.. 117

7.2     The link between the mirror neuron system and grasp learning. 118

7.2.1      Two Visual Control Hypotheses. 119

7.2.2      Mirror neurons in feed-forward control (alternative I) 121

7.2.3      Mirror Neurons in feedback control (alternative II) 122

7.2.4      The target of implementation.. 122

7.2.5      The visual servo task. 123

7.2.6      The feed-forward model learning. 123

7.3     Implementation: F5 manual visual control circuit for 2D arm... 123

7.3.1      A leaky integrator model for F5 manual visual feedback circuit. 124

7.3.2      Simulation: visual feedback control with leaky integrators. 127

7.3.3      Simulation: feedback and lower motor centers. 128

7.3.4      F5 Feed-forward visual control and mirror neurons. 130

7.3.5      Simulation: trajectory planning and controller performance. 135

7.4     Feed-forward unit activity and mirror neurons. 137

8    CHAPTER VIII: CONCLUSIONS. 142

8.1     Mirror neurons. 142

8.2     Grasp learning: infant development. 144

8.3     Grasp learning: monkey neurophysiology.. 144

8.4     Grasp learning: neural architecture. 145

9    CHAPTER IX: FUTURE WORK.. 146

9.1     Simultaneous learning in MNS model and LGM... 146

9.2     Going beyond grasping: learning the ‘hand state’ 146

9.3     Tactile feedback for grasping. 146

9.4     Learning to extract the right affordance. 147

9.5     More realistic models of the limb and the brain regions. 147

9.6     Sensitivity analyses for the simulation parameters. 147

10      REFERENCES. 149

11      APPENDIX.. 164

11.1       Mirror neuron system model (MNS) 164

11.1.1    Color segmentation.. 164

11.1.2    Reach and grasp schema precision grasp planning and execution.. 165

11.2       Learning to grasp models (ILGM and LGM) 165

 

 

 

MODELING THE MIRROR: GRASP LEARNING AND

ACTION RECOGNITION

 

by

 

Erhan Oztop

_________________________________________________

 

 

A Dissertation Presented to the

FACULTY OF THE GRADUATE SCHOOL

UNIVERSITY OF SOUTHERN CALIFORNIA

In Partial Fulfillment of the

Requirements for the Degree

DOCTOR OF PHILOSOPHY

(COMPUTER SCIENCE)

 

 

August 2002

 

 

 

Copyright 2002                                                                           Erhan Oztop

 


DEDICATION

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

To my Grandmother

 

 


ACKNOWLEDGEMENTS

The time I had at USC during the Ph.D. route was a very enriching period of my life. I had the opportunity to work in an exciting and stimulating research environment, led by Michael Arbib.

I would like to present my deepest gratitude to The Scientific and Technical Research Council of Turkey (TUBITAK) for providing me the scholarship that made it possible for me to attempt and complete the Ph.D. study presented in this thesis. The study would not have been possible if TUBITAK did not provide support for the very first semester and the final semester at USC.

I would like to thank Michael Arbib for guiding and educating me throughout the my years at USC. He has been an extraordinary advisor and mentor, whom I owe all the brain theory I learned. I would also like to present my gratitude to Michael Arbib for providing support via HFSP and USCBP grants. Without HFSP and USCBP support, this study would not be possible.

Stefan Schaal is a great mentor, who has been a constant support and source of inspiration with never-ending energy. Besides introducing me to Robotics, he and his colleague, Sethu Vijayakumar were very influential in maturating the concept of machine learning in my mind.

I also owe a great debt of gratitude to Nina Bradley for being a source positive energy and mentoring me especially in infant motor development. Without her, the thesis would be lacking a major component.

I am also full of gratitude to my Ph.D. qualification exam comittee members Maja Mataric and Christoph von der Malsburg for their guidance and support.

I am greatly indebted to Giacomo Rizzolatti for enabling me to visit his lab and providing the opportunity to communicate with not only himself but also with Vittorio Gallese and Leonardo Fogassi who have provided invaluable insights about mirror neurons. In addition, I am very thankful to Massimo Matelli and Giuseppe Luppino, for the first hand information on the mirror neuron system connectivity, and to Luciano Fadiga for stimulating discussions. I would also like to thank to Christian Keysers and Evelyne (Kohler) for not only actively involving me in their recording sessions but also offering their sincere friendship.

I am very thankful to Hideo Sakata for giving me the opportunity to visit his lab in Tokyo and interact with many researchers including Murata-san with whom I had very stimulating discussions.

I am deeply thankful to Mitsuo Kawato, for giving me the opportunity to interact with various researchers in Japan by having me at ATR during the summer of 2001. My research experience at ATR was very rewarding; I greatly expanded my knowledge on motor control and motor learning. I would like to salute the staff at ATR for all their help. I also would like to present my thanks to the friends at ATR for welcoming me and making me feel at home.

I would like to present my appreciation and thanks to my mentors at Middle East Technical University in Turkey. I present my sincere thanks to my masters advisor Marifi Guler for introducing me to neural computation and to Volkan Atalay for introducing me to computer vision, and supporting my Ph.D. application. Especially, I would like to present my gratitude to Fatos Yarman Vural for her guidance and support during my Masters study and for preparing me for the Ph.D. work presented in this dissertation. Other influential Computer Science professors to whom I am grateful for educating me are Nese Yalabik, Gokturk Ucoluk and Sinan Neftci.

I would like to present my gratitude to Tosun Terzioglu, Albert Ekip, Turgut Onder and Semih Koray who were professors of the Mathematics Department at the Middle East Technical University. They taught me how to think ‘right’ and exposed the beauty of mathematics to me.

Throughout these six years at USC, I had the pleasure to meet several valuable people who contributed to this dissertation. Firstly, I am very thankful to Aude Billard and Auke Jan Ijspeert for all their support and scientific interaction and feedback. They have a huge role in helping me get through the tough times during my Ph.D. work. In addition, I would like to thank Aude Billard for providing me the computing hardware and helping me have a nice working environment, which was very essential for the progress of my Ph.D. study. I am thankful to my great friend Sethu Vijayakumar for his support and stimulating discussions. Jun Nakanishi, Jun Mei Zu, Aaron D’Souza, Jan Peters and Kazunori Okada besides being of great support, were always there to discuss issues and helped me overcome various obstacles. I am deeply thankful to Shrija Kumari for offering not only her smile and friendship but also her energy to proofread my manuscript. I owe a lot to Jun Mei Zu: she has always been there as a great friend and has always offered her help and support when I needed it most. I am indebted to my ex-officemate and a very valuable friend Mihail Bota for his constant support and interactions for improving the thesis and providing me the psychological support to overcome many obstacles throughout my Ph.D. years. Finally, I would like to thank Juan Salvador Yahya Marmol for being a good friend and sharing my workload during various periods of my Ph.D. I count myself very lucky to have these great friends and colleagues whom once again I present my gratitude: Thank you guys!

Not a Hedco Neuroscience inhabitant but a very valuable friend, Lay Khim Ong was always there for offering her help both psychologically and physically (Hey Khim: thank you for your great editing!). I would like to thank other great friends who supported me (in spite of my negligence in keeping in touch with them). Kyle Reynolds, Awe  Vinijkul, Aye Vinijkul Reynolds, Alper Sen, Ebru Dincel: please accept my sincere thanks and appreciation.

I owe deep gratitude to my wife Mika Satomi for her patience in dealing with me in difficult times.  She was always there. Her contribution to this thesis is indispensable. I especially celebrate and thank her for the artistic talents and hard work that she generously offered me throughout the Ph.D. study.

I am greatly indebted to Paulina Tagle and Laura Lopez for their support and help over all these years. I also would like to thank Laura Lopez and Yolanda Solis for their kind friendship and support. My gratitude to Luigi Manna, who helped me with the hardware and software issues during the Ph.D. study.

I would like to thank Laurent Itti for his generous help for improving our lab environment and providing partial support for my research. Also, I would like to salute his lab members for their support and friendship. Florence Miau, in particular, had always offered her warm friendship during her internship at USC.

I would like to present my appreciation to the good things in life particularly, I would like to thank the ocean for comforting and rejuvenating me during difficult times.

Finally, I am deeply indebted to my family, to whom I owe much more than what can be expressed. This work would not be possible without the help of my parents. (Anne ve Baba, Evrim ve Nurdan: Hersey icin cok cok tesekkurler!)

 

 



LIST OF FIGURES

Figure 2.1 Lateral view of macaque brain showing the areas of agranular frontal cortex and posterior parietal cortex (adapted from Geyer et al. 2000). The naming conventions: frontal regions, Matelli et al.(1991); parietal regions, Pandya and Seltzer (1982) 5

Figure 2.2 A canonical neuron response during grasping of various objects in the dark (left to right and top to bottom: plate, ring, cube, cylinder, cone and sphere. The rasters and histograms are aligned with object presentation. Small grey bars in each raster marks onset of key press, go signal, key release, onset of object pulling, release signal, and object release, respectively. The peaks in ring and sphere object cases correspond to the grasping of the object by the monkey (adapted from Murata et al. 1997a) 6

Figure 2.3 The motor responses of the same neuron shown in Figure 2.2. The motor preference of the neuron is also carried over to the visual preference (compare the ring and sphere histograms of both figures) (adapted from Murata et al. 1997a) 7

Figure 2.4 Activity of a cell during action observation (left) and action execution (right). There is no activity in presentation of the object during both initial presentation and bringing the tray towards the monkey. The vertical line over the histogram indicates the hand-object contact onset. (from Gallese et al., 1996). 8

Figure 2.5 Visual response of a mirror neuron. A. Precision grasp B. power grasp C. mimicking of precision grasp. The vertical lines over the histograms indicate the hand-object contact onset. (adapted from Gallese et al., 1996) 9

Figure 2.6 Example of a strictly congruent manipulating mirror neuron: A) The experimenter retrieved the food from a well in a tray. B) Same action, but performed by the monkey. C) The monkey grasped a small piece of food using a precision grip. The vertical lines over the histograms indicate the hand-object contact onset (adapted from Gallese et al., 1996). 9

Figure 2.7 The classification of area F5 neurons derived from published literature (Dipellegrino et al. 1992; Gallese 2002; Gallese et al. 1996; Murata et al. 1997a; Murata et al. 1997b; Rizzolatti et al. 1996a; Rizzolatti and Gallese 2001). All F5 neurons fire in response to some motor action. In addition, canonical neurons fire for object presentation while the mirror neurons fire for action observation. The majority of hand related F5 neurons are purely motor (Gallese 2002)(labelled as Motor Neurons in the figure) 10

Figure 2.8 The macaque parieto-frontal projections from mesial parietal cortex, medial bank of the intraparietal sulcus and the surface of the superior parietal lobule (adapted from Rizzolatti et al. 1998). Note that the Brodmann’s area 7m corresponds to Pandya and Seltzer's (1982) area PGm... 11

Figure 2.9 The intraparietal sulcus opened to show the anatomical location of AIP in the macaque (adapted from Geyer et al. 2000) 13

Figure 2.10 An AIP visual-dominant neuron activity under three task conditions: Object manipulation in the light, object manipulation in the dark and object fixation in the light. The neuron is active during fixation and holding phase when the action is performed in light condition. However, during grasping in dark the neuron shows no activity. The fixation of the object alone without grasping also produces a discharge (adapted from Sakata et al. 1997a) 14

Figure 2.11 Activity the same neuron in Figure 2.10 during fixation of different objects. The neuron show selectivity for horizontal plate (adapted from Sakata et al. 1997a) 14

Figure 2.12 An AIP visual-dominant neuron’s axis orientation tuning and object fixation response is shown. The neuron fires maximally during the fixation of a vertical bar or a cylinder. The tuning is demonstrated in the lower half of the figure (adapted from Sakata et al. 1999) 15

Figure 2.13 Response of an axis-orientation-selective (AOS) neuron in the caudal part of the lateral bank of the intraparietal sulcus (c-IPS) to a luminous bar tilted 45° forward (left) or 45 backward (right) in the sagittal plane. The monkey views the bar with binocular vision. The line segment under the histograms mark the fixation start and the period of 1 second. (adapted from Sakata et al. 1999) 16

Figure 2.14 The response of the same neuron in Figure 2.13, for monocular vision conditions for the left and right eyes. (adapted from Sakata et al. 1999) 16

Figure 2.15 Orientation tuning of a surface-orientation selective (SOS) neuron. First row: Stimuli presented. Middle row: responses of the cell with binocular view. Last row: responses of the cell with monocular view (adapted from Sakata et al. 1997a) 17

Figure 2.16 The reconstructed connectivity of area 7a. The thickness of the arrows represent the strength of the connection. (adapted from Bota 2001) 20

Figure 2.17 The reconstructed connectivity of area 7b. The thickness of the arrows represent the strength of the connection. (adapted from Bota 2001) 21

Figure 2.18 The reconstructed connectivity of area AIP. The thickness of the arrows represent the strength of the connection. (adapted from Bota 2001) 22

Figure 3.1 Lateral view of the monkey cerebral cortex (IPS, STS and lunate sulcus opened). The visuomotor stream for hand action is indicated by arrows (adapted from Sakata et al., 1997) 27

Figure 3.2 AIP extracts the affordances and F5 selects the appropriate grasp from the AIP ‘menu’. Various biases are sent to F5 by Prefrontal Cortex (PFC) which relies on the recognition of the object by Inferotemporal Cortex (IT). The dorsal stream through AIP to F5 is replicated in the MNS model 28

Figure 3.3 Each of the 3 grasp types here is defined by specifying two "virtual fingers", VF1 and VF2, which are groups of fingers or a part of the hand such as the palm which are brought to bear on either side of an object to grasp it. The specification of the virtual fingers includes specification of the region on each virtual finger to be brought in contact with the object. A successful grasp involves the alignment of two "opposition axes": the opposition axis in the hand joining the virtual finger regions to be opposed to each other, and the opposition axis in the object joining the regions where the virtual fingers contact the object. (Iberall and Arbib 1990) 29

Figure 3.4 The components of hand state F(t) = (d(t), v(t), a(t), o1(t), o2(t), o3(t), o4(t)). Note that some of the components are purely hand configuration parameters (namely v,o3,o4,a) whereas others are parameters relating hand to the object. 31

Figure 3.5 The MNS (Mirror Neuron System) model. (i) Top diagonal: a portion of the FARS model. Object features are processed by cIPS and AIP to extract grasp affordances, these are sent on to the canonical neurons of F5 that choose a particular grasp. (ii) Bottom right. Recognizing the location of the object provides parameters to the motor programming area F4 which computes the reach. The information about the reach and the grasp is taken by the motor cortex M1 to control the hand and the arm. (iii) New elements of the MNS model: Bottom left are two schemas, one to recognize the shape of the hand, and the other to recognize how that hand is moving. (iv) Just to the right of these is the schema for hand-object spatial relation analysis. It takes information about object features, the motion of the hand and the location of the object to infer the relation between hand and object. (v) The center two regions marked by the gray rectangle form the core mirror circuit. This complex associates the visually derived input (hand state) with the motor program input from region F5canonical neurons during the learning process for the mirror neurons. The grand schemas introduced in section 3.2 are illustrated as the following. The “Core Mirror Circuit” schema is marked by the center grey box; The “Visual Analysis of Hand State” schema is outlined by solid lines just below it, and the “Reach and Grasp” schema is outlined by dashed lines. (Solid arrows: established connections; dashed arrows: postulated connections) 32

Figure 3.6 (a) For purposes of simulation, we aggregate the schemas of the MNS (Mirror Neuron System) model of Figure 3.5 into three "grand schemas" for Visual Analysis of Hand State, Reach and Grasp, Core Mirror Circuit. (b) For detailed analysis of the Core Mirror Circuit, we dispense with simulation of the other two grand schemas and use other computational means to provide the three key inputs to this grand schema. 34

Figure 3.7 (Left) The final state of arm and hand achieved by the reach/grasp simulator in executing a power grasp on the object shown. (Right) The hand state trajectory read off from the simulated arm and hand during the movement whose end-state is shown at left. The hand state components are: d(t), distance to target at time t; v(t), tangential velocity of the wrist; a(t), Index and thumb finger tip aperture; o1(t), cosine of the angle between the object axis and the (index finger tip – thumb tip) vector; o2(t), cosine of the angle between the object axis and the (index finger knuckle – thumb tip) vector; o3(t), The angle between the thumb and the palm plane; o4(t), The angle between the thumb and the index finger. 37

Figure 3.8 Grasps generated by the simulator. (a) A precision grasp. (b) A power grasp. (c) A side grasp.. 38

Figure 3.9 (a) Training the color expert, based on colored images of a hand whose joints are covered with distinctively colored patches. The trained network will be used in the subsequent phase for segmenting image. (b) A hand image (not from the training sample) is fed to the augmented segmentation program. The color decision during segmentation is done by consulting to the Color Expert. Note that a smoothing step (not shown) is performed before segmentation.. 40

Figure 3.10 Illustration of the model matching system. Left: markers located by feature extraction schema. Middle and Right: initial and final stages of model matching. After matching is performed a number of parameters for the Hand configuration are extracted from the matched 3D model 41

Figure 3.11 The scaling of an incomplete input to form the full spatial representation of the hand state As an example, only one component of the hand state, the aperture is shown. When the 66 percent of the action is completed, the pre-processing we apply effectively causes the network to receive the stretched hand state (the dotted curve) as input as a re-representation of the hand state information accessible to that time (represented by the solid curve; the dashed curve shows the remaining, unobserved part of the hand state) 44

Figure 3.12 The solid curve shows the effective input that the network receives as the action progresses. At each simulation cycle the scaled curves are sampled (30 samples each) to form the spatial input for the network. Towards the end of the action the networks input gets closer to the final hand state. 45

Figure 3.13 (a) A single grasp trajectory viewed from three different angles to clearly show its 3D pattern. The wrist trajectory during the grasp is marked by square traces, with the distance between any two consecutive trace marks traveled in equal time intervals. (b) Left: The input to the network. Each component of the hand state is labelled. (b) Right: How the network classifies the action as a power grasp: squares: power grasp output; triangles: precision grasp; circles: side grasp output. Note that the response for precision and side grasp is almost zero. 47

Figure 3.14 Power and precision grasp resolution. The conventions used are as in the previous figure. (a) The curves for power and precision cross towards the end of the action showing the change of decision of the network. (b) The left shows the initial configuration and the right shows the final configuration of the hand   48

Figure 3.15: (Top) Strong precision grip mirror response for a reaching movement with a precision pinch. (Bottom) Spatial location perturbation experiment. The mirror response is greatly reduced when the grasp is not directed at a target object. (Only the precision grasp related activity is plotted. The other two outputs are negligible.) 48

Figure 3.16 Altered kinematics experiment. Left: The simulator executes the grasp with bell-shaped velocity profile. Right: The simulator executes the same grasp with constant velocity. Top row shows the graphical representation of the grasps and the bottom row shows the corresponding output of the network. (Only the precision grasp related activity is plotted. The other two outputs are negligible.) 49

Figure 3.17 Grasp and object axes mismatch experiment. Rightmost: the change of the object from cylinder to a plate (an object axis change of 90 degrees). Leftmost: the output of the network before the change (the network turns on the precision grip mirror neuron). Middle: the output of the network after the object change. (Only the precision grasp related activity is plotted. The other two outputs are negligible.) 50

Figure 3.18 The plots show the level of mirror responses of the explicit affordance coding object for an observed precision pinch for four cases (tiny, small, medium, big objects). The filled circles indicate the precision activity while the empty squares indicate the power grasp related activity.. 52

Figure 3.19 The solid curve: the precision grasp output, for the non-explicit affordance case, directed to a tiny object. The dashed curve: the precision grasp output of the model to the explicit affordance case, for the same object. 52

Figure 3.20: Empty squares indicate the precision grasp related cell activity, while the filled squares represent the power grasp related cell activity. The grasps show the effect of changing the object affordance, while keeping a constant hand state trajectory. In each case, the hand-state trajectory provided to the network is appropriate to the medium-sized object, but the affordance input to the network encodes the size shown. In the case of the biggest object affordance, the effect is enough to overwhelm the hand state’s precision bias. 53

Figure 3.21 The graph is drawn to show the decision switch time versus object size. The minimum is not at the boundary, that is, the network will detect a precision pinch quickest with a medium object size. Note that the graph does not include a point for "Biggest object" since there is no resolution point in this case (see the final panel of Figure 3.19) 54

Figure 3.22 The precision grasp action used to test our visual system is depicted by superimposed frames (not all the frames are shown) 54

Figure 3.23 The video sequence used to test the visual system is shown together with the 3D hand matching result (over each frame). Again not all the frames are shown.. 55

Figure 3.24 The plot shows the output of the MNS model when driven by the visual recognition system while observing the action depicted in Figure 3.22. It must be emphasized that the training was performed using the synthetic data from the grasp simulator while testing is performed using the hand state extracted by the visual system only. Dashed line: Side grasp related activity; Solid line: Precision grasp related activity. Power grasp activity is not visible as it coincides with the time axis. 56

Figure 4.1 The elevated circular region corresponds to the area defined by the equation (x*x+y*y) <0.25. The environment returns +1 as the reward if the action falls into the circular region, otherwise –1 is returned. 64

Figure 4.2 The adaptation of the firing potential of the stochastic units are shown as a series of evolving 3D maps. (left to right and top to bottom) 65

Figure 4.3. The normalized histogram of the actions generated over 60000 samples. Note that the actions generated captured the environment’s reward distribution (see Figure 4.1). 66

Figure 4.4 The stochastic environment’s double peaked reward distribution (see text for the explanation) 66

Figure 4.5 Some snapshots showing the phases of learning of the layer in the stochastic environment where the reward distribution has two peaks (see Figure 4.4). 67

Figure 4.6. The normalized histogram of 60000 data points (actions) generated by the trained layer in the stochastic environment depicted in Figure 4.4. 68

Figure 5.1 Infant grip configurations can be divided in two categories: power and precision grips. Infants tend to switch from power grips to precision grips as they grow (adapted from Butterworth et al. 1997) 73

Figure 5.2 The structure of the Infant Learning to Grasp Model. The individual layers are trained based on somatosensory feedback. 76

Figure 5.3 Hand Position layer specifies the approach direction of the hand towards the object. The representation is allocentric (centred on the object). Geometrically the space around the object can be uniquely specified with the vector (azimuth, elevation, radius). The Hand Position layer generates the vector by a local population vector computation. The locus of the local neighbourhood is determined by the probability distribution represented in the firing potential of Hand Position layer neurons (see Chapter 4, for details) 77

Figure 5.4:The grasp stability we used in the simulations is illustrated for a hypothetical precision pinch grip (note that this is a simplified, the actual hand used in the simulations has five fingers) 79

Figure 5.5 The trained model’s Hand Position layer is shown as a 3D plot. One dimension is summed to reduce the 4D map to a 3D map. Intuitively the map says: ‘when the object is above the shoulder and in front grasp it from the bottom’ 81

Figure 5.6: The output of the trained model’s target position layer is shown as a 3D plot. One dimension is summed to reduce the 4D map to a 3D map. The object is on the left side of the (right handed) arm. Intuitively, the map says ‘when the object is on the left side grasp it from the right side of the object’ 81

Figure 5.7 The learning evolution of the distribution of the Hand Position layer is shown as a 3D plot. Note that the 1000 neurons shown represent the probability distribution of approach directions. Initially, the layer is not trained and responds in a random fashion to the given input. As the learning progresses, the neurons gain specificity for this object location. 82

Figure 5.8 ILGM planned and performed a power grasp after learning. Note the supination (and to a lesser extent extension) of the wrist required to grasp the object from the bottom side. 83

Figure 5.9 Two learned precision grips (left: three fingered; right four fingered) are shown. Note that the wrist configuration for each case. ILGM learned to combine the wrist location with the correct wrist rotations to secure the object. 83

Figure 5.10 ILGM was able to generate two fingered precision grips. However these were less than the three or four finger grips. 84

Figure 5.11 The cube on the table simulation set up. ILGM interacts with the object with the physical constraint that it has to avoid collision with the table. 85

Figure 5.12 ILGM learned a ‘menu’ of precision grips with the common property that the wrist was placed well above the object. The orientation of the hand and the contact points on the object showed some variability. Two example precision grips are shown in the figure. 85

Figure 5.13. ILGM acquired thumb opposing index finger precision grips. 86

Figure 5.14 The three cylinder orientations and grasp attempts by the poor vision condition. 87