Reading ‘what you see’: Reconstruction of visual images using brain activity patterns
Summary
Imagine being able to replay your dreams and fantasies on a TV screen. This technique is called “visual image reconstruction”. Visual image reconstruction allows us to reconstruct images that a person sees, displaying complex perceptual contents as they are represented in the brain.
Our laboratory has developed “neural decoding” techniques that we can use to predict the presented visual images that a person sees from the brain activity patterns. Neural decoding is performed using pattern recognition programs, also known as “decoders,” that learn statistical relationships between several pre-specified visual images and the corresponding brain activity patterns. Although this approach can predict which stimulus is presented among a set of pre-specified categories (e.g., horizontal or vertical grating), the prediction is just categorical and is not an ‘image’ itself.
In this study, we developed a method to reconstruct visual images by combining local image decoders that predict local image contrast at multiple spatial scales, using brain activity patterns measured by fMRI scanners. Using this method, we can reconstruct arbitrary visual images, including geometric figures and alphabet letters, which are not used to train the decoders. Reconstruction can also be used to identify the presented image among hundreds of millions of candidate images. Furthermore, we succeeded in replaying a sequence of presented visual images as a movie by reconstructing each visual image from 2-s individual fMRI scanning data.
Although we currently only reconstruct the visual images that a person actually sees, it is possible that the same principle can be used to visualize subjective perception such as mental images and dreams. Our approach may serve as an innovative tool to reveal neural mechanisms of the mind as well as practical applications in the field of medical engineering.
Warning: Undefined variable $image in /home/cns/web/dni/htdocs/wp-content/plugins/flv-embed/flv-embed.php on line 1123
Warning: Undefined variable $shownavigation in /home/cns/web/dni/htdocs/wp-content/plugins/flv-embed/flv-embed.php on line 1123
Warning: Undefined variable $type in /home/cns/web/dni/htdocs/wp-content/plugins/flv-embed/flv-embed.php on line 1123
A movie was created by concatenating images reconstructed from 2-s single-volume fMRI data of four runs presenting geometric shapes. Since fMRI data reflects changes in blood flow, it slightly delays from neural activity in the brain. The delay of the reconstruction from the image presentation corresponds to this hemodynamic response delay.
Data
The fMRI data used in this research can be downloaded freely. The data download site is here.
Tool
The sparse logistic regression (SLR) developed by O. Yamashita is used as a decoder in this research. SLR toolbox site is here.
Reference
Yoichi Miyawaki, Hajime Uchida, Okito Yamashita, Masa-aki Sato, Yusuke Morito, Hiroki C. Tanabe, Norihiro Sadato, Yukiyasu Kamitani (2008). “Visual image reconstruction from human brain activity using a combination of multi-scale local image decoders” Neuron 60, 915-929.
Collaborators
Department of Neuroinformatics CNS ATR
Yoichi Miyawaki, Hajime Uchida, Yukiyasu Kamitani
Department of Computational Brain Imaging CNS ATR
Okito Yamashita, Masa-aki Sato
National Institute of Information and Communications Technology
Yoichi Miyawaki
Nara Institute of Science and Technology
Hajime Uchida, Yukiyasu Kamitani
National Institute for Physiological Sciences
Yusuke Morito, Hiroki Tanabe, Norihiro Sadato
Contact
ATR
2-2-2 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-0288
Phone: 0774-95-1172, FAX: 0774-95-1172
http://www.atr.jp/index_j.html