{"id":84,"date":"2009-10-26T09:16:29","date_gmt":"2009-10-26T00:16:29","guid":{"rendered":"http:\/\/www.cns.atr.jp\/dni\/?page_id=84"},"modified":"2015-07-14T11:45:24","modified_gmt":"2015-07-14T02:45:24","slug":"reconstruction-visual-images","status":"publish","type":"page","link":"https:\/\/bicr.atr.jp\/dni\/en\/research-projects\/reconstruction-visual-images\/","title":{"rendered":"Reading \u2018what you see\u2019: Reconstruction of visual images using brain activity patterns"},"content":{"rendered":"<h2><strong>Summary<\/strong><\/h2>\n<p>Imagine being able to replay your dreams and fantasies on a TV screen. This technique is called \u201cvisual image reconstruction\u201d. Visual image reconstruction allows us to reconstruct images that a person sees, displaying complex perceptual contents as they are represented in the brain.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignright size-full wp-image-213\" title=\"neuroncoversm\" src=\"https:\/\/bicr.atr.jp\/dni\/dni-content\/uploads\/2009\/10\/neuroncoversm.jpg\" alt=\"neuroncoversm\" width=\"250\" height=\"325\" srcset=\"https:\/\/bicr.atr.jp\/dni\/dni-content\/uploads\/2009\/10\/neuroncoversm.jpg 250w, https:\/\/bicr.atr.jp\/dni\/dni-content\/uploads\/2009\/10\/neuroncoversm-230x300.jpg 230w\" sizes=\"auto, (max-width: 250px) 100vw, 250px\" \/>Our laboratory has developed \u201cneural decoding\u201d techniques that we can use to predict the presented visual images that a person sees from the brain activity patterns. Neural decoding is performed using pattern recognition programs, also known as \u201cdecoders,\u201d that learn statistical relationships between several pre-specified visual images and the corresponding brain activity patterns. Although this approach can predict which stimulus is presented among a set of pre-specified categories (e.g., horizontal or vertical grating), the prediction is just categorical and is not an \u2018image\u2019 itself.<\/p>\n<p>In this study, we developed a method to reconstruct visual images by combining local image decoders that predict local image contrast at multiple spatial scales, using brain activity patterns measured by fMRI scanners. Using this method, we can reconstruct arbitrary visual images, including geometric figures and alphabet letters, which are not used to train the decoders. Reconstruction can also be used to identify the presented image among hundreds of millions of candidate images. Furthermore, we succeeded in replaying a sequence of presented visual images as a movie by reconstructing each visual image from 2-s individual fMRI scanning data.<\/p>\n<p>Although we currently only reconstruct the visual images that a person actually sees, it is possible that the same principle can be used to visualize subjective perception such as mental images and dreams. Our approach may serve as an innovative tool to reveal neural mechanisms of the mind as well as practical applications in the field of medical engineering.<br \/>\n<p id=\"player1\" style=\"display:none\"><a href=\"http:\/\/www.macromedia.com\/go\/getflashplayer\">Get the latest Flash Player<\/a> to see this player.<\/p>\r\n<noscript><p>[Javascript required to view Flash movie, please turn it on and refresh this page]<\/p><\/noscript>\r\n\r\n<script type=\"text\/javascript\">\r\n\tdocument.getElementById(\"player1\").style.display = \"\";\r\n\t\r\n\tvar s1 = new SWFObject(\"https:\/\/bicr.atr.jp\/dni\/wp-content\/plugins\/flv-embed\/flvplayer.swf\",\"player1\",\"480\",\"320\",\"7\");\r\n\ts1.addParam(\"wmode\",\"transparent\");\r\n\ts1.addParam(\"allowscriptaccess\",\"always\");\r\n\ts1.addParam(\"allowfullscreen\",\"true\");\r\n\ts1.addVariable(\"height\",\"320\");\r\n\ts1.addVariable(\"width\",\"480\"); \n\ts1.addVariable(\"displayheight\",\"320\");\n\ts1.addVariable(\"file\",\"https:\/\/bicr.atr.jp\/dni\/dni-content\/uploads\/2009\/10\/S1.flv\");\n\ts1.addVariable(\"backcolor\",\"0xFF2100\");\n\ts1.addVariable(\"screencolor\",\"0x222222\");\n\ts1.addVariable(\"link\",\"https:\/\/bicr.atr.jp\/dni\/dni-content\/uploads\/2009\/10\/S1.flv\");\n\ts1.addVariable(\"showstop\",\"true\");\n\ts1.addVariable(\"showdownload\",\"true\");\n\ts1.addVariable(\"autostart\",\"true\");\r\n\ts1.write(\"player1\");\r\n<\/script><br \/>\nA movie was created by concatenating images reconstructed from 2-s single-volume fMRI data of four runs presenting geometric shapes. Since fMRI data reflects changes in blood flow, it slightly delays from neural activity in the brain. The delay of the reconstruction from the image presentation corresponds to this hemodynamic response delay.<\/p>\n<h2><strong>Data<\/strong><\/h2>\n<p>The fMRI data used in this research can be downloaded freely. The data download site is <a href=\"https:\/\/bicr.atr.jp\/dni\/en\/downloads\/fmri-data-set-for-visual-image-reconstruction\/\">here<\/a>.<\/p>\n<h2><strong>Tool<\/strong><\/h2>\n<p>The sparse logistic regression (SLR) developed by O. Yamashita is used as a decoder in this research.  SLR toolbox site is <a href=\"https:\/\/bicr.atr.jp\/%7Eoyamashi\/SLR_WEB.html\">here<\/a>.<\/p>\n<h2><strong>Reference<\/strong><\/h2>\n<p>Yoichi Miyawaki, Hajime Uchida, Okito Yamashita, Masa-aki Sato, Yusuke Morito, Hiroki C. Tanabe, Norihiro Sadato, Yukiyasu Kamitani (2008). \u201cVisual image reconstruction from human brain activity using a combination of multi-scale local image decoders\u201d Neuron 60, 915-929.<\/p>\n<h2><strong>Collaborators<\/strong><\/h2>\n<p>Department of Neuroinformatics CNS ATR<br \/>\nYoichi Miyawaki, Hajime Uchida, Yukiyasu Kamitani<\/p>\n<p>Department of Computational Brain Imaging CNS ATR<br \/>\nOkito Yamashita, Masa-aki Sato<\/p>\n<p>National Institute of Information and Communications Technology<br \/>\nYoichi Miyawaki<\/p>\n<p>Nara Institute of Science and Technology<br \/>\nHajime Uchida, Yukiyasu Kamitani<\/p>\n<p>National Institute for Physiological Sciences<br \/>\nYusuke Morito, Hiroki Tanabe, Norihiro Sadato<\/p>\n<h2><strong>Contact<\/strong><\/h2>\n<p>ATR<br \/>\n2-2-2 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-0288<br \/>\nPhone: 0774-95-1172, FAX: 0774-95-1172<br \/>\nhttp:\/\/www.atr.jp\/index_j.html<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Summary Imagine being able to replay your dreams and fantasies on a TV screen. This technique is called \u201cvisual image reconstruction\u201d. Visual image reconstruction allows us to reconstruct images that a person sees, displaying complex perceptual contents as they are represented in the brain. Our laboratory has developed \u201cneural decoding\u201d techniques that we can use to predict the presented visual images that a person sees from the brain activity patterns. Neural decoding is performed using pattern recognition programs, also known as \u201cdecoders,\u201d that learn statistical relationships between several pre-specified visual images and the corresponding brain activity patterns. Although this approach can predict which stimulus is presented among a set of [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":78,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-84","page","type-page","status-publish","hentry"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/bicr.atr.jp\/dni\/en\/wp-json\/wp\/v2\/pages\/84","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/bicr.atr.jp\/dni\/en\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/bicr.atr.jp\/dni\/en\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/bicr.atr.jp\/dni\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/bicr.atr.jp\/dni\/en\/wp-json\/wp\/v2\/comments?post=84"}],"version-history":[{"count":11,"href":"https:\/\/bicr.atr.jp\/dni\/en\/wp-json\/wp\/v2\/pages\/84\/revisions"}],"predecessor-version":[{"id":236,"href":"https:\/\/bicr.atr.jp\/dni\/en\/wp-json\/wp\/v2\/pages\/84\/revisions\/236"}],"up":[{"embeddable":true,"href":"https:\/\/bicr.atr.jp\/dni\/en\/wp-json\/wp\/v2\/pages\/78"}],"wp:attachment":[{"href":"https:\/\/bicr.atr.jp\/dni\/en\/wp-json\/wp\/v2\/media?parent=84"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}