# Deep Image Reconstruction ## Original paper Shen, G., Horikawa, T., Majima, K., and Kamitani, Y. (2017). Deep image reconstruction from human brain activity. bioRxiv. ## Dataset ### MRI data This dataset contains fMRI data from from three subjects ('sub-01', 'sub-02', and 'sub-03'). Each subject data contains five types of MRI data collected over multiple scanning sessions. - 'ses-perceptionNaturalImageTraining': fMRI data from the training natural image sessions in the image presentation experiment (120 runs; 15 sessions) - 'ses-perceptionNaturalImageTest': fMRI data from the test natural image sessions in the image presentation experiment (24 runs; 3 sessions) - 'ses-perceptionArtificalImage': fMRI data from the geometric shape sessions in the image presentation experiment (20 runs; 2-3 sessions) - 'ses-perceptionLetterImage': fMRI data from the alphabetical letter sessions in the image presentation experiment (12 runs; 1 session) - 'ses-imagery': fMRI data in the imagery experiment (20 runs; 3-4 sessions) Each scanning session consisted of functional (EPI) and anatomical (inplane T2) images. The functional EPI images covered the entire brain (TR, 2000 ms; TE, 43 ms; flip angle, 80°; voxel size, 2 × 2 × 2 mm; FOV, 192 × 192 mm; number of slices, 76; slice gap, 0 mm; multiband factor, 4) and inplane T2-weighted anatomical images were acquired with the same slices used for the EPI (TR, 11,000 ms; TE, 59 ms; flip angle, 160°; voxel size, 0.75 × 0.75 × 2.0 mm; FOV, 192 × 192 mm). The dataset also includes a T1-weighted anatomical reference image for each subject (TR, 2250 ms; TE, 3.06 ms; TI, 900 ms; flip angle, 9°; voxel size, 1.0 × 1.0 × 1.0 mm; FOV, 256 × 256 mm). The T1-weighted images were scanned only once for each subject in a separate scanning session and are stored in 'ses-anatomy' directories. The T1-weighted images were defaced by pydeface (). All DICOM files are converted to Nifti-1 files by mri_convert in FreeSurfer. ### Task event files Task event files ('sub-\*_ses-\*_task-\*_run-\*_events.tsv') contains recorded event (stimuli presentation, subject responses, etc.) during fMRI runs. In task event files for image presentation experiments ('task-perception'), each column represents: - 'onset': onset time of an event (sec) - 'duration': duration of the event (sec) - 'event_type': type of the event (1: Stimulus presentation block, 2: Repetition block, -1, -2, and -3: Initial, inter-trial, and post rest blocks without visual stimulus) - 'stimulus_id': ID for the stimulus presented in the block - 'response_time': time of button press in the block, elapsed time from the beginning of each run (sec; '0' means that a subject did not press the button in the block) In the natural image presentation experiments, stimulus ID is a number in which the integer part indicates WordNet ID for the synset (category) and the decimal part indicates image ID. For example, '1518878.005958' represents 'image 5958 in synset n01518878 (‘ostrich’)'. Due to license issues, we do not include the stimulus images in the data repository. A script downloading the stimulus images are available at . Downloaded image files are named as 'XXXX-YYYY.JPEG', where 'XXXX' and 'YYYY' represents the WordNet ID and image ID, respectively (e.g., 'n01518878_5958.JPEG'). In the geometric shape and alphabetical letter presentation experiments, the stimuli consisted of a total of 40 combinations of 5 shapes (filled square, small opened square, large opened square, +, and X) and 8 colors (red, green, blue, cyan, magenta, yellow, white, and black). The stimulus ID indicates: - 1-5: red, 6-10: green, 11-15: blue, 16-20: yellow, 21-25: magenta, 26-30: cyan, 31-35: white, and 36-40: black, - in each color group, the 1st, 2nd, 3rd, 4th and 5th stimuli are filled square, small opened square, large opened square, +, and X, respectively. In the alphabetical letter presentation experiments, stimulus ID of 1 to 10 represent 'A', 'C', 'E', 'I', 'N', 'O', 'R', 'S', 'T', and 'U', respectively. In task event files for imagery experiments ('task-imagery'), each column in represents: - 'onset': onset time (sec) of an event - 'duration': duration (sec) of the event - 'trial_no': trial (block) number of the event - 'event_type': type of the event (-1: rest block, 1: cue presentation period, 2: imagery period, 3: evaluation of imagery quality period, 4: inter-trial and post rest period) - 'category_id': ImageNet/WordNet synset ID of a synset (category) which the subject was instructed to imagine in the block - 'response_time': time of button press for imagery quality evaluation at the block, elapsed time from the beginning of each run (sec) - 'evaluation': vividness of their mental imagery evaluated by the subject (5: very vivid, 4: fairly vivid, 3: rather vivid, 2: not vivid, 1: cannot recognize the target image)