Author: Natalia Brody, Emory University, Class of 2019
Project Partner: Luca Donini, University of Cambridge
Mentors: Ralph Adolphs, Ph.D. and Juri Minxha, Caltech
Abstract
In the past, neuroscientists have used a single experimental paradigm to
study the way we interact with and process the world around us: a subject is shown a stimulus while some form of brain activity (ex. EEG, fMRI, single-units) is recorded. However, the stimulus is typically something simple (i.e., an image, a sound, or a video) and does not reflect the richness of the real world. This project attempts to address this problem by creating a more engaging platform for neuroscience research via virtual reality. Using the game engine Unity, several virtual environments were designed with threat and reward tasks meant to incentivize spatial exploration. Ultimately, virtual reality is shown to be a valid platform for neuroscience research: it elicits realistic responses, gives access to abundant and rich data, and can be applied to niches difficult to study in real-life scenarios (i.e., fear).
Introduction
Conventional ways of understanding the brain largely depend on 2D stimuli such as pictures and videos. However, brain functions such as emotion and social cognition require richer stimuli and a more trackable user experience in order to be adequately studied. For example, to begin addressing how PTSD affects the brain it would be necessary to 1) elicit a realistic emotional response that is expected to be altered by PTSD and 2) closely monitor the behavioral and psychophysiological responses associated with this response. This example is applicable to more than PTSD—Autism Spectrum Disorders and common disorders such as ADHD, phobias, and anxiety also require rich stimuli and the ability to closely monitor an individual’s experience in order to be best studied. Our project proposes that virtual reality is a fitting tool for studying these types of conditions. In addition to the ability to better evaluate human behavior, data collected in virtual reality (VR) could be compiled to create diagnostic models for neurological disorders. These models could be created by comparing the individual’s experience of a certain virtual environment with the experience of someone suspected to have a disorder, or comparing the experience of someone known to have a disorder with someone suspected to have it.
The main goal of this project was to create a more engaging and immersive platform for stimuli via virtual reality that could be used as an alternative to the traditional approach. To achieve this goal, a library of virtual environments and 360 ̊videos were created and presented to subjects in order to examine aspects of VR people respond to and to investigate whether certain stimuli (such as visual or auditory cues) affect decision making in this context. The project was conducted in two parts. The first part of the project was a pilot program in which numerous draft virtual environments were built and used to evaluate various aspects of VR. The second part of the project, “Experimental Study” consisted of a finalized game environment that was presented to subjects in a formal experimental settings.
Methods
Preliminary Game Design
Unity game engine is a platform used to develop video games for PC, consoles, mobile devices, and websites. Over the course of this project, Unity was used to build a library of virtual environments that could then be displayed in an Oculus Rift virtual reality headset. To begin, several drafts of potential environments for use in our study were
built. Audio, object, and animation packages were purchased from the Unity asset store (an online store through which users can package and distribute their Unity creations to other users) to include in the scenes. Because there are so many possibilities and factors of interest available for manipulation in VR, multiple draft environments were created. Across the various virtual environments, a theme of fear was maintained that would serve as a metric for measuring whether VR was realistic enough and its ability to elicit a strong emotional response (in this case, fear) from a subject. For example, a subject could encounter a frightening monster and, if they had an adverse reaction to this encounter, it could help demonstrate VR as an engaging form of stimuli that evokes reactions parallel to those that would occur in the real world.
The environments created for our library all have similar structure as exploratory spaces that contain both threats and rewards. Threats vary from wolves to zombies and other monsters, while the reward, collectible gold coins, remains the same across environments. This threat and reward system was included in order to incentivize spatial exploration (i.e. wanting to wander the space in order to obtain rewards) and to demonstrate that although virtual, components of virtual reality can influence decision making and the way one explores his environment. Additionally, 360 ̊ videos were included in the project library. These videos were sent to the Oculus Rift through Vrideo, an application that distributes immersive videos for use in virtual reality. The purpose of these videos was to give a broader understanding of how subjects interact with and respond to certain aspects of VR.
Pilot Program
After the project library was completed, a pilot program was created so that members of my lab could explore the environments and videos and give feedback. To collect feedback on the library environments, questionnaires were created using Qualtrics. Qualtrics is a tool that creates surveys for research purposes and offers a wide range of features for creating survey questions and analyzing response data. There were a total of three Qualtrics questionnaires: one to review an environment, one to review a video, and one to make comparisons about all of the environments and videos explored. Through the feedback collected, which features of VR are most and least effective were identified. This information was then used to refine current virtual environments and, in the future, can be used as a guide for research motivated VR game design.
Secondary Game Design
After the pilot program, a more complex virtual environment was created for the second phase of the project, a formal experimental study. Thus, a partnership with White Door Games game developer was started and permission to use their VR video game, Dreadhalls, for the project was obtained. The terms of this partnership allowed for access to and modification to the game’s source-code in order to meet the project’s needs.
Modifications included adding a timer, a coin counter, a map with customized visibility, and variability to maze structure. Unlike the free-roaming and spacious environments built for the first pilot program, Dreadhalls restricted user mobility. The game had a maze-like structure that forced the subject to travel down narrow hallways and make frequent decisions as they traverse from one side to the other. This format allowed for the investigation of decision-making in the presence of threat OR of threats in the envi-
ronment, and to understand how certain conditioned stimuli (such as a seemingly innocuous sound which becomes predictive of threat) can affect how subjects explore the maze. Overall, this data driven approach aimed to extrapolate trends regarding strategy and decision making in order to further verify that VR engages the user and provides insightful data on behavior.
Experimental Study
A second trial was created, through which Dreadhalls would again be used to further evaluate the suitability of VR as a platform for neuroscience research. Like the pilot program, participants would experience the environment(s) being tested and then provide feedback on a Qualtrics questionnaire. For this pilot program, a new set of questionnaires specific to Dreadhalls were provided. Subjects were recruited by sending emails to the Caltech “houses” (dormitories).
Upon arrival, subjects were administered game instructions and game controller instructions. Each subject was given 45 minutes to attempt the game as many times as they wanted. Each participant received a minimum of $10 for participation, but could increase their award based on how many coins they collected (COINS bonus) and
whether they reached the maze exit within the game (EXIT bonus). After the participant finished, they completed the questionnaire and received their monetary reward.
CURJ 