Introduction
Extended reality’s unique ability to construct a spatial environment allows it to place the user in an infinite number of contexts. As part of my Karp Library Fellowship last semester, I conducted research into a specific aspect of this wide world of possibilities: how these environments affect and enhance storytelling. For the purposes of this research, “narrative contexts” refers to any kind of storytelling, be that news, story-based games, VR films, recreating memories to share with others, or the conveyance of any other kind of narrative. In my research, using extended reality (XR) to deliver narratives was found to have notable effects on immersion, cognition, and overall impact on the viewer when compared to other storytelling mediums.
Effects on immersion
Narratively speaking, VR is unique from all other mediums in how it visually reconstructs the story around the viewer. A study from Xi’an Jiaotong at Liverpool University compared viewers’ engagement and arousal when viewing on a computer screen, movie theater-style project, and virtual reality headset. Across the board, VR produced the highest engagement and arousal in all participants, largely attributed to a greater feeling of immersion and spatial awareness. (11)
Cognition (fiction, nonfiction, and therapeutic)
Part of why spatial awareness and its effects on overall immersion are so important is because of the reshaping of cognitive processes within the viewer. Researcher Panagiotis Kourtesis employs the “extended mind theory,” which posits that cognition is not limited to the brain but includes external tools such as touch and interaction. For example, when viewing a film, the characters carry out all cognitive processes such as problem-solving and decision-making. This means the viewer’s cognitive processes are limited to processing the visual and auditory information of the film—the viewer is only completing cognitive processing within the brain and not utilizing external cognitive tools. Conversely, when engaging with a story in a virtual world, the viewer uses external tools by physically interacting with the scene, doing their own problem-solving, decision-making, and social interaction. By making engagement with the story active instead of passive, users “exhibit behavioral and cognitive changes that reflect the same patterns they would display in the real world.” (5) The immersion of XR creates a unique sense of embodiment within the story and suggests that immersive experiences could have more effective learning outcomes than viewing traditional documentary films.
A study done with middle school students found that by allowing students to recreate aspects of their life in VR and relive them, a process called reflective learning was deeply engaging. Reflective learning encourages self-reflection and can deepen understanding, which suggests that VR could be used to enhance knowledge retention. (10) The unique ability to fully embody a narrative has also found applications in mental health therapies, where it has been used to benefit PTSD patients. By using VR to reconstruct certain traumatic events, patients can examine the situation more holistically and change their viewpoints to be more productive. It also allows them to change certain aspects of these narratives before reliving them, helping to reshape their cognitive response to the event. (4)
Empathy and working beyond empathy tourism
Re-immersion in one’s own experiences can encourage reflection, but immersion in other’s experiences can have major pitfalls. VR allows 360º video to be experienced as if the user were the recorder. Many experiences specifically designed for the purposes of evoking “empathy” exist, offering the user a glimpse into topics such as living with a disability, living in poverty, or living in a warzone. While these experiences can provide perspective, many claim to be educational but are not directed or designed by the people they claim to represent. Researchers such as Lisa Nakamura at the University of Michigan have identified VR’s ability to serve as a so-called “empathy machine” that facilitates the phenomenon of “identity tourism,” where users can try on hardships, feel as though they have lived it, then take the headset off and return to their lives. (13) This allows the user to decide for themselves whether or not an experience is worthy of sympathy instead of taking members of marginalized groups at their word.
It is important to note that while VR improves cognition, it too is limited to visual, auditory, and spatial information. We should be wary of VR experiences that claim to “embody” complex lived experiences—such as those shaped by physical disabilities, mental health challenges, systemic oppression, or environmental injustice. These experiences, often rooted in marginalized identities related to race, gender, sexuality, or socioeconomic status, cannot be fully replicated or understood through VR alone. Moreover, users can always remove the headset—an option not available to those whose realities are being represented. Educational or “empathetic” VR should be created with care and humility, prioritizing ethical storytelling, community input, and real-world action rather than passive consumption or emotional tourism.
Immersive filmmaking
One such example of an educational immersive filmmaker is Vancouver-based Paisley Smith, who combines the therapeutic and artistic benefits of VR storytelling in their works centered on various social issues. Their interactive VR film Homestay is centered around their own grief after a Japanese exchange student named Taro, who was living with their family at the time, took his own life. In Homestay, the viewer is placed inside the Nitobe Memorial Garden at the University of British Columbia, paying homage to Taro’s country of origin and the country he came to study in, while interacting with Smith’s narration of their grief and attempts to understand the gulf between differing cultural experiences. Smith has cited the process as key in helping them process their own grief, as well as inviting the user to experience the author’s emotions and learn from them in a more direct way. Some of Smith’s contemporaries, Sutu and Tamara Shogaolu, have developed projects that invite the viewer to engage in VR Aboriginal “Dreamtime” meditation and place the viewer directly inside the story of a displaced Egyptian lesbian couple, respectively. (7) Since research has shown that diverse video games (including VR) can enhance prosocial emotional states, closely tied to empathy, it seems promising that telling stories in this manner might help players break down stereotypes and better empathize with other groups. (9)
Immersive journalism
Smith studied under Nonny de la Peña, often called the “Godmother of VR” and perhaps the foremost proponent of immersive journalism, the use of XR to place users directly inside news stories both as viewers and participants. De la Peña has designed numerous immersive journalistic experiences, but a particularly notable example is Gone Gitmo, which simulated imprisonment in an offshore military base, using audio from real military interrogations. Using Gone Gitmo, she conducted a study analyzing the “Response As If Real” (RAIR) factor in both traditional and immersive forms of journalism. A high RAIR quotient is massively sought after in journalism as it indicates that a story has been told effectively enough for the reader to understand it. De la Peña’s study found that even in low-fidelity VR environments, RAIR massively increased as compared to reading or even watching footage of the story on a 2D screen. The study also noted significantly improved retention of the story when compared to reading an article. With appropriate wariness about “empathy machines,” immersive journalism could be used to tell stories with more sticking power.
Another study analyzing map-based immersive news from the University of Bergen also found viewers reported significantly increased immersion as opposed to 2D environments; it also found that viewers were more inclined to explore the area in XR. This kind of map-based immersion could be used to help users better understand the physical location of news stories far away from where they reside, as well as understanding certain geographic-specific issues such as major weather events or border conflicts.
History
In a similar vein, VR has been used to great success in visualizing historical practices and events as well. A study of the Santorini Tomato Museum in Greece found that visitors better understood human presence and action in historical areas when presented with VR portrayals of a historic factory. (1) The study also analyzed using VR to visualize the full extent of now-damaged or destroyed monuments to help viewers better understand the context of the site they were visiting. Combined with De la Peña’s findings on memory retention, historical recreations in XR could significantly strengthen historical education.
UX constraints and best practices
At the end of the day, if VR can offer so many benefits, why isn’t it more widespread? Accessibility is a huge factor, with both the cost of headsets being extremely prohibitive as well as the learning curve of the technology. Since XR is still in a much earlier stage of development compared to other technologies, it can suffer uniquely in setup challenges and relative discomfort. All immersion-based benefits of XR rely deeply on a factor often called “flow state.” Flow is described as an “experience where action and awareness merge, there is high concentration on task and little attention is paid to time or self.” (12) When users are comfortable with the technology, flow is found to be higher in VR environments than other technologies.
Flow is crucial to immersion in VR states. However, the University of Bergen study found that while in VR environments, performing actions specifically associated with technology such as clicking, typing, and resizing windows reduced immersion and broke the user’s flow state. Subjects also regarded immersive experiences as entirely cosmetic if they were not interactive. In line with Kourtesis’ research on the extended mind theory, mimicking natural movements, such as grabbing or throwing, were found to improve immersion and usability. When considering this in tandem with De la Peña’s research on immersive journalism, it would seem that whether visuals are low or high fidelity is not as relevant to the effectiveness of VR-based environments, but that intuitive UX is crucial. When subpar user experience breaks immersion, it eliminates exactly what makes VR such an effective medium.
Conclusion
As VR becomes more accessible, we can encourage a uniquely deep immersion in new worlds, old worlds, and worlds otherwise outside of our reach. As we move forward to apply these contexts, we must strive not only to make these technologies accessible, but to properly and fully harness their potential by listening to users and making them intuitive for as broad of an audience as possible.
About this Author
Elizabeth Hogrefe, Karp Library Fellow, Narrative Designer
Photography by Hammer Chen
Citations
(1) Sylaiou, Stella, Panagiotis Dafiotis, Christos Fidas, Elia Vlachou, and Vani Nomikou. 2024. "Evaluating the Impact of XR on User Experience in the Tomato Industrial Museum “D. Nomikos”" Heritage 7, no. 3: 1754-1768. https://doi.org/10.3390/heritage7030082
(2) Sinlapanuntakul P, Korentsides J and Chaparro BS (2023) Exploring the user experience (UX) of a multi-window augmented reality environment. Front. Virtual Real. 4:1194019. doi: 10.3389/frvir.2023.1194019
(3) Sviland, V. (2023.) Designing Map-Based Visual Storytelling for News Articles in Mixed Reality [Master’s thesis, University of Bergen].
(4) Georgieva, Iva, and Georgi V Georgiev. “Reconstructing Personal Stories in Virtual Reality sas a Mechanism to Recover the Self.” International journal of environmental research and public health vol. 17,1 26. 18 Dec. 2019, doi:10.3390/ijerph17010026
(5) Kourtesis, P. (2024). The Extended Mind & Body in Extended Realities: A Scoping Review of XR Applications and Risks in the Metaverse.
(6) Wilson, K. (2018, September 26). “Homestay” VR film explores what it means to be an international student in Canada. The Georgia Straight.
(7) Warren, M. (2019, October 29). Balancing social justice and Immersive Entertainment. Film Independent. https://www.filmindependent.org/blog/balancing-social-justice-and-immersive-entertainment/
(8) De la Peña, N., Weil, P., Llobera, J., Spanlang, B., Friedman, D., Sanchez-Vives, M. V., & Slater, M. (2010). Immersive Journalism: Immersive Virtual Reality for the First-Person Experience of News. In Presence: Teleoperators and Virtual Environments (Vol. 19, Issue 4, pp. 291–301). MIT Press - Journals. https://doi.org/10.1162/pres_a_00005
(9) Tarrant, J. (2024, September 24). VR for empathy. Healium. https://tryhealium.com/science-behind-healium/vr-for-empathy
(10) Kim, D., Coenraad, M. & Park, H. R. (2021). Digital storytelling as a tool for reflection in virtual reality projects. Journal of Curriculum Studies Research, 3 (1), 101-121.
(11) Yu, Z., Lo, C., Niu, M. & Liang, H. (2023). Comparing Cinematic Conventions through Emotional Responses in Cinematic VR and Traditional Mediums.
(12) Hassan, L., Jylhä, H., Sjöblom, M. & Hamari, J. (2020). Flow in VR: A Study on the Relationships Between Preconditions, Experience and Continued Use. Proceedings of the 53rd Hawaii International Conference on System Sciences.
(13) Cantos, M. (2024, October 31). Nonny de la Peña has created journalism’s new reality. Hispanic Executive. https://hispanicexecutive.com/nonny-de-la-pena-emblematic-group/
(14) Cheng, K., & Verboord, M. (2024). The diffusion of immersive journalism as media innovation from media professionals’ perspectives. Journalism, 0 (0). https://doi.org/10.1177/14648849241282510
(15) Knowles, L., Stelzer, E., Jovel, K. & O’Connor, M. A pilot study of virtual support for grief: Feasibility, acceptability, and preliminary outcomes. Computers in Human Behavior, vol 73, 2017.
