Exploring the effects of interactions with intelligent agents in immersive systems

Supervisors: Assoc Prof Selen Turkay & Joel Harman

Virtual Reality (VR) applications have seen increased usage in several fields, including immersive experiences such as games, educational tools and training due to the accessibility of powerful consumer grade hardware. Despite this growth, VR tutorials used for introducing Players have not significantly changed, still following basic tutorial methods such as instructional screens. We aimed to determine Player’s struggles with VR tutorials/onboarding experiences and explore a prototype modular system for VR experiences utilising an LLM to provide personalised assistance to Players. We determined Player struggles by surveying VR Developers and Showcasers, additionally, we developed a basic tutorial experience in Unity to determine the viability of a modular tutorial experience. We found that the primary issues for Players included object grabbing, teleporting and steering, as well as the usage of inputs such as triggers and grip buttons due to spatial/sensory issues while in VR. For these issues, we propose tutorial features such as guided tutorials and requested assistance through an NPC agent, as well as spatial callouts for controls to assist with spatial issues. This research finds strong support for personalised assistance via an LLM for the improvement of tutorial experiences for Players, as well as strong support for a modular tool to reduce VR development time.