A URMC Motion Labs & Studio X Collaboration
By: Mila Paymukhina '24, Karp Library Fellow XR Developer
Throughout the nearly three years I have worked at Studio X, I have had the opportunity to take part in several XR projects at the intersection of education and gaming. From creating an alchemical laboratory for Aurum VR to expanding a fun, interactive tutorial for new users to learn about VR and Studio X through Studio Xplorer, each project has provided invaluable insights. Yet, I was curious to understand XR's impact beyond education and gaming, and I got the chance to do just that through a collaboration between Studio X and URMC Motion Labs.
On Motion Labs
Motion Labs is a brand-new addition to the orthopedics department at URMC, aiding patients with movement disorders and athletes recovering from injuries to unlock their physical potential. The space is equipped with state-of-the-art technology primed for motion analysis and personalized exercise regimens. However, one of the current challenges of the space is the need for innovative software applications that leverage the lab’s equipment to reach its full potential.
Lab Set-Up
My work with the facility was set in a new lab space—initially barely used due to the absence of software integrating its advanced technology. This space boasts a screen spanning an entire wall, a treadmill with dual force plate detectors to monitor weight shifts and varying speeds for each leg, and eight motion-capturing cameras strategically positioned throughout the room.
The goal of the project
My project aims to enhance physical therapy for patients regaining range of motion in their shoulders by leveraging immersive technology. While walking on the treadmill, patients are immersed in a virtual environment projected on a screen in front of them. They are tasked with catching virtual balls thrown from different locations. The game captures the real-time position of their hands to project them onto the virtual hand avatars.
Development process
The project consisted of two primary phases:
- Establishing a real-time connection between sensors on the patients and the game development software
To utilize information from the Vicon cameras that track live motion data from sensors attached to the patients, I collaborated with the engineers from the Vicon team. Together, we established a live data transfer from Vicon software to Unity, a game engine that I used to develop the virtual experience. With the help of the Motion Labs team and Studio X full-time XR developer Liam, I calibrated the virtual hand avatars in the experience so that they seamlessly reflect the movements of patients' hands in real life.
2. Developing an endless-runner game to simulate a virtual walk through a park
After having the connection between the sensors and the game development environment established, I created an endless-runner game with Liam’s mentorship. For the minimal viable product (MVP), the game included the possibility for a physician to throw a ball from any location on the screen, speed up or slow down the pace to match the speed of the treadmill in the lab, manual calibration of height depending on the height of the patients, ability to pause and resume the game at any time, and a starting screen choosing the arm the patient needs to exercise.
Personal Impact
This project offered me an opportunity for hands-on learning, mirroring the intensity and depth of a real-world internship. I'm especially appreciative of the chance to work with the Vicon team. Vicon, renowned for its award-winning motion capture systems utilized by industry giants like Sony and Disney, provided valuable insights into their problem-solving methodologies and their technical expertise. Moreover, it was extremely rewarding to work with third-party stakeholders and have an opportunity to apply the skills I learned through my work at Studio X to bring the vision of the Motion Labs team into a (virtual) reality.
Future of the project
Throughout the spring semester, I developed an MVP of the project that is now ready to be tested on patients. Looking ahead, I envision a future where this project evolves with additional functionalities by successive developers, further augmenting the landscape of physical therapy.
