Sign for Safety: The Ethical Integration of Motion Capture Animation and Signing Digital Avatars into Emergency Systems for Deaf Communities

Supervisors: Dr. Maria Zelenskaya & Dr. Jessica Korte

With over 16,000 Deaf Australians and only 747 certified Auslan interpreters, developing accurate and ethical AI signing digital avatars and motion capture processes is crucial to supporting Deaf individuals’ communication, improving accessibility, and preserving Deaf cultural identity. This research focuses on enhancing motion capture, animation, and AI technology to accurately capture the phonological features and Non-Manual Features of Australian Sign Language (Auslan), such as, handshapes, hand positioning relative to the body, head movements, eye contact, and facial expressions. We utilized Unreal Engine, StretchSense gloves, and Xsens MVN software with a seasoned Deaf performer*, gathering real-time insights on sign positioning, facial expressions, eye contact, and timing. Surveys within Deaf communities provided qualitative and quantitative perspectives on the animation quality and motion capture processes.

Based our survey answers and contemporary research findings, results found that:

1. Emotion and expression in digital avatars require significant improvement, especially for emergency communication. AI advancements in motion capture software, such as Chameleon, enhance accuracy and timing, while manual animation improves quality but faces human error and language challenges.
2. Precise phonological and non-manual features are essential for clarity in Sign Languages. Multiple camera angles and full-body captures are vital.
3. Qualitative data surveys that seek to understand Deaf perspectives often lack accessibility. Improvements can be made through signed captions, plain English instructions, and the option for participants to sign responses anonymously.

To better engage Deaf communities, research environments must accommodate Auslan and provide ethical communication methods. Potential pathways for further research include AIpowered signing surveys to maintain anonymity, consistent feedback through Zoom meetings, and marker-less capture software like Chameleon to enhance clarity, cultural preservation, and efficiency. Further research is recommended to explore AI for public announcements and refining survey processes.

*Please note: When referring to “Deaf performers” in the context of animation and motion capture, “performer” is used to refer to a person that is having their movements or motion data recorded. The term “Deaf performer” was regarded as most appropriate at the time of research and during the data capture, especially after consulting multiple Deaf individuals within similar fields.