Sign Language Technologies: Recognising Spatial Referencing Structures

Supervisor: Dr Jessica Korte

Sign languages rely heavily on spatial referencing to convey grammatical and contextual information through partially lexical signs (PLSs) such as depicting signs, pointing signs, and fragment buoys. However, current sign language recognition (SLR) systems predominantly focus on fully lexical signs (FLSs), leading to an underrepresentation of these spatial structures in popular datasets despite their prevalence in natural signing. This study investigates the capacity of existing SLR models to recognise spatially dependent PLSs across different signed languages, using Belissen et al.’s (2020) framework—a pose estimation and bidirectional LSTM model initially developed for French Sign Language (LSF)—applied to the British Sign Language (BSL) Corpus. The research evaluates the model’s generalisability by analysing its performance in categorising FLSs and PLSs, with aims for improving systems targeting Australian Sign Language (Auslan). Results revealed that while the model achieved comparable accuracy for FLSs (F1: 0.12 on BSL vs. 0.15 on LSF) and pointing signs (F1: 0.52 vs. 0.54), depicting signs (F1: 0.14 vs. 0.30) and fragment buoys (F1: 0.00 vs. 0.10) showed markedly lower performance on BSL, correlating with their minimal representation in training data, and inconsistent pose estimation results. These disparities highlight the impact of data scarcity on underrepresented PLS categories and the need for linguistically diverse, balanced corpora to improve model robustness. Future work will focus on adapting the framework to Auslan-specific data to further evaluate the significance of spatial referencing in the production of accurate SLR translations, and an exploration of alternative pose estimation techniques to enhance tracking precision during rapid movements.