根據世界衛生組織（WHO）[1]，全球超過5%的人口需要對他們的聽力障礙進行康復和援助。手語是聽障和聾啞社群的主要交流方式，但現有的手語識別和教學產品的效果有限。現有模型在識別手語的複雜語義上下文和細微手指動作方面面臨挑戰。本論文提出了一個手語教學和評分系統（STSS），結合了Siamese長短期記憶（LSTM）進行粗粒度分類和卷積LSTM（ConvLSTM）進行細粒度分類。Siamese LSTM分析經過時間和空間規範化預處理的關鍵點數據，快速計算樣本視頻與標準化視頻數據集之間的相似性。ConvLSTM對相似性結果高於某一門檻的數據點進行進一步分析。本論文所提出的STSS，與其他機制進行比較，在精確率、召回率和F1-Score方面均表現出色。

According to the World Health Organization (WHO) [1], over 5% of the global population requires assistance for their hearing loss. Sign language is the primary communication method for the deaf community, but recognition technologies are limited in their effectiveness. Existing models are challenged by the complex contextual relationships of sign language gestures and recognition of subtle finger movements. This dissertation proposed a Sign Language Teaching and Scoring System (STSS) which combines coarse-grained classification using Siamese Long Short-Term Memory (LSTM) and a fine-grained classification utilizing the Convolutional LSTM (ConvLSTM) model. The Siamese LSTM analyzes spatially and temporally normalized key point data, and quickly calculates the similarity between sample and standard sign language videos. It utilizes an adaptive contrastive loss function that dynamically adjusts according to similarity measures. The contrastive loss function helps the model focus on more challenging gestures that are very similar, but distinct. The ConvLSTM conducts further analysis on datapoints where similarity results rise above a certain threshold. The proposed STSS is then compared with other mechanisms, showing outperformance with respect to precision, recall, and F1-Score.

Outline
Outline	IV
List of Figures	VI
Chapter 1. Introduction	1
1.1	Research Goals	3
1.2	Organization of the Dissertation	5
Chapter 2. Related work	6
2.1	Machine Learning	6
2.1.1	Hidden Markov Model (HMM)	6
2.1.2	K-nearest neighbor (KNN)	7
2.1.3	Support Vector Machine (SVM)	8
2.2	Deep Learning	9
2.2.1	Convolutional Neural Network (CNN)	9
2.2.2	Graph Convolutional Network (GCN)	11
2.2.3	Long short-term memory (LSTM)	12
2.2.4	Hybrid networks	13
2.2.5	Principal Component Analysis Network (PCANet)	16
Chapter 3. Preliminary	18
3.1	MediaPipe for Key Point Recognition	18
3.2	Siamese Neural Network Architecture	20
3.3	Long Short-Term Memory (LSTM) Network	20
3.4	Convolutional LSTM (ConvLSTM) Network	22
Chapter 4. Notations, Assumptions, Problem Description	24
4.1	Notations and Assumptions	24
4.2	Problem Description	24
4.3	Objective	26
Chapter 5. The Proposed Sign Language Teaching and Scoring System (STSS)	29
5.1	Data Preprocessing	30
5.1.1	Input Video Segmentation	30
5.1.2	Key Point Extraction	32
5.1.3	Temporal Normalization	32
5.1.4	Spatial Normalization	33
5.2	Coarse-Grained Classification using Siamese LSTM Model	34
5.3	Fine-Grained Classification using ConvLSTM Model	37
5.4	Summary	40
Chapter 6. Performance Evaluation	41
6.1	Dataset	41
6.2	Simulation Results	42
6.3	Summary	51
Chapter 7. Conclusion and Future Work	52
References	53

List of Figures
Fig. 3.1. Key point coordinates extracted for each hand with Mediapipe	19
Fig. 3.2. Key point coordinates extracted for the face and upper body	19
Fig. 3.3. LSTM Cell Structure	21
Fig. 3.4. Convolutional kernel operations over an image	23
Fig. 5.1. The architecture of proposed STSS mechanism	29
Fig. 5.2. Input Video Segmentation process	31
Fig. 5.3. Architecture of Siamese LSTM	35
Fig. 5.4. Architecture of ConvLSTM	38
Fig. 6.1. Training set distribution	41
Fig. 6.2. Testing set distribution	42
Fig. 6.3. Impact of sampling frame interval on accuracy and average classification time	44
Fig. 6.4. Confusion Matrix of each sign language category	45
Fig. 6.5. Varying threshold and layer counts in relation to recall, precision, and F1-Score	47
Fig. 6.6. ROC Curves for proposed STSS and TAM models	48
Fig. 6.7. Comparison of the proposed STSS, ML-CNN, and TAM in terms of precision, recall, and F1-Score	50

References
[1]	World Health Organization, “Deafness and Hearing Loss,” World Health Organization, https://www.who.int/news-room/fact-sheets/detail/deafness-and-hearing-loss, 2024.
[2]	Kudrinko, Karly, et al. "Wearable sensor-based sign language recognition: A comprehensive review." IEEE Reviews in Biomedical Engineering, vol. 14, pp. 82-97, 2020.
[3]	L. E. Baum and T. Petrie, “Statistical inference for probabilistic functions of finite state Markov chains,” Ann. Math. Statistics, vol. 37, no. 6, pp. 1554–1563, 1966.
[4]	T. Starner, J. Weaver, and A. Pentland, “Real-time American sign language recognition using desk and wearable computer based video,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 20, no. 12, pp. 1371–1375, 1998.
[5]	H.-L. Lou, “Implementing the Viterbi algorithm,” Proc. IEEE Signal Process. Mag., pp. 42–52, 1995.
[6]	X. Liu et al., “3D skeletal gesture recognition via hidden states exploration,” IEEE Trans. Image Process., vol. 29, pp. 4583–4597, 2020.
[7]	G. Fang, W. Gao, X. Chen, C. Wang, and J. Ma, ‘‘Signer-independent 843 continuous sign language recognition based on SRN/HMM,’’ Proc. Int. 844 Gesture Workshop, pp. 76–85, 2001.
[8]	Rung-Huei Liang and Ming Ouhyoung, "A real-time continuous gesture recognition system for sign language," Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition, Nara, Japan, pp. 558-567, 1998.
[9]	N. Tubaiz, T. Shanableh, and K. Assaleh, ‘‘Glove-based continuous Arabic 865 sign language recognition in user-dependent mode,’’ IEEE Trans. Human- 866 Mach. Syst., vol. 45, no. 4, pp. 526–533, 2015.
[10]	E. Alpaydin, Introduction to Machine Learning, MIT Press, 2010.
[11]	J. Wu, L. Sun, and R. Jafari, “A wearable system for recognizing American sign language in real-time using IMU and surface EMG sensors,” IEEE J. Biomed. Heal. Informat., vol. 20, no. 5, pp. 1281–1290, 2016.
[12]	W. Aly, S. Aly ,and S. Almotairi, ‘‘User-independent American Sign Language Alphabet Recognition based on Depth Image and PCANet features,’’ IEEE Access, vol. 7, pp. 123138–123150, 2019.
[13]	Luqman, Hamzah. "An efficient two-stream network for isolated sign language recognition using accumulative video motion." IEEE Access, vol. 10, pp. 93785-93798, 2022.
[14]	O. Koller, N. C. Camgoz, H. Ney, and R. Bowden, “Weakly Supervised Learning with Multi-Stream CNN-LSTM-HMMs to Discover Sequential Parallelism in Sign Language Videos,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 42, no. 9, pp. 2306–2320, 2020.
[15]	J. Forster, C. Schmidt, T. Hoyoux, O. Koller, U. Zelle, J. Piater, and H. Ney, “RWTH-PHOENIX-Weather: A large vocabulary sign language recognition and translation corpus,” Proc. Int. Conf. Language Resources Eval., pp. 3785–3789, 2012.
[16]	Lin, K., Wang, X., Zhu, L., Zhang, B. and Yang, Y., “SKIM: Skeleton-Based Isolated Sign Language Recognition With Part Mixing,” IEEE Transactions on Multimedia, vol. 26, pp.4271-4280, 2024.
[17]	J. Huang, W. Zhou, H. Li, and W. Li, ‘‘Attention-based 3D-CNNs for large-vocabulary sign language recognition,’’ IEEE Trans. Circuits Syst. Video Technol., vol. 29, no. 9, pp. 2822–2832, 2019.
[18]	M. Al-Hammadi, G. Muhammad, W. Abdul, M. Alsulaiman, M. A. Bencherif and M. A. Mekhtiche, "Hand Gesture Recognition for Sign Language Using 3DCNN," IEEE Access, vol. 8, pp. 79491-79509, 2020.
[19]	Wang, Zhibo, et al. "Hear sign language: A real-time end-to-end sign language recognition system," IEEE Transactions on Mobile Computing, vol. 21, no. 7, pp. 2398-2410, 2022.
[20]	Bencherif, Mohamed A., et al., "Arabic sign language recognition system using 2D hands and body skeleton data," IEEE Access, vol. 9, pp. 59612-59627, 2021.
[21]	Hao Zhou, Wengang Zhou, Yun Zhou, Houqiang Li, "Spatial-temporal multi-cue network for sign language recognition and translation," IEEE Transactions on Multimedia, vol. 24, pp. 768-779, 2021.
[22]	A. Graves and J. Schmidhuber, “Framewise phoneme classification with bidirectional LSTM and other neural network architectures,” Neural Netw., vol. 18, no. 5–6, pp. 602–610, 2005.
[23]	K. Cho et al., “Learning phrase representations using RNN encoder- decoder for statistical machine translation,” Proc. Conf. Empir. Meth- ods Nat. Lang. Process., pp. 1724–1734, 2014.
[24]	B. Fang, J. Co, and M. Zhang, ‘‘DeepASL: Enabling ubiquitous and non- intrusive word and sentence-level sign language translation,’’ Proc. 15th ACM Conf. Embedded Netw. Sensor Syst., pp. 1–13, 2017.
[25]	E. Rakun, A. M. Arymurthy, L. Y. Stefanus, A. F. Wicaksono, and I. W. W. Wisesa, ‘‘Recognition of sign language system for Indonesian language using long short-term memory neural networks,’’ Adv. Sci. Lett., vol. 24, no. 2, pp. 999–1004, 2018.
[26]	N. Heidari, and Iosifidis, “Temporal attention-augmented graph convolutional network for efficient skeleton-based human action recognition,” 25th IEEE International Conference on Pattern Recognition, Milan, Italy, pp. 7907-7914, 2021.
[27]	G A. Prasath, and K. Annapurani, “Prediction of sign language recognition based on multi layered CNN,” Multimedia Tools and Applications, vol. 82, no. 19, pp. 29649-29669, 2023.
[28]	Yang, Ti et al., “Articulated pose estimation with flexible mixtures-of-parts,” Proc. CVPR, pp.1385-1392, 2011.
[29]	Basavarajaiah, Madhushree, “6 basic things to know about Convolution,” Medium.com, https://medium.com/@bdhuma/6-basic-things-to-know-about-convolution-daef5e1bc411, 2019.
[30]	Pisa, Ivan et al., “Denoising Autoencoders and LSTM-Based Artificial Neural Networks Data Processing for Its Application to Internal Model Control in Industrial Environments—The Wastewater Treatment Plant Control Case,” Sensors, vol. 20, no. 13, p. 3743, 2020.
[31]	Google AI,“MediaPipe Solutions Guide,” Google AI for Developers, https://ai.google.dev/edge/mediapipe/solutions/vision/hand_landmarker, 2024.
[32]	Google AI,“MediaPipe Solutions Guide,” Google AI for Developers,  https://ai.google.dev/edge/mediapipe/solutions/vision/pose_landmarker, 2024.