多人西式划船運動非常重視整組選手划槳動作的一致性與協調性，整齊的划船動作能夠創造出更好的成績。本研究在獲得多位選手划槳訓練影片，並由OpenPose系統取得人體關節點資訊後，將兩位選手之划槳影片進行配對分析，經過圖神經網路(Graph Neural Network)的計算後，皆獲得一個相似度數值，由該數值高低可以判斷該組之兩位選手的協調程度，數值高表示該對選手可以在划船上取得不錯的協調性，能增加划船的速度。反之，若相似度數值較低表示該對選手不適合作為一組，協調性恐怕較差無法發揮最好的划船效率。
本研究在專家的建議下，擷取划船練習影片二十秒到六十秒間的划船動作進行配對，並採用區段相似度進行最佳配對計算，實驗結果顯示，二維圖嵌入模型的區段相似度，具有最佳的影片配對結果，對於教練的訓練方向與選手配對可以提供重要的參考依據。

Consistency among players are the most important issue in Rowing sport. Coordinated rowing movements always create better athletic results. In this study, we first obtain the rowing video of each player and then apply the video to acquire OpenPose system to measure human joint points on each frame. The human joint points information are then applied to measure the consistence of two players. After the matching calculation of Graph Neural Network, each group of two player can obtain one similarity to represent the degree of consistence. Higher similarity leads to better consistency. 
By some experts’ suggestion, our matching process measures from 20 seconds to 60 seconds. The proposed regional similarity with 2-D Graph Embedding Model acquires best similarity acquisition. The proposed model can acquire the best Rowing match among many players and the results are efficient for coach to improve athletes’ training methods.

論文提要內容 i
Abstract ii
目錄 iii
圖目錄 v
表目錄 vii
第一章 緒論 1
1.1 研究背景與動機 1
1.2 研究目的 2
1.3 論文架構 2
第二章 相關研究 4
2.1 人體動作識別研究探討 4
2.2 OpenPose系統介紹 4
2.3 OpenPose系統演算法介紹 8
2.4 圖神經網路介紹 12
2.4.1 GNN的輸入類型 13
2.4.2 GNN的應用輸出類型 14
2.4.3 GNN方法種類 15
2.4.3.1 Recurrent Graph Neural Networks (RecGNNs) 15
2.4.3.2 Spatial-temporal graph neural networks (STGNNs) 17
2.4.4 圖嵌入模型與圖配對網路 18
2.4.4.1 圖編碼層 19
2.4.4.2 傳播層 20
2.4.4.3 聚合層 22
2.4.4.5 訓練 23
第三章 選手動作相似度分析 25
3.1 系統流程架構 25
3.2 資料集 27
3.3 取得選手肢體座標資訊 30
3.4 動作相似度分析 31
3.4.1 一維算法 32
3.4.2 二維算法 32
3.5 區段相似度(period similarity) 33
3.6 影片分析起始位置 33
第四章 選手動作分析實驗結果 34
4.1 划槳姿勢相似度評估 34
4.2 划槳姿勢分析 37
4.3 實驗結果驗證 49
4.4 實驗結果總結 54
第五章 結論與未來研究方向 56
5.1 結論 56
5.2 未來研究方向 56
參考文獻 58
 
圖目錄
圖 1、OpenPose系統的兩種關節點[42] 5
圖 2、遮擋情況下的姿態評估[3] 5
圖 3、OpenPose系統手指(a)及臉部(b)關節點[42] 6
圖 4、DeepCut流程示意圖[20] 8
圖 5、汽車骨架預測[3] 8
圖 6、一個人的姿態評估示意圖 8
圖 7、多人的姿態評估[3] 9
圖 8、流程圖[3] 10
圖 9、PAF設定[3] 11
圖 10、OpenPose系統模型架構圖[4] 11
圖 11、(a)歐幾里得空間、(b)非歐幾里得空間[25] 13
圖 12、(a)有向圖，(b)無向圖，(c) 時空圖 14
圖 13、分析分子的物理結構[25] 14
圖 14、(a)分析社群網路上的人際關係、(b)一張影像中的物件關係[25] 14
圖 15、Hidden state更新 16
圖 16、模型學習，更新參數 17
圖 17、非同步進行 17
圖 18、基於CNN的STGNN方法[28] 18
圖 19、圖嵌入模型(a)和圖配對網路(b)[38] 18
圖 20、系統流程圖 26
圖 21、選手使用划船機 27
圖 22、選手在畫面上太小而無法被辨識到 28
圖 23、錯誤將水面上選手的倒影判斷成選手的腿部 28
圖 24、完成一週期划槳 29
圖 25、完成姿態評估的一週期划槳 31
圖 26、Graph Embedding Model一維與二維運算0~3600畫面折線圖 39
圖 27、Graph Matching Network一維與二維運算0~3600畫面折線圖 40
圖 28、3號、11號與12號選手0~3600畫面折線圖 42
圖 29、4號、6號與7號選手0~3600畫面折線圖 42
圖 30、7號、10號與12號選手0~3600畫面折線圖 43
圖 31、8號、7號與11號選手0~3600畫面折線圖 43
圖 32、10號、7號與12號選手0~3600畫面折線圖 44
圖 33、11號、3號與12號選手0~3600畫面折線圖 44
圖 34、14號、0號與3號選手0~3600畫面折線圖 45
圖 35、1號、2號與11號選手600~1800畫面折線圖 45
圖 36、3號、0號與1號選手600~1800畫面折線圖 46
圖 37、4號、7號與13號選手600~1800畫面折線圖 46
圖 38、6號、8號與9號選手600~1800畫面折線圖 47
圖 39、7號、8號與10號選手600~1800畫面折線圖 47
圖 40、12號、3號與9號選手600~1800畫面折線圖 48
圖 41、14號、3號與1號選手600~1800畫面折線圖 48
圖 42、選手0號和3號，在0個畫面之後每15個畫面做比對 50
圖 43、選手0號和3號，在500個畫面之後每15個畫面做比對 51
圖 44、選手0號和14號，在0個畫面之後每15個畫面做比對 53
圖 45、選手0號和14號，在500個畫面之後每15個畫面做比對 54
 
表目錄
表 1、比較WL kernel、Graph Embedding Model (GNN)和Graph Matching Network (GMN)準確度 [38] 24
表 2、0~3600畫面使用Graph Embedding Model一維輸入實驗結果 35
表 3、0~3600畫面使用Graph Embedding Model二維輸入實驗結果 35
表 4、0~3600畫面使用Graph Matching Network一維輸入實驗結果 36
表 5、0~3600畫面使用Graph Matching Network二維輸入實驗結果 36
表 6、各運算方法運算時間 41

[1] C. Reinschmidt, A. J. van-den-Bogert, B. M. Nigg, A. Lundbrg, and N. Murphy, “Effect of Skin Movement on the Analysis of Skeletal Knee Joint Motion During Running,” Journal of Biomechanics, vol. 30, pp. 729-732, 1997.
[2] Y. Fujimori, and Y. Ohmura, T. Harada, “Wearable motion capture suit with full-body tactile sensors,” In IEEE International Conference on Robotics and Automation, pp. 3186-3193, 2009.
[3] Z. Cao, G. Hidalgo, T. Simon, S. -E. Wei and Y. Sheikh, “OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, no. 1, pp. 172-186, 2017.
[4] Z. Cao, T. Simon, S. E. Wei, and Y. Sheikh, “Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields,” In Conference Computer Vision and Pattern Recognition, pp. 7291-7299, 2017.
[5] S. Yan, Y. Xiong, and D. Lin, “Spatial Temporal Graph Convolutional Network for Skeleton-Based Action Recognition,” In Thirty-Second AAAI Conference on Artifificial Intelligence, AAAI18-Vision, pp. 7774-7452, 2018.
[6] M. M, and P. S Mohan, “A Real-Time Embedded System for Human Action Recognition Using Template Matching,” In International Conference on Electrical, Instrumentation and Communication Engineering, pp. 1-5, 2017.
[7] B. Zhang, C. Y. Chen, L. Cw Chan and W. Fok, “Intelligent Sports performance Scoring and Analysis system Based on Deep Learning Network,” In 2020 3rd International Conference on Artificial Intelligence and Big Data, pp. 17-21, 2020.
[8] Y. Cho and K. Yoon, “PaMM: Pose-Aware Multi-Shot Matching for Improving Person Re-Identification,” IEEE Transactions on Image Processing, vol. 27, no. 8, pp. 3739-3752, 2017.
[9] R. Theagarajan, F. Pala, X. Zhang, B. Bhanu, “Soccer:Who Has the Ball? Generating Visual Analytics and Player Statistics,” In Computer Vision and Pattern Recognition, pp. 1830-18308, 2018.
[10] T. Tan and Y. Z. Zhao, “Neural Network Model Of Message Passing in Team Cooperation,” In 2020 International Conference on Computer Engineering and Application, pp. 396-399, 2020.
[11] E. Tiwari, P. Sardar and S. Jain, “Football Match Result Prediction Using Neural Networks and Deep Learning,” In 2020 8th International Conference on Reliability, Infocom Technologies and Optimization, pp. 229-231, 2020.
[12] H. Jiang, Y. Lu and J. Xue, “Automatic Soccer Video Event Detection Based On A Deep Neural Network Combined CNN and RNN,” In 2016 IEEE 28th International Conference on Tools with Artificial Intelligence, pp. 490-494, 2016.
[13] J. Wang and L. Yu, “Use the combination of the decision tree and the artificial neural networks to predict the outcome of table tennis matches,” In 2010 Sixth International Conference on Natural Computation, pp. 1929-1933, 2010.
[14] P. Edouard Sarlin, D. DeTone, T. Malisiewicz, and A. Rabinovich, “SuperGlue: Learn Feature Matching with Graph Neural Networks,” In Conference on Computer Vision and Pattern Recognition, pp. 4938-4947, 2020.
[15] J. Li, and J. Li, “An Improved Pose Matching Method Based on Deep Learning,” In IEEE 3rd International Conference on Civil Aviation Safety and Information Technology, pp. 369-371, 2019.
[16] S. Huang, J. Yuan and H. Chen, “Human actions segmentation and matching based on 3D skeleton model,” In Proceedings of the 32nd Chinese Control Conference, pp. 5877-5882, 2013.
[17] M. Chikano, F. Tomiyasu, S. Awai, Y. Hirai and T. Konno, “Person Matching Technology using Gait Information of 2D Pose Estimation,” In 2020 IEEE 2nd Global Conference on Life Sciences and Technologies, pp. 140-144, 2020.
[18] B. Fernando, S. Shirazi and S. Gould, “Unsupervised Human Action Detection by Action Matching,” In Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pp. 1604-1612, 2017.
[19] A. Toshev and C. Szegedy, “DeepPose: Human Pose Estimation via Deep Neural Networks,” In Conference on Computer Vision and Pattern Recognition, pp. 1653- 1660, 2013.
[20] L. Pishchulin, E. Insafutdinov, S. Tang, B. Andres, M. Andriluka, P. V. Gehler, B. Schiele, “DeepCut: Joint Subset Partition and Labeling for Multi Person Pose Estimation,” In Conference on Computer Vision and Pattern Recognition, pp. 4929-4937, 2016.
[21] S. E. Wei, V. RamaKrishna, T. Kanade, Y. Sheikh, “Convolutional Pose Machines,” In Conference on Computer Vision and Pattern Recognition, pp. 4724-4732, 2016.
[22]V. Ramakrishna, D. Munoz, M. Hebert, J. A. Bagnell, Y. Sheikh, “Pose Machines: Articulated Pose Estimation via Inference Machines,” In European Conference on Computer Vision, pp. 33-47, 2014.
[23] S. Qiao, Y. Wang, J. Li, “Real-Time Human Gesture Grading Based on OpenPose,” In 10th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics, pp. 1-6, 2017.
[24] W. Lina and J. Ding, “Behavior detection method of OpenPose combined with Yolo network,” In 2020 International Conference on Communications, Information System and Computer Engineering, pp. 326-330, 2020.
[25] J. Zhou , G. Cui , Z. Zhang, C. Yang, Z. Liu, L. Wang, C. Li, M. Sun, “Graph Neural Networks: A Review of Methods and Applications, ” AI Open 2021, Vol 1, pp. 57-81, 2021.
[26] A. Sperduti and A. Starita, “Supervised neural networks for the classification of structures,” IEEE Transactions on Neural Networks, vol. 8, no. 3, pp. 714–735, 1997.
[27] M. Gori, G. Monfardini, and F. Scarselli, “A new model for learning in graph domains,” International Joint Conference on Neural Networks, vol. 2, pp. 729–734, 2005.
[28] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S. Yu, “A Comprehensive Survey on Graph Neural Networks,” IEEE Transactions on Neural Networks and Learning Systems, vol. 32, no. 1, pp. 4–24, 2019.
[29] F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Monfardini, “The Graph Neural Network Model,” IEEE Transactions on Neural Networks, vol. 20, no. 1, pp. 61–80, 2008.
[30] C. Gallicchio and A. Micheli, “Graph Echo State Networks,” In International Joint Conference on Neural Networks. IEEE, pp. 1–8, 2010.
[31] Y. Li, D. Tarlow, M. Brockschmidt, and R. Zemel, “Gated Graph Sequence Neural Networks,” In International Conference on Learning Representations, San Juan, Puerto Rico, United States, 2016.
[32] H. Dai, Z. Kozareva, B. Dai, A. Smola, and L. Song, “Learning Steady-States of Iterative Algorithms over Graphs,” In International Conference on Machine Learning, pp. 1114–1122, 2018.
[33] J. Zhang, X. Shi, J. Xie, H. Ma, I. King, and D.-Y. Yeung, “GaAN: Gated Attention Networks for Learning on Large and Spatiotemporal Graphs,” In Uncertainty in Artificial Intelligence, California, United States, 2018.
[34] Y. Seo, M. Defferrard, P. Vandergheynst, and X. Bresson, “Structured Sequence Modeling with Graph Convolutional Recurrent Networks,” In International Conference on Neural Information Processing. Springer, pp. 362–373, 2018.
[35] Y. Li, R. Yu, C. Shahabi, and Y. Liu, “Diffusion Convolutional Recurrent Neural Network: Data-Driven Traffic Forecasting,” In International Conference on Learning Representations, Vancouver, Canada, 2018.
[36] M. Defferrard, X. Bresson, and P. Vandergheynst, “Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering,” In Neural Information Processing Systems, pp. 3844–3852, 2016.
[37] A. Jain, A. R. Zamir, S. Savarese, and A. Saxena, “Structural-RNN: Deep Learning on Spatio-Temporal Graphs,” In Conference on Computer Vision and Pattern Recognition, pp. 5308–5317, 2016.
[38] Y. Li, C. Gu, T. Dullien, O. Vinyals, and P. Kohli, “Graph Matching Networks for Learning the Similarity of Graph Structured Objects,” In International Conference on Machine Learning, New Orleans, Louisiana, United States, 2019.
[39] Y. Li, D. Tarlow, M. Brockschmidt, and R. Zemel “Gated graph sequence neural networks,” In International Conference on Learning Representations, San Juan, Puerto Rico, United States, 2016.
[40] N. Shervashidze, P. Schweitzer, E. Leeuwen, K. Mehlhorn, and K. M. Borgwardt, “Weisfeiler-Lehman Graph Kernels,” In Journal of Machine Learning Research, pp. 2539–2561, 2011.
[41] 陳文和. “定性分析在划船測功儀技術上之應用,” 真理大學運動知識學報, 第十一期, pp. 21-33, 2014.
[42] Openpose Doc. https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/doc/02_output.md