本文提出一網路架構，透過深度學習方法，實現了僅使用視覺影像資訊進行一全向移動機械臂平台的移動與夾取控制。實驗一開始，透過立體視覺攝影機拍攝前方區域，判斷目標物與障礙物位置後，再進行全向移動平台的導引控制。當平台移動到目標物前方後，接著進行目標物的夾取控制。
所提出的系統分成兩個子系統，第一個子系統是全向移動平台控制系統，其以立體視覺攝影機所擷取到的工作環境之立體影像為輸入訊號，決定出移動平台最佳的移動行為。第二個子系統是機械手臂夾取控制系統，其透過立體攝影機所擷取的前方物體之立體影像為輸入訊號，決定出機械手臂最佳的夾取控制命令。在訓練資料庫的收集上，我們透過人工操控方式，操控移動機械臂進行避障移動及物體夾取控制，並將立體影像資訊及控制命令記錄下來。在類神經網路訓練階段，透過監督式模仿學習方式進行學習，讓機器人學會如何使用影像資訊直接操控全向移動平台的移動控制及機械手臂的夾取控制。實驗結果顯示，所提出的視覺導引控制系統在12.5(cm/s)下可達到平均78%的夾取成功率。

This thesis presents a novel neural network design for the application of visual guidance and picking control of an omnidirectional mobile manipulator platform through deep learning. In the experimental setting, a stereo camera was used to capture the front area of the mobile platform. Next, the proposed neural network used the stereo image as input to determine the best motion behavior of the platform. This procedure was performed recursively until the mobile platform arrives in front of the target. Finally, the proposed neural network calculated a six-degree-of-freedom (6-DoF) picking control command to the 6-DoF manipulator to pick up the target.
We divide the proposed system into two sub-systems, one is the omnidirectional mobile platform control system, and the other one is the manipulator control system. We trained two convolutional neural networks (CNNs) separately and used them to control the omnidirectional mobile platform and the 6-DoF manipulator, respectively. A training database that contains stereo images of the scene and the corresponding control commands was manually recorded. Through a supervised imitation learning, the proposed two control systems learn the best motion and picking control strategies to visually guide the platform and to visually pick up the target, respectively. Experimental results the proposed visually guidance control system achieves a picking success rate about 78% in average.

目錄
中文摘要	I
英文摘要	II
目錄	III
圖目錄	V
表目錄	VII
第一章 序論	1
1.1  研究背景	1
1.2  研究動機與目的	3
1.3  論文架構	5
第二章 相關研究與論文流程架構	7
2.1 端到端學習	7
2.2 強化學習	10
2.3 遷移學習	10
2.4 模仿學習	11
2.5 文獻總結	14
2.6 論文方法流程架構	15
第三章 立體視覺影像處理	17
3.1 各方法說明	17
3.2 特徵點獲取	18
3.3 立體視覺特徵點	21
第四章 基於視覺資料驅動控制	23
4.1 全向移動平台控制	26
4.2機械手臂控制	31
第五章 實驗結果與分析	37
5.1軟硬體介紹	37
5.2訓練資料	38
5.3測試結果	39
5.3實驗分析	49
第六章 結論與未來展望	52
參考文獻	53

圖目錄
圖 1.1 現有方法所需相關技術	4
圖 2.1 PilotNet系統架構圖	8
圖 2.2 手眼協調方法系統架構圖	9
圖 2.3 傳統學習與遷移學習架構比較圖	11
圖 2.4 基於模仿學習之行為複製方法	13
圖 2.5 論文流程架構圖	16
圖 3.1 VGG19架構	20
圖 3.2 本論文立體視覺取特徵	21
圖 3.3 合併後取特徵	21
圖 4.1 典型CNN架構	24
圖 4.2 論文CNN架構	24
圖 4.3 全向移動平台控制架構	27
圖 4.4 全向移動平台新策略訓練	29
圖 4.5 馬達位置呈現	31
圖 4.6 機械手臂控制架構	32
圖 4.7 Sigmoid函數圖	34
圖 5.1 ZED Stereo Camera	37
圖 5.2 實驗平台	38

表目錄
表 2.1 遷移學習細節表	11
表 3.2 模型資訊表	18
表 3.3 論文手臂資料庫驗證結果	18
表 3.4 VGG網路架構	19
表 3.5 VGG16與VGG19比較	20
表 3.6 先合併與後合併比較表	22
表 4.1 典型CNN與論文CNN比較	25
表 4.2 全向移動平台動作策略表	27
表 4.3 各馬達呈現動作	31
表 5.1 電腦規格表	37
表 5.2 ZED Stereo Camera技術規格表	38
表 5.3 資料庫數量呈現	38
表5.4 各實驗條件	40
表 5.5 實驗一結果呈現	41
表 5.6 實驗二結果呈現	42
表 5.7 實驗三結果呈現	43
表 5.8 實驗四結果呈現	44
表 5.9 實驗五結果呈現	45
表 5.10 實驗六結果呈現	46
表 5.11 實驗七結果呈現	48
表 5.12 速度8.5(cm/s)實驗測試結果	50
表 5.13 速度10.5(cm/s)實驗測試結果	50
表 5.14 速度12.5(cm/s)實驗測試結果	51
表 5.15 速度15(cm/s)實驗測試結果	51

[1]	K. Benali, J.-F. Brethe, F. Guerin and M. Gorka, “Dual arm robot manipulator for grasping boxes of different dimensions in a logistics warehouse, ”International Conference on Industrial Technology (ICIT), pp. 147-152, 2018
[2]	G. Devineau, W. Xi, F. Moutarde and J. Yang, “Deep Learning for Hand Gesture Recognition on Skeletal Data,” International Conference on Automatic Face & Gesture Recognition (FG 2018),pp. 106-113, 2018
[3]	Amazonrobotics, https://www.amazonrobotics.com
[4]	Simbe Robotics, http://www.simberobotics.com/
[5]	A. Krizhevsky, I. Sutskever, and G. Hinton, “ ImageNet classification with deep convolutional neural networks,” Neural Information Processing Systems, pp. 1106-1114, 2012.
[6]	J. Weisz and P. K. Allen, “Pose error robust grasping from contact wrench space metrics,” IEEE International Conference on Robotics and Automation, pp. 557-562, 2012
[7]	A. Rodriguez, M. T. Mason and S. Ferry, “From caging to grasping,” The International Journal of Robotics Research, vol. 31, no. 7, pp. 886-900, 2012.
[8]	A. Herzog, P. Pastor, M. Kalakrishnan, L. Righetti, J. Bohg, T. Asfour and S. Schaal, “Learning of grasp selection based on shape-templates,” Autonomous Robots, vol. 36, no. 1, pp. 51-65, 2014.
[9]	I. Lenz, H. Lee, and A. Saxena, “Deep learning for detecting robotic grasps,” The International Journal of Robotics Research, vol. 34, no. 4-5, pp. 705-724, 2015.
[10]	S. Levine, P. Pastor, A. Krizhevsky and D. Quillen, “Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection,” arXiv:1603.02199, 2016
[11]	Y. Wang , G. Zhang, H. Lang and B. Zuo, “A modified image-based visual servo controller with hybrid camera configuration for robust robotic grasping,” Robotics and Autonomous Systems, vol. 62, no. 10, 2014.
[12]	V. H. Andaluz , F. Roberti, L. Salinas, J. Toibero and R. Carelli ,”Passivity-based visual feedback control with dynamic compensation of mobile manipulators: Stability and L2-gain performance analysis,” Robotics and Autonomous Systems, vol. 66, 2015.
[13]	V. H. Andaluz , F. Roberti, L. Salinas, J. Toibero and R. Carelli , “Visual control with adaptive dynamical compensation for 3D target tracking by mobile manipulators,” Mechatronics, vol.22 ,no.4, 2012.
[14]	A. Krizhevsky, I. Sutskever and G. E. Hinton,” ImageNet Classification with Deep Convolutional Neural Networks,” Neural Information Processing Systems, pp.1097-1105,2012
[15]	Y. Zhu, R. Mottaghi and E. Kolve,” Target-driven visual navigation in indoor scenes using deep reinforcement learning,” International Conference on Robotics and Automation (ICRA),pp.3357-3364,2017
[16]	A. Attia and S. Dayan,” Global overview of Imitation Learning,” arXiv:1801.06503, 2018
[17]	M. Bojarski, P. Yeres, A. Choromanska, K. Choromanski, B. Firner, L. Jackel and U. Muller” Explaining How a Deep Neural Network Trained with End-to-End Learning Steers a Car,” arXiv:1704.07911, 2017
[18]	S. Levine, P. Pastor, A. Krizhevsky, J. Ibarz and D. Quillen, “Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection,” The International Journal of Robotics Research, vol. 37, no. 4-5, pp. 421-436, 2018.
[19]	S.-J Pan and Q. Yang,”A Survey on Transfer Learning,” IEEE Transactions on Knowledge and Data Engineering ,pp.1345-1359 , 2009
[20]	K. Simonyan and A. Zisserman, ”Very Deep Convolutional Networks for Large-Scale Image Recognition,”arXiv:1409.1556, 2014
[21]	K. He, X. Zhang, S. Ren and J. Sun” Deep Residual Learning for Image Recognition,” arXiv:1512.03385, 2015
[22]	C. M. Bishop, “Pattern Recognition and Machine Learning,” Springer, 2006.
[23]	I. Tsochantaridis, T. Joachims, T. Hofmann, and Y. Altun, “Large margin methods for structured and interdependent output variables,” Journal of Machine Learning Research, vol. 6, pp. 1453–1484, 2005.
[24]	S. Banerjee, S.-S Chaudhuri and S. Roy,” Fuzzy logic based vision enhancement using sigmoid function,”  IEEE Calcutta Conference , pp.41-45, 2018
[25]	S. Ioffe and C. Szegedy,” Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift,” arXiv:1502.03167, 2015