本文使用深度確定決策梯度模型（DDPG）來控制具有假液位的鍋爐程序。根據以往的經驗，僅使用線上互動訓練，該模型就無法控制具有時間延遲和噪音的鍋爐，且極端的控制輸出不能滿足工廠既平穩又穩健的需求。因此，本文假設擁有工廠的歷史控制數據，並使用這些數據對模型進行預訓練，然後再使用線上互動訓練的方式對其進行增強，該方法稱為PT-DDPG。本文使用了多種獎勵函數，並比較了使用不同獎勵函數的DDPG和PT-DDPG控制性能，從分析結果可以看出，使用單一獎勵PT-DDPG的控制性能是最佳方法。根據這種方法，本文提出以多個時間步長的資訊作為狀態，來解決DDPG無法控制過爐具有時間延遲和噪音的問題，從IAE分析結果可以得知，使用三個時間步長的資訊作為狀態有最佳的表現，因此將該方法稱為Three Step PT-DDPG。從過衝率的比較結果可以看出，Three Step PT-DDPG消除了設定點改變時的過衝，而且受擾動的影響不明顯。最後，針對控制輸出和響應曲線的分析表明，Three Step PT-DDPG模型的控制輸出與PI控制類似，由此可知Three Step PT-DDPG的控制輸出是基於歷史PI控制數據的，但是響應時間比PI控制短且動作更準確，因此該方法改善了DDPG模型無法應用於工廠的問題也提升的DDPG模型的控制性能。

This paper using the Deep Deterministic policy gradient model (DDPG) control the boiler process with fake liquid level. Based on past experience, only using online interactive training cause the model cannot control the boiler with time delay and noise, and the extreme control output cannot satisfy the demand of the plant, which is smooth and stable. So this paper assume that the historical control data is available, and use the data to pre-train the model and then enhanced by online interactive training, this method is called PT-DDPG. This paper used many kinds of reward function and compares the control performance of DDPG and PT-DDPG, it can be seen from the analysis result that the control performance of using single reward PT-DDPG is the best method. According to this method, this paper proposed using multiple time steps information as the state to overcome the problem which is DDPG cannot control the boiler with time delay and noise, it can be seen from the IAE analysis result, using the information from three time steps as the state is quite robust, this method is called 3 steps PT-DDPG. It can be seen from the result of the comparison of the overshoot percentage, 3 steps PT-DDPG eliminated the overshoot of set point change, and disturbed by the disturbance is not significant. Finally, the analysis of the control output and response curve shows that the control output of the 3 steps PT-DDPG model is similar to the PI control, but the response time is shorter than PI control and the action is more accurate. It can be seen that the 3 steps PT-DDPG is based on historical control data to control the boiler, therefore, this method improved the problem that DDPG cannot be applied to the plant.

目錄
中文摘要	I
英文摘要	II
目錄	IV
圖目錄	VI
表目錄	VIII
第一章	緒論	1
1-1	研究背景	1
1-2	文獻回顧	4
1-2.1	傳統控制方法	4
1-2.2	深度學習應用於程序控制的方法	6
1-3	研究目的	10
第二章	介紹	11
2-1	鍋爐模型介紹	11
2-2	DDPG模型介紹	12
2-3	模型訓練方法介紹	14
2-3.1	線上直接互動訓練	14
2-3.2	線上優先級經驗回放訓練	16
2-3.3	歷史數據預先訓練	18
2-4	模型的參數設定與獎勵函數介紹	21
2-4.1	模型參數設定介紹	21
2-4.2	獎勵函數介紹	21
2-5	訓練與測試環境介紹	23
2-5.1	訓練環境介紹	23
2-5.2	測試環境介紹	24
2-6	多時間步的資訊作為狀態的選用介紹	24
第三章	研究方法	26
第四章	結果與討論	28
4-1	參數影響分析	29
4-2	訓練方法比較	35
4-3	多時間步的資訊作為狀態的選用分析	38
4-4	響應曲線的比較與分析	42
4-5	控制輸出分析	48
第五章	結論	51
第六章	參考資料	52
 
圖目錄
圖 1 鍋爐液位的響應曲線	11
圖 2 DDPG模型結構與更新流程圖	13
圖 3 求和樹	17
圖 4 鍋爐液位的三衝量控制架構	18
圖 5 Actor學習率的參數影響分析	30
圖 6 Critic學習率的參數影響分析	31
圖 7 gamma值的參數影響分析	32
圖 8 設定點改變的過衝量分析	33
圖 9 各種訓練方法的比較	36
圖 10 訓練時間成本之分析	37
圖 11 狀態選用比較圖	39
圖 12 使用不同時間步的資訊作為狀態時的設定點改變過衝率分析	41
圖 13 使用不同時間步的資訊作為狀態的擾動響應過衝率分析	41
圖 14 在設定點為0 mm時受到不同蒸氣擾動的響應曲線(a)受到蒸氣擾動為12 t/h時，(b)受到蒸氣擾動為-12 t/h時，(c)受到蒸氣擾動為20 t/h時，(d)受到蒸氣擾動為-20 t/h時的響應曲線。	42
圖 15在設定點為±25 mm時受到不同蒸氣擾動的響應曲線(a)SP=25mm, D=12 t/h時，(b)SP=25mm, D=-12 t/h時，(c)SP=25mm, D=20 t/h時，(d)SP=25mm, D=-20 t/h時，(e)SP=-25mm, D=12 t/h時，(f)SP=-25mm, D=-12 t/h時，(g)SP=-25mm, D=20 t/h時，(h) SP=-25mm, D=-20 t/h時的響應曲線。	44
圖 16在設定點為±50 mm時受到不同蒸氣擾動的響應曲線(a)SP=50 mm, D=12 t/h時，(b)SP=50 mm, D=-12 t/h時，(c)SP=50 mm, D=20 t/h時，(d)SP=50 mm, D=-20 t/h時，(e)SP=-50 mm, D=12 t/h時，(f)SP=-50 mm, D=-12 t/h時，(g)SP=-50 mm, D=20 t/h時，(h)SP=-50 mm, D=-20 t/h時的響應曲線。	45
圖 17 在SP=0 mm時受到各種蒸汽擾動時的控制輸出分析	48
圖 18 在SP=±25 mm時受到各種蒸汽擾動時的控制輸出分析	49
圖 19 在SP=±50 mm時受到各種蒸汽擾動時的控制輸出分析	49

 
表目錄
表 1 各獎勵函數的最佳參數設定	33
表 2 三種控制架構針對具有延遲與噪音的鍋爐IAE比較	47

[1] C.E. Garcia, D.M. Prett, M. Morari, Model predictive control: theory and practice—a survey, Automatica 25 (1989) 335-348.
[2] M.L. Wu, L.L. Wei, J.K. Huang, M.Y. Wu, The Cascade Three-Elements Fuzzy Auto-Adapted PID Control System for Boiler,  Advanced Materials Research, Trans Tech Publ, 2010, pp. 1919-1923.
[3] U.-C. Moon, K.Y. Lee, Step-response model development for dynamic matrix control of a drum-type boiler–turbine system, IEEE Transactions on Energy Conversion 24 (2009) 423-430.
[4] T.P. Lillicrap, J.J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, D. Wierstra, Continuous control with deep reinforcement learning, arXiv preprint arXiv:1509.02971 (2015).
[5] S. Spielberg, R. Gopaluni, P. Loewen, Deep reinforcement learning approaches for process control,  2017 6th international symposium on advanced control of industrial processes (AdCONIP), IEEE, 2017, pp. 201-206.
[6] S. Steven Pon Kumar, B. Gopaluni, P. Loewen, Process Control using Deep Reinforcement Learning⋆.
[7] M. Xu, S. Li, W. Cai, Cascade generalized predictive control strategy for boiler drum level, ISA transactions 44 (2005) 399-411.
[8] N. Yu, W. Ma, M. Su, Application of adaptive Grey predictor based algorithm to boiler drum level control, Energy conversion and management 47 (2006) 2999-3007.
[9] L. Chen, C. Wang, Y. Yu, Y. Zhao, The research on boiler drum water level control system based on self-adaptive fuzzy-PID,  2010 Chinese Control and Decision Conference, IEEE, 2010, pp. 1582-1584.
[10] P. Bhowmik, P. Dutta, S. Dhar, M. Dey, Design and Analysis of a Three Element Boiler Drum Level Control System.
[11] H.R. Berenji, A reinforcement learning—based architecture for fuzzy logic control, International Journal of Approximate Reasoning 6 (1992) 267-292.
[12] J. Hoskins, D. Himmelblau, Process control via artificial neural networks and reinforcement learning, Computers & chemical engineering 16 (1992) 241-251.
[13] J. Si, Y.-T. Wang, Online learning control by association and reinforcement, IEEE Transactions on Neural networks 12 (2001) 264-276.
[14] M. Sedighizadeh, A. Rezazadeh, Adaptive PID controller based on reinforcement learning for wind turbine control,  Proceedings of world academy of science, engineering and technology, Citeseer, 2008, pp. 257-262.
[15] P. Kittisupakorn, P. Thitiyasook, M. Hussain, W. Daosud, Neural network based model predictive control for a steel pickling process, Journal of Process Control 19 (2009) 579-590.
[16] M.A. Berger, J.V. da Fonseca Neto, Neurodynamic Programming Approach for the PID Controller Adaptation, IFAC Proceedings Volumes 46 (2013) 534-539.
[17] T. Schaul, J. Quan, I. Antonoglou, D. Silver, Prioritized experience replay, arXiv preprint arXiv:1511.05952 (2015).
[18] Y. Hou, L. Liu, Q. Wei, X. Xu, C. Chen, A novel ddpg method with prioritized experience replay,  2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC), IEEE, 2017, pp. 316-321.
[19] E. Reichensdörfer, J. Günther, K. Diepold, Recurrent neural networks for pid auto-tuning, (2017).
[20] S.S.P. Kumar, A. Tulsyan, B. Gopaluni, P. Loewen, A deep learning architecture for predictive control, IFAC-PapersOnLine 51 (2018) 512-517.
[21] Q. Shi, H.-K. Lam, B. Xiao, S.-H. Tsai, Adaptive PID controller based onQ-learning algorithm, CAAI Transactions on Intelligence Technology 3 (2018) 235-244.
[22] O. Vinyals, I. Babuschkin, W.M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D.H. Choi, R. Powell, T. Ewalds, P. Georgiev, Grandmaster level in StarCraft II using multi-agent reinforcement learning, Nature 575 (2019) 350-354.