遞迴神經網路已被廣泛應用於語音辨識與聲紋辨識，且由於其數據結構具有時序性與化工廠製程數據非常相似，故本研究比較人工類神經網路、遞迴類神經網路與長短期記憶網路在化工製程故障診斷上的差異。
本研究以田納西伊士曼製程作為研究個案，並使用不同層數與不同神經元個數的人工類神經網路、遞迴類神經網路與長短期記憶網路進行故障診斷，並接著分析三種類神經網路之間的差異。
結果顯示長短期記憶遞迴神經網路擁有最優異的故障分類準確率。基本人工類神經網路與傳統機器學習的動態主成分分析法一樣都是利用故障特徵平均來進行分類，使其無法將相似的故障區分出來，而遞迴類神經網路則是使用初始隱藏狀態的更新來克服這個問題。至於長短期記憶網路則是因為擁有兩個關鍵的邏輯閘，以利於模型過濾掉不重要的資訊進而提升故障分類準確率。
此外我們發現長短期記憶模型中的遺忘閘的做動範圍非常小，故我們使用另一種遞迴類神經網路的變形—閘控型遞迴單元，並進行故障分類。結果顯示閘控型遞迴單元可以使用更少的訓練參數量與訓練時間就達到95.9%的故障分類準確率。

Recurrent neural network has been widely used in speech recognition and voiceprint recognition because of the similarity of data structure with chemical engineering process data. This study intends to compare the differences between ANN, RNN and LSTM applied on the fault diagnosis classification in the chemical process.
This study uses the Tennessee Eastman process as a case study and uses different structure of ANNs, RNNs and LSTMs to classify the fault, and then analyzes the difference between them.
The results show that the LSTM have the best fault classification accuracy. ANN and DPCA of traditional machine learning use the average feature of faults to classify them so that they cannot distinguish similar faults. While RNN uses the update of the initial hidden state to overcome this problem. As for the LSTM, there are two key logic gates, so that the model can filter out the unimportant information and improve the fault classification accuracy.
In addition, we find that forget gates in the LSTM is normally open, so we use another RNN’s deformation, GRU. The results show that the GRU can achieve 95.9% fault classification accuracy with less training parameter amount and training time.

目錄	III
圖目錄	V
表目錄	XI
第一章 介紹	1
1.1研究背景	1
1.2文獻回顧	3
1.3研究動機與目的	5
第二章 理論與模型介紹	6
2.1田納西伊士曼製程與數據描述	6
2.2類神經網路模型介紹	12
2.2.1人工類神經網路(ANN)	12
2.2.2遞迴類神經網路(RNN)	14
2.2.3長短期記憶網路(LSTM)	16
2.3類神經網路訓練和測試與數據降維可視化方式	23
2.3.1 類神經網路訓練與測試方式	23
2.3.2 故障分類準確率定義	26
2.3.3 故障特徵可視化方式	28
第三章 結果討論	29
3.1故障分類結果	29
3.1.1	5筆數據訓練ANN之分類結果	29
3.1.2	40筆數據訓練ANN之分類結果	30
3.1.3 5筆數據訓練RNN之分類結果	31
3.1.4 40筆數據訓練RNN之分類結果	32
3.1.5 5筆數據訓練LSTM之分類結果	33
3.1.6 40筆數據訓練LSTM之分類結果	34
3.2故障分類結果分析	35
3.2.1 分類結果綜合比較	35
3.2.2 分類結果之混淆矩陣	37
3.2.3 ANN與RNN之可視化比較	40
3.2.4 RNN與LSTM之可視化比較	50
3.3	LSTM邏輯閘運作方式分析	57
3.3.1 LSTM輸入閘之分析	58
3.3.2 LSTM遺忘閘之分析	66
3.3.3 LSTM輸出閘之分析	74
3.3.4 LSTM與RNN隱藏狀態傳遞方式比較	83
3.3.5 閘控型遞迴單元(Gated Recurrent Unit)	85
第四章 結論	89
第五章 參考文獻	91
	
 
圖目錄
圖2-1 田納西伊士曼製程流程圖...........................................................7
圖2-2 生物神經網路(左)與人工神經網路之神經元(右)...................12
圖2-3 類神經網路.................................................................................13
圖 2-4 單層RNN輸出所有時間點下的數值(上)，單層RNN輸出最後一個時間點的數值(下)......................................................................14
圖2-5 雙曲正切函數.............................................................................15
圖2-6 長短期記憶網路結構圖.............................................................17
圖2-7 雙曲函數.....................................................................................18
圖2-8 LSTM遺忘閘.............................................................................19
圖2-9 LSTM輸入閘.............................................................................20
圖2-10 LSTM更新單元狀態…............................................................21
圖2-11 LSTM輸出閘...........................................................................22
圖2-12 故障分類架構圖.......................................................................23
圖2-13 製程數據強度圖；左上(故障0)；中上(故障3)；右上(故障9)；左下(故障15)；右下(故障16).....................................................24
圖2-14 類神經網路訓練與測試流程...................................................25
 
圖3-1 使用5筆數據訓練ANN之分類結果......................................29
圖3-2 使用40筆數據訓練ANN之分類結果....................................30
圖3-3 使用5筆數據訓練RNN之分類結果......................................31
圖3-4 使用40筆數據訓練RNN之分類結果....................................32
圖3-5 使用5筆數據訓練LSTM之分類結果....................................33
圖3-6 使用40筆數據訓練LSTM之分類結果..................................34
圖3-7 ANN、RNN與LSTM分類結果比較......................................35
圖3-8 40筆數據訓練ANN之分類混淆矩陣.....................................37
圖3-9 40筆數據訓練RNN之分類混淆矩陣.....................................38
圖3-10 40筆數據訓練LSTM之分類混淆矩陣.................................39
圖3-11 ANN隱藏層輸出投影至二維平面之故障特徵分布.............40
圖3-12 RNN隱藏層輸出投影至二維平面之故障特徵分布.............41
圖3-13 DPCA在二維平面上故障特徵的歐氏距離分布圖...............43
圖3-14 ANN在二維平面上故障特徵的歐氏距離分布圖.................43
圖3-15 RNN在二維平面上故障特徵的歐氏距離分布圖.................44
圖3-16 RNN故障0之特徵在二維平面之移動路徑.........................45
圖3-17 RNN故障1之特徵在二維平面之移動路徑.........................46
圖3-18 RNN故障3之特徵在二維平面之移動路徑….....................46
 
圖3-19 RNN故障8之特徵在二維平面之移動路徑….....................47
圖3-20 RNN故障9之特徵在二維平面之移動路徑.........................47
圖3-21 RNN故障13之特徵在二維平面之移動路徑.......................48
圖3-22 RNN故障14之特徵在二維平面之移動路徑.......................48圖3-23 RNN故障15之特徵在二維平面之移動路徑.......................49
圖3-24 RNN故障16之特徵在二維平面之移動路徑.......................49
圖3-25 LSTM故障0之特徵在二維平面之移動路徑……...............50
圖3-26 LSTM故障1之特徵在二維平面之移動路徑.......................51
圖3-27 LSTM故障3之特徵在二維平面之移動路徑.......................51圖3-28 LSTM故障8之特徵在二維平面之移動路徑.......................52
圖3-29 LSTM故障9之特徵在二維平面之移動路徑.......................52
圖3-30 LSTM故障13之特徵在二維平面之移動路徑.....................53
圖3-31 LSTM故障14之特徵在二維平面之移動路徑.....................53
圖3-32 LSTM故障15之特徵在二維平面之移動路徑.....................54
圖3-33 LSTM故障16之特徵在二維平面之移動路徑.....................54
圖3-34二維平面上故障特徵的歐氏距離分RNN(上)LSTM(下).....55
圖3-35 故障0(上)、故障1(中)與故障3(下)輸入閘數值對批次時間圖............................................................................................................59 
圖3-36 故障8(上)、故障9(中)與故障13(下)輸入閘數值對批次時間圖............................................................................................................60圖3-37 故障14(上)、故障15(中)與故障16(下)輸入閘數值對批次時間圖........................................................................................................61
圖3-38 故障0(上)、故障1(中)與故障3(下)輸入閘數值分布圖......62
圖3-39 故障8(上)、故障9(中)與故障13(下)輸入閘數值分布圖....63
圖3-40 故障14(上)、故障15(中)與故障16(下)輸入閘數值分布圖............................................................................................................64
圖3-41 各故障輸入閘數值之四分位距盒狀圖……...........................65
圖3-42 故障0(上)、故障1(中)與故障3(下)遺忘閘數值對批次時間圖............................................................................................................67
圖3-43 故障8(上)、故障9(中)與故障13(下)遺忘閘數值對批次時間圖.............................................................................................................68
圖3-44 故障14(上)、故障15(中)與故障16(下)遺忘閘數值對批次時間圖........................................................................................................69
圖3-45 故障0(上)、故障1(中)與故障3(下)遺忘閘數值分布圖............................................................................................................70
 
圖3-46 故障8(上)、故障9(中)與故障13(下)遺忘閘數值分布圖............................................................................................................71
圖3-47 故障14(上)、故障15(中)與故障16(下)遺忘閘數值分布圖............................................................................................................72
圖3-48 各故障遺忘閘數值之四分位距盒狀圖...................................73
圖3-49 故障0(上)、故障1(中)與故障3(下)輸出閘數值對批次時間圖............................................................................................................75
圖3-50 故障8(上)、故障9(中)與故障13(下)輸出閘數值對批次時間圖............................................................................................................76
圖3-51 故障14(上)、故障15(中)與故障16(下)輸出閘數值對批次時間圖........................................................................................................77
圖3-52 故障0(上)、故障1(中)與故障3(下)輸出閘數值分布圖............................................................................................................79
圖3-53 故障8(上)、故障9(中)與故障13(下)輸出閘數值分布圖............................................................................................................80
圖3-54 故障14(上)、故障15(中)與故障16(下)輸出閘數值分布圖............................................................................................................81
圖3-55 各故障輸出閘數值之四分位距盒狀圖...................................82
 
圖3-56 GRU結構圖............................................................................86
圖3-57 40筆數據訓練GRU之分類混淆矩陣..................................87
 
表目錄
表2-1 田納西伊士曼製程故障類型總表 ...............................................8
表2-2 田納西伊士曼製程操作變數表....................................................9
表2-3 田納西伊士曼製程測量變數表(1).............................................10
表2-4 田納西伊士曼製程測量變數表(2).............................................10
表2-5 田納西伊士曼製程測量變數表(3).............................................11
表2-6 田納西伊士曼製程測量變數表(4).............................................11
表2-7 分類樣本個數之混淆矩陣..........................................................26
表2-8 分類準確率之混淆矩陣..............................................................27
表3-1 TEP故障分類結果比較表..........................................................36
表3-2 LSTM邏輯閘數值平均變動範圍表..........................................83
表3-3 LSTM與GRU訓練參數與時間比較表.....................................88

[1] M.R. Maurya, R. Rengaswamy, V. Venkatasubramanian, Fault diagnosis using dynamic trend analysis: A review and recent developments, Engineering Applications of artificial intelligence
 20 (2007) 133-146.
[2] H. Vedam, Venkatasubramanian, PCA-SDG based process monitoring and fault diagnosis, Control engineering practice 7 (1999) 903-917.
[3] M. Misra, H.H. Yue, S.J. Qin, C. Ling, Multivariate process monitoring and fault diagnosis by multi-scale PCA, Computers chemical engineering 26 (2002) 1281-1293.
[4] S. Yoon, J.F. MacGregor, Principal‐component analysis of multiscale data for process monitoring and fault diagnosis, American Institute of Chemical Engineers journal 50 (2004) 2891-2903.
[5] N. Zhong, X. Deng, Multimode non‐Gaussian process monitoring based on local entropy independent component analysis, The Canadian Journal of Chemical Engineering
 95 (2017) 319-330.
[6] J.F. MacGregor, C. Jaeckle, C. Kiparissides, Koutoudi, Process monitoring and diagnosis by multiblock PLS methods, American Institute of Chemical Engineers journal 40 (1994) 826-838.
[7] S. Yin, X. Zhu, O. Kaynak, Improved PLS focused on key-performance-indicator-related fault diagnosis, Institute of Electrical and Electronics Engineers 62 (2014) 1651-1658.
[8] J.M. Lee, S.J. Qin, I.B.j. Lee, Fault detection and diagnosis based on modified independent component analysis, American Institute of Chemical Engineers journal 52 (2006) 3501-3514.
[9] C. Jing, J. Hou, SVM and PCA based fault classification approaches for complicated industrial process, Neurocomputing 167 (2015) 636-642.
[10] R. Novak, Y. Bahri, D.A. Abolafia, J. Pennington, J. Sohl-Dickstein, Sensitivity and generalization in neural networks: an empirical study, 2018 International Conference on Learning Representations (2018).
[11] F. Lv, C. Wen, M. Liu, Z. Bao, Weighted time series fault diagnosis based on a stacked sparse autoencoder, Journal of Chemometrics 31 (2017) e2912.
[12] Z. Zhang, J. Zhao, A deep belief network based fault diagnosis model for complex chemical processes, Computers chemical engineering 107 (2017) 395-407.
[13] P. Malhotra, L. Vig, G. Shroff, P. Agarwal, Long short term memory networks for anomaly detection in time series,  Proceedings, Presses universitaires de Louvain, 2015, pp. 89.
[14] Z.C. Lipton, D.C. Kale, C. Elkan, R. Wetzel, Learning to diagnose with LSTM recurrent neural networks, 2016 International Conference on Learning Representations
 (2015).
[15] H. Zhao, S. Sun, B. Jin, Sequential fault diagnosis based on lstm neural network, Institute of Electrical and Electronics Engineers 6 (2018) 12929-12939.
[16] G.M. Xavier, J.M. de Seixas, Fault Detection and Diagnosis in a Chemical Process using Long Short-Term Memory Recurrent Neural Network,  2018 International Joint Conference on Neural Networks 2018, pp. 1-8.
[17] P. Filonov, F. Kitashov, A. Lavrentyev, Rnn-based early cyber-attack detection for the tennessee eastman process, 2017 International Conference on Machine (2017).
[18] Z.C. Lipton, J. Berkowitz, C. Elkan, A critical review of recurrent neural networks for sequence learning, 2016 International Conference on Learning Representations (2015).
[19] I. Sutskever, O. Vinyals, Q.V. Le, Sequence to sequence learning with neural networks,  Advances in neural information processing systems, 2014, pp. 3104-3112.
[20] A. Karpathy, J. Johnson, L. Fei-Fei, Visualizing and understanding recurrent networks, 2016 International Conference on Learning Representations (2015).
[21] R. Pascanu, T. Mikolov, Y. Bengio, On the difficulty of training recurrent neural networks,  International conference on machine learning, 2013, pp. 1310-1318.
[22] S. Hochreiter, J. Schmidhuber, Long short-term memory, Neural computation 9 (1997) 1735-1780.
[23] J.J. Downs, E.F. Vogel, A plant-wide industrial process control problem, Computers chemical engineering 17 (1993) 245-255.
[24] C.A. Rieth, B.D. Amsel, R. Tran, M.B. Cook, Issues and Advances in Anomaly Detection Evaluation for Joint Human-Automated Systems,  International Conference on Applied Human Factors and Ergonomics, Springer, 2017, pp. 52-63.
[25] K. Suzuki, Artificial neural networks: methodological advances and biomedical applications, BoD–Books on Demand2011.
[26] F. Chollet, Keras, 2015.
[27] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, Tensorflow: A system for large-scale machine learning,  12th  Symposium on Operating Systems Design and Implementation, 2016, pp. 265-283.
[28] D.P. Kingma, J. Ba, Adam: A method for stochastic optimization, 2015 International Conference on Learning Representations (2014).
[29] L. Prechelt, Early stopping-but when?, Neural Networks: Tricks of the trade, Springer1998, pp. 55-69.
[30] L.v.d. Maaten, G. Hinton, Visualizing data using t-SNE, Journal of machine learning research
 9 (2008) 2579-2605.
[31] K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, Y. Bengio, Learning phrase representations using RNN encoder-decoder for statistical machine translation, Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) (2014) pages 1724–1734.