近年來將深度學習(Deep Learning)的技術應用於電腦視覺領域受到許多注目，然而相較於物件分類的單影像分類問題，由於異常事件的未知性與現實的多變化，應用非監督式學習的深度學習技術於監控影片異常偵測至今仍是困難挑戰。本論文提出了結合三維卷積神經網路 (3D Convolutional Neural Network, C3D)和單類別卷積神經網路(One-Class Convolutional Neural Network, OC-CNN)，以實現視訊監控系統之異常偵測。我們使用公開的事件資料集訓練C3D使其學習出的區域特徵具有緊湊性，以利後續的分類器學習，同時為了避免特徵過於集中而失去辨別性，我們使用人類動作行為的公開資料集UCF輔助學習C3D 網路，以改善C3D的特徵分類能力。最後我們將C3D正常事件特徵輸入各個獨立的區域分類器，並輔以高斯雜訊建成的偽異常資料，獨立訓練分類器並進行各區域的異常偵測。透過本文所提出之神經網路模型，可從正常事件訓練集學習具空間性與時間性的特徵，使這些隱藏特徵成功學習區域特性並應用在監控影片之異常區域偵測上，最後透過OC-CNN分類器偵測出未曾見過的異常事件。本論文所提出的方法，在公開廣泛使用的資料集上，與過去常使用的深度學習相關技術相比也有著優秀的表現。

In recent years, the application of Deep Learning technology in computer vision field has attracted a lot of attention. However, in comparison with the single image objects classification, the deep learning technology of unsupervised learning is still a challenge for surveillance videos due to the realistic changes and unforeseen anomalies. In this thesis we proposed a combination of the 3D Convolution neural network (C3D) and the One-Class Convolutional Neural Network (OC-CNN) to perform anomaly detections of surveillance videos. We use two different datasets to train the system that it is capable to be compacted on “intra-class” but separated on “inter-class.” The former is a public (normal) event training dataset and the latter is the public dataset of human behavior dataset called UCF. The C3D network is adopted as the baseline architecture for training to extract features so that it learns the regional features with compactness as well as with high descriptive capability. In classifying events into normal vs. abnormal, a classifier is trained on each region independently. Normal features extracted from the before mentioned C3D network and pseudo abnormal data of Gaussian noise are used as negative and positive training samples to train such classifier. Finally, we input the C3D features of normal event into each independent regional classifier, and supplemented with the pseudo anomaly data built by Gaussian noise to independently train the classifier and perform anomaly detection in each area. Through the network we proposed, the spatial-temporal hidden features can be learned only from a normal event training set. Furthermore, these hidden features successfully learned the regional characteristics, and then be applied to the regional surveillance video anomaly detection by OC-CNN classifiers. The experiments show the method proposed in this thesis had great performance on two widely used public datasets in comparison with the deep learning related techniques that had been commonly used in anomaly detection of videos.

Table of Contents

Table of Contents	IV
List of Tables and Figures	V
Chapter 1 Introduction	1
Chapter 2 Related Work	4
Chapter 3 Introduction to Related Theoretical Framework	9
3.1 3D Convolutional Neural Network	9
3.2 Deep Features for One-Class Neural Network	12
3.3 One-Class Convolutional Neural Network	16
Chapter 4 Proposed Abnormal Event Detection	18
4.1 C3D Feature Learning	19
4.2 One-Class Classifier Learning	23
4.3 Anomaly Detection	26
Chapter 5 Experimental Results and Discussions	29
5.1 Implementation	32
5.2 UCSD Results	33
5.3 CUHK Avenue Results	35
5.4 Experiments and discussion	38
Chapter 6 Conclusions, Limitations and Future Research	40
References	41
 
List of Tables and Figures
Table 1. The C3D network structure implemented in this thesis	21
Table 2. The colors for confusion matrix	28
Table 3. Experimental results which are compared with the experiments cited from [21].	39

Figure 1. Examples of anomaly detection by the proposed intelligent monitoring system	2
Figure 2. Comparison of 2D and 3D convolution, cited from [34]	10
Figure 3. 2D and 3D convolution diagram	11
Figure 4. C3D network structure, cited from [34]	11
Figure 5. The widely applied deep learning model for classification, cited from [28]	13
Figure 6. Example of normal and abnormal chair image and the extracted features under the different situations, cited from [28]	13
Figure 7. Learning one-class classifier proposed by Perera et al., cited from [28]	14
Figure 8. One-Class Convolutional Neural Network, cited from [36]	17
Figure 9. The flowchart of the proposed abnormal event detection based on C3D and OC-CNN	19
Figure 10. Flowchart of training for the C3D network in this study	21
Figure 11. The training process and structure of each OC-CNN	24
Figure 12. The flowchart of anomaly detection	26
Figure 13. Example of anomaly detection on regions	28
Figure 14. UCSD Dataset	30
Figure 15. CUHK Avenue Dataset	30
Figure 16. The example of image and anomaly label of ground truth	31
Figure 17. Examples of anomaly detection on UCSD Ped1: (a)-(d) bicycling, (e)-(h) driving cars, and (i)-(l) other human-like objects with some failed detections	34
Figure 18. Examples of anomaly detection on UCSD Ped2	34
Figure 19. Examples of anomaly detection on Avenue dataset	36
Figure 20. Examples of anomaly detection on Avenue for an event of throwing a bag	37
Figure 21. Examples of anomaly detection on Avenue for an event of throwing papers	37

[1]	V. Chandola, A. Banerjee and V. Kumar, “Anomaly Detection: A Survey,” ACM Comput, Surveys, 41(3), pp. 1 - 58, 2009.
[2]	O.P. Popoola and K. Wang, “Video-Based Abnormal Human Behavior Recognition—A Review,” IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews, 42(6), pp. 865 – 878, 2012.
[3]	T. Li, H. Chang, M. Wang, B. Ni, R. Hong and S. Yan, “Crowded Scene Analysis: A Survey,” IEEE Transactions on Circuits and Systems for Video Technology, 25(3), pp. 367 – 386, March 2015.
[4]	C. Lu, J. Shi and J. Jia, “Abnormal Event Detection at 150 FPS in MATLAB,” Computer Vision (ICCV), 2013 IEEE International Conference, pp. 2720 – 2727, December 2013.
[5]	Y. Cong, J. Yuan and Y. Tang, “Video Anomaly Search in Crowded Scenes via Spatio-Temporal Motion Context,” Information Forensics and Security, IEEE Transactions on, 8(10), pp. 1590 – 1599. 2013.
[6]	S. S. Pathan, A. Al-Hamadi and B. Michaelis, “Incorporating Social Entropy for Crowd Behavior Detection Using SVM,” Advances in Visual Computing, pp. 153 – 162, Springer Berlin Heidelberg, 2010.
[7]	M. J. Roshtkhari and M. D. Levine, “Multiple Object Tracking Using Local Motion Patterns,” Journal of Computer Vision, 107(2), pp. 203 – 217, 2014.
[8]	H. Wang, A. Kläser, C. Schmid and C. L. Liu, “Action Recognition by Dense Trajectories,” Computer Vision and Pattern Recognition (CVPR), IEEE Conference, pp. 3169 – 3176, June 2011.
[9]	X. Cui, Q. Liu, M. Gao and D. N. Metaxas, “Abnormal Detection Using Interaction Energy Potentials,” Computer Vision and Pattern Recognition (CVPR), IEEE Conference, pp. 3161 – 3167, June 2011.
[10]	Y. Zhang, L. Qin, H. Yao and Q. Huang, “Abnormal Crowd Behavior Detection Based on Social Attribute-Aware Force Model,” Image Processing (ICIP), 2012 19th IEEE International Conference, pp. 2689 – 2692, September 2012.
[11]	Tutorial on Deep Learning for Vision (CVPR2014) https://sites.google.com/site/deeplearningcvpr2014/
[12]	M. D. Zeiler, G. W. Taylor and R. Fergus, “Adaptive Deconvolutional Networks for Mid and High Level Feature Learning,” ICCV, 2011.
[13]	Y-L. Boureau, F. Bach, Y. LeCun and J. Ponce, “Learning Mid-Level Features for Recognition,” IEEE Conference on Computer Vision and Pattern Recognition, pp. 2559 – 2566, 2010. 
[14]	T. B. Moeslund, A. Hilton and V. Kru¨ger, “A Survey of Advances in Vision-Based Human Motion Capture and Analysis,” Computer Vision and Image Understanding, vol. 104, no. 2, pp. 90–126, 2006.
[15]	P. Turaga, R. Chellappa, V. S. Subrahmanian and O. Udrea, “Machine Recognition of Human Activities: A Survey,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 18, no. 11, pp. 1473–1488, 2008.
[16]	P. Schlachter, Y. Liao and B. Yang, “Deep One-Class Classification Using Data Splitting,” arXiv preprint arXiv:1902.01194, 2019.
[17]	R. Chalapathy, A. K. Menon and S. Chawla, “Anomaly Detection Using One-Class Neural Networks.” arXiv:1802.06360 [cs.LG], Feb. 2018.
[18]	A. Krizhevsky, I. Sutskever and G. E. Hinton, “Imagenet Classification with Deep Convolutional Neural Networks,” Advances in neural information processing systems, 2012.
[19]	J. Y.-H. Ng, Joe, M. Hausknecht, S. Vijayanarasimhan and O. Vinyals, “Beyond Short Snippets: Deep Networks for Video Classification,” Proceedings of the IEEE conference on computer vision and pattern recognition, 2015.
[20]	Z. Luo, B. Peng, D.-A. Huang, A. Alahi and Fei-Fei Li, “Unsupervised Learning of Long-Term Motion Dynamics for Videos,” arXiv preprint arXiv:1701.01821 2, 2017.
[21]	B. R. Kiran, D. M. Thomas and R. Parakkal, “An Overview of Deep Learning Based Methods for Unsupervised and Semi-Supervised Anomaly Detection in Videos,” Journal of Imaging 4.2, 2018.
[22]	C. Zhou, and R. C. Paffenroth, “Anomaly Detection with Robust Deep Autoencoders,” Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 2017.
[23]	J. R. Medel and A. Savakis, “Anomaly Detection Using Predictive Convolutional Long Short-Term Memory Units,” arXiv preprint arXiv:1612.00390, 2016.
[24]	Y. S. Chong and Y. H. Tay, “Abnormal Event Detection in Videos Using Spatiotemporal Autoencoder,” International Symposium on Neural Networks. Springer, Cham, 2017.
[25]	W. Sultani, C. Chen, and S. Mubarak, “Real-World Anomaly Detection in Surveillance Videos,” Center for Research in Computer Vision (CRCV), University of Central Florida (UCF), 2018.
[26]	M. Sabokrou, M. Frayyaz, M. Faithy, Z. Moayedd and R. Klette, “Deep-Anomaly: Fully Convolutional Neural Network for Fast Anomaly Detection in Crowded Scenes,” Computer Vision and Image Understanding, 2018.
[27]	S. M. Erfani, S. Rajasegarar, S. Karunasekera and C. Leckie, “High-Dimensional and Large-Scale Anomaly Detection Using a Linear One-Class SVM with Deep Learning,” Pattern Recognition 58: 121-134, 2016.
[28]	P. Perera, and V. M. Patel, “Learning Deep Features for One-Class Classification,” arXiv preprint arXiv:1801.05365 , 2018.
[29]	C. Liu, “Beyond Pixels: Exploring New Representations and Applications for Motion Analysis,” Doctoral Thesis. Massachusetts Institute of Technology, May 2009.
[30]	C. Cortes and V. Vapnik, “Support-Vector Networks,” Machine learning 20.3: 273-297, 1995.
[31]	B. Schölkopf, J. C. Platt, J. Shawe-Taylor, A. J. Smola and R. C. Williamson,  “Estimating the Support of a High-Dimensional Distribution,” Neural Computation, 13(7), pp. 1443 – 1471, 2001
[32]	D. M. J. Tax and R. P. W. Duin, “Support Vector Data Description,” Machine Learning, 54(1), pp. 45 – 66, 2004.
[33]	USCD database http://www.svcl.ucsd.edu/projects/anomaly/dataset.htm
[34]	D. Tran, L. Bourdev, R. Fergus, L. Torresani and M. Paluri, “Learning Spatiotemporal Features with 3d Convolutional Networks,” in Proceedings of the IEEE international conference on computer vision, pp. 4489–4497, 2015.
[35]	X. Shi, Z. Chen, H. Wang, D.-Y. Yeung, W. K. Wong and W.-C. Woo, “Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting,” In Advances in neural information processing systems, pp. 802-810, 2015.
[36]	P. Oza, and V. M. Patel, “One-Class Convolutional Neural Network,” IEEE Signal Processing Letters, 26(2), 277-281, 2018.
[37]	P. Schlachter, Y. Liao and B. Yang, “Deep One-Class Classification Using Data Splitting,” arXiv preprint arXiv:1902.01194, 2019.
[38]	K, Simonyan, and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” In ICLR, 2015.