淡江大學覺生紀念圖書館 (TKU Library)
進階搜尋


系統識別號 U0002-1410201314325900
中文論文名稱 以影格中的不變特徵為基礎的全景修補演算法
英文論文名稱 Panoramic Inpainting based on Invariant Features of Video Frames
校院名稱 淡江大學
系所名稱(中) 資訊工程學系博士班
系所名稱(英) Department of Computer Science and Information Engineering
學年度 102
學期 1
出版年 103
研究生中文姓名 張仕明
研究生英文姓名 Shih-Ming Chang
學號 898410047
學位類別 博士
語文別 英文
口試日期 2013-10-14
論文頁數 71頁
口試委員 指導教授-顏淑惠
委員-施國琛
委員-洪啟舜
委員-顏淑惠
委員-許輝煌
委員-林其誼
中文關鍵字 全景影像  動態修補  自動化物件切割 
英文關鍵字 Panorama  Motion Inpainting  Automatic GrabCut 
學科別分類 學科別應用科學資訊工程
中文摘要 全景成像技術是近年來十分重要的議題,到目前為止已有許多全景成像的技術發展在各式各樣的產品上。從最早期開發在個人電腦中以及近年來開發在數位相機與可攜式裝置中都可以看到這類技術的運用。在傳統的全景成像技術中,使用者可以使用一連串的影像或是一小段影片來建立同一場景的全景圖。但在傳統的全景成像技術中,如果影像或影片內有移動的物件時,所建立出來的全景圖背景會保留其物件,同時當全景圖的素材中有較為明顯的人物時也容易造成人物結構上的模糊。因此本論文針對此問題提出一個運用人臉辨識、物件切割、動量背景修補以及有效的全景合成技術的演算法。針對背景有移動人物的場景,將其人物與背景先進行分離,在此部份本論文先使用著名的物件切割方法-GrabCut演算法來切割物件並且將其改良為自動化程序,讓使用者可以不用經過手工的標記物件即可達到物件切割的效果。接著使用動量背景修補方法將分離出來的區域填補成原先可能的背景,在此部分本論文根據傳統的影像修補方法並加入動量的參考依據讓背景也跟著在移動的情況下也可以順利地得到正確的修補結構。最後再將所有的素材合成為一張全景影像,本論文所提出的全景圖製作方法加入了影像能量圖與影像縫線的概念,讓接合的縫線落在重疊區域內較不明顯的結構中,如此一來可以有效地解決人物鬼影和因為重疊位置的關係所形成的結構模糊的問題。而人臉辨識的運用除了定義出人物的大概區域外,也可以用來判斷針測到的人物是否是使用者認識的人,如果認定是陌生人時即可利用上述的方法進行移除的動作,進一步達到過濾陌生人的功能。本論文所提出的方法除了人物區域定義是可以進行手動設定,其他的步驟皆為自動化程序,如此可有效減少使用者操作上的負擔。
英文摘要 Panoramic photography is becoming very popular within the general users, skilled photographers and in many useful computer and Internet based application domains like 3D virtual reality. With the introduction of panoramic photography support in the general purpose digital cameras and smart phones, users and applications that use the panoramic photos are also increasing. In traditional panoramic photography, moving objects or as referred in this thesis - the strangers, in the background should be eliminated since those strangers obscure the scenery that we want to retain in our photograph. This thesis discusses a novel method to remove the strangers (moving objects) whose face data is not available in the face database of camera) from the background of the focused area and compose a panoramic image. In the proposed system the object segmentation is automation and based on GrabCut algorithm. The method of motion inpainting of background can be repair background on moving background, effectively. The method of panorama creation is using concept of energy map and image seam that avoided ghost problem in panorama and maintained the structure of human. The proposed of panorama creation system is fully automatic except that the user required marking the unidentified moving objects in the object segmentation phase.
論文目次 Table of Content
Chapter 1 Introduction 1
1.1 Motivation 1
1.2 Overview of Panorama 2
1.3 Organization of this Dissertation 2
Chapter 2 Related Works 5
2.1 Feature and Object Extraction 5
2.2 Panorama Creation 7
2.3 Seam Carving 10
Chapter 3 Object Segmentation 13
3.1 Face Detection 13
3.2 Object Location Definition 15
3.3 Object Segmentation 17
3.4 The Dynamic Background Inpainting 20
3.4.1 Motion Estimation 21
3.4.2 Background Structure Inpainting 24
Chapter 4 Proposed Panorama Creation 28
4.1 Feature Matching 28
4.2 Overlapping Inpainting 32
4.3 Optimal Seam and Combination 37
4.3.1 Image calibration 37
4.3.2 Find Optimal Seam 41
Chapter 5 Experimental Results 48
Chapter 6 Conclusion and Future Work 66
Reference 68

Table of Figure
Figure 1. The flowchart of proposed approach 3
Figure 2. The flowchart of proposed method 4
Figure 3. The result of SIFT and ASIFT algorithms 6
Figure 4. The example of panoramic creation 7
Figure 5. Results obtained using the Autostitch 9
Figure 6. The result of image retargeting via seam carving 12
Figure 7. The result of face detection 14
Figure 8. The result of Object Location Definition 16
Figure 9. The example of GrabCut algorithm 17
Figure 10. The result of Object Segmentation 19
Figure 11. The sample of video inpainting [32] 20
Figure 12. Direction definition of motion estimation 21
Figure 13. The schematic diagram of step 1 22
Figure 14. The schematic diagram of step 2 22
Figure 15. The schematic diagram of step 3 23
Figure 16. The schematic diagram of step 4 24
Figure 17. The schematic diagram of background structure inpainting 25
Figure 18. The inpainting result of each frame 26
Figure 19. The final result of background structure inpainting 27
Figure 20. The human result of SIFT and ASIFT algorithm 29
Figure 21. The results of image combination 30
Figure 22. The poor result of human in panoramic creation 32
Figure 23. The disarray structure of human 33
Figure 24. The integrity structure of human 35
Figure 25. The schematic diagram of overlapping inpainting 36
Figure 26. The cylindrical projection in different frames 38
Figure 27. The result of easy combination 38
Figure 28. The schematic diagram of optimal seam 43
Figure 29. The example of shortest path 44
Figure 30. The energy map of optimal seam 46
Figure 31. Image combination (top: original images, bottom: combined photo with overlapped region is marked in red) 47
Figure 32. Main interface of the system 49
Figure 33. Steps of the tool and result 50
Figure 34. Examples of main structure maintain 55
Figure 35. Performance chart of table 1 57
Figure 36. Experimental results (left: one frame from source video, right: resultant panorama) 65
參考文獻 [1]. M. Brown, D. G. Lowe (2007) Automatic Panoramic Image Stitching using Invariant Features. International Journal of Computer Vision. 74(1):59-73.
[2]. Saeid Fazli, Hamed Moradi Pour, Hamed Bouzari, (2009) Particle Filter based Object Tracking with Sift and Color Feature. International Conference on Machine Vision. pp: 89-93.
[3]. H. Bay, T. Tuytelarrs, L. J. V. Gool, (2006) SURF: Speeded Up Robust Features. European Conference on Computer Vision. 3951:404-417.
[4]. H. Bay, A. Ess, T. Tuytelarrs, L. J. V. Gool, (2008) Speeded-Up Robust Features (SURF). Computer Vision and Image Understanding. 110:346-359.
[5]. Kekre, Hemant B., Sudeep D. Thepade. (2008) Rotation Invariant Fusion of Partial Image Parts in Vista Creation using Missing View Regeneration. WASET International Journal of Electrical Computer and Systems Engineering (IJECSE) 47: 660.
[6]. Helmut Dersch (2007) Panorama Tools. Open Source Software for Immersive Imaging International VR Photography Conference, 2007.
[7]. Wang Meng (2009) Panorama Painting: With a Bare Digital Camera. In Image and Graphics, 2009. ICIG'09. Fifth International Conference on. pp: 82-86.
[8]. Song, Baosen, Yongqing Fu, Jinlin Wang (2011) Automatic panorama creation using multi-row images. Information Technology Journal 10:1977-1982.
[9]. Yingen Xiong, Kari Pulli (2010) Fast image stitching and editing for panorama painting on mobile phones. IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). pp: 47-52.
[10]. Wen-Yan Lin, Siying Liu, Yasuyuki Matsushita, Tian-Tsong Ng, Loong-Fah Cheong. (2011) Smoothly varying affine stitching. Computer Vision and Pattern Recognition (CVPR) 2011 IEEE Conference on. pp: 345-352.
[11]. M. Brown, D. G. Lowe (2003) Recognising Panoramas. In Proceedings of the 9th International Conference on Computer Vision (ICCV2003). pp: 1218-1225.
[12]. I. Laurent, K. Christof, and N. Ernstr (1998) A Model of Saliency Based Visual Attention for Rapid Scene Analysis, IEEE transactions on pattern analysis and machine intelligence. 20(11): 1254-1259.
[13]. F. Liu and M. Gleicher (2005) Automatic image retargeting with fisheye-view warping. In Proc. ACM Symposium on User Interface Software and Technology. pp: 153-162.
[14]. Shai Avidan, Ariel Shamir (2007) Seam Carving for Content-Aware Image Resizing. ACM Transactions on Graphics (TOG) - Proceedings of ACM SIGGRAPH. 26(3):Article 10.
[15]. Y. S. Wang, C. L. Tai, O. Sorkine and T. Y. Lee (2008). Optimized scale-and-stretch for image resizing. ACM Transactions on Graphics (TOG)-Proceedings of ACM SIGGRAP Asia 2008. 27(5): Article 118.
[16]. S. Battiato (2012). Content-based image resizing on mobile devices. In International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISAPP). pp: 87-90.
[17]. Chenjun Tao, Jiaya Jia and Hanqiu Sun (2007). Active window oriented dynamic video retargeting. In Proceedings of the Workshop on Dynamical Vision. pp: 1-12.
[18]. Sunghyun Cho, Hanul Choi, Matsushita, Y. and Seungyong Lee (2009). Image retargeting using importance diffusion. Image Processing (ICIP), 2009 16th IEEE International Conference on. pp: 977-980.
[19]. Li-Qun Chen, Xing Xie, Xin Fan, Wei-Ying Ma, Hong-Jiang Zhang and He-Qin Zhou (2003). A visual attention model for adapting images on small displays. Multimedia System. 9(4):353-364.
[20]. Xin Fan, Xing Xie, He-Qin Zhou and Wei-Ying Ma (2003). Looking into video frames on small displays. In Proc. Eleventh ACM Int. Conf. Multimedia. pp: 247-250.
[21]. Jianping Xiao, Xuecheng Zou, Zhenglin Liu and Xu Guo (2007). A novel adaptive interpolation algorithm for image resizing. International Journal of Innovative Computing, Information and Control. 3(6):1335-1345.
[22]. Anthony Santella, Maneesh Agrawala, Doug DeCarlo, David Salesin and Michael Cohen (2006). Gaze-based interaction for semi-automatic photo cropping. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. pp: 771-780.
[23]. Yunfeng Zhang, Shanshan Gao, Caiming Zhang and Jing Chi (2009). Application of a bivariate rational interpolation in image zooming. , International Journal of Innovative Computing, Information and Control. 5(11):4299-4307.
[24]. Jian Yao, Jean-Marc Odobez (2008) Fast human detection from videos using covariance features. Presented at: The Eighth International Workshop on Visual Surveillance (VS2008). http://hal.inria.fr/inria-00325628/. Accessed 29 Septembre 2008.
[25]. Yu-Ting Chen, Chu-Song Chen (2008) Fast Human Detection Using a Novel Boosted Cascading Structure With Meta Stages. IEEE Transactions on Image Processing. 17(8):1452-1464.
[26]. C. Rother, V. Kolmogorov, A. Blake (2004) "GrabCut": interactive foreground extraction using iterated graph cuts. ACM Trans. Graph. 23:309-314.
[27]. S. Vicente, V. Kolmogorov, C. Rother (2008) Graph cut based image segmentation with connectivity priors. In IEEE Conference on Computer Vision and Pattern Recognition. pp: 23-28.
[28]. V. Vezhnevets, V. Konouchine (2005) Grow-Cut - Interactive Multi-Label N-D Image Segmentation. Proc. Graphicon. pp: 150-156.
[29]. Yuri Y. Boykov, Marie-Pierre Jolly (2001) Interactive Graph Cuts for Optimal Boundary & Region Segmentation of Objects in N-D Images. Eighth International Conference on Computer Vision (ICCV'01). 1:105-112.
[30]. Patwardhan KA, Sapiro G, Bertalmio M. (2007) Video Inpainting Under Constrained Camera Motion. IEEE Transactions on Image Processing. pp: 545-553.
[31]. Y. Wexler, E. Shechtman, M. Irani (2007) Space-Time Completion of Video. IEEE Trans. on Pattern Analysis and Machine Intelligence. 29:463-476.
[32]. Yu-Ting Chen, Chu-Song Chen (2008) Fast Human Detection Using a Novel Boosted Cascading Structure With Meta Stages. IEEE Transactions on Image Processing. 17(8):1452-1464.
[33]. C. Kokaram, S. J. Godsill (1997) Joint Detection, Interpolation, Motion and Parameter Estimation for Image Sequences with Missing Data. In International Conference on Image Processing. pp: 191-194.
[34]. L. -M. Po, W. C. Ma (1996) A Novel Four-Step Search Algorithm for Fast Block Motion Estimation. IEEE Trans. Circuits Syst. Video Technol. 6(3):313-317.
[35]. R. Li, B. Zeng, M. L. Liou, (1994) A New Three-Step Search Algorithm for Block Motion Estimation. IEEE Trans. Circuits Syst. Video Technol. 4(4):438-442.
[36]. C. H. Cheung, L. M. Po (2005) Novel cross-diamond-hexagonal search algorithms for fast block motion estimation. IEEE Transactions on Multimedia. 7(1):16-22.
[37]. S. Vassiliadis, E. A. Hakkennes, J. S. S. M. Wong, G. G. Pechanek (1998) The Sum-Absolute-Difference Motion Estimation Accelerator. In the 24th. EUROMICRO Conference. pp: 559-566.
[38]. Bruno Postle (2011) Panorama Tools. Open Source Software for Immersive Imaging. http://panotools.sourceforge.net/. Accessed 22 February 2011.
[39]. Wan-Lei Zhao, Chong-Wah Ngo (2009) Scale-Rotation Invariant Pattern Entropy for Keypoint-Based Near-Duplicate Detection. Image Processing of IEEE Transactions. 18(2):412-423.
[40]. Yifan Lu, Lei Wang, Hartley, R.,Hongdong Li, Chunhua Shen (2008) Multi-view Human Motion Capture with an Improved Deformation Skin Model. Computing: Techniques and Applications (DICTA) Digital Image. pp: 420-427.
[41]. J. M. Morel, G. Yu (2009) ASIFT: A New Framework for Fully Affine Invariant Image Comparison. SIAM Journal on Imaging Sciences. 2(2):438-469.
[42]. A. Criminisi, P. Perez, K. Toyama (2004) Region Filling and Object Removal by Exemplar-Based Image Inpainting. IEEE Trans. On Image Processing. 13:1200-1212.
[43]. A Criminisi, I Reid, A Zisserman (1999) A plane measuring device. Image and Vision Computing. 17(8):625-634.
[44]. B. Heigl, R. Koch, M. Pollefeys, J. Denzler, L. Van Gool (1999) Plenoptic modeling and rendering from image sequences taken by hand-held camera. In Mustererkennung. pp: 94-101.
論文使用權限
  • 同意紙本無償授權給館內讀者為學術之目的重製使用,於2018-10-29公開。
  • 同意授權瀏覽/列印電子全文服務,於2018-10-29起公開。


  • 若您有任何疑問,請與我們聯絡!
    圖書館: 請來電 (02)2621-5656 轉 2281 或 來信