本文提出一個以學習樣本為依據的超解析人臉影像合成系統。這樣的問題是困難的，因為往往兩個在低解析影像看似相似的人於高解析會有相當大的鑑別性。為了有效率的使用有限的訓練樣本集以學習低和高解析之間的關係，過去的系統按不同的目標需求發展出不同的人臉影像特徵表達方式；如：直接處理整張人臉影像以維持整體的人臉幾何架構、局部人臉區塊為處理單位以解決訓練樣本數不足，或是設計不同影像一維二維Glocal的表達形式來提高資料分析上的效率。特別是即使是基於同一個核心演算法也會因影像的表達方式不同而有不同的合成結果。
本文利用boosting的概念；首先，我們將訓練樣本分為正與負樣本群；其中正樣本群的每一個樣本為同一個人的兩張不同解析度人臉影像對（意即：輸入的低解析影像與目標的高解析影像），負樣本為兩張不同人的高低解析影像對。所提出的系統將不斷的迭代挑選好的特徵表達方式以提高合成能力和人臉間的鑑別率。明確來說，每一特徵會對應到一個以這個特徵為基礎的學習回歸方程，最佳的特徵會使得正負樣本群分的最開。同時，被誤判的樣本群將藉由調整樣本權重與回歸模型於下一時間點越來越被重視。這樣的過程，挑選的特徵具有相當多的變化性，舉例來說：有些是著重於整體幾何架構、局部人臉五官特性、有些強調輪廓資訊等，這使得我們所提出的系統不只適用於與訓練樣本一樣的資料庫中，更能處理不同角度或表情或真實情況下的人臉影像合成問題。

This study develops a face hallucination system based on a novel two-dimensional direct combined model (2DDCM) algorithm that employs a large collection of low-resolution/high-resolution facial pairwise training examples.  This approach uses a formulation that directly combines the pairwise example in a 2D combined matrix while completely preserving the geometry-meaningful facial structures and the detailed facial features.  Such a representation would be expected to yield a useful transformation for face reconstruction.  Our algorithm achieves this goal by addressing four key issues.  First, we establish the 2D combination representation that defines two structure-meaningful vector spaces to respectively describe the vertical and the horizontal facial-geometry properties.  Second, we directly combine the low-resolution and high-resolution pairwise examples to completely model their relationship, thereby preserving their significant features.  Third, we develop an optimization framework that finds an optimal transformation to best reconstruct the given low-resolution input.  The 2D combination representation makes the transformation more powerful than other approaches.  Fourth, specific to our framework, we will appropriately apply the proposed 2DDCM algorithm for modeling global and local properties of the facial image.  Our approach is demonstrated by extensive experiments with high-quality hallucinated faces.

目錄
第一章 緒論	1
1.1 研究目的與動機	1
1.2 相關研究	3
1.3 本文貢獻	6
第二章 組合式超解析人臉影像合成系統	9
2.1 系統架構	9
2.2 訓練部分	11
2.2-1 訓練樣本收集	11
2.2-2 以二維混合模型為基礎的轉換公式	13
2.2-3 影像特徵表達	15
2.2-4 以Boosting方式為基礎的影像特徵挑選機制	22
2.3 高解析影像合成	30
第三章 實驗結果與分析	32
3.1 實驗設定	32
3.1-1 系統設定	32
3.1-2 人臉資料庫	32
3.1-3 比較方法	33
3.1-4 評估公制與辨識率	33
3.2 實驗結果	34
3.2-1 系統評估	34
3.2-2 結果分析與比較	37
3.2-3 應用於不同情況	42
3.2-4 應用於真實影像	45
第四章 結論	47
參考文獻	49
附錄：英文論文	54

圖目錄
圖1 本系統的訓練部分示意圖	10
圖2 本系統的測試部分示意圖	11
圖3 正負樣本影像對示意圖	12
圖4 影像表達示意圖	17
圖5 旋轉區塊影像表達	19
圖6 交叉局部區塊影像表達	19
圖7 Glocal影像表達的作法示意圖	21
圖8 正負樣本Val值(影像力)的分佈示意圖	27
圖9 最佳特徵與權重的挑選分析	29
圖10 系統迭代合成結果	31
圖11 系統迭代評估(T=1~10)	34
圖12 分類器篩選合成差異圖	35
圖13 不同方法的公制評估	35
圖14 超解析合成結果	36
圖15 資料庫交叉合成結果	39
圖16 不同解析的合成結果(縮小2倍)	40
圖17 不同解析的合成結果(縮小8倍)	40
圖18 不同Pose的合成結果	41
圖19 不同表情的合成結果(微笑表情)	41
圖20 錯誤教正的合成結果	42
圖21 不同訓練樣本個數的合成結果	44
圖22 真實影像的合成結果(一)	45
圖23 真實影像的合成結果(二)	46

表目錄
表1 本系統Boosting流程步驟  22
表2 人臉辨識率(合成結果)  35

[1]	M. Davis, M. Smith, J. Canny, N. Good, S. King, and R. Janakiraman, “Towards    context-aware face recognition,” ACM International Conference on Multimedia, 2005, pp. 483-486.
[2]	A. Sharma and D.W. Jacobs, “Bypassing Synthesis: PLS for Face Recognition with Pose, Low-Resolution and Sketch," IEEE Conference on Computer Vision and Pattern Recognition, 2011, pp. 593-600.
[3]	W. W. W. Zou and P. C. Yuen, “Very Low Resolution Face Recognition Problem,” IEEE Transactions on Image Processing, 2012, Vol. 21, pp. 327-340.
[4]	S. Biswas, G. Aggarwal, P. J. Flynn, and K. W. Bowyer, “Pose-Robust Recognition of Low-Resolution Face Images,” IEEE Trans. Pattern Anal Mach Intell, 2013,  Vol. 35, no. 12, pp. 3037-3049.
[5]	K. Jia and S. Gong, “Multi-Resolution Patch Tensor for Facial Expression Hallucination,” IEEE Conference on Computer Vision and Pattern Recognition, 2006, pp. 395-402.
[6]	L. Zhang, H. Yuxiao, L. Mingjing, M. Weiying, and Z. Hongjiang, “Efficient Propagation for Face Annotation in Family Albums,” ACM International Conference on Multimedia, 2004.
[7]	K. Li, Q. Dai, R. Wang, Y. Liu, F. Xu, and J. Wang, “A Data-Driven Approach for Facial Expression Retargeting in Video,” IEEE Transactions on Multimedia, 2014, pp. 299-310.
[8]	K. Liu and J. Ostermann,”Realistic Facial Expression Synthesis for an Image-based Talking Head,” IEEE Conference on Multimedia and Expo, 2011.
[9]	F. Yang, L. D. Bourdev, E. Shechtman, J. Wang, and D. N. Metaxas, “Facial Expression Editing in Video Using A Temporally-Smooth Factorization,” IEEE Conference on Computer Vision and Pattern Recognition, 2012, pp. 861-868.
[10]	J. Allebach, and P. W. Wong, “Edge-directed interpolation,” IEEE Conference on Image Processing, 1996, pp. 707-710.
[11]	H. Demirel and G. Anbarjafari, “Image Resolution Enhancement by Using Discrete and Stationary Wavelet decomposition,” IEEE Trans. Image Process, 2011, vol. 20, no. 5, pp. 1458-1460.
[12]	S. Mallat and G.Yu, “Super-Resolution with Sparse Mixing Estimators,” IEEE Trans. on Image Processing, 2010, Vol. 19, No.11, pp. 2889-2900.
[13]	K. Zhang, X. Gao, D. Tao, and X. Li, “Single Image Super-resolution with Non-Local Means and Steering Kernel Regression,” IEEE Trans. Image Processing, 2012, vol. 21, no. 11, pp. 4544-4556.
[14]	M. D. Robinson, C. A. Toth, J. Y. Lo, and S. Farsiu, “Efficient Fourier Wavelet Super Resolution,” IEEE Trans. Image Process, 2010, vol. 19, no. 10, pp. 2669-2681.
[15]	F. Zhou, W. Yang, and Q. Liao, “Interpolation-Based Image Super- Resolution Using Multi-surface Fitting,” IEEE Transactions On Image Processing, 2012, Vol. 21, No. 7, PP. 3312-3318.
[16]	C. Hong, Y. Yan, and X. Yimin, “Super-Resolution Through Neighbor Embedding,” IEEE Conference on Computer Vision and Pattern Recognition, 2004, pp. 1-10.
[17]	K.I. Kim and Y. Kwon, “Example-based Learning for Single-Image Super-Resolution and JPEG Artifact Removal,” Pattern Recognition, 2008, pp.456-465.
[18]	K.I. Kim and Y. Kwon, “Single-Image Super-Resolution Using Sparse Regression and Natural Image Prior,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, vol. 32, no. 6, pp. 1127-1133.
[19]	X. Gao, K. Zhang, D. Tao, and X. Li, “Joint Learning for Single-Image Super-Resolution via a Coupled Constraint,” IEEE Transactions On Image Processing, 2012, Vol. 21, No. 2.
[20]	S. Baker and T. Kanade, “Limits on super-resolution and how to break them,” IEEE Transactions s on Pattern Analysis and Machine Intelligence, 2002, pp. 1167 -1183.
[21]	X. G. Wang and X. O. Tang, “Hallucinating Face by Eigen-transformation,” IEEE Transactions on Systems, 2005, pp. 425-434.
[22]	L. Wei, L. Dahua, and T. Xiaoou, “Face Hallucination Through Dual Associative Learning,” IEEE Conference on Image Processing, 2005.
[23]	O. Takahiro and H. Miki, “ Adaptive example-based super-resolution using kernel PCA with a novel classification approach,” Signal Processing, 2011.
[24]	X. Ma, J. Zhang, and C. Qi, “Hallucinating face by position-patch,” Pattern Recognition, 2010, pp. 2224-2236.
[25]	Y. Liang, J.H. Lai, P. C. Yuen, W. W. W. Zou, and Z. Cai, “Face Hallucination with Imprecise-Alignment Using Iterative Sparse Representation,” Pattern Recognition, 2014, pp. 3327-3342.
[26]	H. Chang, D. Yeung, and Y. Xiong, ”Super resolution through Neighbor Embedding,”  IEEE Conference on Computer Vision and Pattern Recognition, 2004, pp. 1275-1282.
[27]	J. Yang, J. Wright, T. Huang, and Yi Ma, “Image Super-Resolution as Sparse Representation of Raw Image Patches,” IEEE Conference on Computer Vision and  Pattern Recognition, 2008, pp.1-8.
[28]	C. Liu, H. Y. Shum, and C. S. Zhang, A two-step approach to hallucinating faces: global parametric model and local nonparametric model, IEEE Conference on Computer Vision and Pattern Recognition, 2001, pp. 192-198.
[29]	H. M. M. Naleer, L. Yao, and A. Yaozu, “The face Hallucinating two-step framework using hallucinated high-resolution residual,” IEEE Conference on Digital Image Processing, 2011.
[30]	W. T. Freeman, “Learning low-level vision,” IEEE Conference on International Computer Vision, 1999, pp. 1182-1189.
[31]	Y. Zhuang, J. Zhang, and F. Wu, “Hallucinating Faces: LPH Super Resolution and Neighbor Reconstruction for Residue Compensation,” Pattern Recognition, 2007, pp. 3178-3194.
[32]	H. Huang, H. He, X. Fan, and J. Zhang, “Super-resolution of human face image using canonical correlation analysis,” Pattern Recognition, 2010, pp. 2532-2543.
[33]	Le An and B. Bhanu, "Face Image Super-Resolution using 2D CCA," Signal Processing, 2014.
[34]	Z. Wen and C. W. Kuen, “Learning-based Face Hallucination in DCT Domain,” IEEE Conference on Computer Vision and Pattern Recognition, 2008, pp. 1-8.
[35]	P. J. Seon and L. S. Whan, “An Example-Based Face Hallucination Method for Single-Frame, Low-Resolution Facial Images,” IEEE Transactions on Image Processing, 2008, pp. 1806-1816.
[36]	Y. C. Yuan, L. Sifei, and Y. M. Hsuan, “Structured Face Hallucination,” IEEE Conference on Computer Vision and Pattern Recognition, 2013, pp. 1099-1106.
[37]	F. Tappen and L. Ce, “A Bayesian approach to alignment-based image hallucination,” European conference on Computer Vision, 2012, pp. 236-249.
[38]	L. Yongchao, C. Cheng, Q. Guoping, and L. K. Man, “Face hallucination based on sparse local-pixel structure,” Pattern Recognition, 2014, pp. 1261-1270.
[39]	Y. Hu, K.M. Lam, G. Qiu, and T. Shen, “From local pixel structure to global image super-resolution: a new face hallucination framework,” IEEE Transactions on Image Processing, 2011, pp. 433-445.
[40]	C. Y.n Yang and M. H. Yang, “Fast Direct Super-Resolution by Simple Functions, “ In IEEE International Conference on Computer Vision, 2013, pp. 561-568.
[41]	L. S. Ho and C. Seungjin, “Two-Dimensional Canonical Correlation Analysis,” IEEE Transactions s on Signal Processing Letters, 2007, pp. 735-738.
[42]	Y. Hua, S. Jiande, and D. Lina, “Face Hallucination Based on Independent Residual Features,” IEEE Conference on Signal Processing (ICSP), 2010, pp. 1074-1077.
[43]	L. Wei, L. Dahua, and T. Xiaoou, “Neighbor Combination and Transformation for Hallucinating Faces,” IEEE Conference on Multimedia and Expo, 2005.
[44]	Karl, S. Ni, and T. Q. Nguyen, “Image Super resolution Using Support Vector Regression,” IEEE Transactions on Image Processing, 2007, pp. 1596-1610.
[45]	C. T. Tu, K. H. Lin, and T. L. Liu, “A Two-dimensional Direct Combined Model for Facial Hallucination,” IAPR International Conference on Machine Vision Applications, 2013, pp. 149-152.
[46]	X. Gao, K. Zhang, D. Tao, and X. Li, “Joint learning for single-image super resolution via a coupled constraint,” IEEE Transactions on Image Processing, 2012, pp. 469-480.
[47]	C.T. Tu and J. J. Lien, “Automatic Location of Facial Feature Points and Synthesis of Facial Sketches Using Direct Combined Model,” IEEE Transactions on Systems, Man, and Cybernetics, Part B, 2010, pp. 1158-1169.
[48]	T. Sun and S. Chen, “Locality preserving CCA with applications to data visualization and pose estimation,” Image and Vision Computing, 2007, pp. 531-543.
[49]	Z. Gou and C. Fyfe, “A canonical correlation neural network for multi collinearity and functional data,” Neural Networks, pp. 285-293.
[50]	W. Yang, D. Yi, Z. Lei, J. Sang, and S. Li, “2D–3D face matching using CCA,”  IEEE Conference on Automatic Face Gesture Recognition, 2008, pp. 1-6.
[51]	S. Caifen, G. Shaogang, and W. Peter, “Capturing Correlations Among Facial Parts for Facial Expression Analysis,” Proceedings of the British Machine Vision, 2007.
[52]	L. Ce, S. Y. Heung, and W. T. Freeman, “Face Hallucination: Theory and Practice,” International Journal of Computer Vision, 2007, pp. 115-134.
[53]	P. Philips, H. Moon, P. Pauss, and S. Rivzvi, “The feret evaluation methodology for face-recognition algorithms,” IEEE Conf. Computer Vision and Pattern Recognition, 1997, pp. 137-143.
[54]	C. H. Tzong, L. T. Luh, and F. C. Shann, “Learning Effective Image Metrics from Few Pairwise Examples,” IEEE International Conference on Computer Vision, 2005, pp. 1371-1378.
[55]	B. G. Vijay Kumar and R. Aravind “A 2D Model for Face Superresolution,” International Conference on Pattern Recognition, 2008, pp. 1-4.
[56]	C. H. Tzong, L. T. Luh, and F. C. Shann, “Learning Effective Image Metrics from Few Pairwise Examples,” IEEE International Conference on Computer Vision, 2005, pp. 1371-1378. 
[57]	A. Martinez and R. Benavente. The ar face database. Technical report, CVC Techinical Report No.24, June 1998.
[58]	Z. Wang, A. Bovik, H. Sheikh, and E. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE Transactions on Image Processing, 2004, pp. 600-612.
[59]	S. Thakur, J. K. Sing, D. K. Basu, and M. Nasipuri, “Face Recognition Using Principal Component Analysis and RBF Neural Networks,” CETET '08. First International Conference on Emerging Trends in Engineering and Technology, 2008, pp. 16-18.
[60]	C. C. Liu and S. Y. Chin, “A face identification algorithm using support vector machine based on binary two dimensional principal component analysis,” IET International Conference on Technologies and Applications, 2010, pp. 193-198.