本論文主要研究隨意風格轉換，針對 X. Li等人[15]所提出的線性風格轉換系統（Linear Style Transformation，LST）做探討並以各種面相做改進。LST希望利用深度神經網路來學習模擬白化再著色（Whitening - Coloring Transform，WCT）的運算，以達到風格轉換的目的。WCT中的「白化」與「著色」算是互逆的運算，然而在LST的訓練過程中並未特別訓練這兩個運算的互逆性。本論文的研究內容之一即是針對這一部分做改進，期望能改善轉換結果。另外在LST中的轉換矩陣T是全域轉換，然而考慮到每個通道應同時具有一些獨立性與相關性，我們提出分割再轉換的方式，能在網路參數量大幅降低的架構下，不只取得通道間的獨立性與全域性之平衡，還能得到更好的轉換結果。最後我們利用在轉換當中加入隨機雜訊的方式，讓網路能生成多樣的轉換結果。

We propose two schemes to improve the linear style transfer system (LST) proposed by Li et al., which employs deep neural networks to learn the Whitening and Coloring Transform (WCT) for style transfer. The schemes are proposed for tackling the following two problems from LST: (1) Whitening and coloring operations are inverse operations of each other, but the inverse of these two operations is not emphasized in the training process of LST, and (2) Different channels in the content feature might have not only some correlation but also some independence. The transformation matrix learned by the network in LST being used to transform the whole content feature is considered too global. To strengthen the invertibility between the whitening and coloring operations we propose an additional identity loss. To balance between globality and locality of the trained transformation, we propose a split-and-transform scheme. Experimental results show that the proposed schemes not only greatly reduce the amount of network parameters, but also helps yielding better transferred results.

目錄
目錄    III 
圖目錄    V 
表目錄    VI 
第一章、 研究背景與目的    1 
第二章、 相關研究    3 
2.1 Gram Matrix    3 
2.2白化再著色轉換（WCT）    3 
2.3線性風格轉換（Linear Style Transformation，LST）    5 
2.4 隨機多樣性轉換    7 
2.4.1 特徵期望值不變之雜訊    7 
2.4.2 正交雜訊    7 
第三章、 研究方法    9 
3.1 互逆性訓練    9 
3.2 分割再轉換    10 
3.3 多樣性轉換結果    12 
3.3.1 期望值1雜訊    12 
3.3.2 正交雜訊    13 
第四章、 實驗結果    15 
4.1互逆性訓練    15 
4.1.1 還原影像測試    15 
4.1.2 白化的結果    16 
4.1.3 風格轉換的結果    17 
4.2 分割轉換    19 
4.2.1 分割轉換的結果    20 
4.2.2 LST縮小轉換矩陣的結果    27 
4.3 加入雜訊    29 
4.3.1 期望值1雜訊    29
4.3.2 正交雜訊    31
4.4 不同程度的轉換    32
第五章、 結論    34
參考文獻    35
附錄：英文論文    38

圖目錄
圖一LST網路架構    6
圖二LST    損失函數7
圖三互逆性強化架構圖    10
圖四分割再轉換架構圖    11
圖五加入雜訊後的整體架構圖    13
圖六互逆性強化還原影像的結果    16
圖七白化的結果    17
圖八互逆性訓練的風格轉換結果    18
圖九不同分割架構的轉換結果    20
圖十LST 與 S1 架構比較    21
圖十一S1加入互逆性訓練的結果I    22
圖十二S1加入互逆性訓練的結果II    22
圖十三S2架構中四個分割區塊不同組合的部分轉換結果I    23
圖十四S2架構中四個分割區塊不同組合的部分轉換結果II    25
圖十五S3架構中八個分割區塊一些不同組合的部分轉換結果I    26
圖十六S3架構中八個分割區塊一些不同組合的部分轉換結果II    27
圖十七LST縮小轉換矩陣的結果 29
圖十八多樣性的生成結果I    30
圖十九多樣性的生成結果II    30
圖二十多樣性的生成結果III    31
圖二十一多樣性的生成結果IV    31
圖二十二加入正交雜訊後的轉換結果    32
圖二十三使用不同α值產生的結果    33
表目錄
表一分割網路的網路參數量    19
表二LST縮小T尺寸的網路參數量    28

[1].T. C. Wang, M. Y. Liu, J. Y. Zhu, A. Tao, J. Kautz, and B.Catanzaro, “High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs,” arXiv:1711.11585 [cs.CV], Aug., 2018.
[2].T. Miyato and M. Koyama, “CGANs with Projection Discriminator,” arXiv:1802.05637 [cs.LG], Aug., 2018.
[3].J. Y. Zhu, R. Zhang, D. Pathak, T. Darrell, A. A. Efros, O. Wang, and E. Shechtman, “Toward Multimodal Image-to-Image Translation,” arXiv:1711.11586 [cs.CV], Oct., 2018.
[4].J. Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks,” arXiv:1703.10593 [cs.CV], Nov., 2018
[5].P. Isola, J. Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-Image Translation with Conditional Adversarial Networks,” arXiv:1611.07004 [cs.CV], Nov., 2018.
[6].T. Park, M. Y. Liu, T. C. Wang, and J. Y. Zhu, “Semantic Image Synthesis with Spatially-Adaptive Normalization,” arXiv:1903.07291 [cs.CV], Nov., 2019.
[7].D. Kotovenko, A. Sanakoyeu, P. Ma, S. Lang, and B. Ommer, “A Content Transformation Block for Image Style Transfer,” arXiv:2003.08407 [cs.CV], Mar., 2020.
[8].L. A. Gatys, A. S. Ecker, and M. Bethge, “A Neural Algorithm of Artistic Style,” arXiv:1508.06576 [cs.CV], Sep., 2015.
[9].K. Simonyan and A. Zisserman, “Very Deep Convolutional Networks for Large-Scale Image Recognition,” arXiv:1409.1556v6 [cs.CV], April, 2015.
[10].J. Johnson, A. Alahi, and F. F. Li, “Perceptual Losses for Real-Time Style Transfer and Super-Resolution,” arXiv:1603.08155 [cs.CV], Mar., 2016.
[11].Y. Li, C. Fang, J. Yang, Z. Wang, X. Lu, and M. H. Yang, “Universal Style Transfer via Feature Transforms,” arXiv:1705.08086 [cs.CV], Nov., 2017.
[12].Z. Xu, M. Wilber, C. Fang, A. Hertzmann, and H. Jin, “Learning from Multi-Domain Artistic Images for Arbitrary Style Transfer,” arXiv:1805.09987 [cs.CV], May, 2018.
[13].L. Sheng, Z. Lin, Ji. Shao, and X. Wang, “Avatar-Net Multi-Scale Zero-Shot Style Transfer by Feature Decoration,” arXiv:1805.03857 [cs.CV], May, 2018.
[14].X. Huang and S. Belongie, “Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization,” arXiv:1703.06868 [cs.CV], Jul., 2018.
[15].X. Li, S. Liu, J. Kautz, and M. H. Yang, “Learning Linear Transformations for Fast Arbitrary Style Transfer,” arXiv:1808.04537 [cs.CV], Aug., 2018.
[16].D. Y. Park and K. H. Lee, “Arbitrary Style Transfer with Style-Attentional Networks,” arXiv: 1812.02342 [cs.CV], May, 2019.
[17].J. An, H. Xiong, J. Luo, J. Huan, and J. Ma, “Fast Universal Style Transfer for Artistic and Photorealistic Rendering,” arXiv:1907.03118v1 [cs.CV] , Jul., 2019.
[18].A. D. Nguyen, S. Choi, W. Kim, and S. Lee, “A Simple Way of Multimodal and Arbitrary Style Transfer,” IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2019).
[19].Z. Wang, L. Zhao, H. Chen, L. Qiu, Q. Mo, S. Lin, W. Xing, and D. Lu, “Diversified Arbitrary Style Transfer via Deep Feature Perturbation,” arXiv:1909.08223 [cs.CV], Nov., 2019.
[20].Y. Zhang, C. Fang, Y. Wang, Z. Wang, Z. Lin, Y. Fu, and J. Yang, “Multimodal Style Transfer via Graph Cuts,” arXiv:1904.04443 [cs.CV], Jan., 2020.
[21].P. Baldi, “Autoencoders, Unsupervised Learning, and Deep Architectures,” in Proc. 2011 ICML Workshop on Unsupervised and Transfer Learning, 27:37–50, 2012.
[22].Y. Jin, Y. Yang, Z. Feng, J. Ye, Y. Yu, and M. Song, “Neural Style Transfer: A Review,” arXiv:1705.04058 [cs.CV], Oct., 2018.
[23].T. Y. Lin, M. Maire, S. Belongie, L. Bourdev, R. Girshick, J. Hays, P. Perona, D. Ramanan, C. L. Zitnick, and P. Dollr, Microsoft coco: Common objects in context. In ECCV, 2014.
[24].K. Nichol. Painter by numbers, wikiart. https://www.kaggle.com/c/painter-by-numbers, 2016.