問答系統(Question answering)主要是在解決給定一道問題，透過機器閱讀(Machine Reading)的方式讓系統能夠理解這一道題目後進行回答。問答系統通常包含了問題分析(Question Analysis)、文件檢索(Document Retrieval)、答案抽取(Answer Extraction)、答案驗證(Answer Validation)。
　　在過去文獻中有相當多的問答系統相關研究，但是並未對問答系統中答案驗證不平衡資料集與平衡資料集進行深入探討。本研究目的會透過機器學習完整分析不平衡資料集與平衡資料集。
　　本研究使用 NTCIR-12 QA-Lab2  日本大學入學考試世界歷史資料集，此資料集與以往問答系統比較不同的地方在於是系統必須先理解一篇短文之後，才能夠回應接下來相關的問題。
　　本研究針對不平衡資料集與平衡資料集提出了許多的模型，藉由最佳化參數與交叉驗證後，實驗結果顯示在不平衡資料集中，最佳模型的正確率達到了 90%。本論文主要貢獻為提出了一套問答系統，並且在答案驗證階段透過不平衡資料集與平衡資料集證實，不平衡資料集所建構出來之模型顯著性較高。

Question Answering is a system that can process and answer a given question. Question Answering system usually consists of four stages: Question Analysis, Document Retrieval, Answer Extraction and Answer Validation.
Although a considerable number of studies have been made on Question Answering system, little is known about the power of Imbalanced datasets and balanced datasets for Answer Validation from Question Answering.
The purpose of this paper is to provide a comprehensive analysis of Imbalanced datasets and balanced datasets through machine learning.
In this paper, we used datasets from NTCIR-12 QA-Lab2 Japanese university entrance exams on the subject of "World History". The difference between this datasets and previous ones lies in the different processing method that the system needed to understand a context provided by the present research’s datasets and answered the following related questions.
The study presented many Imbalanced datasets and Balanced datasets models by using f.select and Cross Validation. The results show the best performance of our system achieved an accuracy of 90% in the Imbalanced datasets model.
The main contribution of this study was in proposing a question answering system for Japanese university entrance exams and providing evidence that the Imbalanced datasets model outperformed the balanced datasets model for Answer Validation.

目錄
第一章　緒論	1
1.1	研究背景	1
1.2	研究動機	3
1.3	問題定義	4
1.4	研究目的	4
1.5	論文架構	5
第二章　文獻探討	7
2.1	問答系統	7
2.2	答案驗證	10
2.3	不平衡資料集	12
2.4	機器學習	14
2.5	本章小節	15
第三章　研究方法與系統架構	16
3.1	研究方法	16
3.2	系統架構	19
3.2.1	問題分析	20
3.2.2	文件檢索	22
3.2.3	答案抓取	23
3.2.4	答案驗證	25
3.2.5	答案產生	27
3.3	資料集	28
3.4	資料前處理	30
3.4.1	XML資料集擷取	31
3.4.2	特徵處理	35
3.5	機器學習	37
3.6	模型建置	40
第四章　研究結果與討論	44
4.1	實驗資料分配與評估方式	44
4.1.1	實驗資料	44
4.1.2	實驗評估方式	45
4.2	答案驗證分析評估	47
4.2.1	平衡資料集	47
4.2.2	不平衡資料集	50
4.3	問答系統之答案驗證分析評估	53
4.3.1	平衡資料集	53
4.3.2	不平衡資料集	56
4.4	實驗結果總結	57
第五章　結論與貢獻	58
5.1	結論	58
5.2	研究貢獻 	59
5.3	管理意涵 	60
5.4	未來研究方向	60
參考文獻 	61

表目錄
表 1　NTCIR12 QA-Lab2 資料集問題類型定義之分析	28
表 2　NTCIR-12 QA-Lab2 日本大學入學考試訓練資料集之分析	29
表 3　NTCIR-12 QA-Lab2 日本大學入學考試測試資料集之分析	29
表 4  NTCIR-12 QA-Lab2 2009年日本大學入學考試內容	30
表 5　12組特徵	35
表 6　日本大學入學考試世界歷史題目之訓練資料集分析	40
表 7　日本大學入學考試世界歷史題目之測試資料集分析	41
表 8  SVM實驗數據內容(範例顯示2009年資料集第一題)	43
表 9　不平衡訓練資料集與平衡訓練資料集之分析	45
表 10  NTCIR-12 QA-Lab2 全部訓練資料集與測試資料集中，七組平衡資料集之10-fold交叉驗證分析結果	47
表 11  全部訓練資料集與兩組測試資料集之開放測試正確率分析	49
表 12  NTCIR-12 QA-Lab2 隨機從訓練資料集中，挑出五組平衡資料集之10-fold交叉驗證分析結果	49
表 13  NTCIR-12 QA-Lab2 隨機從訓練資料集中，挑出八組不平衡資料集之10-fold交叉驗證分析結果	51
表 14  八組不平衡訓練資料集與不平衡測試資料集之開放測試分析	52
表 15  NTCIR-12 QA-Lab2 全部訓練資料集與測試資料集中透過問答系統中七組平衡資料集之10-fold交叉驗證分析結果	53
表 16  NTCIR-12 QA-Lab2 隨機從訓練資料集中透過問答系統選出五組平衡資料集之10-fold交叉驗證分析結果	55
表 17  NTCIR-12 QA-Lab2 隨機從訓練資料集中透過問答系統挑出八組不平衡資料集之10-fold交叉驗證分析結果	56

圖目錄
圖 1　本研究之研究架構流程	6
圖 2　系統開發的研究生命週期循環圖	17
圖 3　系統發展研究方法論流程圖	18
圖 4　系統架構(System Architecture)	19
圖 5　Stanford POS tagger 與 NER 範例	20
圖 6　JA&EN Translator範例	21
圖 7　問題分析(Question Analysis)之結果	21
圖 8　文件檢索(Document Retrievals)之結果	22
圖 9　答案抓取(Answer Extraction)之結果	24
圖 10  答案驗證(Answer Validation)之結果	26
圖 11　Essay 題目之範例	27
圖 12　NTCIR12 QA-Lab2日本大學入學考試日文原始資料	31
圖 13　NTCIR12 QA-Lab2日本大學入學考試英文原始資料	32
圖 14　NTCIR-12 QA-Lab2 2009年大學入學考試英文XML格式資料集	33
圖 15　整理NTCIR-12 QA-Lab2整理訓練資料集後之格式	33
圖 16　經過文件檢索後所得到的句子整理之格式	34
圖 17　LibSVM訓練資料與測試資料建構模型之流程	38
圖 18　LibSVM經過最佳參數化之結果圖	39
圖 19　日本大學入學考試世界歷史資料集分析圖	41
圖 20　日本大學入學考試世界歷史題目之測試資料集分析圖	42
圖 21　LibSVM訓練集格式	42
圖 22　混淆矩陣之圖示說明	46
圖 23　NTCIR-12 QA-Lab2全部訓練資料集與測試資料集中，七組平衡資料集交叉驗證結果之圖表	48
圖 24　NTCIR-12 QA-Lab2五組平衡資料集之圖表	50
圖 25　NTCIR-12 QA-Lab2 八組不平衡資料集之交叉驗證分析之圖表	51
圖 26　NTCIR-12 QA-Lab2全部訓練資料集與測試資料集中透過問答系統中七組平衡資料集交叉驗證結果之圖表	54
圖 27　NTCIR-12 QA-Lab2透過問答系統五組平衡資料集之圖表	55
圖 28　NTCIR-12 QA-Lab2 透過問答系統八組不平衡資料集之交叉驗證分析之圖表	56

[1] Abbasi, A., Chen, H., & Salem, A. (2008). Sentiment analysis in multiple languages: Feature selection for opinion classification in web forums. ACM Transactions on Information Systems (TOIS), 26(3), 12. 
[2] Anand, A., Pugalenthi, G., Fogel, G. B., & Suganthan, P. (2010a). An approach for classification of highly imbalanced data using weighting and undersampling. Amino Acids, 39(5), 1385-1391. 
[3] Anand, A., Pugalenthi, G., Fogel, G. B., & Suganthan, P. (2010b). An approach for classification of highly imbalanced data using weighting and undersampling. Amino Acids, 39(5), 1385-1391. 
[4] Batuwita, R., & Palade, V. (2012). Adjusted geometric-mean: A novel performance measure for imbalanced bioinformatics datasets learning. Journal of Bioinformatics and Computational Biology, 10(04), 1250003. 
[5] Braschler, M., & Schauble, P. (2000). Experiments with the eurospider retrieval system for clef 2000. Cross-language information retrieval and evaluation (pp. 140-148) Springer.
[6] Chang, C., & Lin, C. (2011). LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST), 2(3), 27. 
[7] Dang, H. T., Kelly, D., & Lin, J. (2007). Overview of the TREC 2007 question answering track.
[8] Gmez-Adorno, H., Sidorov, G., Pinto, D., & Gelbukh, A. (2014). Graph-based approach to the question answering task based on entrance exams. Notebook for PAN at CLEF, , 1395-1403. 
[9] Greene, S., & Resnik, P. (2009). More than words: Syntactic packaging and implicit sentiment. Paper presented at the Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, 503-511. 
[10] Haque, M. M., Skinner, M. K., & Holder, L. B. (2014). Imbalanced class learning in epigenetics. Journal of Computational Biology, 21(7), 492-507. 
[11] Harman, D. (1993). Overview of the first TREC conference. Paper presented at the Proceedings of the 16th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 36-47. 
[12] Lee, C., Day, M., Sung, C., Lee, Y., Jiang, T., Wu, C., . . . Hsu, W. (2007). Chinese-chinese and english-chinese question answering with ASQA at NTCIR-6 CLQA. Proceedings of NII-NACSIS Test Collection for Information Retrieval Systems (NTCIR’07), , 175-181. 
[13] Magnini, B., Negri, M., Prevete, R., & Tanev, H. (2002). Is it the right answer?: Exploiting web redundancy for answer validation. Paper presented at the Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, 425-432. 
[14] Magnini, B., Romagnoli, S., Vallin, A., Herrera, J., Penas, A., Peinado, V., . . . de Rijke, M. (2004). The multiple language question answering track at clef 2003. Comparative evaluation of multilingual information access systems (pp. 471-486) Springer.
[15] Malakasiotis, P., & Androutsopoulos, I. (2007). Learning textual entailment using SVMs and string similarity measures. Paper presented at the Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, 42-47. 
[16] Martineau, J., & Finin, T. (2009). Delta TFIDF: An improved feature space for sentiment analysis. Icwsm, 9, 106. 
[17] Matsuyoshi, S., Miyao, Y., Shibata, T., Lin, C., Shih, C., Watanabe, Y., & Mitamura, T. (2014). Overview of the ntcir-11 recognizing inference in text and validation (rite-val) task. Paper presented at the Proceedings of the 11th NTCIR Conference, 223-232. 
[18] Nanni, L., Fantozzi, C., & Lazzarini, N. (2015). Coupling different methods for overcoming the class imbalance problem. Neurocomputing, 158, 48-61. 
[19] Napierala, K., & Stefanowski, J. (2012). BRACID: A comprehensive approach to learning rules from imbalanced data. Journal of Intelligent Information Systems, 39(2), 335-373. 
[20] Nunamaker Jr, J. F., Chen, M., & Purdin, T. D. (1990). Systems development in information systems research. Journal of Management Information Systems, 7(3), 89-106. 
[21] Okita, T., & Liu, Q. (2014). The question answering system of DCUMT in NTCIR-11 QA lab. Paper presented at the Proceedings of the 11th NTCIR Conference, 
[22] Peñas, A., Forner, P., Sutcliffe, R., Rodrigo, Á, Forăscu, C., Alegria, I., . . . Osenova, P. (2010). Overview of ResPubliQA 2009: Question answering evaluation over european legislation. Multilingual information access evaluation I. text retrieval experiments (pp. 174-196) Springer.
[23] Peñas, A., Hovy, E. H., Forner, P., Rodrigo, Á, Sutcliffe, R. F., Forascu, C., & Sporleder, C. (2011). Overview of QA4MRE at CLEF 2011: Question answering for machine reading evaluation. Paper presented at the CLEF (Notebook Papers/Labs/Workshop), 1-20. 
[24] Peñas, A., Rodrigo, Á, & Verdejo, F. (2008). Overview of the answer validation exercise 2007. Advances in multilingual and multimodal information retrieval (pp. 237-248) Springer.
[25] Ravichandran, D., & Hovy, E. (2002). Learning surface text patterns for a question answering system. Paper presented at the Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, 41-47. 
[26] Sakamoto, K., Matsui, H., Matsunaga, E., Jin, T., Shibuki, H., Mori, T., . . . Kando, N. (2014). Forst: Question answering system using basic element at NTCIR-11 QA-lab task. Paper presented at the Proceedings of the 11th NTCIR Conference, 
[27] Shibuki, H., Sakamoto, K., Kano, Y., Mitamura, T., Ishioroshi, M., Itakura, K. Y., . . . Kando, N. (2014). Overview of the NTCIR-11 QA-lab task. Paper presented at the Ntcir, 
[28] Shibuki, H., Sakamoto, K., Kano, Y., Mitamura, T., Ishioroshi, M., Mori, T., & Kando, N. (2015). NTCIR-12 QA-lab task second pilot. NTCIR-12 Kick-Off Event, 
[29] Shima, H., Kanayama, H., Lee, C., Lin, C., Mitamura, T., Miyao, Y., . . . Takeda, K. (2011). Overview of ntcir-9 rite: Recognizing inference in text. Paper presented at the Proceedings of the 9th NII Test Collection for Information Retrieval Workshop (NTCIR’11), 291-301. 
[30] Sokolova, M., Japkowicz, N., & Szpakowicz, S. (2006). Beyond accuracy, F-score and ROC: A family of discriminant measures for performance evaluation. AI 2006: Advances in artificial intelligence (pp. 1015-1021) Springer.
[31] Sun, Z., Song, Q., Zhu, X., Sun, H., Xu, B., & Zhou, Y. (2015). A novel ensemble method for classifying imbalanced data. Pattern Recognition, 48(5), 1623-1637. 
[32] Tomar, D., Singhal, S., & Agarwal, S. (2014). Weighted least square twin support vector machine for imbalanced dataset. International Journal of Database Theory and Application, 7(2), 25-36. 
[33] Turban, E., Sharda, R., & Delen, D. (2007). Decision support and business intelligence systems Pearson Education India.
[34] Voorhees, E. M. (1999). The TREC-8 question answering track report. Paper presented at the Trec, , 99 77-82. 
[35] Wang, D., Boytsov, L., Araki, J., Patel, A., Gee, J., Liu, Z., . . . Mitamura, T. (2014). CMU multiple-choice question answering system at NTCIR-11 QA-lab. Paper presented at the Proceedings of the 11th NTCIR Conference, 
[36] Zhao, Z., Zhong, P., & Zhao, Y. (2011). Learning SVM with weighted maximum margin criterion for classification of imbalanced data. Mathematical and Computer Modelling, 54(3), 1093-1099.