台灣的大學EFL寫作課程大多採用傳統教學方式並以教師人工批改作文，加上班級學生數較多，造成學生們進行寫作練習時較為耗時且成效有限，教師們則因批改作文量過大且費時而疲累。針對類似情形，國外部分ESL/EFL課程採用自動化寫作評估系統 (簡稱AWE) 解決這些問題。因此本研究以台灣某大學應用外語系中級寫作課程33位學生為研究對象，採用Pearson的自動化寫作評估系統WriteToLearn，作為寫作學習工具，進行為期一學年的研究藉以探討於台灣大學EFL寫作課程使用自動化寫作評估系統的可能性，期望透過本研究為台灣EFL寫作課程的授課教師與學生們提供更有效的寫作學習方式。本研究採用量化和質性研究兩種方式，以WriteToLearn蒐集的資料以及開放式問卷彙整的答案為主，輔以教師/研究者的觀察結果和研究對象在WriteToLearn的作文，討論自動化寫作評估系統作為寫作學習工具對EFL學習者的寫作能力是否具成效以及EFL學習者對使用自動化寫作評估系統做為學習工具的看法和優缺點。研究結果顯示研究對象的作文分數雖未顯著提升，但逾半數研究對象皆認為WriteToLearn作為寫作練習工具對提升寫作能力有幫助。此外，研究對象針對WriteToLearn作為寫作練習工具提出七大優點–文法/拼字挑錯、立即性回饋、寫作分析、提供評語、類似正式考試, 寫作指引和用字建議，以及六大缺點–評語籠統、沒有限制寫作時間、缺乏可信度、錯誤判定有誤、遺漏錯誤未指出和其他缺點(例如電腦使用問題)。建議未來EFL教師可將寫作課以傳統授課指導搭配使用自動化寫作評估系統作為學生們的寫作學習工具，而未來研究者可以英文程度更高的學習者為研究對象更深入探討自動化寫作評估系統於EFL寫作課內作為學習工具的情形與結果。

Most of the university EFL writing classes in Taiwan adopt traditional instruction methods and human raters (teachers) would grade student essays. Due to large class sizes, it is time-consuming and ineffective for students to practice writing essays in class. At the same time, it is also time-consuming and tiring for teachers to grade In order to solve the problem, some ESL/EFL classes abroad have adopted Automated Essay Evaluation systems (AWE). The participants of this research were 33 foreign-students from the Department of Applied Foreign Languages at a university in Taiwan. The setting was in an intermediate writing class. The research instrument, WriteToLearn, an AWE system by Pearson, was used as an essay-writing learning tool for a full school year in order to evaluate its application in university-level EFL writing classes in Taiwan. It was hoped this research would provide a more effective essay-writing learning method. This research used quantitative and qualitative methods. Data were collected from WriteToLearn and open-ended questionnaires. The data were analyzed along with the teacher/researcher’s observation and the essays submitted to WriteToLearn to see whether AWE systems was an effective learning tool on EFL learners’ essay-writing abilities. The other research question answered was EFL learners’ perception on the usage of AWE system as a learning tool for writing and its advantages and disadvantages. The research showed that although participants’ essay scores did not improve significantly, over half of them believed that AWE systems (such as WriteToLearn) as an essay-writing learning tool were effective in improving writing abilities. Moreover, participants pointed out seven strengths of WriteToLearn – Grammar / Spelling Correction, Immediacy, Writing Analysis, Comment Providing, Similar to Formal Tests, Idea Guiding and Word Choice Suggestion. They also identified six major weaknesses – Unclear Comments, Unlimited Writing Time, Score Reliability, Wrongful Error Detection, Unable to Identify All Mistakes and Others. It was recommended that EFL teachers could run writing classes as a combination of teaching with traditional methods and using AWE systems as a learning tool for student essay-writing. It was also suggested that future researchers could further investigate the effectiveness of AWE systems in EFL writing classes with learners of better English comprehension.

Table of Contents
Acknowledgements ................................................................................................................... i
Chinese Abstract ...................................................................................................................... ii
English Abstract ...................................................................................................................... iii
Table of Contents ..................................................................................................................... v
List of Tables .......................................................................................................................... viii
List of Figures .......................................................................................................................... ix
Chapter One Introduction....................................................................................................... 1
Statement of the Problem ................................................................................................ 4
Purpose of the Study ........................................................................................................ 6
Significance of the Study ................................................................................................. 7
Chapter Two Literature Review ............................................................................................. 8
Introduction ...................................................................................................................... 8
Prior Research on Automated Writing Evaluation ....................................................... 8
Prior Research of Criterion ........................................................................................... 10
Prior Research of MY Access! ....................................................................................... 11
Prior Research on WriteToLearn ................................................................................... 12
Prior Research on Teachers’ Perspective and Students’ Perspective toward AWE . 15
Limitations of Automated Scoring ............................................................................... 17
Chapter Three Methodology ................................................................................................. 18
Introduction .................................................................................................................... 18
Participants ..................................................................................................................... 18
Instruments ..................................................................................................................... 19
Automated Writing Evaluation (AWE) system ................................................... 19
Open-ended questionnaires ................................................................................... 21
Researcher’s observation ...................................................................................... 22
Procedure ........................................................................................................................ 22
Collection of data from WriteToLearn .................................................................. 25
Data collection from open-ended questionnaires ................................................ 28
Data Analysis .................................................................................................................. 29
Data from WriteToLearn ........................................................................................ 29
Data from open-ended questionnaires ................................................................. 31
Chapter Four Results ............................................................................................................. 33
Introduction .................................................................................................................... 33
Data from WriteToLearn ................................................................................................ 33
Number of students choosing essay topics in different genres ........................... 34
The highest and lowest scores for essays in each genre ...................................... 36
Average scores for essays in different genres ....................................................... 38
The individual average scores for each essay in different genres calculated
according to the 6 assessment criteria .................................................................. 39
Participant’s level scores in WriteToLearn for individual participant’s first and
last essays as compared to that of all participants .............................................. 42
Data from Open-Ended Questionnaire ........................................................................ 45
Advantages of using WriteToLearn ....................................................................... 47
Disadvantages of using WriteToLearn................................................................... 48
Chapter Five Discussion ........................................................................................................ 52
Introduction .................................................................................................................... 52
Effectiveness on EFL Learners’ Writing Abilities While Adopting AWE system as A
Learning Tool ................................................................................................................. 52
Discussion of WriteToLearn Data .................................................................................. 53
Discussion of Open-Ended Questionnaire ................................................................... 60
Advantages of using WriteToLearn ....................................................................... 60
Disadvantages of WriteToLearn ............................................................................. 66
Discussion by Observer/Researcher ............................................................................. 74
Chapter 6 Conclusion ............................................................................................................ 97

Introduction .................................................................................................................... 97
Summary of the Study ................................................................................................... 97
Pedagogical Implications ............................................................................................... 98
Limitation of the Study .................................................................................................. 99
Suggestions for Future Studies ................................................................................... 100
References ............................................................................................................................. 102
Appendix A: Questionnaire ................................................................................................. 110
Appendix B: Number of Students Choosing Each Essay Topic in Different Genres ..... 111
Appendix C: Overall Scores of Essays in Different Genres ............................................. 112
Appendix D: Average Trait Scores for Narrative Essays ................................................. 113
Appendix E: Average Trait Scores for Cause-Effect Essays ............................................ 114
Appendix F: Average Trait Scores for Comparison and Contrast Essays ...................... 115
Appendix G: Average Trait Scores for Argument Essays (Maximum 6) ........................ 116
Appendix H: Average Trait Scores for Argument essays (Maximum 4) ......................... 116
Appendix I: Advantages of Using WriteToLearn ............................................................... 117
Appendix J: Disadvantages of Using WriteToLearn .......................................................... 118
Appendix K: Student Essay Sample “Your Role Model” ................................................. 119
Appendix L: Comments on Essay Sample “Your Role Model” (Graded a 4 in Ideas) . 120

List of Tables
TABLE 1 HIGHEST AND LOWEST SCORES FOR ESSAYS IN EACH GENRE ..................................................................... 37
TABLE 2 AVERAGE SCORE FOR ALL THE ESSAYS IN INDIVIDUAL GENRES .................................................................. 39
TABLE 3 AVERAGE TRAIT SCORES FOR NARRATIVE ESSAYS ....................................................................................... 40
TABLE 4 AVERAGE TRAIT SCORES FOR CAUSE-EFFECT ESSAYS ................................................................................ 41
TABLE 5 AVERAGE TRAIT SCORES FOR COMPARISON AND CONTRAST ESSAYS ............................................................ 41
TABLE 6 AVERAGE TRAIT SCORES FOR ARGUMENT ESSAYS (MAXIMUM 6) ................................................................ 42
TABLE 7 AVERAGE TRAIT SCORES FOR ARGUMENT ESSAYS (MAXIMUM 4) ................................................................. 42
TABLE 8 PARTICIPANTS’ PERCEIVED ADVANTAGES OF WRITETOLEARN ................................................................... 47
TABLE 9 PARTICIPANTS’ PERCEIVED DISADVANTAGES OF WRITETOLEARN ............................................................. 48
List of Figures
FIGURE 1 SCREENSHOT OF WRITETOLEARN’S TOPIC LIST .................................................................................... 21
FIGURE 2 QUESTIONNAIRE (FOR ENGLISH TRANSLATION, PLEASE REFER TO APPENDIX A) ........................................ 22
FIGURE 3 SCREENSHOT OF WRITETOLEARN’S SCOREBOARD ................................................................................. 26
FIGURE 4 SCREENSHOT OF WRITETOLEARN’S TEACHER TOOLS MENU .................................................................. 30
FIGURE 5 STUDENT GRADE COUNTS FOR THE FIRST AND LAST ESSAYS WRITTEN WITH WRITETOLEARN.................. 43
FIGURE 6 DEGREE OF HELP ON IMPROVING WRITING SKILL BY WRITETOLEARN ................................................... 46
FIGURE 7 STUDENT LU’S ESSAY SAMPLE “ESSAY ABOUT SCHOOLS” (FIFTH EDITION) ............................................. 76
FIGURE 8 WRITETOLEARN RESULTS FOR STUDENT LU’S ESSAY SAMPLE “ESSAY ABOUT SCHOOLS” ........................ 77
FIGURE 9 STUDENT LI’S ESSAY SAMPLE “TWO PRODUCTS” (FIRST DRAFT) ............................................................ 78
FIGURE 10 WRITETOLEARN RESULTS FOR STUDENT LI’S ESSAY SAMPLE “TWO PRODUCTS” (REVISED) ................. 79
FIGURE 11 SCORES OF STUDENT WU’S ESSAY – A “FIRST” IN YOUR LIFE ............................................................... 81
FIGURE 12 EXCERPT OF STUDENT WU’S ESSAY – A “FIRST” IN YOUR LIFE (FIRST DRAFT) ...................................... 82
FIGURE 13 EXCERPT OF STUDENT WU’S ESSAY – A “FIRST” IN YOUR LIFE (THIRD VERSION) .................................. 83
FIGURE 14 SCORES OF STUDENT WU’S ESSAY- EFFECTS OF PEER PRESSURE .......................................................... 84
FIGURE 15 EXCERPT OF STUDENT WU’S ESSAY – EFFECTS OF PEER PRESSURE (FIRST DRAFT) ............................... 85
FIGURE 16 EXCERPT OF STUDENT WU’S ESSAY – EFFECTS OF PEER PRESSURE (SECOND VERSION) ........................ 85
FIGURE 17 SCORES OF STUDENT BAI’S ESSAY – ESSAY ABOUT SCHOOLS .................................................................. 86
FIGURE 18 STUDENT BAI’S ESSAY – ESSAY ABOUT SCHOOLS (FIRST DRAFT)) ........................................................... 87
FIGURE 19 STUDENT BAI’S ESSAY – ESSAY ABOUT SCHOOLS (THIRD VERSION) ........................................................ 88
FIGURE 20 WRITETOLEARN SCORECARD SAMPLE ................................................................................................. 90
FIGURE 21 ORGANIZATION SCORE: 4 ..................................................................................................................... 91
FIGURE 22 ORGANIZATION SCORE 4: ESSAY EXAMPLE ............................................................................................ 92
FIGURE 23 SCREENSHOT OF STUDENT TSENG’S ESSAY COMMENTS.......................................................................... 93
FIGURE 24 SCREENSHOT OF “COMPARE AND CONTRAST TWO LITERARY CHARACTERS” PROMPT TEXT .................... 94

References
Baron, D. (2005, May 6). The College Board’s new essay reverses decades of progress toward literacy. The Chronicle of Higher Education, P. B14.
Burstein, J. (2013). Automated Essay Evaluation and Scoring. In Chapelle, C.A. (Ed.), The Encyclopedia of Applied Linguistics (pp.309-315). West Sussex, UK: Wiley-Blackwell.
Calfee, R. (2000). To grade or not to grade. IEEE Intelligent Systems 15(5), 35–37.
Chen, C.-F.E., & Cheng, W.-Y.E. (2008). Beyond the Design of Automated Writing Evaluation: Pedagogical Practices and Perceived Learning Effectiveness in EFL Writing Classes. Language Learning & Technology, 12(3), 94-112.
Condon, W. (2013). Large-Scale Assessment, Locally-Developed Measures, and Automated Scoring of Essays: Fishing for red herrings? Assessing Writing, 18, 100-108.
Deane, P. (2013). On the Relation between Automated Essay Scoring and Modern Views of the Writing Construct. Assessing Writing, 18(1), 7–24.
Deane, P., Williams, F., Weng, V., & Trapani, C. S. (2013). Automated Essay Scoring in Innovative Assessments of Writing from Sources. Journal of Writing Assessment, 6(1).
Elliot, N., Gere, A.R., Gibson, G., Toth, C., Whithaus, C., & Presswood, A. (2013). Uses and Limitations of Automated Writing Evaluation Software. WPA- CompPile Research Bibliographies, 23.
Elliot, N., Perelman, L. (Eds.) (2012). Writing Assessment in the 21st Century: Essays in honor of Edward M. White (pp. 219–232). New York, NY: Hampton Press.
Fang, Y. (2010). Perceptions of the Computer-Assisted Writing Program among EFL College Learners. Educational Technology & Society, 13(3), 246-256.
Foltz, P. W., Rosenstein, M., Lochbaum, K. E., & Davis, L. (2012). Improving Reliability throughout the Automated Scoring Process. Paper presented at the National Council on Measurement in Education Conference, Vancouver, BC.
Foltz, P.W., Laham, D., and Landauer, T.K. (1999). The Intelligent Essay Assessor: Applications to Educational Technology. Interactive Multimedia Electronic Journal of Computer-Enhanced Learning, 1(2). Accessed on http://imej.wfu.edu/articles/1999/2/04/ index.asp.
Grant, L., & Ginther, A. (2000). Using Computer-Tagged Linguistic Features to Describe L2 Writing Differences. Journal of Second Language Writing, 9, 123-145.
Grimes, D., & Warschauer, M. (2010). Utility in a Fallible Tool: A Multi-Site Case Study of Automated Writing Evaluation. The Journal of Technology, 8(6).
Hearst. (2000). The Debate on Automated Essay Grading. Intelligent Systems and Their Applications, 15(5), 22-37.
Hidayati, K. H. (2018). Teaching Writing to EFL Learners: An Investigation of Challenges Confronted by Indonesian Teachers. Langkawi, 4(1), 21-31.
Jarvis, S., Grant, L., Bikowskia, D., & Ferris, D. (2003). Exploring Multiple Profiles of Highly Rated Learner Compositions. Journal of Second Language Writing, 12(4), 377-403.
Landauer, T. K., Laham, D., & Foltz, P. W. (2003). Automated Scoring and Annotation of Essays with the Intelligent Essay Assessor. Automated essay scoring: A cross-disciplinary perspective, 87–112. Mahwah, New Jersey: Lawrence Erlbaum Associates.
Landauer, T. K., Laham, D., & Foltz, P. W. (2003). Automatic Essay Assessment. Assessment in Education, 10, 295-308.
Landauer, T. K., Lockbaum, K. E., & Dooley, S. (2009). A New Formative Assessment Technology for Reading and Writing. Theory into Practice 48(1):44-52. DOI: 10.1080/00405840802577593.
Landauer, T., Laham, D., & Foltz, P. (2000). The Intelligent Essay Assessor. Intelligent Systems, IEEE 15, 27-31.
Li, J., Link, S., & Hegelheimer, V. (2015). Rethinking the Role of Automated Writing Evaluation (AWE) Feedback in ESL Writing Instruction. Journal of Second Language Writing, 27, 1-18.
Liu, S., & Kunnan, A. J. (2016). Investigating the Application of Automated Writing Evaluation to Chinese Undergraduate English Majors: A Case Study of WriteToLearn. Computer Assisted Language Instruction Consortium, 33(1), 71-91.
Lochbaum, K. E., Rosenstein, M., Foltz, P. W., & Derr, M. A. (2013). Detection of Gaming in Automated Scoring of Essays with the IEA. Paper presented at the National Council on Measurement in Education Conference (NCME), San Francisco, CA.
Lu, X. (2019). An Empirical Study on the Artificial Intelligence Writing Evaluation System in China CET. Big Data, 7(2), 121-129. http://doi.org/10.1089/big.2018.0151.
McCurry, D. (2010). Can machine scoring deal with broad and open writing tests as well as human readers? Assessing Writing, 15(2), 118-129.
McNamara, D. S., Crossley, S. A., Roscoe, R. D., Allen, L. K., & Dai, J. (2015). A Hierarchical Classification Approach to Automated Essay Scoring. Assessing Writing, 23, 35-39. http://dx.doi.org/10.1016/j.asw.2014.09.002.
Page, E. B. (1994). Computer Grading of Student Prose Using Modern Concepts and Software. Journal of Experimental Education, 62, 127-142.
Page, E. B. (2003). Project Essay Grade: PEG. In M.D. Shermis & J.C. Burstein (Eds.) Automated Essay Scoring: A Cross-Disciplinary Perspective (pp.43-54). Mahwah. New Jersey: Lawrence Erlbaum Associates.
Pearson Education Inc. (2007). General Overview of WriteToLearn and Its Components. Retrieved from: https://cdn2.hubspot.net/hubfs/559254/WTL/resources/WTL_WhitePaper_GeneralOverview_WTL_Components_r1.pdf?t=1520621384043.
Pearson Education Inc. (2013). WriteToLearn Common Core Report. Retrieved from: http://assets.pearsonschool.com/correlations/Write_to_Learn_CCSS%20Curriculum%20Guide_071013%5B1%5D.pdf.
Pearson Education Inc. (2019). WriteToLearn. Accessed 2019/5/5 on: https://www.pearsonassessments.com/professional-assessments/products/programs/write-to-learn.html?tab=faqs.
Perelman, L. (2014). When ‘the State of the Art’ is counting words. Assessing Writing, 21, 104-111.
Peterson, S. S., & McClay, J. (2010). Assessing and Providing Feedback for Student Writing in Canadian Classrooms. Assessing Writing, 15(2), 86-99.
Powers, D. E., Burstein, J., Chodorow, M. S., Fowles, M. E., & Kukich, K. (2002). Comparing the validity of automated and human scoring of essays. Journal of Educational Computing Research, 26(4), 407-425.
Shermis, M. & Bernstein, J. (2003). Automated Essay Scoring: A Cross-Disciplinary Perspective. Mahwah, NJ: Lawrence Erlbaum Publishers.
Shermis, M.D., Burstein, J.C., & Bliss, L. (2004). The Impact of Automated Essay Scoring on High Stakes Writing Assessments. Paper presented at the Annual Meetings of the National Council on Measurement in Education, San Diego, CA.
Tang, J. & Rich, S. S. (2017). Automated Writing Evaluation in an EFL Setting: Lessons from China. JALT CALL Journal, 13(2), 117-146.
Topol, B., Olson, J., & Roeber, E. (2011). The Cost of New Higher Quality Assessments: A Comprehensive Analysis of the Potential Costs for Future State Assessments. Stanford Center for Opportunity Policy in Education.
Vantage Learning McCann (2019). My Access!® Writing and Assessment Solution: Vantage Learning. Accessed 2019/5/5 on http://www.vantagelearning.com/products/my-access-school-edition/.
Wang, J. & Brown, M.S. (2007). Automated Essay Scoring Versus Human Scoring: A Comparative Study. Journal of Technology, Learning, and Assessment, 6(2). Retrieved [2018/05/21] from http://www.jtla.org.
Wang, J., & Brown, M. S. (2008). Automated Essay Scoring Versus Human Scoring: A Correlational Study. Contemporary Issues in Technology and Teacher Education, 8(4), 310-325.
Wang, P. (2015). Effects of an Automated Writing Evaluation Program: Student Experiences and Perceptions. Electronic Journal of Foreign Language Teaching, 12(1), 79-100.
Wang, W. & Wen, Q. (2002). L1 use in the L2 composing process: An exploratory study of 16 Chinese EFL writers. Journal of Second Language Writing, 11(3), 225-246.
Warschauer, M., & Grimes, D. (2008). Automated Writing Assessment in the Classroom. Pedagogies, 3(1).
Warschauer, M., & Ware, P. (2006). Automated Writing Evaluation: Defining the Classroom Research Agenda. Language teaching research, 10 (2), 157–180. http://dx.doi.org/10.1191/1362168806lr190oa.
Weigle, S. C. (2013). English Language Learners and Automated Scoring of Essays: Critical considerations. Assessing Writing, 18, 85–99. http://dx.doi.org/10.1016/j.asw.2012.10.006.
Williamson, D. M., Bennett, R., Lazer, S., Bernstein, J., Foltz, P. W., Landauer, T. K., Rubin, D., Way, D., & Sweeney, K. (2010). Automated Scoring for the Assessment of Common Core Standards.
Yang, Y., Buckendahl, C.W., Juszkiewicz, P.J., & Bhola, D.S. (2002). A Review of Strategies for Validating Computer Automated Scoring. Applied Measurement in Education, 15(4), 391–412.