績效評估矩陣是透過問卷，從使用者或顧客端，了解系統之運作 績效，進而找出須改善的要項以提升系統運作績效。由於問卷調查有 抽樣誤差與受訪者模糊語意資料蒐集的複雜度的問題，本文藉由提出 一判別指標，同時應用統計推論推導判別指標信賴區間並參考Buckley 的模糊檢定方法建構模糊隸屬函數，進而提出模糊評估準則，找出績效評估矩陣(performance evaluation matrix)內的關鍵改善服務要項。

本文所提方法的優點乃是維持李克特量表的簡單填答模式，維持 資料蒐集的效能，接著透過統計推論和模糊檢定解決抽樣誤差和降低 模糊不確定性的影響，並以數位學習系統(E-learning System)之電腦輔 助語言教學系統(Computer-assisted language learning system, CALL System)為研究案例來說明本文所提方法的應用。

而為了驗證績效改善的成效，本文進一步發展一個驗證模式，並 使用數值案例來說明所提出的驗證模型的應用。

Performance evaluation matrix based on questionnaires collected from users to pinpoint the performance of system operation, and further to locate the items to improve for upgrading the performances of system. To achieve this, this dissertation proposes a discrimination index and applies statistics inference to deduce confidence intervals of discrimination index. Meanwhile, we refer to Buckley’s fuzzy testing method to construct fuzzy membership function and address fuzzy evaluation criterion for exploring the items considered critical to quality to overcome the complicated problems of the sampling error and interviewees’ fuzzy linguistics.

The advantages of the method in this dissertation are to keep simple filling pattern of Likert’s scale and efficacy of data collection. Subsequently, we reduce fuzzy linguistics and sampling error by statistics inference and fuzzy hypothesis testing.

Then,   we   use   E-learning   System   as   case   study   with   the computer-assisted language learning system (CALL system) to demonstrate application of the proposed method. In order to confirm the effectiveness of the performance improvement, this dissertation further develops a verification   model   and   uses   a   numerical   example   to   demonstrate application of the proposed verification model.

Table of Contents	III
Table of Figures	V
Table of Tables	VI
Chapter 1 Introduction	1
1.1	Research Motivation	1
1.2	Research Issues	2
1.3	Research Objectives	3
1.4	Research Framework and Organization	5
Chapter 2 Literature Review	7
2.1	Measurement of Online Learning Service Quality	7
2.2	Likert Scale	8
2.3	Performance Evaluation Matrix	10
2.4	Web-based E-learning System	11
2.5	Computer-assisted Language Learning	13
Chapter 3 Development and Application of a Performance Evaluation Model	15
3.1	Evaluating the Critical to Quality through PEM	15
3.2	Performance Indices and Performance Evaluation Matrix	18
3.3	Fuzzy Estimator	21
3.4	Fuzzy Hypothesis Testing	27
3.5	Case Study	32
Chapter 4 Construction and Application of Performance Improvement Verification Method	37
4.1	The Fuzzy Estimation on the Difference of Customer Satisfaction Index Before and After Improvement	37
4.2	Improvement Verification Method	42
4.3	Numerical Example	46
Chapter 5 Conclusions and Future Research	49
5.1	Conclusions	49
5.2	Future research	51
References	53
Appendix: Notation	64




Table of Figures
Figure 1 Research framework  	6
Figure 2 Matrix of performance evaluation	19
Figure 3  	27
Figure 4 The power of the test	29
Figure 5   and curving triangular fuzzy figure of  	30
Figure 6  	44
Figure 7  	48
Figure 8 Flow chart of the performance evaluation and improvement verification method	51


Table of Tables
Table 1 Importance & Satisfaction Survey for CALL System	34

Allen, E. & Seaman, C. (2007). Likert scales and data analyses. Quality Progress, 40(7), 64-65.

Altuntas, S., Dereli, T. & Yilmaz, M. K. (2012). Multi-criteria decision making methods based weighted SERVQUAL scales to measure perceived service quality in hospitals: A case study from Turkey. Total  Quality  Management  and  Business  Excellence,  23(11-12),
1379-1395.

Amaral, L. A. & Meurers, D. (2006). Where does ICALL fit into foreign language   teaching?   Talk   at CALICO.   University   of   Hawaii. Retrieved December 27, 2017 from http://purl.org/net/icall/handouts/calico06- amaral-meurers.pdf

Amaral,   L.   A.   &   Meurers,   D.   (2011).   On   using   intelligent computer-assisted language learning in real-life foreign language teaching and learning. ReCALL, 23(1), 4-24.

Bowersox,  D.  J.  &  Closs,  D.  J.  (1996). Logistical  management. The Integrated Supply Chain Process. McGraw-Hill, New York.

Bradley, R. A., Katti, S. K. & Coons, I. J. (1962). Optimal scaling for ordered categories. Psychometrika, 27(4), 355-374.

Buckley, J. J. (2005). Fuzzy statistics: Hypothesis testing. Soft Computing,9, 512-518.
 
Carswell, A. D. & Venkatesh, V. (2002). Learner outcomes in an asynchronous distance education environment. International Journal of Human–Computer Studies, 56, 475-494.

Chandran, D. & Kempegowda, S. (2010). Hybrid e-learning platform based on cloud architecture model: A proposal. Proc. International Conference on Signal and Image Processing (ICSIP), 534-537.

Chang, L. (1994). A psychometric evaluation of 4-poing and 6-point Likert-type scales in relation to reliability and validity. Applied Psychological Measurement, 18(3), 205-215.

Chen, H. C., Chang, T. C., Lin, Y. T., Cheng, H. Y., & Chang, T. L. (2013). Application of DMAIC process to enhance health effects in caring institution. Quality & Quantity, 47(4), 2065-2080.

Chen, H. C., Chen, K. S., Chang, T. L., & Hsu, C. H. (2010). An application of six sigma methodology to enhance leisure service quality. Quality and Quantity, 44(6), 1151-1164.

Chen, K. S. & Chen, H. T. (2014). Applying importance-performance analysis with simple regression model and priority indices to assess hotels’ service performance. Journal of Testing and Evaluation, 42(2),455-466.

Chen, K. S., Chen, H. T., & Lin, C. L. (2012). Applying a revised SQPM in the define step of six sigma and a case study. Total Quality Management & Business Excellence, 23(3), 309-324.

Chen, S. H. (2009). Establishment of a performance-evaluation model for service  quality  in  the  banking  industry.  The  Service  Industries Journal, 29(2), 235-247.

Chen, S. H. (2012). Integrating service quality evaluation model to improve employees' satisfaction for high-tech iindustry. Human Factors and Ergonomics in Manufacturing  & Service Industries, 22(6), 517-527.

Chen, S. H. (2013). Devising appropriate service strategies for customers of different value: An integrated assessment model for the banking industry. International Journal of Human Resource Management, 24(21), 3939-3956.

Chen, K. S., Chang, H. T., & Yu, C. M. (2019). Development and application of performance improvement verification model: A case study of an e-learning system. Total Quality Management & Business Excellence, 30(7-8), 936-952.

Chiou, W. C., Lin, C. C., & Perng, C. (2011). A strategic website evaluation of online travel agencies. Tourism Management, 32(6), 1463-1473.

Churchill, Jr. G. A. & Surprenant, C. (1982). An investigation into the determinants   of   customer   satisfaction.   Journal   of   Marketing Research (JMR), 19, 491-504.

DeLone, W. H. & McLean, E.  R. (2003). The DeLone and McLean model of information systems success: A ten-year update. Journal of Management Information Systems, 19(4), 9-30.
 
DeVellis, R. F. (2003). Scale development: Theory and applications (2nd ed.). Newbury Park, CA: Sage.

Dickinson, M. & Herring, J. (2008). Developing online ICALL exercises for Russian. Proceedings of the Third Workshop on Innovative Use of NLP for Building Educational Applications (pp. 1-9), Columbus, OH.

Etzel,  M.  J.,  Walker,  B.  J.  &  Stanton,  W.  J.  (2001).  Marketing management (12th ed.). Boston, MA: McGraw Hill-Irwin.

Fornell,  C.  (1992).  A  national  customer  satisfaction  barometer:  The Swedish experience. Journal of Marketing, 56(1), 6-21.

Gamper, J. & Knapp, J. (2002). A review of intelligent CALL systems. Computer Assisted Language Learning, 15(4), 329–342.

Gelderman, M. (1998). The relation between user satisfaction, usage of information systems and performance. Information and Management, 34(1), 11–18.

Global Market Insights (2017). E-Learning Market Size By Application (SMBs,  Large  Enterprises,  Education),  By  Technology  (Online e-learning, Learning Management System (LMS), Mobile e-learning, Rapid   e-learning,   Virtual   Classroom),   By   Provider   (Service, Content), Industry Analysis Report, Regional Outlook (U.S., Canada, UK, Germany, France, Italy, Russia, Spain, China, India, Japan, Australia, Brazil, Mexico, GCC, South Africa), Growth Potential,Price Trends, Competitive Market Share & Forecast, 2017 – 2024, Retrieved  from https://www.gminsights.com/industry-analysis/elearning-market-size

Grewal, A., Rai, S., Phillips, R., Fung, C. C. (2005). The e-learning lifecycle and its services: the web services approach. Proceedings of the  Second  International  Conference  on  E-Learning  for Knowledge-Based Society, 8, 4.1-4.8.

Grönroos, C. (2001). The perceived service quality concept – a mistake?. Managing Service Quality, 11(3), 150-152.

Hamburg, H. & Lindecke, C. (2003). Social aspects of e-learning and blending learning methods. Proceedings of the 4th European Conference E-Comm-Line, Bucharest, Romania.

Hodge, D. R. & Gillespie, D. (2003). Phrase completions: An alternative to Likert scales. Social Work Research, 27(1), 45-55.

Hooper, D., Coughlan J. & Mullen M. R. (2008). Structural Equation Modelling: Guidelines for Determining Model Fit,” Electronic Journal of Business Research Methods, 6(1), 53-60.

Hossain, M. J. & Ahmed, S. M. Z. (2013). Developing a service performance assessment system to improve service quality of academic libraries, Business Information Review, 30(4), 210-221.

Hsu, T. H., Chu, K. M. & Chan, H. (2001). A Study on fuzzy linguistic scale. Journal of Business dministration, 51, 27-51.
 
Hung, Y. H., Huang, M. L. & Chen, K. S. (2003). Service quality evaluation by service quality performance matrix. Total Quality Management & Business Excellence, 14(1), 79-89.

Hwang, G. J. & Tsai, C. C. (2011). Research trend in mobile and ubiquitous learning: a review of publications in selected journal from 2001  to  2010,  British  Journal  of  Education  Technology,  42(4), E65-E70.

Jamieson,  S.  (2004).  Likert  scales:  How  to  (ab)use  them.  Medical Education, 38(12), 1217-1218.

Jiang, J. J., Klein, G., Roan, J., & Lin, J. T. M. (2001). IS service performance: Self-perceptions and user perceptions. Information & Management, 38(8), 499-506

Kao, C. (1998). Performance of several nonlinear programming software packages on microcomputers. Computers & Operations Research, 25(10), 807–816.

Karat,  J.  (1988).  Software  evaluation  methodologies.  In  M. Helander (Ed.), Handbook of human – computer interaction (pp. 891–903). Amsterdam, Netherland: Elsevier Science.

Koivumaki,  T.,  Ristola,  A.,  &  Kesti,  M.  (2008).  The  effects  of information quality of mobile information services on user satisfaction and service acceptance-empirical evidence from Finland. Behaviour & Information Technology, 27(5), 375-385.
 
Kotler, P. (2000). Marketing Management: Analysis, Planning, Implementation and Control (10th ed.). Englewood Cliffs, New Jersey: Prentice-Hall (2000)

Kyj,  L.  S.  &  &  Kyj,  M.  J.  (1994).  Customer   service:  product fifferentiation in international markets. International Journal of Physical Distribution & Logistics Management, 24(4), 41-49.

Lalonde, B. J. & Zinszer, P. H. (1976). Customer Service: Meaning and Measurement. National Council of Physical Distribution Management, Chicago.

Lambert, D. M. & Sharma, A. (1990). A customer-based competitive analysis for logistics decisions. International Journal of Physical Distribution and Logistics Management, 20(1), 17-24.

Leung, S. O. (2011). A comparison of psychometric properties and normality in 4-, 5-, 6-, and 11-point likert scales. Journal of Social Service Research, 37(4), 412-421

Li, Q. (2013). A novel Likert scale based on fuzzy sets theory. Expert Systems with Applications, 40(5), 1609-1618.

Likert, R. (1932). A technique for the measurement of attitudes. Archives of Psychology, 140, 1-55.

Lin, C. S., Wu, S., & Tsai, R. J. (2005). Integrating perceived playfulness into expectation-confirmation model for web portal context. Information & Management, 42(5), 683-693.
 
Lin, H.-H. & Wang, Y.-S. (2006). An examination of the determinants of customer loyalty in mobile commerce contexts. Information & Management, 43(3), 271-282.

Lu, J., Yu, C. S., Liu, C. (2003). Learning style, learning patterns, and learning performance in a WebCT-based MIS course. Information & Management, 40, 497-507.

Manhas, P. S. & Tukamushaba, E. K. (2015). Understanding service experience and its impact on brand image in hospitality sector. International Journal of Hospitality Management, 45, 77-87.

Markovic´, S. & Jankovic´, S. R. (2013). Exploring the relationship between service quality and customer satisfaction in Croatian hotel industry. Tourism and Hospitality Management, 19(2), 149-164.

Masud, M. A. H. & Huang, X. (2012). An e-learning system architecture based on cloud computing. World Academy of Science, Engineering and Technology, 62, 74-78.

Mazis, M. B., Antola, O. T., & Klippel, R. E. (1975). A comparison of four multi-attribute models in the prediction of consumer attitudes. Journal of Consumer Research, 2(1), 38-52.

Melone, N. P. (1990). A theoretical assessment of the user-satisfaction construct  in  information  systems  research.  Management  Science, 36(1), 76-91.

Morris, L., Wu, S. & Finnegan, C. (2005). Predicting retention in online general  education  courses.  The  American  Journal  of  Distance Education, 19, 23-36.

Orvik, J. M. (1972). Social desirability for the individual, his group, and society. Multivariate Behavioral Research, 7, 3-32.

Parasuraman, A., Zeithaml, V. A. & Berry, L. L. (1985). A conceptual model of service quality and its implications for future research. Journal of Marketing, 49, 41-50.

Petrides, L. A. (2002). Web-based technologies for distributed (or distance)  learning:  creating  learning-centered  educational experiences in the higher education classroom. International Journal of Instructional Media, 29(1), 69-77.

Ray, J. J. (1990). Acquiescence and problems with forced-choice scale. Journal of Social Psychology, 130(3), 397-399.

Raymond, L. (1987). Validating and applying user satisfaction as a measure of MIS success in small organization. Information and Management, 12(4), 173-179.

Ribbink, D., Riel, A., Liljander, V., & Streukens, S. (2004). Comfort your online customer: Quality, trust and loyalty on the internet. Managing Service Quality, 14(6), 446-456.

Russell, C. J. & Bobko, P. (1992). Moderated regression analysis and Likert scales: Too coarse for comfort. Journal of Applied Psychology, 77(3), 336-342.
 
Shee,  D.  Y.  &  Wang,  Y.  S.  (2008).  Multi-criteria  evaluation  of  the web-based e-learning system: A methodology based on learner satisfaction  and  its  applications.  Computers  &  Education,  50(3),
894-905.

Smith,  C.  U.  &  Williams,  L.  G.  (1999).  A  performance  model interchange format. Journal of Systems and Software, 49(1), 63-80.

Sureshchandar, G. S., Rajendran, C., & Anantharaman, R. N. (2002).Determinants of customer‐ perceived service quality: A confirmatory factor analysis approach. Journal of Services Marketing, 16(1), 9-34.

Tao, C. J., Chen, S. C. & Chang, L. (2009). Apply 6-sigma methodology in measuring the competition quality of satisfaction performance – an example of ISP Industry. Quality & Quantity, 43(4), 677-694.

Wang, K. J., Chang, T. C., & Chen, K. S. (2015). Determining critical service quality from the view of performance influence. Total Quality Management & Business Excellence, 26(3-4), 368-384.

Wang, A. Y. & Newlin, M. H. (2002). Predictors of web-student performance:  The  role  f  self-efficacy  and  reasons  for  taking  an on-line class. Computers in Human Behavior, 18, 151-163.

Whitten, J. L. & Bentley, L. D. (2004). System analysis & design methods. New York: McGraw-Hill.

Wolfinbarger, M. & Gilly, M. C. (2003). ETailQ: Dimensionalizing, measuring and predicting etail quality. Journal of Retailing, 79(3), 183-198.
 
Woodruff,  R.,  Cadotte,  E.  R.,  &  Jenkins,  R.  L.  (1983).  Modeling consumer satisfaction processes using experience-based norms. Journal of Marketing Research, 20(3), 296-304.

Wu, W. H., Jim Wu, Y. C., Chen, C. Y., Kao, H. Y., Lin, C. H., & Huang, S. H. (2012). Review of trends from mobile learning studies: A meta-analysis. Computers & Education, 59(2), 817-827.

Yu,  C.  M.,  Chang,  H.  T.,  &  Chen,  K.  S.  (2018).  Developing  a performance evaluation matrix to enhance the learner satisfaction of an	e-learning  system.  Total  Quality  Management  &  Business Excellence, 29(7-8), 727-745.

Yu, C. M., Chang, T. C., & Hu, I. H. (2018). Developing a discriminant index to determine critical service attributes of continuous performance  improvement.  Journal  of  Service  Science  Research,
10(2), 145-165.