||The Construction and Application of Performance Evaluation and Improvement Verification Method Based on the Fuzzy Nature of Linguistic Information: A Case Study of E-Learning System
||Doctoral Program, Department of Management Sciences
fuzzy hypothesis testing
improvement verification model
||績效評估矩陣是透過問卷，從使用者或顧客端，了解系統之運作 績效，進而找出須改善的要項以提升系統運作績效。由於問卷調查有 抽樣誤差與受訪者模糊語意資料蒐集的複雜度的問題，本文藉由提出 一判別指標，同時應用統計推論推導判別指標信賴區間並參考Buckley 的模糊檢定方法建構模糊隸屬函數，進而提出模糊評估準則，找出績效評估矩陣(performance evaluation matrix)內的關鍵改善服務要項。
本文所提方法的優點乃是維持李克特量表的簡單填答模式，維持 資料蒐集的效能，接著透過統計推論和模糊檢定解決抽樣誤差和降低 模糊不確定性的影響，並以數位學習系統(E-learning System)之電腦輔 助語言教學系統(Computer-assisted language learning system, CALL System)為研究案例來說明本文所提方法的應用。
||Performance evaluation matrix based on questionnaires collected from users to pinpoint the performance of system operation, and further to locate the items to improve for upgrading the performances of system. To achieve this, this dissertation proposes a discrimination index and applies statistics inference to deduce confidence intervals of discrimination index. Meanwhile, we refer to Buckley’s fuzzy testing method to construct fuzzy membership function and address fuzzy evaluation criterion for exploring the items considered critical to quality to overcome the complicated problems of the sampling error and interviewees’ fuzzy linguistics.
The advantages of the method in this dissertation are to keep simple filling pattern of Likert’s scale and efficacy of data collection. Subsequently, we reduce fuzzy linguistics and sampling error by statistics inference and fuzzy hypothesis testing.
Then, we use E-learning System as case study with the computer-assisted language learning system (CALL system) to demonstrate application of the proposed method. In order to confirm the effectiveness of the performance improvement, this dissertation further develops a verification model and uses a numerical example to demonstrate application of the proposed verification model.
||Table of Contents III
Table of Figures V
Table of Tables VI
Chapter 1 Introduction 1
1.1 Research Motivation 1
1.2 Research Issues 2
1.3 Research Objectives 3
1.4 Research Framework and Organization 5
Chapter 2 Literature Review 7
2.1 Measurement of Online Learning Service Quality 7
2.2 Likert Scale 8
2.3 Performance Evaluation Matrix 10
2.4 Web-based E-learning System 11
2.5 Computer-assisted Language Learning 13
Chapter 3 Development and Application of a Performance Evaluation Model 15
3.1 Evaluating the Critical to Quality through PEM 15
3.2 Performance Indices and Performance Evaluation Matrix 18
3.3 Fuzzy Estimator 21
3.4 Fuzzy Hypothesis Testing 27
3.5 Case Study 32
Chapter 4 Construction and Application of Performance Improvement Verification Method 37
4.1 The Fuzzy Estimation on the Difference of Customer Satisfaction Index Before and After Improvement 37
4.2 Improvement Verification Method 42
4.3 Numerical Example 46
Chapter 5 Conclusions and Future Research 49
5.1 Conclusions 49
5.2 Future research 51
Appendix: Notation 64
Table of Figures
Figure 1 Research framework 6
Figure 2 Matrix of performance evaluation 19
Figure 3 27
Figure 4 The power of the test 29
Figure 5 and curving triangular fuzzy figure of 30
Figure 6 44
Figure 7 48
Figure 8 Flow chart of the performance evaluation and improvement verification method 51
Table of Tables
Table 1 Importance & Satisfaction Survey for CALL System 34
||Allen, E. & Seaman, C. (2007). Likert scales and data analyses. Quality Progress, 40(7), 64-65.
Altuntas, S., Dereli, T. & Yilmaz, M. K. (2012). Multi-criteria decision making methods based weighted SERVQUAL scales to measure perceived service quality in hospitals: A case study from Turkey. Total Quality Management and Business Excellence, 23(11-12),
Amaral, L. A. & Meurers, D. (2006). Where does ICALL fit into foreign language teaching? Talk at CALICO. University of Hawaii. Retrieved December 27, 2017 from http://purl.org/net/icall/handouts/calico06- amaral-meurers.pdf
Amaral, L. A. & Meurers, D. (2011). On using intelligent computer-assisted language learning in real-life foreign language teaching and learning. ReCALL, 23(1), 4-24.
Bowersox, D. J. & Closs, D. J. (1996). Logistical management. The Integrated Supply Chain Process. McGraw-Hill, New York.
Bradley, R. A., Katti, S. K. & Coons, I. J. (1962). Optimal scaling for ordered categories. Psychometrika, 27(4), 355-374.
Buckley, J. J. (2005). Fuzzy statistics: Hypothesis testing. Soft Computing,9, 512-518.
Carswell, A. D. & Venkatesh, V. (2002). Learner outcomes in an asynchronous distance education environment. International Journal of Human–Computer Studies, 56, 475-494.
Chandran, D. & Kempegowda, S. (2010). Hybrid e-learning platform based on cloud architecture model: A proposal. Proc. International Conference on Signal and Image Processing (ICSIP), 534-537.
Chang, L. (1994). A psychometric evaluation of 4-poing and 6-point Likert-type scales in relation to reliability and validity. Applied Psychological Measurement, 18(3), 205-215.
Chen, H. C., Chang, T. C., Lin, Y. T., Cheng, H. Y., & Chang, T. L. (2013). Application of DMAIC process to enhance health effects in caring institution. Quality & Quantity, 47(4), 2065-2080.
Chen, H. C., Chen, K. S., Chang, T. L., & Hsu, C. H. (2010). An application of six sigma methodology to enhance leisure service quality. Quality and Quantity, 44(6), 1151-1164.
Chen, K. S. & Chen, H. T. (2014). Applying importance-performance analysis with simple regression model and priority indices to assess hotels’ service performance. Journal of Testing and Evaluation, 42(2),455-466.
Chen, K. S., Chen, H. T., & Lin, C. L. (2012). Applying a revised SQPM in the define step of six sigma and a case study. Total Quality Management & Business Excellence, 23(3), 309-324.
Chen, S. H. (2009). Establishment of a performance-evaluation model for service quality in the banking industry. The Service Industries Journal, 29(2), 235-247.
Chen, S. H. (2012). Integrating service quality evaluation model to improve employees' satisfaction for high-tech iindustry. Human Factors and Ergonomics in Manufacturing & Service Industries, 22(6), 517-527.
Chen, S. H. (2013). Devising appropriate service strategies for customers of different value: An integrated assessment model for the banking industry. International Journal of Human Resource Management, 24(21), 3939-3956.
Chen, K. S., Chang, H. T., & Yu, C. M. (2019). Development and application of performance improvement verification model: A case study of an e-learning system. Total Quality Management & Business Excellence, 30(7-8), 936-952.
Chiou, W. C., Lin, C. C., & Perng, C. (2011). A strategic website evaluation of online travel agencies. Tourism Management, 32(6), 1463-1473.
Churchill, Jr. G. A. & Surprenant, C. (1982). An investigation into the determinants of customer satisfaction. Journal of Marketing Research (JMR), 19, 491-504.
DeLone, W. H. & McLean, E. R. (2003). The DeLone and McLean model of information systems success: A ten-year update. Journal of Management Information Systems, 19(4), 9-30.
DeVellis, R. F. (2003). Scale development: Theory and applications (2nd ed.). Newbury Park, CA: Sage.
Dickinson, M. & Herring, J. (2008). Developing online ICALL exercises for Russian. Proceedings of the Third Workshop on Innovative Use of NLP for Building Educational Applications (pp. 1-9), Columbus, OH.
Etzel, M. J., Walker, B. J. & Stanton, W. J. (2001). Marketing management (12th ed.). Boston, MA: McGraw Hill-Irwin.
Fornell, C. (1992). A national customer satisfaction barometer: The Swedish experience. Journal of Marketing, 56(1), 6-21.
Gamper, J. & Knapp, J. (2002). A review of intelligent CALL systems. Computer Assisted Language Learning, 15(4), 329–342.
Gelderman, M. (1998). The relation between user satisfaction, usage of information systems and performance. Information and Management, 34(1), 11–18.
Global Market Insights (2017). E-Learning Market Size By Application (SMBs, Large Enterprises, Education), By Technology (Online e-learning, Learning Management System (LMS), Mobile e-learning, Rapid e-learning, Virtual Classroom), By Provider (Service, Content), Industry Analysis Report, Regional Outlook (U.S., Canada, UK, Germany, France, Italy, Russia, Spain, China, India, Japan, Australia, Brazil, Mexico, GCC, South Africa), Growth Potential,Price Trends, Competitive Market Share & Forecast, 2017 – 2024, Retrieved from https://www.gminsights.com/industry-analysis/elearning-market-size
Grewal, A., Rai, S., Phillips, R., Fung, C. C. (2005). The e-learning lifecycle and its services: the web services approach. Proceedings of the Second International Conference on E-Learning for Knowledge-Based Society, 8, 4.1-4.8.
Grönroos, C. (2001). The perceived service quality concept – a mistake?. Managing Service Quality, 11(3), 150-152.
Hamburg, H. & Lindecke, C. (2003). Social aspects of e-learning and blending learning methods. Proceedings of the 4th European Conference E-Comm-Line, Bucharest, Romania.
Hodge, D. R. & Gillespie, D. (2003). Phrase completions: An alternative to Likert scales. Social Work Research, 27(1), 45-55.
Hooper, D., Coughlan J. & Mullen M. R. (2008). Structural Equation Modelling: Guidelines for Determining Model Fit,” Electronic Journal of Business Research Methods, 6(1), 53-60.
Hossain, M. J. & Ahmed, S. M. Z. (2013). Developing a service performance assessment system to improve service quality of academic libraries, Business Information Review, 30(4), 210-221.
Hsu, T. H., Chu, K. M. & Chan, H. (2001). A Study on fuzzy linguistic scale. Journal of Business dministration, 51, 27-51.
Hung, Y. H., Huang, M. L. & Chen, K. S. (2003). Service quality evaluation by service quality performance matrix. Total Quality Management & Business Excellence, 14(1), 79-89.
Hwang, G. J. & Tsai, C. C. (2011). Research trend in mobile and ubiquitous learning: a review of publications in selected journal from 2001 to 2010, British Journal of Education Technology, 42(4), E65-E70.
Jamieson, S. (2004). Likert scales: How to (ab)use them. Medical Education, 38(12), 1217-1218.
Jiang, J. J., Klein, G., Roan, J., & Lin, J. T. M. (2001). IS service performance: Self-perceptions and user perceptions. Information & Management, 38(8), 499-506
Kao, C. (1998). Performance of several nonlinear programming software packages on microcomputers. Computers & Operations Research, 25(10), 807–816.
Karat, J. (1988). Software evaluation methodologies. In M. Helander (Ed.), Handbook of human – computer interaction (pp. 891–903). Amsterdam, Netherland: Elsevier Science.
Koivumaki, T., Ristola, A., & Kesti, M. (2008). The effects of information quality of mobile information services on user satisfaction and service acceptance-empirical evidence from Finland. Behaviour & Information Technology, 27(5), 375-385.
Kotler, P. (2000). Marketing Management: Analysis, Planning, Implementation and Control (10th ed.). Englewood Cliffs, New Jersey: Prentice-Hall (2000)
Kyj, L. S. & & Kyj, M. J. (1994). Customer service: product fifferentiation in international markets. International Journal of Physical Distribution & Logistics Management, 24(4), 41-49.
Lalonde, B. J. & Zinszer, P. H. (1976). Customer Service: Meaning and Measurement. National Council of Physical Distribution Management, Chicago.
Lambert, D. M. & Sharma, A. (1990). A customer-based competitive analysis for logistics decisions. International Journal of Physical Distribution and Logistics Management, 20(1), 17-24.
Leung, S. O. (2011). A comparison of psychometric properties and normality in 4-, 5-, 6-, and 11-point likert scales. Journal of Social Service Research, 37(4), 412-421
Li, Q. (2013). A novel Likert scale based on fuzzy sets theory. Expert Systems with Applications, 40(5), 1609-1618.
Likert, R. (1932). A technique for the measurement of attitudes. Archives of Psychology, 140, 1-55.
Lin, C. S., Wu, S., & Tsai, R. J. (2005). Integrating perceived playfulness into expectation-confirmation model for web portal context. Information & Management, 42(5), 683-693.
Lin, H.-H. & Wang, Y.-S. (2006). An examination of the determinants of customer loyalty in mobile commerce contexts. Information & Management, 43(3), 271-282.
Lu, J., Yu, C. S., Liu, C. (2003). Learning style, learning patterns, and learning performance in a WebCT-based MIS course. Information & Management, 40, 497-507.
Manhas, P. S. & Tukamushaba, E. K. (2015). Understanding service experience and its impact on brand image in hospitality sector. International Journal of Hospitality Management, 45, 77-87.
Markovic´, S. & Jankovic´, S. R. (2013). Exploring the relationship between service quality and customer satisfaction in Croatian hotel industry. Tourism and Hospitality Management, 19(2), 149-164.
Masud, M. A. H. & Huang, X. (2012). An e-learning system architecture based on cloud computing. World Academy of Science, Engineering and Technology, 62, 74-78.
Mazis, M. B., Antola, O. T., & Klippel, R. E. (1975). A comparison of four multi-attribute models in the prediction of consumer attitudes. Journal of Consumer Research, 2(1), 38-52.
Melone, N. P. (1990). A theoretical assessment of the user-satisfaction construct in information systems research. Management Science, 36(1), 76-91.
Morris, L., Wu, S. & Finnegan, C. (2005). Predicting retention in online general education courses. The American Journal of Distance Education, 19, 23-36.
Orvik, J. M. (1972). Social desirability for the individual, his group, and society. Multivariate Behavioral Research, 7, 3-32.
Parasuraman, A., Zeithaml, V. A. & Berry, L. L. (1985). A conceptual model of service quality and its implications for future research. Journal of Marketing, 49, 41-50.
Petrides, L. A. (2002). Web-based technologies for distributed (or distance) learning: creating learning-centered educational experiences in the higher education classroom. International Journal of Instructional Media, 29(1), 69-77.
Ray, J. J. (1990). Acquiescence and problems with forced-choice scale. Journal of Social Psychology, 130(3), 397-399.
Raymond, L. (1987). Validating and applying user satisfaction as a measure of MIS success in small organization. Information and Management, 12(4), 173-179.
Ribbink, D., Riel, A., Liljander, V., & Streukens, S. (2004). Comfort your online customer: Quality, trust and loyalty on the internet. Managing Service Quality, 14(6), 446-456.
Russell, C. J. & Bobko, P. (1992). Moderated regression analysis and Likert scales: Too coarse for comfort. Journal of Applied Psychology, 77(3), 336-342.
Shee, D. Y. & Wang, Y. S. (2008). Multi-criteria evaluation of the web-based e-learning system: A methodology based on learner satisfaction and its applications. Computers & Education, 50(3),
Smith, C. U. & Williams, L. G. (1999). A performance model interchange format. Journal of Systems and Software, 49(1), 63-80.
Sureshchandar, G. S., Rajendran, C., & Anantharaman, R. N. (2002).Determinants of customer‐ perceived service quality: A confirmatory factor analysis approach. Journal of Services Marketing, 16(1), 9-34.
Tao, C. J., Chen, S. C. & Chang, L. (2009). Apply 6-sigma methodology in measuring the competition quality of satisfaction performance – an example of ISP Industry. Quality & Quantity, 43(4), 677-694.
Wang, K. J., Chang, T. C., & Chen, K. S. (2015). Determining critical service quality from the view of performance influence. Total Quality Management & Business Excellence, 26(3-4), 368-384.
Wang, A. Y. & Newlin, M. H. (2002). Predictors of web-student performance: The role f self-efficacy and reasons for taking an on-line class. Computers in Human Behavior, 18, 151-163.
Whitten, J. L. & Bentley, L. D. (2004). System analysis & design methods. New York: McGraw-Hill.
Wolfinbarger, M. & Gilly, M. C. (2003). ETailQ: Dimensionalizing, measuring and predicting etail quality. Journal of Retailing, 79(3), 183-198.
Woodruff, R., Cadotte, E. R., & Jenkins, R. L. (1983). Modeling consumer satisfaction processes using experience-based norms. Journal of Marketing Research, 20(3), 296-304.
Wu, W. H., Jim Wu, Y. C., Chen, C. Y., Kao, H. Y., Lin, C. H., & Huang, S. H. (2012). Review of trends from mobile learning studies: A meta-analysis. Computers & Education, 59(2), 817-827.
Yu, C. M., Chang, H. T., & Chen, K. S. (2018). Developing a performance evaluation matrix to enhance the learner satisfaction of an e-learning system. Total Quality Management & Business Excellence, 29(7-8), 727-745.
Yu, C. M., Chang, T. C., & Hu, I. H. (2018). Developing a discriminant index to determine critical service attributes of continuous performance improvement. Journal of Service Science Research,