在可解釋人工智慧（Explainable AI, XAI）領域中，提升人工智慧決策過程的
理解度與透明度至關重要，特別是在信任感為核心需求的敏感應用場景中。本
研究旨在探討影像處理任務中，稀疏模型（Sparse Models）與密集模型（Dense
Models）之間的比較分析證實了在影像分類任務中，透過適當的模型設計，可以
在保持效能的同時滿足透明度的基本需求。

In Explainable Artificial Intelligence (XAI), understanding and transparency in AI decision-making processes are critical, especially in sensitive applications where trust is essential. This research proposal aims to conduct a comparative analysis of sparse and dense models within the context of image processing tasks, using the well-known ImageNet dataset. We compare dense and sparse models for complexity, accuracy, and interpretability, focusing on the trade-offs between these factors. Dense models, represented by advanced architectures like deep neural networks, deliver high accuracy but can lack transparency due to their complex structures. In contrast, sparse models, known for their simplicity and fewer parameters, potentially offer greater interpretability. This study utilizes Neural Concept Activation Vectors (NCAVs) to introduce an interpretable XAI layer. NCAVs extract meaningful concepts from images, which are then used to train interpretable sparse models using techniques like Lasso. The effectiveness of these models is evaluated based on F1 scores and accuracy. Our results indicate that while dense models generally achieve higher accuracy, sparse models trained on extracted concepts offer competitive performance with significantly improved interpretability. This research contributes significant insights into the deployment of AI technologies, demonstrating that it is possible to balance performance with the imperative need for transparency in image classification tasks.

Contents
Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 The Rise of Explainable AI . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Research Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2 Literature Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.1 Concept-Based Explanations . . . . . . . . . . . . . . . . . . . . . . . 7
2.1.1 Sparse and Dense Models in XAI . . . . . . . . . . . . . . . . 9
2.1.2 Hybrid Models Combining Sparse and Dense Approaches . . . 9
2.1.3 User Trust and Model Transparency . . . . . . . . . . . . . . 10
3 Interpretable Deep Learning with NCAV-Lasso . . . . . . . . . . 13
3.1 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.1.1 Non-negative Concept Activation Vectors (NCAV) . . . . . . 13
3.1.2 Integration of Lasso Regularization . . . . . . . . . . . . . . . 14
3.1.3 Modified Feature Extraction Pipeline . . . . . . . . . . . . . . 15
3.1.4 Implementation Details . . . . . . . . . . . . . . . . . . . . . 16
3.1.5 Training Procedure . . . . . . . . . . . . . . . . . . . . . . . . 16
4 Experimental Setup and Results . . . . . . . . . . . . . . . . . . . . 19
4.1 Experimental Design . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
4.1.1 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
4.1.2 Independent Variables . . . . . . . . . . . . . . . . . . . . . . 20
4.1.3 Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
4.1.4 Experimental Procedure . . . . . . . . . . . . . . . . . . . . . 21
4.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
4.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.3.1 Comparison with Baseline CNN . . . . . . . . . . . . . . . . . 23
4.3.2 Advantages of the Proposed Method . . . . . . . . . . . . . . 24
4.3.3 Implementation Challenges . . . . . . . . . . . . . . . . . . . 24
5 Conclusion and related work . . . . . . . . . . . . . . . . . . . . . . 25
5.1 Exploring Alternative Interpretability Methods . . . . . . . . . . . . 25
5.2 Reordering Model Components . . . . . . . . . . . . . . . . . . . . . 25
5.3 Combining Interpretability Techniques . . . . . . . . . . . . . . . . . 26
5.4 Applications to Other Domains . . . . . . . . . . . . . . . . . . . . . 26
5.5 Automated Hyperparameter Tuning . . . . . . . . . . . . . . . . . . . 26
5.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
Bibliography
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

List of Figures
1.1 Concept visualization using NCAV for the digit ’3’. . . . . . . . . . . 3
2.1 Examples of concept-based explanations and model interpretability
techniques. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
3.1 Architecture of the NCAV+Lasso Framework. The diagram shows
how NCAV and Lasso layers are embedded within a ResNet-50 model
as additional end layers. The NCAV layer extracts concepts from the
CNN features, while the Lasso layer promotes sparsity in these con-
cepts to enhance interpretability. This design enables interpretable
outputs while maintaining the original CNN architecture’s struc-
ture.enhance interpretability by promoting sparsity in the learned
concepts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
4.1 Average accuracy (%) by NCAV number and Lasso parameter. This
result shows the performance variation as the NCAV count and Lasso
regularization parameter λ change, highlighting the trade-off between
sparsity and accuracy. . . . . . . . . . . . . . . . . . . . . . . . . . . 22

[1] Sirur, S., Nurse, J. R. C., and Webb, H. (2018). Are We There Yet? Under-
standing the Challenges Faced in Complying with the General Data Protection
Regulation (GDPR). Proceedings of the 2nd International Workshop on Multi-
media Privacy and Security, 88-95.
[2] Antwarg, L., Mindlin, R., Shapira, B., and Rokach, L. (2021). Explaining
anomalies detected by autoencoders using Shapley Additive Explanations. Ex-
pert Systems With Applications, 186, 115736.
[3] Zafar, M. R., and Khan, N. (2021). Deterministic Local Interpretable Model-
Agnostic Explanations for Stable Explainability. Machine Learning and Knowl-
edge Extraction, 3(2), 525-541.
[4] Kukreja, S. L., Löfberg, J., and Brenner, M. J. (2006). A Least Absolute Shrink-
age and Selection Operator (LASSO) for Nonlinear System Identification. 14th
IFAC Symposium on System Identification, 814-819.
[5] Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. (2009). Im-
ageNet: A Large-Scale Hierarchical Image Database. Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition, 248-255.
[6] Cranmer, M., et al. (2021). Disentangled Sparsity Networks for Explainable AI.
Advances in Neural Information Processing Systems.
[7] Ghorbani, A., Wexler, J., Zou, J., and Kim, B. (2019). Towards Automatic
Concept-based Explanations. Advances in Neural Information Processing Sys-
tems, 9277-9286.
[8] Zhang, R., Madumal, P., Miller, T., Ehinger, K., and Rubinstein, B. (2021). In-
vertible Concept-based Explanations for CNN Models with Non-negative Con-
cept Activation Vectors. Proceedings of the AAAI Conference on Artificial In-
telligence, 13543-13551.
[9] Rudin, C., et al. (2022). Interpretable Machine Learning: Fundamental Princi-
ples and 10 Grand Challenges. Statistics Surveys, 16, 1-85.
[10] West, D. (2000). Neural network credit scoring models. Computers & Operations
Research, 27(11-12), 1131-1152.
[11] Ustun, B., and Rudin, C. (2016). Supersparse Linear Integer Models for Opti-
mized Medical Scoring Systems. Machine Learning, 102(3), 349-391.
[12] Sahoh, B., and Choksuriwong, A. (2023). The Role of Explainable Artificial
Intelligence in High-Stakes Decision-Making Systems. Journal of Ambient In-
telligence and Humanized Computing.
[13] Wang, C., Han, B., Patel, B., and Rudin, C. (2023). In Pursuit of Interpretable,
Fair and Accurate Machine Learning for Criminal Recidivism Prediction. Jour-
nal of Quantitative Criminology.
[14] Wu, M., et al. (2018). Beyond Sparsity: Tree Regularization of Deep Models for
Interpretability. Proceedings of the AAAI Conference on Artificial Intelligence,
1670-1678.
[15] Zhang, R., et al. (2021). Sparse, Physics-Based, and Partially Interpretable
Neural Networks for PDEs. Advances in Neural Information Processing Sys-
tems.
[16] Alterovitz, R., et al. (2016). Robot Planning in the Real World: Research
Challenges and Opportunities. IEEE Robotics & Automation Magazine, 23(4),
42-50.
[17] Abdar, M., et al. (2021). A Review of Uncertainty Quantification in Deep Learn-
ing: Techniques, Applications and Challenges. Information Fusion, 76, 243-297.
[18] Chen, Z., Bei, Y., and Rudin, C. (2020). Concept Whitening for Interpretable
Image Recognition. Nature Machine Intelligence, 2(12), 772-782.
[19] Shin, D. (2021). The Effects of Explainability and Causability on Perception,
Trust, and Acceptance. International Journal of Human-Computer Studies, 146,
102551.
[20] Zhang, R., et al. (2023). Interpretable Directed Diversity: Leveraging Model
Explanations for Iterative Crowd Ideation. Advances in Neural Information
Processing Systems.
[21] van Zyl, C., Xianming, Y., and Naidoo, R. (2024). Harnessing Explainable
Artificial Intelligence for Feature Selection in Time Series Energy Forecasting:
A Comparative Analysis of Grad-CAM and SHAP. Applied Energy, 353, 122079.
[22] He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep Residual Learning for
Image Recognition. Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition, 770-778.
[23] Hurley, N., and Rickard, J. (2009). Comparing Measures of Sparsity. IEEE
Transactions on Information Theory, 55(10), 4723-4741.
[24] Bock, S., and Weiß, M. (2019). A Proof of Local Convergence for the Adam
Optimizer. International Joint Conference on Neural Networks (IJCNN), 1-8.