本研究針對無線供電通訊網路（WPCN）中混合式接取點（HAP）輔助之非正交多重存取（NOMA）系統，探討能量效率最大化問題。為抑制多用戶干擾並改善雙重遠近效應，提出一種結合空分多址（SDMA）與NOMA的動態用戶分群架構。由於時間配置、功率控制與用戶分群高度耦合且具動態特性，本文引入Soft Actor-Critic（SAC）強化學習演算法，於連續動作空間中自適應學習WET/WIT時間分配、傳輸功率與用戶分群策略。模擬結果顯示，所提方法相較於傳統OMA與固定分群NOMA，能有效提升HAP端能量效率，並具備良好收斂性與環境適應能力。

Wireless Powered Communication Networks (WPCNs) enable energy-constrained devices to operate sustainably through wireless energy transfer and data transmission. In HAP-assisted Non-Orthogonal Multiple Access (NOMA) networks, however, multi-user interference, energy imbalance, and the doubly near-far problem significantly limit system energy efficiency.
This thesis proposes a reinforcement learning-based dynamic user clustering framework for HAP-NOMA networks to maximize energy efficiency. A hierarchical interference management scheme combining SDMA and NOMA is adopted, where users with high channel correlation are grouped for SIC-based NOMA transmission, while users with low correlation are separated into different spatial beams to suppress inter-cluster interference.
Due to the non-convex and highly coupled nature of joint time allocation, power control, and user clustering, this work employs the Soft Actor-Critic (SAC) algorithm to learn optimal transmission strategies under dynamic channel and energy conditions. The SAC agent jointly optimizes WET/WIT time allocation, NOMA user grouping, and transmission power in continuous action spaces.
Simulation results show that the proposed SAC-based scheme outperforms conventional OMA, fixed NOMA clustering, and heuristic methods in terms of energy efficiency, demonstrating strong adaptability and stable convergence in dynamic WPCN environments.

目錄
第一章	緒論	1
1.1前言	1
1.2文獻回顧	4
1.3論文貢獻	6
1.4論文架構	9
第二章	背景知識	10
2.1無線感測網路(WPCN)	11
2.2非正交多重存取（Non-OrthogonalMultipleAccess,NOMA）	12
2.3強化學習（ReinforcementLearning,RL）	14
2.3.1 SoftActor-Critic,SAC軟性行動評論家	14
2.4空間分多址（Spatial Division Multiple Access,SDMA）	16
第三章	問題表述	18
3.1研究目標	19
3.2傳統優化方法之限制	21
3.3環境說明	24
3.4相關假設	26
第四章	系統模型	29
4.1系統建模	29
4.2傳輸與分組模型（Hybrid SDMA/NOMA）	31
4.3能量模型與能量效率目標	32
4.4訓練流程與策略更新	33
4.4.1訓練流程說明	35
4.4.2獎勵回饋與狀態轉移	35
4.4.3策略更新機制（SAC）	36
第五章	實驗結果與分析	38
5.1模擬場景及參數設定	38
5.1.1主要模擬參數設定	38
5.1.2動態用戶設定方式（Dynamic WD Configuration）	39
5.2	實驗結果與分析	41
5.2.1用戶數量變動與可擴展性分析	42
5.2.2與其他論文比較	43
5.2.3訓練收斂性與學習穩定度	44
5.2.4動態用戶分群收斂行為分析	45
5.2.5與其他RL方式比較	49
第六章	結論	51
6.1未來工作	52
參考文獻	54
圖目錄
圖 1 HTT架構示意圖	12
圖 2 上行鏈路傳輸過程[1]	13
圖 3 Soft Actor-Critic,SAC流程圖	16
圖 4 SDMA空間分多址圖	17
圖 5 訓練流程圖	34
圖 6 能量效率隨無線裝置數量變化情形	42
圖 7 500個time step下節點變化	43
圖 8 跟其他論文比較動態節點	43
圖 9 訓練隨訓練步數逐步收斂	44
圖 10 50% 分群變化圖	46
圖 11 60% 分群變化圖	46
圖 12 70% 分群變化圖	47
圖 13 80% 分群變化圖	47
圖 14 100% 分群變化圖	48
圖 15 能量效率收斂曲線	48
圖 16 與其他方式比較	49
表目錄
表 1 強化學習環境	39
表 2 動態環境建置	41

[1]	S. M. R. Islam, N. Avazov, O. A. Dobre, and K. Kwak, “Power-Domain Non-Orthogonal Multiple Access (NOMA) in 5G Systems: Potentials and Challenges,” IEEE Commun. Surv. Tutor., vol. 19, no. 2, pp. 721–742, 2017, doi: 10.1109/COMST.2016.2621116.
[2]	L. Zhu, Z. Xiao, X.-G. Xia, and D. Oliver Wu, “Millimeter-Wave Communications With Non-Orthogonal Multiple Access for B5G/6G,” IEEE Access, vol. 7, pp. 116123–116132, 2019, doi: 10.1109/ACCESS.2019.2935169.
[3]	M. B. Dadi and B. C. Rehaimi, “Performance of downlink MIMO-NOMA system in 5G Networks,” in 2023 IEEE International Workshop on Mechatronic Systems Supervision (IW_MSS), Hammamet, Tunisia: IEEE, Nov. 2023, pp. 1–4. doi: 10.1109/IW_MSS59200.2023.10368947.
[4]	S. K. Saraswat and D. Singh, “Analysis of Optimization of Rate in Power Domain NOMA Schemes for MIMO,” in 2020 International Conference on Power Electronics & IoT Applications in Renewable Energy and its Control (PARC), Mathura, India: IEEE, Feb. 2020, pp. 481–484. doi: 10.1109/PARC49193.2020.236660.
[5]	R. Shen, X. Wang, and Y. Xu, “Weighted Sum-Rate Maximized Power Allocation in Downlink MIMO-NOMA Systems,” in 2019 IEEE 19th International Conference on Communication Technology (ICCT), Xi’an, China: IEEE, Oct. 2019, pp. 679–684. doi: 10.1109/ICCT46805.2019.8947035.
[6]	X. Chen, F.-K. Gong, G. Li, H. Zhang, and P. Song, “User Pairing and Pair Scheduling in Massive MIMO-NOMA Systems,” IEEE Commun. Lett., vol. 22, no. 4, pp. 788–791, Apr. 2018, doi: 10.1109/LCOMM.2017.2776206.
[7]	X. Liu and X. Wang, “Efficient Antenna Selection and User Scheduling in 5G Massive MIMO-NOMA System,” in 2016 IEEE 83rd Vehicular Technology Conference (VTC Spring), Nanjing, China: IEEE, May 2016, pp. 1–5. doi: 10.1109/VTCSpring.2016.7504208.
[8]	N. Ye, L. Jiang, X. Li, and A. Wang, “A Novel mmWave MIMO-NOMA Scheme Based on Beam Aggregation,” in 2020 IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB), Paris, France: IEEE, Oct. 2020, pp. 1–2. doi: 10.1109/BMSB49480.2020.9379913.
[9]	A.-T. Tran, T. P. Truong, D. Won, N.-N. Dao, and S. Cho, “Weighted Sum-Rate Maximization in Rate-Splitting MISO Downlink Systems,” IEEE Trans. Netw. Sci. Eng., pp. 1–17, 2025, doi: 10.1109/TNSE.2025.3645935.
[10]	Z. Ding, “NOMA Beamforming in SDMA Networks: Riding on Existing Beams or Forming New Ones?,” IEEE Commun. Lett., vol. 26, no. 4, pp. 868–871, Apr. 2022, doi: 10.1109/LCOMM.2022.3146583.
[11]	H. Al-Obiedollah, A. Hayajneh, H. B. Salameh, K. Cumanan, and A. M. Y. Al-Nimrat, “On The Performance of Multi-Carrier SDMA-NOMA CR-Based Systems,” in 2021 Eighth International Conference on Software Defined Systems (SDS), Gandia, Spain: IEEE, Dec. 2021, pp. 1–5. doi: 10.1109/SDS54264.2021.9732076.
[12]	J. Maeng, M. K. Dahouda, and I. Joe, “Optimal Power Allocation with Sectored Cells for Sum-Throughput Maximization in Wireless-Powered Communication Networks Based on Hybrid SDMA/NOMA,” Electronics, vol. 11, no. 6, p. 844, Mar. 2022, doi: 10.3390/electronics11060844.
[13]	A. Amer, S. Khisa, A. Amhaz, C. Assi, S. Hoteit, and J. B. Othman, “Energy Efficiency Maximization with SIC Power Aware Hybrid SDMA/NOMA Scheme,” in ICC 2025 - IEEE International Conference on Communications, Montreal, QC, Canada: IEEE, Jun. 2025, pp. 1–6. doi: 10.1109/ICC52391.2025.11161976.
[14]	N. Y. Mitsuishi, Y. Ma, and J. B. Coder, “Optimized Power Allocation in Multi-cell 4G/5G Systems using Multi-Agent Deep Reinforcement Learning,” in 2025 IEEE 102nd Vehicular Technology Conference (VTC2025-Fall), Chengdu, China: IEEE, Oct. 2025, pp. 1–6. doi: 10.1109/VTC2025-Fall65116.2025.11310705.
[15]	A. Rabee and I. Barhumi, “Toward Energy-Efficient Dynamic Resource Allocation in Uplink NOMA Systems: Deep Reinforcement Learning for Single and Multi-Cell NOMA Systems,” IEEE Trans. Veh. Technol., vol. 74, no. 6, pp. 9313–9327, Jun. 2025, doi: 10.1109/TVT.2025.3540940.
[16]	A. Rabee and I. Barhumi, “Energy Efficient Resource Allocation Approach for Uplink NOMA Multi-Cell Systems Based on Multi-Agent DRL,” in 2024 IEEE Wireless Communications and Networking Conference (WCNC), Dubai, United Arab Emirates: IEEE, Apr. 2024, pp. 1–6. doi: 10.1109/WCNC57260.2024.10571217.
[17]	K. K. Nguyen, T. Q. Duong, N. A. Vien, N.-A. Le-Khac, and M.-N. Nguyen, “Non-Cooperative Energy Efficient Power Allocation Game in D2D Communication: A Multi-Agent Deep Reinforcement Learning Approach,” IEEE Access, vol. 7, pp. 100480–100490, 2019, doi: 10.1109/ACCESS.2019.2930115.
[18]	Y. Wang, Z. Qian, L. He, R. Yin, and C. Wu, “Intelligent Online Computation Offloading for Wireless-Powered Mobile-Edge Computing,” IEEE Internet Things J., vol. 11, no. 17, pp. 28960–28974, Sep. 2024, doi: 10.1109/JIOT.2024.3406204.
[19]	N. C. Luong et al., “Applications of Deep Reinforcement Learning in Communications and Networking: A Survey,” IEEE Commun. Surv. Tutor., vol. 21, no. 4, pp. 3133–3174, 2019, doi: 10.1109/COMST.2019.2916583.
[20]	H. Khoshkbari, V. Pourahmadi, and H. Sheikhzadeh, “Bayesian Reinforcement Learning for Link-Level Throughput Maximization,” IEEE Commun. Lett., vol. 24, no. 8, pp. 1738–1741, Aug. 2020, doi: 10.1109/LCOMM.2020.2990308.
[21]	C. Chaieb, F. Abdelkefi, and W. Ajib, “Deep Reinforcement Learning for Resource Allocation in Multi-Band and Hybrid OMA-NOMA Wireless Networks,” IEEE Trans. Commun., vol. 71, no. 1, pp. 187–198, Jan. 2023, doi: 10.1109/TCOMM.2022.3225163.
[22]	I. Ahmed, M. K. Shahid, and T. Faisal, “Deep Reinforcement Learning Based Beam Selection for Hybrid Beamforming and User Grouping in Massive MIMO-NOMA System,” IEEE Access, vol. 10, pp. 89519–89533, 2022, doi: 10.1109/ACCESS.2022.3199760.
[23]	A. Rabee and I. Barhumi, “Twin Delayed DRL Approach for Resource Allocation in Multi-User NOMA Systems,” in 2023 IEEE 17th International Conference on Application of Information and Communication Technologies (AICT), Baku, Azerbaijan: IEEE, Oct. 2023, pp. 1–5. doi: 10.1109/AICT59525.2023.10313195.
[24]	A. Schröder, S. Gracla, C. Bockelmann, D. Wübben, and A. Dekorsy, “Model-Free Robust Beamforming in Satellite Downlink Using Reinforcement Learning,” IEEE Open J. Commun. Soc., vol. 6, pp. 10582–10598, 2025, doi: 10.1109/OJCOMS.2025.3644403.
[25]	B. Panda, D. D. Senanayake, and P. Singh, “A Machine Learning Approach with Decision Tree-based Signal Detection for MIMO-NOMA Systems,” in 2024 3rd International Conference on Artificial Intelligence For Internet of Things (AIIoT), Vellore, India: IEEE, May 2024, pp. 1–6. doi: 10.1109/AIIoT58432.2024.10574572.
[26]	I. Altin and M. Akar, “Novel OMA and Hybrid NOMA Schemes for MEC Offloading,” in 2020 IEEE International Black Sea Conference on Communications and Networking (BlackSeaCom), Odessa, Ukraine: IEEE, May 2020, pp. 1–5. doi: 10.1109/BlackSeaCom48709.2020.9235017.