本研究探討以強化學習輔助無人機（UAV）於無線供電通訊網路（WPCN）中執行動態路徑規劃之方法，以降低無人機在完成多目標節點巡航與資料收集任務時的總能量消耗。無人機在具動態變動節點的環境下透過深度Q網路（DQN）學習決策策略，當環境中節點位置或數量發生變動時，透過感測器偵測環境中目標節點的變化，能即時重新規劃巡航路徑，以維持任務的有效執行。本方法結合無線能量傳輸與資料收集需求，使無人機能在有限能量條件下有效完成任務，並提升無線供電通訊網路中無人機巡航路徑規劃於動態環境下之彈性與適應性，展現強化學習於無人機動態路徑規劃應用之可行性。

This study investigates a method for employing reinforcement learning to assist unmanned aerial vehicles (UAVs) in performing dynamic path planning within wireless powered communication networks (WPCNs), aiming to reduce the total energy consumption of UAVs while completing multi-target waypoint patrolling and data collection tasks. In environments with dynamically changing nodes, UAVs utilize a deep Q-network (DQN) to learn decision-making strategies, and through onboard sensors, detect changes in the positions and quantities of target nodes in real time, allowing for immediate replanning of flight paths to maintain effective task execution. This approach integrates wireless energy transfer with data collection requirements, enabling UAVs to accomplish missions efficiently under limited energy conditions while enhancing the flexibility and adaptability of UAV path planning in dynamic environments within WPCNs. The results demonstrate the feasibility of applying reinforcement learning to UAV dynamic path planning in practical WPCN scenarios.

第一章 緒論.... 1
1.1 前言.... 1
1.2 文獻回顧.... 5
1.3 論文貢獻.... 6
第二章 背景知識.... 8
2.1 無線感測網路.... 8
2.2 WPCN ...... 10
2.3 動態環境.. 11
2.4 強化學習.. 12
2.4.1 Q-Learning ... 13
2.4.2 DQN ..... 14
第三章 問題表述.. 15
3.1 問題背景.. 15
3.2 問題描述.. 16
3.3 研究目標.. 17
3.4 問題建模.. 18
3.5 相關假設.. 19
第四章 無線供電通訊網路之動態路徑規劃...... 20
4.1 初步分析.. 20
4.2 使用強化學習的動態環境路徑規劃...... 21
第五章 實驗結果與分析...... 26
5.1 模擬場景及參數...... 26
5.2 實驗結果.. 29
5.3 實驗總結.. 38
第六章 結論.. 39
6.1 未來工作.. 40
參考文獻.. 41

圖目錄
圖1 WSN拓樸示意圖 .. 1
圖2 WPCN架構示意圖 ....... 2
圖3 HTT協議示意圖 ... 3
圖4 UAV作為混和存取點的WPCN架構示意圖 ..... 4
圖5 WSN系統節點多跳傳輸資料流程圖 .. 9
圖6 強化學習框架圖. 13
圖7動態環境示意圖.. 16
圖8使用旅行推銷員規劃路徑.. 21
圖9飛行場景圖.. 26
圖10完成時間比較圖 29
圖11訓練500次路徑規劃結果  33
圖12 訓練500次Cycle 20結果 ...... 34
圖13 Cycle1隨機生成節點結果 ....... 36
圖14總獎勵函數曲線圖.... 36
圖15總損失函數曲線圖.... 37
圖16 平均獎勵函數曲線圖....... 37
圖17 平均損失函數曲線圖....... 38

表目錄
表1 DQN模擬參數 .... 28

參考文獻
[1]	J. Aponte-Luis, J. Gómez-Galán, F. Gómez-Bravo, M. Sánchez-Raya, J. Alcina-Espigado, and P. Teixido-Rovira, “An Efficient Wireless Sensor Network for Industrial Monitoring and Control,” Sensors, vol. 18, no. 1, p. 182, Jan. 2018, doi: 10.3390/s18010182.
[2]	S. Bi, Y. Zeng, and R. Zhang, “Wireless powered communication networks: an overview,” IEEE Wireless Commun., vol. 23, no. 2, pp. 10–18, Apr. 2016, doi: 10.1109/mwc.2016.7462480.
[3]	S. M. A. Huda, M. Y. Arafat, and S. Moh, “Wireless Power Transfer in Wirelessly Powered Sensor Networks: A Review of Recent Progress,” Sensors, vol. 22, no. 8, p. 2952, Apr. 2022, doi: 10.3390/s22082952.
[4]	H. Ju and R. Zhang, “Throughput Maximization in Wireless Powered Communication Networks,” IEEE Trans. Wireless Commun., vol. 13, no. 1, pp. 418–428, Jan. 2014, doi: 10.1109/TWC.2013.112513.130760.
[5]	B. Al Baroomi, T. Myo, M. R. Ahmed, A. Al Shibli, M. H. Marhaban, and M. S. Kaiser, “Ant Colony Optimization-Based Path Planning for UAV Navigation in Dynamic Environments,” in 2023 7th International Conference on Automation, Control and Robots (ICACR), Kuala Lumpur, Malaysia: IEEE, Aug. 2023, pp. 168–173. doi: 10.1109/icacr59381.2023.10314603.
[6]	G. Du, T. Li, G. Cao, P. Li, X. Xu, and Y. Zhang, “Path planning of warehouse UAV automatic inventory under multi-objective constraints,” in 2024 6th International Conference on Robotics, Intelligent Control and Artificial Intelligence (RICAI), Nanjing, China: IEEE, Dec. 2024, pp. 393–398. doi: 10.1109/ricai64321.2024.10911467.
[7]	Z. Zhou, X. Xing, Y. Li, and R. Wang, “Multi-UAV Path Planning Based on Potential Field Dense Reward in Unknown Environments with Static and Dynamic Obstacles,” in 2023 China Automation Congress (CAC), Chongqing, China: IEEE, Nov. 2023, pp. 1289–1294. doi: 10.1109/cac59555.2023.10450792.
[8]	Y. Zhu and S. Wang, “Flying Path Optimization of Rechargeable UAV for Data Collection in Wireless Sensor Networks,” in 2023 IEEE Applied Sensing Conference (APSCON), Bengaluru, India: IEEE, Jan. 2023, pp. 1–3. doi: 10.1109/apscon56343.2023.10101011.
[9]	F. Tossa, Y. Faga, W. Abdou, E. C. Ezin, and P. Gouton, “Wireless Sensor Network Deployment: Architecture, Objectives, and Methodologies,” Sensors, vol. 25, no. 11, p. 3442, May 2025, doi: 10.3390/s25113442.
[10]	S. Bi, C. K. Ho, and R. Zhang, “Wireless powered communication: opportunities and challenges,” IEEE Commun. Mag., vol. 53, no. 4, pp. 117–125, Apr. 2015, doi: 10.1109/MCOM.2015.7081084.
[11]	A. Malik and M. Rao, “Radio Frequency Interference, Its Mitigation and Its Implications for the Civil Aviation Industry,” Electronics, vol. 14, no. 12, p. 2483, June 2025, doi: 10.3390/electronics14122483.
[12]	J. Liang, D. Feng, C. He, G. Qian, C. Guo, and N. Zhang, “Joint Time and Power Allocation in Multi-Cell Wireless Powered Communication Networks,” IEEE Access, vol. 7, pp. 43555–43563, 2019, doi: 10.1109/ACCESS.2019.2908240.
[13]	A. G. Onalan, E. D. Salik, and S. Coleri, “Relay Selection, Scheduling, and Power Control in Wireless-Powered Cooperative Communication Networks,” IEEE Trans. Wireless Commun., vol. 19, no. 11, pp. 7181–7195, Nov. 2020, doi: 10.1109/TWC.2020.3008990.
[14]	H. Ju and R. Zhang, “User cooperation in wireless powered communication networks,” in 2014 IEEE Global Communications Conference, Austin, TX, USA: IEEE, Dec. 2014, pp. 1430–1435. doi: 10.1109/GLOCOM.2014.7037009.
[15]	C. J. C. H. Watkins and P. Dayan, “Q-learning,” Mach Learn, vol. 8, no. 3–4, pp. 279–292, May 1992, doi: 10.1007/BF00992698.
[16]	V. Mnih et al., “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529–533, Feb. 2015, doi: 10.1038/nature14236.
[17]	C.-H. Yu, J. Tsai, and Y.-T. Chang, “Intelligent Path Planning for UAV Patrolling in Dynamic Environments Based on the Transformer Architecture,” Electronics, vol. 13, no. 23, p. 4716, Nov. 2024, doi: 10.3390/electronics13234716.
[18]	J. Le Ny, E. Feron, and E. Frazzoli, “On the Dubins Traveling Salesman Problem,” IEEE Trans. Automat. Contr., vol. 57, no. 1, pp. 265–270, Jan. 2012, doi: 10.1109/TAC.2011.2166311.
[19]	J. Chen, M. Li, Z. Yuan, and Q. Gu, “An Improved A* Algorithm for UAV Path Planning Problems,” in 2020 IEEE 4th Information Technology, Networking, Electronic and Automation Control Conference (ITNEC), Chongqing, China: IEEE, June 2020, pp. 958–962. doi: 10.1109/ITNEC48623.2020.9084806.
[20]	V. Radhakrishnan and W. Wu, “Energy Efficient Communication Design in UAV Enabled WPCN Using Dome Packing Method in Water Distribution System,” Energies, vol. 15, no. 10, p. 3844, May 2022, doi: 10.3390/en15103844.
[21]	D. Xiang, H. Lin, J. Ouyang, and D. Huang, “Combined improved A* and greedy algorithm for path planning of multi-objective mobile robot,” Sci Rep, vol. 12, no. 1, p. 13273, Aug. 2022, doi: 10.1038/s41598-022-17684-0.
[22]	F. Uddin et al., “An Improvement to the 2-Opt Heuristic Algorithm for Approximation of Optimal TSP Tour,” Applied Sciences, vol. 13, no. 12, p. 7339, June 2023, doi: 10.3390/app13127339.