淡江大學覺生紀念圖書館 (TKU Library)
進階搜尋


下載電子全文限經由淡江IP使用) 
系統識別號 U0002-2907201412003100
中文論文名稱 基於頻寬考量優化節能主伺服器之效能
英文論文名稱 Reducing the Power Consumption of Servers with Bandwidth Consideration
校院名稱 淡江大學
系所名稱(中) 電機工程學系碩士班
系所名稱(英) Department of Electrical Engineering
學年度 102
學期 2
出版年 103
研究生中文姓名 林佑璋
研究生英文姓名 Yu-Chang Lin
學號 601450306
學位類別 碩士
語文別 中文
第二語文別 英文
口試日期 2014-07-07
論文頁數 50頁
口試委員 指導教授-李維聰
委員-朱國志
委員-吳庭育
中文關鍵字 分散式運算  評分機  雲端網路  網路頻寬 
英文關鍵字 MapReduce  Benchmark  Cloud Network 
學科別分類 學科別應用科學電機及電子
中文摘要 近年來雲端系統應用越來越成熟,相對的雲端系統應用也變得更廣泛。MapReduce是一種分散計算系統,雲端系統中越來越多人使用。Hadoop是從Google雲端系統演化出來的開放系統,在雲端的環境中透過虛擬機器或實體機器作分配任務、獲取資源和提供服務,藉由網路將這些功能連結串連起來就可以獲取更大量運算資源,因此軟體運算、硬體效能和網路傳輸速率,都會對分散式運算系統的運算效率產生影響。在許多的論文中提出了一些解決方法,有效的管理伺服器,排程工作依照任務資料運算形式選擇伺服器的選擇等等。為了獲得最大的運算資源,通常會同時開啟多台機器加入運算過程,雖然可以讓運算速度加快,但相對的也產生耗能的問題,因為每台伺服器性能都會有所不同,所以運算效率也不同,當分配到一樣的工作量時有些性能好的伺服器會提前結束工作,閒置在那邊等待其他伺服器完成,這些等待的時間就造成不必要能量浪費。雲端應用Green Mater是基於Hadoop MapReduce做出改進,篩選虛擬機器效能,在不嚴重影響整體系統效能下將虛擬機器效能低的關閉不使用,進而達到節能的效果。這篇論文針對Green Master加入網路傳輸資料速度新的變數,將網路頻寬與虛擬機器的系統效能一起列入篩選範圍,挑選出合適的虛擬機器。在MapRduce過程中,當資料被分散後並分配到指定的虛擬機器上做運算時,這些運算虛擬機器是分散在不同地方,須透過網路傳輸將這些分散資料傳送到指定虛擬機,擁有較好的運算能力的虛擬機器但網路傳輸慢,卻必須浪費等待資料的傳輸時間,所以網路頻寬的快慢勢必也會影響到整體運算完成的時間,提出了Dynamic Green Master改善Green Master網路傳輸的部分。
英文摘要 Hadoop evolves from Google cloud system, and is open system. In cloud circumstance Hadoop allocates the tasks by virtual machines or hardware-based computer, obtains the information, and provides service. Hadoop integrates these above-mentioned functions into Internet to gain a lot of computing resource, therefore, computing, hardware performance, and network transfer rate have effect on process efficiency of distributed system. More papers purpose solution to optimize the Hadoop. For example, efficiency server management and according to the process types of data to schedule the tasks, etc. In order to optimize the process speed, Hadoop can start more computers to deal with tasks, but the strategy result in consuming resource overly. Because each server has different performance, their Operational efficiency is also different. If system allocate the tasks of the same workload to each server, the server having great performance can complete the task rapidly, and these servers idle their time to wait other server to complete their tasks, therefore, the waiting time result in wasting unnecessary performance.
Green Master is based on improvement of MapReduce. Green Master can filter the performance of virtual machines, and not influence system performance to turn off the virtual machines of bad performance. The purpose of Green Master can achieve the goal of energy conservation.
This paper is aimed at Green Master to add reference value of Network data transfer rate, and filter the proper virtual machines to increase the performance of cloud system once again.
In procedure of MapReduce data is distributed to assigned virtual machines to be processed. Because cloud system must distribute the data through Network transmission, the virtual machines have great performance, but system transfer data to them slowly in Network. The aforementioned condition idle more time to wait the transfer time so Network bandwidth must influence system performance.We propose Dynamic Green Master to improve the Network transfer of Green Master.
論文目次 目錄
第一章 緒論 1
1.1 研究動機 3
1.2 論文架構 4
第二章 相關研究 6
2.1 Virtual Machine 6
2.2 GFS (Google File System) 9
2.3 MapReduce 10
2.4 Hadoop 14
2.5 Hadoop MapReduce 15
2.5.1 HDFS(Hadoop distributed File System)16
2.5.2 NameNode 18
2.5.3 DataNode 18
2.5.4 Job Tacker 18
2.5.5 TaskNode 18
2.6 Green Master 19
2.6.1 Master 19
2.6.2Green Master (GM) 20
2.6.3 Input File Index 22
2.6.4 Queue 22
2.6.5 Server Information & Benchmark 22
2.6.6 Record 23
2.6.7 Load Balance Optimization 23
2.6.8 Power Saving Algorithm(PSA) 24
2.6.9 Decision 25
2.7 Benchmark 26
2.8 相關論文研究 26
第三章 Dynamic Green Master 30
3.1Dynamic Green Master System 31
3.1.1 Input File Index 32
3.1.2 Queue 32
3.1.3Server Information 32
3.1.4 VM Bandwidth Score 33
3.1.5 New Benchmark Score 34
3.1.6 Sort New Benchmark Score 34
3.1.7 Load Balance Optimization 35
3.1.8 Power Saving Algorithm 35
3.1.9 Decision 35
3.2系統流程圖 36
第四章 實驗結果以及效能分析 38
4.1. 實驗環境 38
4.2. Green Master 網路傳輸的影響 40
4.3. Dynamic Green Master Simulation Result 42
第五章 結論以及未來展望 48
參考文獻 (References) 49
圖目錄
圖2.1、寄宿虛擬化實現方式 7
圖2.2、原生虛擬化實現方式 8
圖2.3、MapReduce簡易結構圖 13
圖2.4、Key/Value示意圖 13
圖2.5、Hadoop環境建置 16
圖2.6、Green Master架構圖 21
圖3.1、Dynamic Green Master Architecture 31
圖3.2、系統流程圖 36
圖4.1、虛擬機器Benchmark Score和網路頻寬 39
圖4.2、Green Master在不同網路頻寬系統時間 40
圖4.3、Green Master在不同網路頻寬和資料量系統時間 41
圖4.4、GM與DGM系統時間 43
圖4.5、GM與DGM單位時間耗能 44
圖4.6、GM與DGM的系統耗能 45
圖4.7、GM、DGM以及網路頻寬優系統耗能 46
圖4.8、耗能比較圖 47

表目錄
表2.4 Google MapReduce與Hadoop MapReduce比較圖 14
表3.1 ε實驗環境表 14
表4.1 Green Master BS排序 42
表4.3 兩台VM在低頻寬實驗環境 47


參考文獻 [1] Ling-Shang Kuo, “MapReduce-based Image Processing System with Priority-based DSRF Algorithm,” 淡江大學, 2012.
[2] Ming-Zhi Wu, Yu-Chang Lin, Wei-Tsong Lee, Yu-Sun Lin, Fong-Hao Liu, “Green Master Based on MapReduce Cluster,” HumanCom and EMC 2013.
[3] Tao Zhu, “Green Scheduling: A Scheduling Policy for Improving the Energy Efficiency of Fair Scheduler,” 2011 12th International Conference on Parallel and Distributed Computing, Applications and Technologies.
[4] Xia Xie, Qingcha Chen, Wenzhi Cao, Pingpeng Yuan, Hai Jin, “Benchmark Object for Virtual Machines,” 2010 Second International Workshop on Education Technology and Computer Science.
[5] Yury Audzevich, “GreenCloud: A Packet-level Simulator of Energy-aware Cloud Computing Data Centers,” This full text paper was peer reviewed at the direction of IEEE Communications Society subject matter experts for publication in the IEEE Globecom 2010 proceedings.
[6] Shiori KURAZUMI ∗, Tomoaki TSUMURA∗, Shoichi SAITO∗ and Hiroshi MATSUO∗, “Dynamic processing slots scheduling for I/O intensive jobs of Hadoop MapReduce,” 2012 Third International Conference on Networking and Computing
[7] Kewen Wang, Xuelian Lin, Wenzhong Tang, “Predator - An Experience Guided Configuration Optimizer for Hadoop MapReduce,” IEEE 4th International Conference on Cloud Computing Technology and Science
[8] Nan Zhu, Xue Liu*, Jie Liu, and Yu Hua, “Towards a Cost-Efficient MapReduce: Mitigating Power Peaks for Hadoop Clusters,” TSINGHUA SCIENCE AND TECHNOLOGY.
[9] Zhang Qing, Jin Yuehu, “Distributed Network Measurement System Based on Hadoop,” Wireless Communications, Networking and Mobile Computing (WiCOM), 2012 8th International Conference on
[10] Anirban Mandal, Yufeng Xin, Ilia Baldine, Paul Ruth, Chris Heerman, “Provisioning and Evaluating Multi-domain Networked Clouds for Hadoop-based Applications,” 2011 Third IEEE International Conference on Coud Computing Technology and Science
[11] URL: http://trac.nchc.org.tw/cloud/wiki/NCHCCloudCourse110718
[12] URL: http://www.microsoft.com/zh-tw/default.aspx
[13] URL: http://www.google.com/intl/zh-TW/drive/features.html
[14] URL: http://www.amazon.com/
[15] Jeffrey Dean and Sanjay Ghemawat, “MapReduce: Simpli_ed Data Processing on Lar Clusters,” OSDO 2004.
[16] 陳信宇, “雲端計算中心力用DVFS技術之工作排程節能演算法,2011.
[17] Yifeng Sun, “Fast Live Cloning of Virtual Machine based on Xen,” 2009 11 th IEEE International Conference on High Performance Computing and Communication.
[18] Jiong Xie, Shu Yin, Xiaojun Ruan, Zhiyang Ding, Yun Tian,, “Improving MapReduce Performance through Data Placement in Heterogeneous Hadoop Cluster,”
[19] Yi Zhao, Wenlong Huang, “Adaptive Distributed Load Balancing Algorithm based on Live Migration of Virtual Machines in Cloud,” 2009 Fifth International Joint Conference on INC, IMS and IDC.
[20] Sandhya S V, Sanjay H A*, Netravathi S J, Sowmyashree M V, Yogeshwari R N, “Fault–Tolerant Master-Workers framework for MapReduce Application,” 2009 International Conference on Advances in Recent Technologies in Communication and Computing.
[21] 鄭瑞明, “在有限資源的雲端下依機器性能優化其虛擬機器的配置,”國立東華大學資訊工程系,2012
論文使用權限
  • 同意紙本無償授權給館內讀者為學術之目的重製使用,於2014-07-30公開。
  • 同意授權瀏覽/列印電子全文服務,於2014-07-30起公開。


  • 若您有任何疑問,請與我們聯絡!
    圖書館: 請來電 (02)2621-5656 轉 2281 或 來信