MapReduce optimization algorithm based on machine learning in heterogeneous cloud environment
LIN Wen-hui LEI Zhen-ming LIU Jun YANG Jie LIU Fang HE Gang WANG Qin
We present an approach to optimize the MapReduce architecture, which could make heterogeneous cloud environment more stable and efficient. Fundamentally different from prev ious methods, our approach in troduces the machine learning technique into MapReduce fram ework, and dynamically improve MapReduce algorithm according to the statistics result of machine learning. There are three main as pects: learning machine pe rformance, reduce task assi gnment algorithm based on learning result, and speculative executio n optimization mechanism. Furthermore, th ere are two important features in our approach.