浪費了“黃金五年”的Java程式設計師，還有救嗎？ >>>   

先來看一下在PairRDDFunctions.scala檔案中reduceByKey和groupByKey的原始碼

/**
 * Merge the values for each key using an associative reduce function. This will also perform
 * the merging locally on each mapper before sending results to a reducer, similarly to a
 * "combiner" in MapReduce. Output will be hash-partitioned with the existing partitioner/
 * parallelism level.
 */
def reduceByKey(func: (V, V) => V): RDD[(K, V)] = {
  reduceByKey(defaultPartitioner(self), func)
}


/**
 * Group the values for each key in the RDD into a single sequence. Allows controlling the
 * partitioning of the resulting key-value pair RDD by passing a Partitioner.
 * The ordering of elements within each group is not guaranteed, and may even differ
 * each time the resulting RDD is evaluated.
 *
 * Note: This operation may be very expensive. If you are grouping in order to perform an
 * aggregation (such as a sum or average) over each key, using [[PairRDDFunctions.aggregateByKey]]
 * or [[PairRDDFunctions.reduceByKey]] will provide much better performance.
 *
 * Note: As currently implemented, groupByKey must be able to hold all the key-value pairs for any
 * key in memory. If a key has too many values, it can result in an [[OutOfMemoryError]].
 */
def groupByKey(partitioner: Partitioner): RDD[(K, Iterable[V])] = {
  // groupByKey shouldn't use map side combine because map side combine does not
  // reduce the amount of data shuffled and requires all map side data be inserted
  // into a hash table, leading to more objects in the old gen.
  val createCombiner = (v: V) => CompactBuffer(v)
  val mergeValue = (buf: CompactBuffer[V], v: V) => buf += v
  val mergeCombiners = (c1: CompactBuffer[V], c2: CompactBuffer[V]) => c1 ++= c2
  val bufs = combineByKey[CompactBuffer[V]](
    createCombiner, mergeValue, mergeCombiners, partitioner, mapSideCombine=false)
  bufs.asInstanceOf[RDD[(K, Iterable[V])]]
}
 

reduceByKey：reduceByKey會在結果傳送至reducer之前會對每個mapper在本地進行merge，有點類似於在MapReduce中的combiner。這樣做的好處在於，在map端進行一次reduce之後，資料量會大幅度減小，從而減小傳輸，保證reduce端能夠更快的進行結果計算。

groupByKey：groupByKey會對每一個RDD中的value值進行聚合形成一個序列(Iterator)，此操作發生在reduce端，所以勢必會將所有的資料通過網路進行傳輸，造成不必要的浪費。同時如果資料量十分大，可能還會造成OutOfMemoryError。

通過以上對比可以發現在進行大量資料的reduce操作時候建議使用reduceByKey。不僅可以提高速度，還是可以防止使用groupByKey造成的記