1. 程式人生 > >對RDD分區的理解

對RDD分區的理解

blog cat success ica sim contex mil ont eight

舉個例子:

val logFile = "file:///home/soyo/桌面/6.txt" 
    val conf = new SparkConf().setAppName("Simple Application")
    val sc = new SparkContext(conf)
val RDD =sc.textFile(logFile)
    RDD.saveAsTextFile("file:////home/soyo/桌面/667.txt")   
    val repartitionedRDD=context.repartition(8) //控制RDD分區的個數
    repartitionedRDD.saveAsTextFile("file:////home/soyo/桌面/668.txt")
 

667.txt:

part-00000 _SUCCESS

668.txt:

part-00000 part-00001 part-00002 part-00003 part-00004 part-00005 part-00006 part-00007 _SUCCESS

如果設置.setMaster("local[2]")默認最少是2個分區,別的類同

對RDD分區的理解