scala集合轉化為DS/DF

 1 case class TestPerson(name: String, age: Long, salary: Double)
 2 val tom = TestPerson("Tom Hanks",37,35.5)
 3 val sam = TestPerson("Sam Smith",40,40.5)
 4 val PersonList = mutable.MutableList[TestPerson]()
 5 PersonList += tom
 6 PersonList += sam
 7 
 8 val personDS = PersonList.toDS()
 
 9 println(personDS.getClass)
10 personDS.show()
11 
12 val personDF = PersonList.toDF()
13 println(personDF.getClass)
14 personDF.show()
15 personDF.select("name", "age").show()

輸出結果：

class org.apache.spark.sql.Dataset

+---------+---+------+
|     name|age|salary|
+---------+---+------+
|Tom Hanks| 37
 
|  35.5|
|Sam Smith| 40|  40.5|
+---------+---+------+

class org.apache.spark.sql.DataFrame

+---------+---+------+
|     name|age|salary|
+---------+---+------+
|Tom Hanks| 37|  35.5|
|Sam Smith| 40|  40.5|
+---------+---+------+

+---------+---+
|     name|age|
+---------+---+
|Tom Hanks| 37
 
|
|Sam Smith| 40|
+---------+---+

參考資料：

https://stackoverflow.com/questions/39397652/convert-scala-list-to-dataframe-or-dataset

Spark:scala集合轉化為DS/DF