2. 程式人生 > >[Spark][Python]DataFrame中取出有限個記錄的例子

[Spark][Python]DataFrame中取出有限個記錄的例子

阿新 • • 發佈：2017-10-05
dep ins pytho rem json.js art hadoop fileinput taskset

[Spark][Python]DataFrame中取出有限個記錄的例子：

sqlContext = HiveContext(sc)

peopleDF = sqlContext.read.json("people.json")

peopleDF.limit(3).show()

===

[[email protected] ~]$ hdfs dfs -cat people.json
{"name":"Alice","pcode":"94304"}
{"name":"Brayden","age":30,"pcode":"94304"}
{"name":"Carla","age":19,"pcoe":"10036"}

{"name":"Diana","age":46}
{"name":"Etienne","pcode":"94104"}
[[email protected] ~]$


In [1]: sqlContext = HiveContext(sc)

In [2]: peopleDF = sqlContext.read.json("people.json")
17/10/05 05:03:11 INFO hive.HiveContext: Initializing execution hive, version 1.1.0
17/10/05 05:03:11 INFO client.ClientWrapper: Inspected Hadoop version: 2.6.0-cdh5.7.0

17/10/05 05:03:11 INFO client.ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0-cdh5.7.0
17/10/05 05:03:14 INFO hive.metastore: Trying to connect to metastore with URI thrift://localhost.localdomain:9083
17/10/05 05:03:14 INFO hive.metastore: Opened a connection to metastore, current connections: 1
17/10/05 05:03:15 INFO hive.metastore: Connected to metastore.
17/10/05 05:03:16 INFO session.SessionState: Created HDFS directory: file:/tmp/spark-99a33db4-b69a-46a9-8032-f87d63299040/scratch/training
17/10/05 05:03:16 INFO session.SessionState: Created local directory: /tmp/4e1c5259-7ae8-482c-ae77-94d3a0c51f91_resources
17/10/05 05:03:16 INFO session.SessionState: Created HDFS directory: file:/tmp/spark-99a33db4-b69a-46a9-8032-f87d63299040/scratch/training/4e1c5259-7ae8-482c-ae77-94d3a0c51f91
17/10/05 05:03:16 INFO session.SessionState: Created local directory: /tmp/training/4e1c5259-7ae8-482c-ae77-94d3a0c51f91
17/10/05 05:03:16 INFO session.SessionState: Created HDFS directory: file:/tmp/spark-99a33db4-b69a-46a9-8032-f87d63299040/scratch/training/4e1c5259-7ae8-482c-ae77-94d3a0c51f91/_tmp_space.db
17/10/05 05:03:16 INFO session.SessionState: No Tez session required at this point. hive.execution.engine=mr.
17/10/05 05:03:16 INFO json.JSONRelation: Listing hdfs://localhost:8020/user/training/people.json on driver
17/10/05 05:03:19 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 251.1 KB, free 251.1 KB)
17/10/05 05:03:20 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 21.6 KB, free 272.7 KB)
17/10/05 05:03:20 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:55073 (size: 21.6 KB, free: 208.8 MB)
17/10/05 05:03:20 INFO spark.SparkContext: Created broadcast 0 from json at NativeMethodAccessorImpl.java:-2
17/10/05 05:03:20 INFO mapred.FileInputFormat: Total input paths to process : 1
17/10/05 05:03:21 INFO spark.SparkContext: Starting job: json at NativeMethodAccessorImpl.java:-2
17/10/05 05:03:21 INFO scheduler.DAGScheduler: Got job 0 (json at NativeMethodAccessorImpl.java:-2) with 1 output partitions
17/10/05 05:03:21 INFO scheduler.DAGScheduler: Final stage: ResultStage 0 (json at NativeMethodAccessorImpl.java:-2)
17/10/05 05:03:21 INFO scheduler.DAGScheduler: Parents of final stage: List()
17/10/05 05:03:21 INFO scheduler.DAGScheduler: Missing parents: List()
17/10/05 05:03:21 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[3] at json at NativeMethodAccessorImpl.java:-2), which has no missing parents
17/10/05 05:03:21 INFO storage.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 4.3 KB, free 277.1 KB)
17/10/05 05:03:21 INFO storage.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.4 KB, free 279.5 KB)
17/10/05 05:03:21 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:55073 (size: 2.4 KB, free: 208.8 MB)
17/10/05 05:03:21 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006
17/10/05 05:03:21 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[3] at json at NativeMethodAccessorImpl.java:-2)
17/10/05 05:03:21 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
17/10/05 05:03:21 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, partition 0,PROCESS_LOCAL, 2149 bytes)
17/10/05 05:03:21 INFO executor.Executor: Running task 0.0 in stage 0.0 (TID 0)
17/10/05 05:03:21 INFO rdd.HadoopRDD: Input split: hdfs://localhost:8020/user/training/people.json:0+179
17/10/05 05:03:21 INFO Configuration.deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id
17/10/05 05:03:21 INFO Configuration.deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
17/10/05 05:03:21 INFO Configuration.deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
17/10/05 05:03:21 INFO Configuration.deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
17/10/05 05:03:21 INFO Configuration.deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
17/10/05 05:03:22 INFO executor.Executor: Finished task 0.0 in stage 0.0 (TID 0). 2354 bytes result sent to driver
17/10/05 05:03:22 INFO scheduler.DAGScheduler: ResultStage 0 (json at NativeMethodAccessorImpl.java:-2) finished in 0.931 s
17/10/05 05:03:22 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 850 ms on localhost (1/1)
17/10/05 05:03:22 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
17/10/05 05:03:22 INFO scheduler.DAGScheduler: Job 0 finished: json at NativeMethodAccessorImpl.java:-2, took 1.388410 s
17/10/05 05:03:23 INFO hive.HiveContext: default warehouse location is /user/hive/warehouse
17/10/05 05:03:23 INFO hive.HiveContext: Initializing metastore client version 1.1.0 using Spark classes.
17/10/05 05:03:23 INFO client.ClientWrapper: Inspected Hadoop version: 2.6.0-cdh5.7.0
17/10/05 05:03:23 INFO client.ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0-cdh5.7.0
17/10/05 05:03:23 INFO spark.ContextCleaner: Cleaned accumulator 2
17/10/05 05:03:23 INFO storage.BlockManagerInfo: Removed broadcast_1_piece0 on localhost:55073 in memory (size: 2.4 KB, free: 208.8 MB)
17/10/05 05:03:25 INFO hive.metastore: Trying to connect to metastore with URI thrift://localhost.localdomain:9083
17/10/05 05:03:25 INFO hive.metastore: Opened a connection to metastore, current connections: 1
17/10/05 05:03:25 INFO hive.metastore: Connected to metastore.
17/10/05 05:03:25 INFO session.SessionState: Created local directory: /tmp/684b38e5-72f0-4712-81d4-4c439e093f5c_resources
17/10/05 05:03:25 INFO session.SessionState: Created HDFS directory: /tmp/hive/training/684b38e5-72f0-4712-81d4-4c439e093f5c
17/10/05 05:03:25 INFO session.SessionState: Created local directory: /tmp/training/684b38e5-72f0-4712-81d4-4c439e093f5c
17/10/05 05:03:25 INFO session.SessionState: Created HDFS directory: /tmp/hive/training/684b38e5-72f0-4712-81d4-4c439e093f5c/_tmp_space.db
17/10/05 05:03:25 INFO session.SessionState: No Tez session required at this point. hive.execution.engine=mr.

In [3]: peopleDF.limit(3).show()
17/10/05 05:04:09 INFO storage.MemoryStore: Block broadcast_2 stored as values in memory (estimated size 65.5 KB, free 338.2 KB)
17/10/05 05:04:10 INFO storage.MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 21.4 KB, free 359.6 KB)
17/10/05 05:04:10 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:55073 (size: 21.4 KB, free: 208.8 MB)
17/10/05 05:04:10 INFO spark.SparkContext: Created broadcast 2 from showString at NativeMethodAccessorImpl.java:-2
17/10/05 05:04:10 INFO storage.MemoryStore: Block broadcast_3 stored as values in memory (estimated size 251.1 KB, free 610.7 KB)
17/10/05 05:04:11 INFO storage.MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 21.6 KB, free 632.4 KB)
17/10/05 05:04:11 INFO storage.BlockManagerInfo: Added broadcast_3_piece0 in memory on localhost:55073 (size: 21.6 KB, free: 208.7 MB)
17/10/05 05:04:11 INFO spark.SparkContext: Created broadcast 3 from showString at NativeMethodAccessorImpl.java:-2
17/10/05 05:04:12 INFO mapred.FileInputFormat: Total input paths to process : 1
17/10/05 05:04:12 INFO spark.SparkContext: Starting job: showString at NativeMethodAccessorImpl.java:-2
17/10/05 05:04:12 INFO scheduler.DAGScheduler: Got job 1 (showString at NativeMethodAccessorImpl.java:-2) with 1 output partitions
17/10/05 05:04:12 INFO scheduler.DAGScheduler: Final stage: ResultStage 1 (showString at NativeMethodAccessorImpl.java:-2)
17/10/05 05:04:12 INFO scheduler.DAGScheduler: Parents of final stage: List()
17/10/05 05:04:12 INFO scheduler.DAGScheduler: Missing parents: List()
17/10/05 05:04:12 INFO scheduler.DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[9] at showString at NativeMethodAccessorImpl.java:-2), which has no missing parents
17/10/05 05:04:12 INFO storage.MemoryStore: Block broadcast_4 stored as values in memory (estimated size 5.9 KB, free 638.2 KB)
17/10/05 05:04:12 INFO storage.MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 3.3 KB, free 641.5 KB)
17/10/05 05:04:12 INFO storage.BlockManagerInfo: Added broadcast_4_piece0 in memory on localhost:55073 (size: 3.3 KB, free: 208.7 MB)
17/10/05 05:04:12 INFO spark.SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1006
17/10/05 05:04:12 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (MapPartitionsRDD[9] at showString at NativeMethodAccessorImpl.java:-2)
17/10/05 05:04:12 INFO scheduler.TaskSchedulerImpl: Adding task set 1.0 with 1 tasks
17/10/05 05:04:12 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, localhost, partition 0,PROCESS_LOCAL, 2149 bytes)
17/10/05 05:04:12 INFO executor.Executor: Running task 0.0 in stage 1.0 (TID 1)
17/10/05 05:04:12 INFO rdd.HadoopRDD: Input split: hdfs://localhost:8020/user/training/people.json:0+179
17/10/05 05:04:14 INFO codegen.GenerateUnsafeProjection: Code generated in 1563.240244 ms
17/10/05 05:04:14 INFO codegen.GenerateSafeProjection: Code generated in 182.529448 ms
17/10/05 05:04:15 INFO executor.Executor: Finished task 0.0 in stage 1.0 (TID 1). 2328 bytes result sent to driver
17/10/05 05:04:15 INFO scheduler.DAGScheduler: ResultStage 1 (showString at NativeMethodAccessorImpl.java:-2) finished in 2.549 s
17/10/05 05:04:15 INFO scheduler.DAGScheduler: Job 1 finished: showString at NativeMethodAccessorImpl.java:-2, took 2.852393 s
17/10/05 05:04:15 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 2547 ms on localhost (1/1)
17/10/05 05:04:15 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool
+----+-------+-----+-----+
| age| name|pcode| pcoe|
+----+-------+-----+-----+
|null| Alice|94304| null|
| 30|Brayden|94304| null|
| 19| Carla| null|10036|
+----+-------+-----+-----+


In [4]:

[Spark][Python]DataFrame中取出有限個記錄的例子

相關推薦

[Spark][Python]DataFrame中取出有限個記錄的例子

dep ins pytho rem json.js art hadoop fileinput taskset [Spark][Python]DataFrame中取出有限個記錄的例子： sqlContext = HiveContext(sc) peopleDF = sql

[Spark][Python]DataFrame select 操作例子

pytho partition rac executor logs part ner man add [Spark][Python]DataFrame中取出有限個記錄的例子 的 繼續 In [4]: peopleDF.select("age")Out[4]: Data

[Spark][Python]DataFrame where 操作例子

frame .sh data mit spark dia where () limit [Spark][Python]DataFrame中取出有限個記錄的例子 的 繼續 [15]: myDF=peopleDF.where("age>21") In [16]: my

[Spark][Python]DataFrame的左右連接例子

pytho ont logs state codes name () class clas [Spark][Python]DataFrame的左右連接例子 $ hdfs dfs -cat people.json {"name":"Alice","pcode":

[Spark][Python][DataFrame][SQL]Spark對DataFrame直接執行SQL處理的例子

hdfs temp div python people data name where afr [Spark][Python][DataFrame][SQL]Spark對DataFrame直接執行SQL處理的例子 $cat people.json {"name":"

js從數組中取出n個不重復的數據

arr rda ray array clas RR gpo cti gin /** * 首先，針對這個數組做一個去重處理，避免你在後面取數據的時候，因為取到相同的元素而又要多去取一次隨機數 * 將獲取到的不重復的數組，再到這裏樣本裏面去取隨機數 * 每取到

CSU 1323 求一堆數中取出2個異或使得異或值大於某個數

  zzy養了一隻小怪獸和N只凹凸曼，單挑的話每隻凹凸曼都不是小怪獸的對手，所以必須由兩隻凹凸曼合作來和小怪獸戰鬥。凹凸曼A和凹凸曼B合作的戰鬥力為他們戰鬥力的異或值。現在由zzy從N只凹凸曼中選出兩隻來和小怪獸戰鬥。請問zzy能否選出兩隻凹凸曼使他們能夠戰勝小怪獸(他們的戰鬥力比小怪獸大)。

js隨機從陣列中取出幾個元素

JS如何從一個數組中隨機取出一個元素或者幾個元素。假如陣列為var items = ['1','2','4','5','6','7','8','9','10'];1.從陣列items中隨機取出一個元素/

【演算法-分治】從陣列中取出n個元素的所有組合（需要深入理解遞迴）

本文為轉載，原文章出處http://www.cnblogs.com/shuaiwhu/archive/2012/04/27/2473788.html 如陣列為{1, 2, 3, 4, 5, 6}，那麼從它中取出3個元素的組合有哪些，取出4個元素的組合呢？ 比如取3個元素

求一堆數中取出2個異或使得異或值大於某個數

#include<stdio.h> int n,m; struct haha { int next[3]; int nm; }node[100005*4]; int pos; int makend() { int i; pos++; for(i=0;

SQL從DB中取出多個欄位賦值給多個變數/一個欄位給單個變數賦值

  DECLARE @FIRST_BEGIN    DATETIME DECLARE @FIRST_END        DATETIME DECLARE @SECOND_BEGIN    DATETIME DECLARE @SECOND_END        DATETI

iOS 從allCount個元素中取出myCount個元素的所有排列組合

#import "CombinationModel.h" @implementation CombinationModel {     NSMutableArray *_usedArr;//識別     NSMutableArray *_resultArr;//結

給定陣列，試求能否從陣列中取出n個不復用的數的和為sum

這是之前在一次面試中遇到的一道題，當時因為太久沒擼演算法，太久沒刷題，所以gg，現在回顧一下。 問題描述： 實現一個函式，傳3個引數，指定陣列（有小數、正負數），n（取出個數），sum（指定和），輸出是否能找到這幾個數。 這和經典的湊硬幣問題其實本質上是相

Java實現從m個各不相同的元素中取出n個，並進行全排列。

　　最近在看面試題的時候發現，一些基礎的演算法都記不住了，只是能大概說出個原理….為了加深記憶，這裡對一些簡單的演算法題進行一個歸納。 　　下面的程式碼主要解決的問題是：從m個各不相同的元素中取出n個，進行全排列，得到所有可能的結果。 　　即：輸入為字元陣列

Java實現集合的組合（從組合中取出K個元素進行組合的所有情況）

1.利用遞迴進行取出資料: public static <T> List<List<T>> combinations(List<T> list, int k) { if (k == 0 ||

從m中取出n個全組合

import java.util.ArrayList; import java.util.Arrays; import java.util.List; public class mAn { public static int count = 0

python系列（五）centos6.x中部署多個python版本

python pyenv centos6.x virtualenv 博主QQ：819594300博客地址：http://zpf666.blog.51cto.com/有什麽疑問的朋友可以聯系博主，博主會幫你們解答，謝謝支持！使用pyenv+virtualenv方式部署python多版本pyenv

python筆記10-切片（從list或字符串中取幾個元素）

-1 下標 功能 切片 name 字符 list python 筆記 name1 = ‘zcl,pyzyz‘names = [‘zcl‘,‘py‘,‘zyz‘]#切片的意思就是從list裏面或者字符串裏面取幾個元素#切片操作對字符串也是完全適用的# print(names[

Python 字典中一鍵對應多個值

列表 檢查 span mark blog class efault remove one #字典的一鍵多值 print‘方案一 list作為dict的值 值允許重復‘ d1={} key=1 value=2 d1.setdefault(key

Win10系統中新增的快捷鍵，做個記錄

png chm height col 視圖 分享 setting wid 記錄 Win10系統中新增的快捷鍵，做個記錄 1、Win+Q或者Win+S 打開下面搜索框 2、Win+T 切換任務欄上程序； 3、