2. 程式人生 > >pandas io tools（使用python處理資料時候經常用到）

pandas io tools（使用python處理資料時候經常用到）

阿新 • • 發佈：2018-12-27

CSV & Text files

The two workhorse functions for reading text files (a.k.a. flat files) are read_csv() and read_table(). They both use the same parsing code to intelligently convert tabular data into a DataFrame object. See thecookbook for some advanced strategies

They can take a number of arguments:

  • filepath_or_buffer: Either a string path to a file, or any object with a read method (such as an open file or StringIO).
  • sep or delimiter: A delimiter / separator to split fields on. read_csv is capable of inferring the delimiter automatically in some cases by “sniffing.” The separator may be specified as a regular expression; for instance you may use ‘|\s*’ to indicate a pipe plus arbitrary whitespace.
  • delim_whitespace: Parse whitespace-delimited (spaces or tabs) file (much faster than using a regular expression)
  • compression: decompress 'gzip' and 'bz2' formats on the fly.
  • dialect: string or csv.Dialect instance to expose more ways to specify the file format
  • dtype: A data type name or a dict of column name to data type. If not specified, data types will be inferred.
  • header: row number to use as the column names, and the start of the data. Defaults to 0 if no names passed, otherwise None. Explicitly pass header=0 to be able to replace existing names.
  • skiprows: A collection of numbers for rows in the file to skip. Can also be an integer to skip the first n rows
  • index_col: column number, column name, or list of column numbers/names, to use as the index (row labels) of the resulting DataFrame. By default, it will number the rows without using any column, unless there is one more data column than there are headers, in which case the first column is taken as the index.
  • names: List of column names to use as column names. To replace header existing in file, explicitly pass header=0.
  • na_values: optional list of strings to recognize as NaN (missing values), either in addition to or in lieu of the default set.
  • true_values: list of strings to recognize as True
  • false_values: list of strings to recognize as False
  • keep_default_na: whether to include the default set of missing values in addition to the ones specified in na_values
  • parse_dates: if True then index will be parsed as dates (False by default). You can specify more complicated options to parse a subset of columns or a combination of columns into a single date column (list of ints or names, list of lists, or dict) [1, 2, 3] -> try parsing columns 1, 2, 3 each as a separate date column [[1, 3]] -> combine columns 1 and 3 and parse as a single date column {‘foo’ : [1, 3]} -> parse columns 1, 3 as date and call result ‘foo’
  • keep_date_col: if True, then date component columns passed into parse_dates will be retained in the output (False by default).
  • date_parser: function to use to parse strings into datetime objects. If parse_dates is True, it defaults to the very robust dateutil.parser. Specifying this implicitly sets parse_dates as True. You can also use functions from community supported date converters from date_converters.py
  • dayfirst: if True then uses the DD/MM international/European date format (This is False by default)
  • thousands: sepcifies the thousands separator. If not None, then parser will try to look for it in the output and parse relevant data to integers. Because it has to essentially scan through the data again, this causes a significant performance hit so only use if necessary.
  • comment: denotes the start of a comment and ignores the rest of the line. Currently line commenting is not supported.
  • nrows: Number of rows to read out of the file. Useful to only read a small portion of a large file
  • iterator: If True, return a TextParser to enable reading a file into memory piece by piece
  • chunksize: An number of rows to be used to “chunk” a file into pieces. Will cause an TextParser object to be returned. More on this below in the section on iterating and chunking
  • skip_footer: number of lines to skip at bottom of file (default 0)
  • converters: a dictionary of functions for converting values in certain columns, where keys are either integers or column labels
  • encoding: a string representing the encoding to use for decoding unicode data, e.g. 'utf-8` or 'latin-1'.
  • verbose: show number of NA values inserted in non-numeric columns
  • squeeze: if True then output with only one column is turned into Series
  • error_bad_lines: if False then any lines causing an error will be skipped bad lines

Consider a typical CSV file containing, in this case, some time series data:

In [1021]: print open('foo.csv').read()
date,A,B,C
20090101,a,1,2
20090102,b,3,4
20090103,c,4,5

The default for read_csv is to create a DataFrame with simple numbered rows:

In [1022]: pd.read_csv('foo.csv')
Out[1022]: 
       date  A  B  C
0  20090101  a  1  2
1  20090102  b  3  4
2  20090103  c  4  5

In the case of indexed data, you can pass the column number or column name you wish to use as the index:

In [1023]: pd.read_csv('foo.csv', index_col=0)
Out[1023]: 
          A  B  C
date             
20090101  a  1  2
20090102  b  3  4
20090103  c  4  5

In [1024]: pd.read_csv('foo.csv', index_col='date')
Out[1024]: 
          A  B  C
date             
20090101  a  1  2
20090102  b  3  4
20090103  c  4  5

You can also use a list of columns to create a hierarchical index:

In [1025]: pd.read_csv('foo.csv', index_col=[0, 'A'])
Out[1025]: 
            B  C
date     A      
20090101 a  1  2
20090102 b  3  4
20090103 c  4  5

The dialect keyword gives greater flexibility in specifying the file format. By default it uses the Excel dialect but you can specify either the dialect name or a csv.Dialect instance.

Suppose you had data with unenclosed quotes:

In [1026]: print data
label1,label2,label3
index1,"a,c,e
index2,b,d,f

By default, read_csv uses the Excel dialect and treats the double quote as the quote character, which causes it to fail when it finds a newline before it finds the closing double quote.

We can get around this using dialect

In [1027]: dia = csv.excel()

In [1028]: dia.quoting = csv.QUOTE_NONE

In [1029]: pd.read_csv(StringIO(data), dialect=dia)
Out[1029]: 
       label1 label2 label3
index1     "a      c      e
index2      b      d      f

All of the dialect options can be specified separately by keyword arguments:

In [1030]: data = 'a,b,c~1,2,3~4,5,6'

In [1031]: pd.read_csv(StringIO(data), lineterminator='~')
Out[1031]: 
   a  b  c
0  1  2  3
1  4  5  6

Another common dialect option is skipinitialspace, to skip any whitespace after a delimiter:

In [1032]: data = 'a, b, c\n1, 2, 3\n4, 5, 6'

In [1033]: print data
a, b, c
1, 2, 3
4, 5, 6

In [1034]: pd.read_csv(StringIO(data), skipinitialspace=True)
Out[1034]: 
   a  b  c
0  1  2  3
1  4  5  6

The parsers make every attempt to “do the right thing” and not be very fragile. Type inference is a pretty big deal. So if a column can be coerced to integer dtype without altering the contents, it will do so. Any non-numeric columns will come through as object dtype as with the rest of pandas objects.

Specifying column data types

Starting with v0.10, you can indicate the data type for the whole DataFrame or individual columns:

In [1035]: data = 'a,b,c\n1,2,3\n4,5,6\n7,8,9'

In [1036]: print data
a,b,c
1,2,3
4,5,6
7,8,9

In [1037]: df = pd.read_csv(StringIO(data), dtype=object)

In [1038]: df
Out[1038]: 
   a  b  c
0  1  2  3
1  4  5  6
2  7  8  9

In [1039]: df['a'][0]
Out[1039]: '1'

In [1040]: df = pd.read_csv(StringIO(data), dtype={'b': object, 'c': np.float64})

In [1041]: df.dtypes
Out[1041]: 
a      int64
b     object
c    float64
dtype: object

Handling column names

A file may or may not have a header row. pandas assumes the first row should be used as the column names:

In [1042]: from StringIO import StringIO

In [1043]: data = 'a,b,c\n1,2,3\n4,5,6\n7,8,9'

In [1044]: print data
a,b,c
1,2,3
4,5,6
7,8,9

In [1045]: pd.read_csv(StringIO(data))
Out[1045]: 
   a  b  c
0  1  2  3
1  4  5  6
2  7  8  9

By specifying the names argument in conjunction with header you can indicate other names to use and whether or not to throw away the header row (if any):

In [1046]: print data
a,b,c
1,2,3
4,5,6
7,8,9

In [1047]: pd.read_csv(StringIO(data), names=['foo', 'bar', 'baz'], header=0)
Out[1047]: 
   foo  bar  baz
0    1    2    3
1    4    5    6
2    7    8    9

In [1048]: pd.read_csv(StringIO(data), names=['foo', 'bar', 'baz'], header=None)
Out[1048]: 
  foo bar baz
0   a   b   c
1   1   2   3
2   4   5   6
3   7   8   9

If the header is in a row other than the first, pass the row number to header. This will skip the preceding rows:

In [1049]: data = 'skip this skip it\na,b,c\n1,2,3\n4,5,6\n7,8,9'

In [1050]: pd.read_csv(StringIO(data), header=1)
Out[1050]: 
   a  b  c
0  1  2  3
1  4  5  6
2  7  8  9

Filtering columns (usecols)

The usecols argument allows you to select any subset of the columns in a file, either using the column names or position numbers:

In [1051]: data = 'a,b,c,d\n1,2,3,foo\n4,5,6,bar\n7,8,9,baz'

In [1052]: pd.read_csv(StringIO(data))
Out[1052]: 
   a  b  c    d
0  1  2  3  foo
1  4  5  6  bar
2  7  8  9  baz

In [1053]: pd.read_csv(StringIO(data), usecols=['b', 'd'])
Out[1053]: 
   b    d
0  2  foo
1  5  bar
2  8  baz

In [1054]: pd.read_csv(StringIO(data), usecols=[0, 2, 3])
Out[1054]: 
   a  c    d
0  1  3  foo
1  4  6  bar
2  7  9  baz

Dealing with Unicode Data

The encoding argument should be used for encoded unicode data, which will result in byte strings being decoded to unicode in the result:

In [1055]: data = 'word,length\nTr\xe4umen,7\nGr\xfc\xdfe,5'

In [1056]: df = pd.read_csv(StringIO(data), encoding='latin-1')

In [1057]: df
Out[1057]: 
      word  length
0  Träumen       7
1    Grüße       5

In [1058]: df['word'][1]
Out[1058]: u'Gr\xfc\xdfe'

Some formats which encode all characters as multiple bytes, like UTF-16, won’t parse correctly at all without specifying the encoding.

Index columns and trailing delimiters

If a file has one more column of data than the number of column names, the first column will be used as the DataFrame’s row names:

In [1059]: data = 'a,b,c\n4,apple,bat,5.7\n8,orange,cow,10'

In [1060]: pd.read_csv(StringIO(data))
Out[1060]: 
        a    b     c
4   apple  bat   5.7
8  orange  cow  10.0

In [1061]: data = 'index,a,b,c\n4,apple,bat,5.7\n8,orange,cow,10'

In [1062]: pd.read_csv(StringIO(data), index_col=0)
Out[1062]: 
            a    b     c
index                   
4       apple  bat   5.7
8      orange  cow  10.0

Ordinarily, you can achieve this behavior using the index_col option.

There are some exception cases when a file has been prepared with delimiters at the end of each data line, confusing the parser. To explicitly disable the index column inference and discard the last column, pass index_col=False:

In [1063]: data = 'a,b,c\n4,apple,bat,\n8,orange,cow,'

In [1064]: print data
a,b,c
4,apple,bat,
8,orange,cow,

In [1065]: pd.read_csv(StringIO(data))
Out[1065]: 
        a    b   c
4   apple  bat NaN
8  orange  cow NaN

In [1066]: pd.read_csv(StringIO(data), index_col=False)
Out[1066]: 
   a       b    c
0  4   apple  bat
1  8  orange  cow

Specifying Date Columns

To better facilitate working with datetime data, read_csv() and read_table() uses the keyword arguments parse_dates and date_parser to allow users to specify a variety of columns and date/time formats to turn the input text data into datetime objects.

The simplest case is to just pass in parse_dates=True:

# Use a column as an index, and parse it as dates.
In [1067]: df = pd.read_csv('foo.csv', index_col=0, parse_dates=True)

In [1068]: df
Out[1068]: 
            A  B  C
date               
2009-01-01  a  1  2
2009-01-02  b  3  4
2009-01-03  c  4  5

# These are python datetime objects
In [1069]: df.index
Out[1069]: 
<class 'pandas.tseries.index.DatetimeIndex'>
[2009-01-01 00:00:00, ..., 2009-01-03 00:00:00]
Length: 3, Freq: None, Timezone: None

It is often the case that we may want to store date and time data separately, or store various date fields separately. the parse_dates keyword can be used to specify a combination of columns to parse the dates and/or times from.

You can specify a list of column lists to parse_dates, the resulting date columns will be prepended to the output (so as to not affect the existing column order) and the new column names will be the concatenation of the component column names:

In [1070]: print open('tmp.csv').read()
KORD,19990127, 19:00:00, 18:56:00, 0.8100
KORD,19990127, 20:00:00, 19:56:00, 0.0100
KORD,19990127, 21:00:00, 20:56:00, -0.5900
KORD,19990127, 21:00:00, 21:18:00, -0.9900
KORD,19990127, 22:00:00, 21:56:00, -0.5900
KORD,19990127, 23:00:00, 22:56:00, -0.5900

In [1071]: df = pd.read_csv('tmp.csv', header=None, parse_dates=[[1, 2], [1, 3]])

In [1072]: df
Out[1072]: 
                  1_2                 1_3     0     4
0 1999-01-27 19:00:00 1999-01-27 18:56:00  KORD  0.81
1 1999-01-27 20:00:00 1999-01-27 19:56:00  KORD  0.01
2 1999-01-27 21:00:00 1999-01-27 20:56:00  KORD -0.59
3 1999-01-27 21:00:00 1999-01-27 21:18:00  KORD -0.99
4 1999-01-27 22:00:00 1999-01-27 21:56:00  KORD -0.59
5 1999-01-27 23:00:00 1999-01-27 22:56:00  KORD -0.59

By default the parser removes the component date columns, but you can choose to retain them via the keep_date_col keyword:

In [1073]: df = pd.read_csv('tmp.csv', header=None, parse_dates=[[1, 2], [1, 3]],
   ......:                  keep_date_col=True)
   ......:

In [1074]: df
Out[1074]: 
                  1_2                 1_3     0         1          2  \
0 1999-01-27 19:00:00 1999-01-27 18:56:00  KORD  19990127   19:00:00   
1 1999-01-27 20:00:00 1999-01-27 19:56:00  KORD  19990127   20:00:00   
2 1999-01-27 21:00:00 1999-01-27 20:56:00  KORD  19990127   21:00:00   
3 1999-01-27 21:00:00 1999-01-27 21:18:00  KORD  19990127   21:00:00   
4 1999-01-27 22:00:00 1999-01-27 21:56:00  KORD  19990127   22:00:00   
5 1999-01-27 23:00:00 1999-01-27 22:56:00  KORD  19990127   23:00:00   
           3     4  
0   18:56:00  0.81  
1   19:56:00  0.01  
2   20:56:00 -0.59  
3   21:18:00 -0.99  
4   21:56:00 -0.59  
5   22:56:00 -0.59

Note that if you wish to combine multiple columns into a single date column, a nested list must be used. In other words, parse_dates=[1, 2] indicates that the second and third columns should each be parsed as separate date columns while parse_dates=[[1, 2]] means the two columns should be parsed into a single column.

You can also use a dict to specify custom name columns:

 
 

            
  

           

              
              

            

            
相關推薦

			   
            
            
            
 

    


    
    
pandas io tools（使用python處理資料時候經常用到）

     
 

CSV & Text files
The two workhorse functions for reading text files (a.k.a. flat files) are read_csv() and read_table().
 They both use the same  



  
 

    


    
    
pandas io tools（使用python處理資料時候經常用到）讀csv，TXT

     
 

CSV & Text files
The two workhorse functions for reading text files (a.k.a. flat files) are read_csv() and read_table().
 They both use the same  



  
 

    


    
    
Python爬蟲視訊教程：教你爬取QQ音樂資料（實戰處理+資料視覺化）-劉宇宙-專題視訊課程...

     
 
                Python爬蟲視訊教程：教你爬取QQ音樂資料（實戰處理+資料視覺化）—704人已學習 課程介紹        本視訊課程主要培訓Python爬蟲入門，資料分析及資料視覺化實戰內容，通過本課的學習，您可以在2小時左右掌握Python基礎程式設計的核心內容，實現Python在爬 



  
 

    


    
    
數字影象處理資料集錦（Python、C++、Matlab）

     
  
 
 一、Numpy的基礎 
 
   python中numpy與matlab的對應關係  
 
 二、 基於skimage數字影象處理 
 
   python數字影象處理（1）：環境安裝與配置  
   python數字影象處理（2）：影象的讀取、顯示與儲存  
   python數字影象處 



  
 

    


    
    
利用正則表示式處理爬取的今日頭條內容資料（Python爬蟲資料清洗）

     
  
 
 
 本次要處理的是抓取的頭條內容，內容如下： 
 content = '''content: '&lt;div&gt;&lt;blockquote&gt;&lt;p&gt;&lt;strong&gt;想要更多科技類資訊，歡迎關注公眾號“ 



  
 

    


    
    
python資料預處理（抽樣、資料集轉換）

     
 
                
博文：python大規模資料處理技巧之二：機器學習中常用操作（http://blog.csdn.net/asdfg4381/article/details/51725424）



1、 資料預處理

隨機化操作


機器學習中的常用隨機化操作中可以使用random包做不重 



  
 

    


    
    
Selenium遇到的問題8 python利用xlwt模組操作xls資料顯示IO錯誤（Python中正反斜槓的用法）

     
 

（一）目錄中的斜槓們

python讀檔案需要輸入的目錄引數，列出以下例子：

path = r"C:\Windows\temp\readme.txt"

path1 = r"c:\windows\temp\readme.txt"

path2 = "c:\\windows\\temp\\readme.t 



  
 

    


    
    
python pandas IO tools 之read_csv檔案讀寫引數詳解

     
 
							
							
							python pandas IO tools 之csv檔案讀寫



讀取csv檔案：pd.read_csv(),寫入csv檔案:pd.to_csv() 
pandas還可以讀取一下檔案： 
read_csv, 
read_excel, 
read_hdf, 
 



  
 

    


    
    
python處理資料的風騷操作[pandas 之 groupby&agg]

     
 
                介紹每隔一段時間我都會去學習、回顧一下python中的新函式、新操作。這對於你後面的工作是有一定好處的。本文重點介紹了pandas中groupby、Grouper和agg函式的使用。這2個函式作用類似，都是對資料集中的一類屬性進行聚合操作，比如統計一個使用者在每個月內的全部花 



  
 

    


    
    
Python處理資料—生成散點圖

     
  
 
 Python作為一門工具，具有極強大的處理資料的能力，今天我們看一下如何用python生成資料散點圖 
 我們先新建一個TXT檔案，裡面儲存10組資料，每組資料4個值，分別代表運動的時間，玩遊戲所佔一天時間的百分比，每週吃垃圾食品的次數，以及個人魅力的分類（1-3），如下： 
 100.00 0.2 



  
 

    


    
    
用10分鐘，搭建影象處理程式設計環境，0失敗！（python語言，windows系統）

     
 
                以前，你可能看過很多的文章，開始搭建一個影象處理的程式設計環境。

結果，按照教程一步一步做的時候，總是出現各種各樣的問題。

就算成功了，後續開發過程中要用到不同版本的opencv，不同版本python，更換特別麻煩。



今天，我以我多年的經驗，給你一個安裝教程。

零 



  
 

    


    
    
python讀取excel（Python處理xls）

     
 
                

前言
在很多應用場景中，程式都需要讀取excel的配置。
比如，在遊戲中，數值策劃用excel 建好了數值表，我們要匯入到遊戲中。

概述
這裡我使用了 pyexcel-xls 這個東西。我個人覺得這個東西匯出的資料結構，結構可讀性很強，資料結構如下：
整個excel檔案 



  
 

    


    
    
python處理資料

     
 
							
							
							1、遇到的情況



print u'%i行，%i列' %(total_data.shape[0], total_data.shape[1])#total_data.shape[0]行數，total_data.shape[1]列數
LTE_total_data  



  
 

    


    
    
python處理資料——去除字串兩端的引號

     
 
							
							
							在用python處理資料，會出現獲得的資料本身兩端帶有引號，而我們需要的是形如xxx,而不是“xxx”否則就會出現問題。比如： 


『解決方法一：』 
使用lstrip()和rsrtip()字串函式

函式說明如下：


  str.lstrip([chars 



  
 

    


    
    
重要的Python庫（利用Python進行資料分析筆記）

     
 
							
							
							NumPy(Numerical Python)


快速高效的多維陣列物件ndarray
用於對陣列執行元素級計算以及直接對陣列執行數學運算的函式
線性代數運算、傅立葉變換，以及隨機數生成
用於將C、C++,Fortran程式碼整合到Python的工具




 



  
 

    


    
    
鐳射雷達資料解析（Python-lidar-data-analysis_V1.0）

     
 
                        2017年9月3號，杭州天氣燥熱，跑這來讀研了，換專業了，一切從頭開始。        上手的第一件事，解析鐳射雷達資料。東西全新，師兄師姐們都不弄這。只能自己硬著頭皮來。想想自己真機智，來學校前在家就先看了python語言（這是個半殘廢的小程式碼工程，最後不 



  
 

    


    
    
計蒜客15430 XOR Queries（Trie處理位運算問題）

     
 ron   二進制   進制   插入   我們   整數   容易   位置   xor   題意：
給出一個長度為n的數組C，回答m個形式為(L, R, A, B)的詢問，
含義為存在多少個不同的數組下標k屬於[L, R]滿足C[k] XOR A >= B（式中XOR為異或運算）。
T組測試數 



  
 

    


    
    
Android簡單的編寫一個txt閱讀器（沒有處理字符編碼），適用於新手學習

     
 選項   tro   源碼   tin   open   打開文件   package   idt   pac   　　本程序只是使用了一些基本的知識點編寫了一個比較簡單粗陋的txt文本閱讀器，效率不高，只適合新手練習。所以大神勿噴。
　　其實想到編寫這種程序源自本人之前喜歡看小說，而很多小說更新太慢，所以本 



  
 

    


    
    
Sublime 3安裝（Python開發插件安裝）

     
 默認   lac   intel   交互   style   rmi   pla   you   下載   Sublime 3安裝（Python開發插件安裝）									
				
										
					
				
									
						1，下載Sublime3https://ww 



  
 

    


    
    
Codeforces 946D Timetable（預處理+分組背包）

     
 鏈接   pre   AI   分組背包   std   inf   time   pro   can   題目鏈接：http://codeforces.com/problemset/problem/946/D
題目大意：有n個字符串，代表n天的課表，1表示這個時間要上課，0表示不要上課，一天在學校時間為第