python工具庫介紹-requests:人性化的HTTP
Requests是Python基於ofollow,noindex">Apache2 Licensed 許可證的人性化HTTP庫。
Python標準庫中urllib2提供了不少HTTP 功能,但API不繫統。它有點過時,完成最簡單的任務也需要大量工作。
下面我們用例項演示訪問github。
>>> import requests >>> r = requests.get('https://api.github.com/user', auth=('[email protected]', 'password')) >>> r.status_code 200 >>> r.headers['content-type'] 'application/json; charset=utf-8' >>> r.encoding 'utf-8' >>> r.text u'{"login":"oychw",...}' >>> r.json() {u'disk_usage': 176, u'private_gists': 0, ...}
Requests為Python處理了所有HTTP/1.1操作, 與Web服務的無縫整合。不需要為URL手動新增查詢字串或POST資料進行表單處理。基於urllib3 , 能自動處理Keep-alive和HTTP連線池。
特點:
-
國際化域名和 URLs
-
Keep-Alive & 連線池
-
持久的 Cookie 會話
-
類瀏覽器的SSL認證
-
基本/摘要式的身份認證
-
優雅的鍵/值 Cookie
-
自動解壓
-
Unicode響應體
-
多段檔案上傳
-
連線超時
-
支援 .netrc
-
適用於 Python 2.6—3.4
-
執行緒安全
使用者手冊
簡介
Requests關注PEP 20的部分:
-
Beautiful is better than ugly.(美麗優於醜陋)
-
Explicit is better than implicit.(明確優於含糊)
-
Simple is better than complex.(簡單優於複雜)
-
Complex is better than complicated.(複雜優於繁瑣)
-
Readability counts.(可讀性)
安裝
-
安裝: 推薦:pip install requests,其次:easy_install requests
-
最新程式碼:
-
下載tar.gz包:curl -OLhttps://github.com/kennethreitz/requests/tarball/master
-
下載zip包:curl -OLhttps://github.com/kennethreitz/requests/zipball/master
-
原始碼安裝:python setup.py install
快速入門
- 傳送請求:
下面獲取Github的公共時間線,並在httpbin演示其他HTTP操作:
>>> import requests >>> r = requests.get('https://github.com/timeline.json') >>> r = requests.post("http://httpbin.org/post") >>> r = requests.put("http://httpbin.org/put") >>> r = requests.delete("http://httpbin.org/delete") >>> r = requests.head("http://httpbin.org/get") >>> r = requests.options("http://httpbin.org/get")
- 在URL中傳遞引數
URL的查詢字串(query string)例如, httpbin.org/get?key=val,在Requests可以用字典的形式構建。比如傳遞key1=value1和key2=value2到 httpbin.org/get:
>>> import requests >>> payload = {'key1': 'value1', 'key2': 'value2'} >>> r = requests.get("http://httpbin.org/get", params=payload) >>> print(r.url) http://httpbin.org/get?key2=value2&key1=value1 >>> payload = {'key1': 'value1', 'key2[]': ['value2', 'value3']} >>> r = requests.get("http://httpbin.org/get", params=payload) >>> print(r.url) http://httpbin.org/get?key1=value1&key2%5B%5D=value2&key2%5B%5D=value3
注意字典裡值為None的鍵會忽略。上面第2個例子訪問的是http://httpbin.org/get?key1=value1&key2[]=value2&key2[]=value3 。注意key後面需要新增中括號對。
- 響應
>>> import requests >>> r = requests.get('https://api.github.com/events') >>> r.text >>> u'[{"id":"2636319727","type":"PullRequestReviewCommentEvent","actor":{"id":1148601,"login":"i ...}] >>> r.encoding 'utf-8' >>> r.encoding = 'ISO-8859-1'
Requests會自動解碼伺服器的返回。大多數unicode字符集都能無縫解碼。請求發出時Requests會基於響應的HTTP頭部推測響應的編碼。同時還可以設定和查詢編碼。改變編碼後,訪問 r.text 將會使用 r.encoding 。
- 二進位制響應
r.content可以以位元組的方式顯示響應。
>>> r.content b'[{"repository":{"open_issues":0,"url":"https://github.com/...
傳輸格式gzip和deflate會自動轉碼。處理圖片例項:
>>> from PIL import Image >>> from StringIO import StringIO >>> i = Image.open(StringIO(r.content))
- Json響應
Requests內建了JSON解碼器:
>>> import requests >>> r = requests.get('https://github.com/timeline.json') >>> r.json() {u'documentation_url': u'https://developer.github.com/v3/activity/events/#list-public-events', u'message': u"Hello there, wayfaring stranger. If you're reading this then you probably didn't see our blog post a couple of years back announcing that this API would go away: http://git.io/17AROg Fear not, you should be able to get what you need from the shiny new Events API instead."}
JSON解碼失敗時r.json 就會丟擲異常。例如, 401 (Unauthorized) , ValueError: No JSON object could be decoded等。
- 原始響應
極端的情況下需要檢視伺服器的原始套接字響應,請求時設定 stream=True:
>>> import requests >>> r = requests.get('https://api.github.com/events', stream=True) >>> r.raw <urllib3.response.HTTPResponse object at 0x7f807dd6f4d0> >>> r.raw.read(10) '\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03'
通常需要存為檔案:
with open(filename, 'wb') as fd: for chunk in r.iter_content(chunk_size): fd.write(chunk)
Response.iter_content 能減少直接使用Response.raw的大量處理,下載流時尤其推薦。
- 自定義頭
>>> import requests >>> import json >>> url = 'https://api.github.com/some/endpoint' >>> payload = {'some': 'data'} >>> headers = {'content-type': 'application/json'} >>> r = requests.post(url, data=json.dumps(payload), headers=headers)
- 更加複雜的POST請求
表單直接以字典形式傳送:
>>> import requests >>> payload = {'key1': 'value1', 'key2': 'value2'} >>> r = requests.post("http://httpbin.org/post", data=payload) >>> print(r.text) { "args": {}, "data": "", "files": {}, "form": { "key1": "value1", "key2": "value2" }, "headers": { "Accept": "*/*", "Accept-Encoding": "gzip, deflate, compress", "Content-Length": "23", "Content-Type": "application/x-www-form-urlencoded", "Host": "httpbin.org", "User-Agent": "python-requests/2.2.1 CPython/2.7.6 Linux/3.13.0-53-generic" }, "json": null, "origin": "119.122.150.177", "url": "http://httpbin.org/post" }
string則會被直接釋出出去。Github API v3中接受編碼為JSON的POST/PATCH資料
>>> import requests >>> import json >>> url = 'https://api.github.com/some/endpoint' >>> payload = {'some': 'data'} >>> r = requests.post(url, data=json.dumps(payload))
- POST複雜編碼的檔案
>>> import requests >>> url = 'http://httpbin.org/post' >>> files = {'file': open('/home/andrew/test.xls', 'rb')} >>>r = requests.post(url, files=files) >>> url = 'http://httpbin.org/post' >>> r.text u'{\n"args": {}, \n"data": "", \n"files": {\n ..."url": "http://httpbin.org/post"\n}\n'
可以顯式地設定檔名,檔案型別和請求頭:
>>> import requests >>> files = {'file': ('report.xls', open('/home/andrew/test.xls', 'rb'), 'application/vnd.ms-excel', {'Expires': '0'})} >>> url = 'http://httpbin.org/post' >>> r = requests.post(url, files=files) >>> r.text u'{\n"args": {}, \n"data": ""..."url": "http://httpbin.org/post"\n}\n'
還可以直接用文字代替檔案:
>>> import requests >>> url = 'http://httpbin.org/post' >>> files = {'file': ('report.csv', 'some,data,to,send\nanother,row,to,send\n')} >>> r = requests.post(url, files=files) >>> r.text u'{\n"args": {}, \n"data": "", \n"files": {\n..."json": null, \n"origin": "14.153.22.104", \n"url": "http://httpbin.org/post"\n}\n'
multipart/form-data不支援特別大的檔案,建議使用requests-toolbelt,參考:toolbelt
- 響應狀態碼
>>> import requests >>> r = requests.get('http://httpbin.org/get') >>> r.status_code 200 >>> r.status_code == requests.codes.ok True >>> bad_r = requests.get('http://httpbin.org/status/404') >>> bad_r.status_code 404 >>> bad_r.raise_for_status() >>> r.raise_for_status()
上面的requests.codes.ok是內建的狀態碼查詢物件。可以使用 Response.raise_for_status()跑出失敗請求(4XX客戶端錯誤或5XX伺服器異常),我們可以通過 Response.raise_for_status() 來丟擲異常。r的返回為200,所以返回None,不產生異常。
- 響應頭
>>> r.headers {'content-length': '275', 'server': 'nginx', 'connection': 'keep-alive', 'access-control-allow-credentials': 'true', 'date': 'Tue, 10 Mar 2015 08:21:36 GMT', 'access-control-allow-origin': '*', 'content-type': 'application/json'} >>> r.headers['Content-Type'] 'application/json' >>> r.headers.get('content-type') 'application/json'
根據 RFC 2616,HTTP頭部不區分大小寫。根據RFC 7230,接收方會對服務端對同一key的不同value進行組合。
- Cookies
可以訪問響應中包含的Cookie:
>>> import requests >>> url = 'http://automationtesting.sinaapp.com/login' >>> r = requests.get(url) >>> r.cookies.keys() ['saeut', 'trac_form_token', 'trac_session'] >>> r.cookies['saeut'] 'CkMPGlT+tfQiXS9uGYviAg=='
使用cookies引數可以傳送你的cookies到伺服器:
>>> import requests >>> url = 'http://httpbin.org/cookies' >>> cookies = dict(cookies_are='working') >>> r = requests.get(url, cookies=cookies) >>> r.text u'{\n"cookies": {\n"cookies_are": "working"\n}\n}\n'
- 重定向與請求歷史
預設對HEAD以外其他所有動作進行位置重定向。Response.history可以看到重定向的記錄。
>>> import requests >>> r = requests.get('http://github.com') >>> r.url u'https://github.com/' >>> r.status_code 200 >>> r.history [<Response [301]>]
GET, OPTIONS, POST, PUT, PATCH 或者 DELETE可以通過allow_redirects引數禁用重定向,這個設定對HEAD也生效:
>>> import requests >>> r = requests.get('http://github.com', allow_redirects=False) >>> r.status_code 301 >>> r.history [] >>> r = requests.head('http://github.com', allow_redirects=True) >>> r.url u'https://github.com/' >>> r.history [<Response [301]>]
- 超時
超時告訴requests在經過timeout引數的秒之後停止等待響應:
>>> import requests >>> requests.get('http://github.com', timeout=0.1) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python2.7/dist-packages/requests/api.py", line 55, in get return request('get', url, **kwargs) File "/usr/lib/python2.7/dist-packages/requests/api.py", line 44, in request return session.request(method=method, url=url, **kwargs) File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 455, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 558, in send r = adapter.send(request, **kwargs) File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 387, in send raise Timeout(e) requests.exceptions.Timeout: (<urllib3.connectionpool.HTTPConnectionPool object at 0x7f807dd6f050>, 'Connection to github.com timed out. (connect timeout=0.1)') >>>
注: 超時不是對整個響應下載的時間限制, 而且指定時間沒有收到伺服器返回就丟擲異常。
- 錯誤與異常
ConnectionError:網路問題(如DNS失敗、拒絕連線等)。
HTTPError: 比較罕見的無效HTTP響應時。
Timeout:請求超時。
TooManyRedirects:超過了設定的最大重定向次數。
requests.exceptions.RequestException是所有具體異常的基類。
高階用法
- Session物件
Session物件能夠跨請求保持引數,Session例項發出的所有請求共享cookies。
Session物件具有主Requests API的所有方法。
>>> s = requests.Session() >>> s.get('http://httpbin.org/cookies/set/sessioncookie/123456789') <Response [200]> >>> r = s.get("http://httpbin.org/cookies") >>> print(r.text) { "cookies": { "sessioncookie": "123456789" } }
Session也可為request方法提供預設資料,新增屬性即可:
>>> import requests >>> s = requests.Session() >>> s.auth = ('user', 'pass') >>> s.headers.update({'x-test': 'true'}) >>> s.get('http://httpbin.org/headers', headers={'x-test2': 'true'}) <Response [200]>
傳遞給request方法的字典都會與已有session層的值合併。方法層的引數會覆蓋會話的引數。在方法層引數中將鍵值設定為None,會被自動忽略key。參考:session api
- 請求(Request)和響應(Response)物件
requests.get()等請求主要做兩件的事情。一為構建Request 物件。二為收到伺服器響應時產生Response 物件。Response物件包含伺服器返回和原來的 Request 物件。
>>> import requests >>> r = requests.get('http://en.wikipedia.org/wiki/Monty_Python') >>> r.headers {'content-length': '67559', ...} >>> r.request.headers {'Connection': 'keep-alive', 'Accept-Encoding': ...}
- 預請求
當從API或會話呼叫接收Response物件時,request屬性實際上是PreparedRequest。如果你需要修改body或header,可以如下方式進行處理:
from requests import Request, Session s = Session() req = Request('GET', url, data=data, headers=header ) prepped = req.prepare() # do something with prepped.body # do something with prepped.headers resp = s.send(prepped, stream=stream, verify=verify, proxies=proxies, cert=cert, timeout=timeout ) print(resp.status_code)
這裡沒有對Request物件進行特殊處理,而是修改PreparedRequest物件。然後用requests.*
或Session.*
.傳送。
上述程式碼沒有Request Session。Session層狀態,如cookie不會使用。用Session.prepare_request()替換
Request.prepare()
即可增加狀態支援:
from requests import Request, Session s = Session() req = Request('GET',url, data=data headers=headers ) prepped = s.prepare_request(req) # do something with prepped.body # do something with prepped.headers resp = s.send(prepped, stream=stream, verify=verify, proxies=proxies, cert=cert, timeout=timeout ) print(resp.status_code)
- SSL證書驗證
使用 verify 引數可以像web瀏覽器一樣為HTTPS請求驗證SSL證書:
參考資料
- requests英文文件
- requests-docs-cn.readthedocs.org
- docs.python-requests.org/en/latest/
- https://github.com/china-testing/python-api-tesing#http
- 本文涉及的python測試開發庫 謝謝點贊!
- 本文相關海量書籍下載