1)requests模塊
阿新 • • 發佈：2018-06-03
字符串 html addition 加密 http1 post請求 ati false bundle 
一：requests 介紹
requests 是使用 Apache2 Licensed 許可證的 基於Python開發的HTTP 庫，其在Python內置模塊的基礎上進行了高度的封裝，
從而使得Pythoner進行網絡請求時，變得美好了許多，使用requests可以輕而易舉的完成瀏覽器可有的任何操作。
二：requests 安裝
pip install requests
三：requests常用方法
response=requests.get(url) #以GET方式請求 response=requests.post(url) #以POST方式請求 response.text #獲取內容 response.content response.encoding #設置編碼格式 response.apparent_encoding#自動獲取編碼 response.code_status #200,404 #返回數據的狀態碼 response.cookies.get_dict() #獲取cookies信息 requests.get(url,cookie={}) requests.get requests.post requests.delete requests.request(‘get‘,#post,get,delete... )
四：requests常用參數
def request(method, url, **kwargs): """Constructs and sends a :class:`Request <Request>`. :param method: method for the new :class:`Request` object. :param url: URL for the new :class:`Request` object. :param params: (optional) Dictionary or bytes to be sent in the query string for the :class:`Request`. :param data: (optional) Dictionary, bytes, or file-like object to send in the body of the :class:`Request`. :param json: (optional) json data to send in the body of the :class:`Request`. :param headers: (optional) Dictionary of HTTP Headers to send with the :class:`Request`. :param cookies: (optional) Dict or CookieJar object to send with the :class:`Request`. :param files: (optional) Dictionary of ``‘name‘: file-like-objects`` (or ``{‘name‘: file-tuple}``) for multipart encoding upload. ``file-tuple`` can be a 2-tuple ``(‘filename‘, fileobj)``, 3-tuple ``(‘filename‘, fileobj, ‘content_type‘)`` or a 4-tuple ``(‘filename‘, fileobj, ‘content_type‘, custom_headers)``, where ``‘content-type‘`` is a string defining the content type of the given file and ``custom_headers`` a dict-like object containing additional headers to add for the file. :param auth: (optional) Auth tuple to enable Basic/Digest/Custom HTTP Auth. :param timeout: (optional) How long to wait for the server to send data before giving up, as a float, or a :ref:`(connect timeout, read timeout) <timeouts>` tuple. :type timeout: float or tuple :param allow_redirects: (optional) Boolean. Set to True if POST/PUT/DELETE redirect following is allowed. :type allow_redirects: bool :param proxies: (optional) Dictionary mapping protocol to the URL of the proxy. :param verify: (optional) whether the SSL cert will be verified. A CA_BUNDLE path can also be provided. Defaults to ``True``. :param stream: (optional) if ``False``, the response content will be immediately downloaded. :param cert: (optional) if String, path to ssl client cert file (.pem). If Tuple, (‘cert‘, ‘key‘) pair. :return: :class:`Response <Response>` object :rtype: requests.Response Usage:: >>> import requests >>> req = requests.request(‘GET‘, ‘http://httpbin.org/get‘) <Response [200]>View Code""" 參數列表
參數詳細說明
#參數 requests.request - method:提交方式 - url:提交地址 - params:在url上傳遞的參數 GET http://www.oldboyedu.com params={"k1":"v1","k2":"v2"} requests.request( method="GET, url="http://www.oldboyedu.com", params={"k1":"v1","k2":"v2"} ) http://www.oldboyedu.com?k1=v1&k2=v2 - data: 在請求體裏面傳遞的數據（字典，字節，字符串）（form表單提交以這種形式） requests.request( method="POST, url="http://www.oldboyedu.com", params={"k1":"v1","k2":"v2"} data={"user1":"alex","pwd":"123"｝ ) #以這種形式傳遞會在請求頭增加 content-type:application/x-www-form-urlencoded #這有什麽作用 在django裏面 request.POST是從request.body提取，就是根據application/x-www-form-urlencoded 判斷，如果你修改了 request.body有值 但是request.POST裏面沒有 #會把數據封裝成 user=alex&pwd=123 - json 在請求體裏傳遞的數據 requests.request( method="POST, url="http://www.oldboyedu.com", params={"k1":"v1","k2":"v2"} json={"user1":"alex","pwd":"123"}#當作字符串發送 ) #請求頭 content-type:application/json #會把數據封裝成字符串 {"user1":"alex","pwd":"123"}轉字符串 #這兩種有什麽區別： data={"user1":"alex","pwd":"123"，“x":[1,2,3]}這種data不行 json可以傳遞字典中嵌套字典時 - headers請求頭 requests.request( method="POST, url="http://www.oldboyedu.com", params={"k1":"v1","k2":"v2"} json={"user1":"alex","pwd":"123"} headers={ "Referer":"http://dig.chouti.com", #上次訪問的地址 "User-Agent":"...",是什麽客戶端發的 } ) - cookies cookies是怎麽發給服務器端，是放在請求頭裏面 - files 上傳文件 requests.request( method="POST, url="http://www.oldboyedu.com", files={ "f1":open("a.txt",‘rb"), 或者"f2":("文件名",open("a.txt",‘rb")) } ) - auth 認證 用的比較多的是路由器如FTP等彈出個彈框，輸入用戶名和密碼，。這種形式頁面輸入和輸出代碼都看不到http://httpbin.org - timeout 超時 - param_timeout #連接和發送的超時param_timeout=（5，1） - allow_redirects:是否重寫向跳轉 - proxies 權重或者代理 requests.post( url="http://www.oldboyedu.com", proxys={ "http":"http://4.19.128.5:8099" } ) 不會直接發oldboyedu，會先發代理，代理在發oldboyedu - stream get是先把東西下載到內在。stream一點一點下載。 - vertify - cert 提供證書 https服務器會先給客戶端發一個證書。服務器加密，客戶端解密 request.get( url="https://" cert="fuck.pem"#自己做的cent,還有第三方的證書 ) request.get( url="https://" cert=("fuck.crt‘,‘xxx.key‘) ) request.get( url="https://" vertify=False,hulei證書 ) - session：用於保存客戶端歷史訪問信息 - proixes #"http":"61.172.249.96:80" #"http":"root:[email protected]:80"
五：請求頭和請求體
##響應是也是有請求頭和請求體 不管是get,post 都需要發送HTTP請求，HTTP請求都包含請求頭和請求體。 請求頭和請求體如何分割 請求頭\r\n\r\n請求體 ###### 請求頭 Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8 Accept-Encoding:gzip, deflate, br Accept-Language:zh-CN,zh;q=0.9 請求頭上面每個KEY-VALUE是如何分割的。以\r\n分割 \r\n\r\n 請求體 上面形成一行，以\r\n或者\r\n\rn分割一起發送 ####如果是get請求，只會發請求頭 有個協議 Http1.1 / GET "/ 就是訪問的URL" #協議 Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8 Accept-Encoding:gzip, deflate, br Accept-Language:zh-CN,zh;q=0.9 #比如http://www.baidu.com?nid=1&v=1 Http1.1 http://www.baidu.com?nid=1&v=1 GET Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8 Accept-Encoding:gzip, deflate, br Accept-Language:zh-CN,zh;q=0.9 \r\n\r\n ###如果是POST請求 比如http://www.baidu.com?nid=1&v=1 Http1.1 / GET Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8 Accept-Encoding:gzip, deflate, br Accept-Language:zh-CN,zh;q=0.9 \r\n\r\n nid=1&v=1 ##響應 普通都是這樣 響應頭 Cache-Control:no-cache Content-Encoding:gzip 響應體 <html> </html> 如果是跳轉（重定向）就沒有響應體 響應頭 Cache-Control:no-cache Content-Encoding:gzip location:http://www.baidu.com #多了個跳轉地址 可以獲取響應碼301/302 或者通過響應頭獲取location 只在響應有有location就要可以跳轉
六：總結
#總結 #get參數 requests.get( url="http://www.baidu.com", params={"k1":"v1","k2":"v2"}, #傳遞的參數http://www.baidu.com？k1=v1&k2=v2 cookies={"c1":"v1","c2":"v2"}, #cookie在請求頭 headers={ "User-Agent": "Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Mobile Safari/537.36" ,#模擬瀏覽器,有些網站會檢查 "Referer":"htt", #瀏覽器上次訪問的地址，有的網站會檢查，如果不帶，網站會認為是爬沖 } ) 擴展 1. HTTP請求 - 頭 - 體 2. cookies - 請求放在請求頭 - 響應在響應頭 3. 重定向 - 響應頭
1)requests模塊