1. 程式人生 > >爬蟲的入門庫random,偽造隨機的請求頭,用法-案例篇

爬蟲的入門庫random,偽造隨機的請求頭,用法-案例篇

   今天,作為爬蟲小白的我,我又被IP了,我就想怎麼不被封,因為比較傻,不知道什麼叫閥,不知道什麼叫請求頭,什麼叫IP地址,就爬啊,爬啊,結果寫了半天的爬蟲被封了,我就問了下度娘,她說我是被IP訪問異常了,要麼手輸入驗證碼,要不然反爬,這些不懂,但是我懂了,一個東西,我的電腦一個東西查半天人家網站不買東西,人家不高興了,所以封了我,要我換個人去,網上說的,換頭,換裝置頭請求頭(相當於身份證的名字名字什麼的),換IP(IP地址,相當於身份證的地址),但是怎麼知道請求頭呢?那我告訴你 “”“”“”“

        那個網站?百度一下首頁!右下角怎麼找的?鍵盤上F12會不?

看到滑鼠位置沒?點選一下網路... >>>>

      左側點選百度網址,點選有“”這樣標誌的,說明這個是個HTML,不要點選圖片什麼的,然後看右邊點選接頭,滑鼠下拉到最後看到的User-Agent。

      怎麼換它呢?

from urllib.request import urlopen

from urllib.request import Request
import random
url ='http://www.baidu.com/'

user_agent_list = [
    "Mozilla/5.0(Macintosh;IntelMacOSX10.6;rv:2.0.1)Gecko/20100101Firefox/4.0.1",
    "Mozilla/4.0(compatible;MSIE6.0;WindowsNT5.1)",
    "Opera/9.80(WindowsNT6.1;U;en)Presto/2.8.131Version/11.11",
    "Mozilla/5.0(Macintosh;IntelMacOSX10_7_0)AppleWebKit/535.11(KHTML,likeGecko)Chrome/17.0.963.56Safari/535.11",
    "Mozilla/4.0(compatible;MSIE7.0;WindowsNT5.1)",
    "Mozilla/4.0(compatible;MSIE7.0;WindowsNT5.1;Trident/4.0;SE2.XMetaSr1.0;SE2.XMetaSr1.0;.NETCLR2.0.50727;SE2.XMetaSr1.0)"
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1",
    "Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11",
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6",
    "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6",
    "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/19.77.34.5 Safari/537.1",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.9 Safari/536.5",
    "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.36 Safari/536.5",
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
    "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_0) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
    "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
    "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
    "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
    "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.0 Safari/536.3",
    "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24"
]

print(random.choice(user_agent_list))
#打印出隨機選取出的偽造請求頭
headers={
    #從上面的列表上隨機取一個
    "User-Agent":random.choice(user_agent_list)
}
request=Request(url,headers=headers)
#帶上請求頭(裝置頭)訪問
response =urlopen(url)
info =response.read()
print(info.decode())