Python爬蟲:使用httpbin.org測試爬蟲請求頭資訊
阿新 • • 發佈:2019-02-16
一般程式寫的爬蟲程式都會自帶請求頭,不知不覺就被網站拒絕了,請求之前可以看看自己的請求頭是什麼,確保不被禁
如果網站太慢打不開,想在本地搭建測試環境,可以在docker環境下啟動:
$ docker run -p 80:80 kennethreitz/httpbin
{
args: { },
headers: {
Accept: "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
Accept-Encoding: "gzip, deflate, br" ,
Accept-Language: "zh-CN,zh;q=0.9,en;q=0.8",
Connection: "close",
Cookie: "_gauges_unique_day=1; _gauges_unique_month=1; _gauges_unique_year=1; _gauges_unique=1",
Host: "httpbin.org",
Upgrade-Insecure-Requests: "1",
User-Agent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
},
origin: "xxx.xxx.xx.xx",
url: "https://httpbin.org/get"
}