1. 程式人生 > >Python爬蟲:使用httpbin.org測試爬蟲請求頭資訊

Python爬蟲:使用httpbin.org測試爬蟲請求頭資訊

一般程式寫的爬蟲程式都會自帶請求頭,不知不覺就被網站拒絕了,請求之前可以看看自己的請求頭是什麼,確保不被禁

如果網站太慢打不開,想在本地搭建測試環境,可以在docker環境下啟動:

$ docker run -p 80:80 kennethreitz/httpbin
{
    args: { },
    headers: {
    Accept: "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
    Accept-Encoding: "gzip, deflate, br"
, Accept-Language: "zh-CN,zh;q=0.9,en;q=0.8", Connection: "close", Cookie: "_gauges_unique_day=1; _gauges_unique_month=1; _gauges_unique_year=1; _gauges_unique=1", Host: "httpbin.org", Upgrade-Insecure-Requests: "1", User-Agent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
}, origin: "xxx.xxx.xx.xx", url: "https://httpbin.org/get" }