使用ip代理池爬蟲時,requests模組get請求出現問題_AttributeError: 'str' object has no attribute 'get'
阿新 • • 發佈:2018-12-21
問題描述:專案使用ip代理池對網頁進行資料爬取,但是requests模組get方法出現問題,出錯如下:
File "E:\project\venv\lib\site-packages\requests\api.py", line 75, in get return request('get', url, params=params, **kwargs) File "E:\project\venv\lib\site-packages\requests\api.py", line 60, in request return session.request(method=method, url=url, **kwargs) File "E:\project\venv\lib\site-packages\requests\sessions.py", line 524, in request prep.url, proxies, stream, verify, cert File "E:\project\venv\lib\site-packages\requests\sessions.py", line 699, in merge_environment_settings no_proxy = proxies.get('no_proxy') if proxies is not None else None AttributeError: 'str' object has no attribute 'get'
程式碼如下:
proxies = "http://" + proxy
res = requests.get(url=url, headers=headers, proxies=proxies)
原因分析:
api.py中get方法定義如下:
def get(url, params=None, **kwargs): r"""Sends a GET request. :param url: URL for the new :class:`Request` object. :param params: (optional) Dictionary, list of tuples or bytes to send in the body of the :class:`Request`. :param \*\*kwargs: Optional arguments that ``request`` takes. :return: :class:`Response <Response>` object :rtype: requests.Response """
:param params: (optional) Dictionary, list of tuples or bytes to send
in the body of the :class:`Request`.
可以知道,傳入的引數必須是字典型別
修改後程式碼:
proxies = {
"http": "http://" + proxy
}
res = requests.get(url=url, headers=headers, proxies=proxies)
執行成功