第1.7章 scrapy之ip代理的使用
阿新 • • 發佈:2019-02-13
1 代理中介軟體
程式碼核心在於隨機選擇一條代理的ip和port,至於代理ip和port的來源,可以是購買的ip代理,也可以從網上爬取的。
# -*- coding: utf-8 -*-
'''
Created on 2017年6月14日
@author: dzm
'''
from eie.middlewares import udf_config
from eie.service.EieIpService import EieIpService
logger = udf_config.logger
eieIpService = EieIpService()
class ProxyMiddleware (object):
'''
ip代理中介軟體
'''
def process_request(self,request,spider):
'''
在請求上新增ip代理
'''
if request.url.startswith('http://'):
proxy = eieIpService.select_rand('HTTP')
if proxy:
proxies = 'http://{}:{}' .format(proxy['ip'],proxy['port'])
else:
raise Exception('沒有找到合適的IP代理')
else:
proxy = eieIpService.select_rand('HTTPS')
if proxy:
proxies = 'https://{}:{}'.format(proxy['ip'],proxy['port'])
else:
raise Exception('沒有找到合適的IP代理')
request.meta['proxy'] = proxies
2 settings配置
這裡需要注意RetryMiddleware和ProxyMiddleware之間的順序,否則會出現Python Scrapy不重試連線超時,scrapy會一致超時重連,我找遍QQ群,也沒人回答我的問題。
中介軟體的順序很重要,詳細描述可以參考:SCRAPY中DOWNLOADER_MIDDLEWARES中介軟體的配置順序
DOWNLOADER_MIDDLEWARES = {
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware':None,
'eie.middlewares.random_user_agent.RandomUserAgent': 100,
'eie.middlewares.proxy_middleware.ProxyMiddleware': 110,
'scrapy.downloadermiddlewares.retry.RetryMiddleware':120,
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 130,
}
可供參考的順序
DOWNLOADER_MIDDLEWARES_BASE
{
'scrapy.contrib.downloadermiddleware.robotstxt.RobotsTxtMiddleware': 100,
'scrapy.contrib.downloadermiddleware.httpauth.HttpAuthMiddleware': 300,
'scrapy.contrib.downloadermiddleware.downloadtimeout.DownloadTimeoutMiddleware': 350,
'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': 400,
'scrapy.contrib.downloadermiddleware.retry.RetryMiddleware': 500,
'scrapy.contrib.downloadermiddleware.defaultheaders.DefaultHeadersMiddleware': 550,
'scrapy.contrib.downloadermiddleware.redirect.MetaRefreshMiddleware': 580,
'scrapy.contrib.downloadermiddleware.httpcompression.HttpCompressionMiddleware': 590,
'scrapy.contrib.downloadermiddleware.redirect.RedirectMiddleware': 600,
'scrapy.contrib.downloadermiddleware.cookies.CookiesMiddleware': 700,
'scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware': 750,
'scrapy.contrib.downloadermiddleware.chunked.ChunkedTransferMiddleware': 830,
'scrapy.contrib.downloadermiddleware.stats.DownloaderStats': 850,
'scrapy.contrib.downloadermiddleware.httpcache.HttpCacheMiddleware': 900,
}
3 檢查ip代理是否成功
從下圖的response可以看出確實是採用了代理地址
4 代理的作用
ip代理及其作用與使用方法
使用代理IP的優缺點?
5 獲取隨機代理
def select_rand_zhimadaili(self):
'''
從芝麻代理中獲取ip代理清單
@see: http://http.zhimadaili.com/index/api/newapi.html
'''
try:
proxy_keys_list = self.redis.keys()
if proxy_keys_list:
key_list = range(1, len(proxy_keys_list))
key = key_list[random.choice(key_list)]
# proxy_list = json.load(self.redis.get(proxy_keys_list[key]))
strs = self.redis.get(proxy_keys_list[key])
return eval(strs)
else:
socket = urllib2.urlopen('http://http.zhimadaili.com/index/api/new_get_use_ips.html?num=20&type=2&pro=0&city=0&yys=0&port=1&time=1')
result = socket.read()
result = json.loads(result)
proxy_list = result['data']
for proxy in proxy_list:
name = '{}:{}'.format(proxy['ip'],proxy['port'])
self.redis.setex(name, proxy, 600)
socket.close()
key_list = range(1, len(proxy_list))
key = key_list[random.choice(key_list)]
return proxy_list[key]
except Exception,e:
logger.error('請求zhimadaili.com獲取代理失敗,失敗原因:%s' % e)
下面的方式是從資料庫中隨機找一條,資料庫是mysql
def select_rand_db(self,types=None):
if types:
sql = "select ip,port,types from eie_ip where types='{}' order by rand() limit 1".format(types)
else:
sql = "select ip,port,types from eie_ip order by rand() limit 1 "
df = pd.read_sql(sql,self.engine)
results = json.loads(df.to_json(orient='records'))
if results and len(results)==1:
return results[0]
return None