Scrapy-拓展類和settings.py配置文件
阿新 • • 發佈：2018-07-21
nbsp string handle receive received 緩存策略 typeerror 響應頭 twisted 
EXTENSIONS = {
# ‘scrapy.extensions.telnet.TelnetConsole‘: None,
‘pro_scrapy.extensions.MyExtend‘: 300,
}
from scrapy import signals class MyExtend: def __init__(self,crawler): self.crawler = crawler # 在指定信號上註冊操作 crawler.signals.connect(self.start, signals.engine_started) crawler.signals.connect(self.close, signals.spider_closed) @classmethoddef from_crawler(cls, crawler): return cls(crawler) def start(self): print(‘signals.engine_started.start‘) def close(self): print(‘signals.spider_closed.close‘)
# -*- coding: utf-8 -*- # Scrapy settings for step8_king projectsettings.py# # For simplicity, this file contains only settings considered important or # commonly used. You can find more settings consulting the documentation: # # http://doc.scrapy.org/en/latest/topics/settings.html # http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html # http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html# 1. 爬蟲名稱 BOT_NAME = ‘step8_king‘ # 2. 爬蟲應用路徑 SPIDER_MODULES = [‘step8_king.spiders‘] NEWSPIDER_MODULE = ‘step8_king.spiders‘ # Crawl responsibly by identifying yourself (and your website) on the user-agent # 3. 客戶端 user-agent請求頭 # USER_AGENT = ‘step8_king (+http://www.yourdomain.com)‘ # Obey robots.txt rules # 4. 禁止爬蟲配置 # ROBOTSTXT_OBEY = False # Configure maximum concurrent requests performed by Scrapy (default: 16) # 5. 並發請求數 # CONCURRENT_REQUESTS = 4 # Configure a delay for requests for the same website (default: 0) # See http://scrapy.readthedocs.org/en/latest/topics/settings.html#download-delay # See also autothrottle settings and docs # 6. 延遲下載秒數 # DOWNLOAD_DELAY = 2 # The download delay setting will honor only one of: # 7. 單域名訪問並發數，並且延遲下次秒數也應用在每個域名 # CONCURRENT_REQUESTS_PER_DOMAIN = 2 # 單IP訪問並發數，如果有值則忽略：CONCURRENT_REQUESTS_PER_DOMAIN，並且延遲下次秒數也應用在每個IP # CONCURRENT_REQUESTS_PER_IP = 3 # Disable cookies (enabled by default) # 8. 是否支持cookie，cookiejar進行操作cookie # COOKIES_ENABLED = True # COOKIES_DEBUG = True # Disable Telnet Console (enabled by default) # 9. Telnet用於查看當前爬蟲的信息，操作爬蟲等... # 使用telnet ip port ，然後通過命令操作 # TELNETCONSOLE_ENABLED = True # TELNETCONSOLE_HOST = ‘127.0.0.1‘ # TELNETCONSOLE_PORT = [6023,] # 10. 默認請求頭 # Override the default request headers: # DEFAULT_REQUEST_HEADERS = { # ‘Accept‘: ‘text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8‘, # ‘Accept-Language‘: ‘en‘, # } # Configure item pipelines # See http://scrapy.readthedocs.org/en/latest/topics/item-pipeline.html # 11. 定義pipeline處理請求 # ITEM_PIPELINES = { # ‘step8_king.pipelines.JsonPipeline‘: 700, # ‘step8_king.pipelines.FilePipeline‘: 500, # } # 12. 自定義擴展，基於信號進行調用 # Enable or disable extensions # See http://scrapy.readthedocs.org/en/latest/topics/extensions.html # EXTENSIONS = { # # ‘step8_king.extensions.MyExtension‘: 500, # } # 13. 爬蟲允許的最大深度，可以通過meta查看當前深度；0表示無深度 # DEPTH_LIMIT = 3 # 14. 爬取時，0表示深度優先Lifo(默認)；1表示廣度優先FiFo # 後進先出，深度優先 # DEPTH_PRIORITY = 0 # SCHEDULER_DISK_QUEUE = ‘scrapy.squeue.PickleLifoDiskQueue‘ # SCHEDULER_MEMORY_QUEUE = ‘scrapy.squeue.LifoMemoryQueue‘ # 先進先出，廣度優先 # DEPTH_PRIORITY = 1 # SCHEDULER_DISK_QUEUE = ‘scrapy.squeue.PickleFifoDiskQueue‘ # SCHEDULER_MEMORY_QUEUE = ‘scrapy.squeue.FifoMemoryQueue‘ # 15. 調度器隊列 # SCHEDULER = ‘scrapy.core.scheduler.Scheduler‘ # from scrapy.core.scheduler import Scheduler # 16. 訪問URL去重 # DUPEFILTER_CLASS = ‘step8_king.duplication.RepeatUrl‘ # Enable and configure the AutoThrottle extension (disabled by default) # See http://doc.scrapy.org/en/latest/topics/autothrottle.html """ 17. 自動限速算法 from scrapy.contrib.throttle import AutoThrottle 自動限速設置 1. 獲取最小延遲 DOWNLOAD_DELAY 2. 獲取最大延遲 AUTOTHROTTLE_MAX_DELAY 3. 設置初始下載延遲 AUTOTHROTTLE_START_DELAY 4. 當請求下載完成後，獲取其"連接"時間 latency，即：請求連接到接受到響應頭之間的時間 5. 用於計算的... AUTOTHROTTLE_TARGET_CONCURRENCY target_delay = latency / self.target_concurrency new_delay = (slot.delay + target_delay) / 2.0 # 表示上一次的延遲時間 new_delay = max(target_delay, new_delay) new_delay = min(max(self.mindelay, new_delay), self.maxdelay) slot.delay = new_delay """ # 開始自動限速 # AUTOTHROTTLE_ENABLED = True # The initial download delay # 初始下載延遲 # AUTOTHROTTLE_START_DELAY = 5 # The maximum download delay to be set in case of high latencies # 最大下載延遲 # AUTOTHROTTLE_MAX_DELAY = 10 # The average number of requests Scrapy should be sending in parallel to each remote server # 平均每秒並發數 # AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0 # Enable showing throttling stats for every response received: # 是否顯示 # AUTOTHROTTLE_DEBUG = True # Enable and configure HTTP caching (disabled by default) # See http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings """ 18. 啟用緩存 目的用於將已經發送的請求或相應緩存下來，以便以後使用 from scrapy.downloadermiddlewares.httpcache import HttpCacheMiddleware from scrapy.extensions.httpcache import DummyPolicy from scrapy.extensions.httpcache import FilesystemCacheStorage """ # 是否啟用緩存策略 # HTTPCACHE_ENABLED = True # 緩存策略：所有請求均緩存，下次在請求直接訪問原來的緩存即可 # HTTPCACHE_POLICY = "scrapy.extensions.httpcache.DummyPolicy" # 緩存策略：根據Http響應頭：Cache-Control、Last-Modified 等進行緩存的策略 # HTTPCACHE_POLICY = "scrapy.extensions.httpcache.RFC2616Policy" # 緩存超時時間 # HTTPCACHE_EXPIRATION_SECS = 0 # 緩存保存路徑 # HTTPCACHE_DIR = ‘httpcache‘ # 緩存忽略的Http狀態碼 # HTTPCACHE_IGNORE_HTTP_CODES = [] # 緩存存儲的插件 # HTTPCACHE_STORAGE = ‘scrapy.extensions.httpcache.FilesystemCacheStorage‘ """ 19. 代理，需要在環境變量中設置 from scrapy.contrib.downloadermiddleware.httpproxy import HttpProxyMiddleware 方式一：使用默認 os.environ { http_proxy:http://root:[email protected]:9999/ https_proxy:http://192.168.11.11:9999/ } 方式二：使用自定義下載中間件 def to_bytes(text, encoding=None, errors=‘strict‘): if isinstance(text, bytes): return text if not isinstance(text, six.string_types): raise TypeError(‘to_bytes must receive a unicode, str or bytes ‘ ‘object, got %s‘ % type(text).__name__) if encoding is None: encoding = ‘utf-8‘ return text.encode(encoding, errors) class ProxyMiddleware(object): def process_request(self, request, spider): PROXIES = [ {‘ip_port‘: ‘111.11.228.75:80‘, ‘user_pass‘: ‘‘}, {‘ip_port‘: ‘120.198.243.22:80‘, ‘user_pass‘: ‘‘}, {‘ip_port‘: ‘111.8.60.9:8123‘, ‘user_pass‘: ‘‘}, {‘ip_port‘: ‘101.71.27.120:80‘, ‘user_pass‘: ‘‘}, {‘ip_port‘: ‘122.96.59.104:80‘, ‘user_pass‘: ‘‘}, {‘ip_port‘: ‘122.224.249.122:8088‘, ‘user_pass‘: ‘‘}, ] proxy = random.choice(PROXIES) if proxy[‘user_pass‘] is not None: request.meta[‘proxy‘] = to_bytes（"http://%s" % proxy[‘ip_port‘]） encoded_user_pass = base64.encodestring(to_bytes(proxy[‘user_pass‘])) request.headers[‘Proxy-Authorization‘] = to_bytes(‘Basic ‘ + encoded_user_pass) print "**************ProxyMiddleware have pass************" + proxy[‘ip_port‘] else: print "**************ProxyMiddleware no pass************" + proxy[‘ip_port‘] request.meta[‘proxy‘] = to_bytes("http://%s" % proxy[‘ip_port‘]) DOWNLOADER_MIDDLEWARES = { ‘step8_king.middlewares.ProxyMiddleware‘: 500, } """ """ 20. Https訪問 Https訪問時有兩種情況： 1. 要爬取網站使用的可信任證書(默認支持) DOWNLOADER_HTTPCLIENTFACTORY = "scrapy.core.downloader.webclient.ScrapyHTTPClientFactory" DOWNLOADER_CLIENTCONTEXTFACTORY = "scrapy.core.downloader.contextfactory.ScrapyClientContextFactory" 2. 要爬取網站使用的自定義證書 DOWNLOADER_HTTPCLIENTFACTORY = "scrapy.core.downloader.webclient.ScrapyHTTPClientFactory" DOWNLOADER_CLIENTCONTEXTFACTORY = "step8_king.https.MySSLFactory" # https.py from scrapy.core.downloader.contextfactory import ScrapyClientContextFactory from twisted.internet.ssl import (optionsForClientTLS, CertificateOptions, PrivateCertificate) class MySSLFactory(ScrapyClientContextFactory): def getCertificateOptions(self): from OpenSSL import crypto v1 = crypto.load_privatekey(crypto.FILETYPE_PEM, open(‘/Users/wupeiqi/client.key.unsecure‘, mode=‘r‘).read()) v2 = crypto.load_certificate(crypto.FILETYPE_PEM, open(‘/Users/wupeiqi/client.pem‘, mode=‘r‘).read()) return CertificateOptions( privateKey=v1, # pKey對象 certificate=v2, # X509對象 verify=False, method=getattr(self, ‘method‘, getattr(self, ‘_ssl_method‘, None)) ) 其他： 相關類 scrapy.core.downloader.handlers.http.HttpDownloadHandler scrapy.core.downloader.webclient.ScrapyHTTPClientFactory scrapy.core.downloader.contextfactory.ScrapyClientContextFactory 相關配置 DOWNLOADER_HTTPCLIENTFACTORY DOWNLOADER_CLIENTCONTEXTFACTORY """ """ 21. 爬蟲中間件 class SpiderMiddleware(object): def process_spider_input(self,response, spider): ‘‘‘ 下載完成，執行，然後交給parse處理 :param response: :param spider: :return: ‘‘‘ pass def process_spider_output(self,response, result, spider): ‘‘‘ spider處理完成，返回時調用 :param response: :param result: :param spider: :return: 必須返回包含 Request 或 Item 對象的可叠代對象(iterable) ‘‘‘ return result def process_spider_exception(self,response, exception, spider): ‘‘‘ 異常調用 :param response: :param exception: :param spider: :return: None,繼續交給後續中間件處理異常；含 Response 或 Item 的可叠代對象(iterable)，交給調度器或pipeline ‘‘‘ return None def process_start_requests(self,start_requests, spider): ‘‘‘ 爬蟲啟動時調用 :param start_requests: :param spider: :return: 包含 Request 對象的可叠代對象 ‘‘‘ return start_requests 內置爬蟲中間件： ‘scrapy.contrib.spidermiddleware.httperror.HttpErrorMiddleware‘: 50, ‘scrapy.contrib.spidermiddleware.offsite.OffsiteMiddleware‘: 500, ‘scrapy.contrib.spidermiddleware.referer.RefererMiddleware‘: 700, ‘scrapy.contrib.spidermiddleware.urllength.UrlLengthMiddleware‘: 800, ‘scrapy.contrib.spidermiddleware.depth.DepthMiddleware‘: 900, """ # from scrapy.contrib.spidermiddleware.referer import RefererMiddleware # Enable or disable spider middlewares # See http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html SPIDER_MIDDLEWARES = { # ‘step8_king.middlewares.SpiderMiddleware‘: 543, } """ 22. 下載中間件 class DownMiddleware1(object): def process_request(self, request, spider): ‘‘‘ 請求需要被下載時，經過所有下載器中間件的process_request調用 :param request: :param spider: :return: None,繼續後續中間件去下載； Response對象，停止process_request的執行，開始執行process_response Request對象，停止中間件的執行，將Request重新調度器 raise IgnoreRequest異常，停止process_request的執行，開始執行process_exception ‘‘‘ pass def process_response(self, request, response, spider): ‘‘‘ spider處理完成，返回時調用 :param response: :param result: :param spider: :return: Response 對象：轉交給其他中間件process_response Request 對象：停止中間件，request會被重新調度下載 raise IgnoreRequest 異常：調用Request.errback ‘‘‘ print(‘response1‘) return response def process_exception(self, request, exception, spider): ‘‘‘ 當下載處理器(download handler)或 process_request() (下載中間件)拋出異常 :param response: :param exception: :param spider: :return: None：繼續交給後續中間件處理異常； Response對象：停止後續process_exception方法 Request對象：停止中間件，request將會被重新調用下載 ‘‘‘ return None 默認下載中間件 { ‘scrapy.contrib.downloadermiddleware.robotstxt.RobotsTxtMiddleware‘: 100, ‘scrapy.contrib.downloadermiddleware.httpauth.HttpAuthMiddleware‘: 300, ‘scrapy.contrib.downloadermiddleware.downloadtimeout.DownloadTimeoutMiddleware‘: 350, ‘scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware‘: 400, ‘scrapy.contrib.downloadermiddleware.retry.RetryMiddleware‘: 500, ‘scrapy.contrib.downloadermiddleware.defaultheaders.DefaultHeadersMiddleware‘: 550, ‘scrapy.contrib.downloadermiddleware.redirect.MetaRefreshMiddleware‘: 580, ‘scrapy.contrib.downloadermiddleware.httpcompression.HttpCompressionMiddleware‘: 590, ‘scrapy.contrib.downloadermiddleware.redirect.RedirectMiddleware‘: 600, ‘scrapy.contrib.downloadermiddleware.cookies.CookiesMiddleware‘: 700, ‘scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware‘: 750, ‘scrapy.contrib.downloadermiddleware.chunked.ChunkedTransferMiddleware‘: 830, ‘scrapy.contrib.downloadermiddleware.stats.DownloaderStats‘: 850, ‘scrapy.contrib.downloadermiddleware.httpcache.HttpCacheMiddleware‘: 900, } """ # from scrapy.contrib.downloadermiddleware.httpauth import HttpAuthMiddleware # Enable or disable downloader middlewares # See http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html # DOWNLOADER_MIDDLEWARES = { # ‘step8_king.middlewares.DownMiddleware1‘: 100, # ‘step8_king.middlewares.DownMiddleware2‘: 500, # } 復制代碼
Scrapy-拓展類和settings.py配置文件