Scrapy提供了一個內建的重複請求過濾器，用於根據網址過濾重複的請求。可以根據業務需求制定規則過濾重複的請求

基於url過濾重複的請求

假設已訪問過

http:://www.abc.com/p/xyz.html?id=1234&refer=4567

我想做的是過濾請求，如：

http:://www.abc.com/p/xyz.html?id=1234&refer=5678

通過編寫自定義中介軟體以進行重複刪除，並將其新增到設定中

import os

from scrapy.dupefilter import RFPDupeFilter
from scrapy.utils.request import
 
 request_fingerprint

class CustomFilter(RFPDupeFilter):
"""A dupe filter that considers specific ids in the url"""

    def __getid(self, url):
        mm = url.split("&refer")[0] #or something like that
        return mm

    def request_seen(self, request):
        fp = self.__getid(request.url)
        if
 
 fp in self.fingerprints:
            return True
        self.fingerprints.add(fp)
        if self.file:
            self.file.write(fp + os.linesep)

然後在settings.py中新增如下程式碼

DUPEFILTER_CLASS = 'scraper.duplicate_filter.CustomFilter'

不過濾重複任何請求

若在 scrapy 中，不過濾任何 request 請求，可以自定義如下檔案

from scrapy.dupefilter import
 
 RFPDupeFilter

class CloseDupefilter(RFPDupeFilter):
    def request_seen(self, request):
        return False

然後在settings.py中新增如下程式碼

DUPEFILTER_CLASS = 'scraper.duplicate_filter.CustomFilter'