方法1： 修改setting.py中的User-Agent

# Crawl responsibly by identifying yourself (and your website) on the user-agent USER_AGENT = ‘Hello World’

方法2. 修改setting中的DEFAULT_REQUEST_HEADERS

# Override the default request headers: DEFAULT_REQUEST_HEADERS = { ‘Accept’: ‘text/html,application/xhtml+xml,application/xml;q=0.9,/

方法3. 在程式碼中修改。

class HeadervalidationSpider(scrapy.Spider): name = ‘headervalidation’ allowed_domains = [‘helloacm.com’]

def start_requests(self):
    header={'User-Agent':'Hello World'}
    yield scrapy.Request(url='http://helloacm.com/api/user-agent/',headers=header)

def parse(self, response):
    print '*'*20
    print response.body
    print '*'*20
 

方法4. 在中介軟體中自定義Header

在專案目錄下新增一個目錄： customerMiddleware，在目錄中新建一個自定義的中介軟體檔案： 檔名隨意為 customMiddleware.py

檔案內容為修改request User-Agent

#--coding=utf-8-- from scrapy.contrib.downloadermiddleware.useragent import UserAgentMiddleware

class CustomerUserAgent(UserAgentMiddleware): def process_request(self, request, spider): ua=‘HELLO World???’ request.headers.setdefault(‘User-Agent’,ua)

在setting中新增下面一句，以便使中介軟體生效。

DOWNLOADER_MIDDLEWARES = { ‘headerchange.customerMiddleware.customMiddleware.CustomerUserAgent’:10 # ‘headerchange.middlewares.MyCustomDownloaderMiddleware’: 543, }