1.1 在settings.py文件中設置USER_AGENT參數即可實現

1.1.1 使用固定User-Agent方法

在settings.py文件中加入如下代碼即可（USER_AGENT是固定寫法），具體使用什麽agent可以自己更改下面藍色字體內容。

USER_AGENT = 'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 2.0.50727; Media Center PC 6.0)
 

1.1.2 使用隨機User-Agent方法

在settings.py文件中通過創建一個agent列表，然後通過從中隨機取出1個賦值給USER_AGENT變量即可

代碼如下：

import random

UserAgentList = [
     "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 2.0.50727; Media Center PC 6.0)",
     "Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 1.0.3705; .NET CLR 1.1.4322)"
 
,
     ]
 
 USER_AGENT = random.choice(UserAgentList)

1.1.3 使用第三方user-agent庫（fake-useragent）實現設置agent方法

在settings.py文件中加入如下代碼：

from fake_useragent import UserAgent
 ua = UserAgent()
 USER_AGENT = ua.random

1.2 通過寫middleware方法實現

1.2.1 通過在setting文件中自己編寫user-agent方法

在setting.py文件中寫一個UserAgentList列表

UserAgentList = [

在setting.py文件中啟用DOWNLOADER_MIDDLEWARES，並且把默認的註釋掉

DOWNLOADER_MIDDLEWARES = {
     #'amazon.middlewares.MyCustomDownloaderMiddleware': 543,
     'amazon.middlewares.MyUserAgentMiddleware': 543,
 }

在middlewares.py文件中增加以下代碼：

rom scrapy.downloadermiddlewares.useragent import UserAgentMiddleware
 from amazon.settings import UserAgentList
 import random

class MyUserAgentMiddleware(UserAgentMiddleware):
     def __init__(self, user_agent=''):
         self.user_agent = user_agent
 
     def process_request(self, request, spider):
         agent = random.choice(list(UserAgentList))
         request.headers['User-Agent'] = agent

1.2.2 利用fake-useragent庫設置User-Agent方法

安裝fake-useragent庫(useragent的庫)

pip install fake-useragent

在middlewares.py文件中導入fake-useragent庫

from fake_useragent import UserAgent

在middlewares.py文件中編寫一個middleware中間件，

from fake_useragent import UserAgent
 from scrapy.downloadermiddlewares.useragent import UserAgentMiddleware

class MyUserAgentMiddleware(UserAgentMiddleware):
     def __init__(self, user_agent=''):
         self.user_agent = user_agent
 
     def process_request(self, request, spider): 
         ua = UserAgent()
         request.headers['User-Agent'] = ua.random

在setting.py文件中啟用DOWNLOADER_MIDDLEWARES，並且把默認的註釋掉

DOWNLOADER_MIDDLEWARES = {
     #'amazon.middlewares.MyCustomDownloaderMiddleware': 543,
     'amazon.middlewares.MyUserAgentMiddleware': 543,
 }

scrapy隨機更改User-Agent方法