爬取豆瓣電影top250，出現以下報錯：

2018-08-11 22:02:16 [scrapy.core.engine] INFO: Spider opened
2018-08-11 22:02:16 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2018-08-11 22:02:16 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2018-08-11 22:02:17 [scrapy.core.engine] DEBUG: Crawled (403) 
 
<GET https://movie.douban.com/robots.txt> (referer: None)
2018-08-11 22:02:17 [scrapy.core.engine] DEBUG: Crawled (403) <GET https://movie.douban.com/top250> (referer: None)
2018-08-11 22:02:17 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 https://movie.douban.com/top250>: HTTP status code is not handled or not
allowed
2018-08-11 22:02:17 [scrapy.core.engine] INFO: Closing spider (finished)
 

防止反爬機制，偽裝user_agent

【1】打開豆瓣top250 ： https://movie.douban.com/top250

【2】F12 打開控制臺->刷新頁面 ->Network->請求頭部找到 User-Agent

在scrapy項目中找到settings.py的 USER_AGENT = ‘ ‘ （把註釋去掉，加以下內容）

USER_AGENT = ‘Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3493.3 Safari/537.36‘

重新執行即可

scrapy crawl douban_spider

INFO: Ignoring response <403 https://movie.douban.com/top250>: HTTP status code is not handled or not allowed