（一）爬取環境

• win10
• python3
• scrapy

（二）豆瓣電影參考標準

主要參考來自神箭手的雲市場

（三）爬取步驟

主要爬取的欄位有：標題，得分，封面url，簡介，全部評論

採用scrapy、ip代理池、ua次進行內容爬取

# -*- coding: utf-8 -*-
# @Time    : 2018/7/22 14:45
# @Author  : 蛇崽
# @Email   : [email protected]
# @File    : doubanmoviespider.py(豆瓣電影)
import json

import
 
 scrapy
from bs4 import BeautifulSoup


class DouBanSpider(scrapy.Spider):
    name = 'adouban_spider'
    start_urls = ['https://movie.douban.com/tag/#/']
    allowed_domains = ['movie.douban.com']

    def parse(self,response):
        for count in range(0,27002,20):
            fir_url = 'https://movie.douban.com/j/new_search_subjects?sort=T&range=0,10&tags=&start={}'
 
.format(count)
            yield scrapy.Request(
                fir_url,
                callback=self.parse_main
            )

    def parse_main(self,response):
        infos = json.loads(response.body.decode('utf-8'))
        '名稱 封面圖片 簡介 評分 短評top100'
        print(infos)
        datas = infos['data'
 
]
        for data in datas:
            title = data['title'] # 名稱
            cover = data['cover'] # 封面圖片
            rate = data['rate']   # 評分
            url = data['url'] # 詳情url
            id = str(data['id'])
            if url:
                print(title,cover,rate,url)
                yield scrapy.Request(url,callback=self.parse_detail,meta={'id':id})

    def parse_detail(self,response):
        strid = response.meta['id']
        base_url = 'https://movie.douban.com/subject/{}/comments?'.format(strid)
        # 全部評論的連結
        b_url = 'https://movie.douban.com/subject/{}/comments?status=P'.format(strid)
        # 前一百條評論
        n_url = 'https://movie.douban.com/subject/4920528/comments?start=0&limit=20&sort=new_score&status=P'
        for page in range(0,100,20):
            n_url = base_url+'start={}&limit=20&sort=new_score&status=P'.format(page)
            print('n_url################# ',n_url)
            if n_url:
                yield scrapy.Request(b_url,callback=self.parse_comment)



        soup = BeautifulSoup(response.body, 'lxml')
        # 簡介(先隱藏後不隱藏的)
        try:
            abstract = soup.find('span', class_='all hidden').get_text()
        except:
            abstract = soup.find('span',attrs={'property':'v:summary'}).get_text()
            pass
        # print('abstract -------- ',abstract)

    def parse_comment(self,response):
        soup = BeautifulSoup(response.body,'lxml')
        shorts = soup.find_all('span',class_='short')
        for short in shorts:
            print('short =============  ',short.get_text())

個人微信：hll643435675（備註：部落格）

陸續優化中，後續會開發更多更好玩的有趣的小工具