爬蟲學習之11:爬取豆瓣電影TOP250並存入資料庫
阿新 • • 發佈:2019-01-27
本次實驗主要測試使用PyMySQL庫寫資料進MySQL,爬取資料使用XPATH和正則表示式,在很多場合可以用XPATH提取資料,但有些資料項在網頁中沒有明顯特徵,用正則表示式反而反而更輕鬆獲取資料。直接上程式碼:
from lxml import etree import requests import re import pymysql import time headers = { 'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36' } conn = pymysql.connect(host='localhost',user='root',passwd='123456',db='crawl',port=3306,charset='utf8') cursor = conn.cursor() def get_movie_url(url): html = requests.get(url,headers=headers) selector = etree.HTML(html.text) movie_hrefs = selector.xpath('//div[@class="hd"]/a/@href') for movie_href in movie_hrefs: get_movie_info(movie_href) def get_movie_info(url): html = requests.get(url,headers=headers) selector = etree.HTML(html.text) try: movie_name = selector.xpath('//*[@id="content"]/h1/span[1]/text()')[0] director = selector.xpath('//*[@id="info"]/span[1]/span[2]/a/text()')[0] actors = selector.xpath('//span[@class="actor"]/span[2]')[0] actor = actors.xpath('string(.)') style = re.findall('<span property="v:genre">(.*?)</span>',html.text,re.S)[0] country = re.findall('製片國家/地區:</span>(.*?)<br/>', html.text, re.S)[0] release_time = re.findall('上映日期:</span>.*?>(.*?)</span>',html.text,re.S)[0] time = re.findall('片長:</span>.*?>(.*?)</span>',html.text,re.S)[0] score = selector.xpath('//*[@id="interest_sectl"]/div[1]/div[2]/strong/text()')[0] cursor.execute( "insert into doubanmovie (movie_name,director,actor,style,country,release_time,time,score) values (%s,%s,%s,%s,%s,%s,%s,%s)", (str(movie_name),str(director),str(actor),str(style),str(country),str(release_time),str(time),str(score)) ) except IndexError: pass if __name__ =='__main__': urls = ['https://movie.douban.com/top250?start={}&filter='.format(str(i)) for i in range(0,250,25)] for url in urls: get_movie_url(url) time.sleep(2) conn.commit() conn.close()
結果已經存入mySQL資料庫: