Python爬蟲之爬取瓜子二手車資訊- requests方法

阿新 • • 發佈：2019-01-20

最近在做二手車市場資料分析，試著爬取瓜子二手車在售車輛資訊，做一下記錄

大致思路如下：

2、從a連結頁面獲取每輛車詳情頁的連結b

3、進入b抓取我需要的車輛資訊：

car_model = Field()             # 車型資訊
registertime = Field()          # 上牌日期
mileage = Field()               # 行駛里程
sec_price = Field()             # 二手車報價
new_price = Field()             # 新車指導價(含稅) daiqueding
registeraddress = Field()       # 所在地
displacement = Field()          # 排放標準

1、頁碼連結構造比較簡單，如圖：

2、詳情頁連結構造如圖

3、詳情資訊結構也比較好找

附上原始碼如下：

# -*- coding: utf-8 -*-
# 僅供交流學習
import requests
from pyquery import PyQuery as pq
import time,random


#請求頭 cookie 必須需要加上,爬前request網址試下可以get到全內容不,不能的話換下cookie
headers = {'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
           'Accept-Encoding':'gzip, deflate,br',
           'Accept-Language':'zh-CN,zh;q=0.9',
           'Cache-Control':'no-cache',
           'Connection':'keep-alive','Cookie':'uuid=02724092-c319-4ba3-89e5-7bf432065f79; ganji_uuid=1367370650065045912399; Hm_lvt_936a6d5df3f3d309bda39e92da3dd52f=1528970077,1528970098; cityDomain=sz; lg=1; clueSourceCode=%2A%2300; sessionid=7273bfb7-b191-4334-f256-30ec9febf860; cainfo=%7B%22ca_s%22%3A%22sem_baiduss%22%2C%22ca_n%22%3A%22bdpc_sye%22%2C%22ca_i%22%3A%22-%22%2C%22ca_medium%22%3A%22-%22%2C%22ca_term%22%3A%22%25E4%25BA%258C%25E6%2589%258B%25E8%25BD%25A6%22%2C%22ca_content%22%3A%22-%22%2C%22ca_campaign%22%3A%22-%22%2C%22ca_kw%22%3A%22-%22%2C%22keyword%22%3A%22-%22%2C%22ca_keywordid%22%3A%2249888355177%22%2C%22scode%22%3A%2210103188612%22%2C%22ca_transid%22%3Anull%2C%22platform%22%3A%221%22%2C%22version%22%3A1%2C%22client_ab%22%3A%22-%22%2C%22guid%22%3A%2202724092-c319-4ba3-89e5-7bf432065f79%22%2C%22sessionid%22%3A%227273bfb7-b191-4334-f256-30ec9febf860%22%7D; antipas=2496334kL545T396w008C21ez41; preTime=%7B%22last%22%3A1531722065%2C%22this%22%3A1528850701%2C%22pre%22%3A1528850701%7D',
           'Host':'www.guazi.com',
           'Pragma':'no-cache',
           'Referer':'https://www.guazi.com/sz/buy/o1r3_16_6/',
           'Upgrade-Insecure-Requests':'1',
           'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36'}
# headers ={
#     'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36',
#     # cookie 必須需要加上,爬前request網址試下可以get到全內容不,不能的話換下cookie
#     'Cookie':'uuid=2276c0ec-9654-440b-b583-788b4a4b7788; ganji_uuid=1526893422532532003974; lg=1; Hm_lvt_936a6d5df3f3d309bda39e92da3dd52f=1531222100; clueSourceCode=10103000312%2300; antipas=2496334IBPB545K39a6U008T21x41; sessionid=94cd5b73-23cf-4515-d5a1-b7b03461d083; cainfo=%7B%22ca_s%22%3A%22pz_baidu%22%2C%22ca_n%22%3A%22tbmkbturl%22%2C%22ca_i%22%3A%22-%22%2C%22ca_medium%22%3A%22-%22%2C%22ca_term%22%3A%22-%22%2C%22ca_content%22%3A%22-%22%2C%22ca_campaign%22%3A%22-%22%2C%22ca_kw%22%3A%22-%22%2C%22keyword%22%3A%22-%22%2C%22ca_keywordid%22%3A%22-%22%2C%22scode%22%3A%2210103000312%22%2C%22ca_transid%22%3Anull%2C%22platform%22%3A%221%22%2C%22version%22%3A1%2C%22client_ab%22%3A%22-%22%2C%22guid%22%3A%222276c0ec-9654-440b-b583-788b4a4b7788%22%2C%22sessionid%22%3A%2294cd5b73-23cf-4515-d5a1-b7b03461d083%22%7D; close_finance_popup=2018-07-16; cityDomain=sz; preTime=%7B%22last%22%3A1531725774%2C%22this%22%3A1531222098%2C%22pre%22%3A1531222098%7D'
# }
# 代理ip
proxies = {
    'http':'http://60.177.231.103:18118',
    'https':'http://60.177.231.103:18118'
}
# proxies = {
#     'http':'http://60.177.226.225:18118',
#     'https':'http://60.177.226.225:18118'
# }
# proxies = {
#     'http':'http://14.118.252.228:6666',
#     'https':'http://14.118.252.228:6666'
# }


class GuaziSpider():

    # 初始化爬蟲
    def __init__(self):
        # 目標url
        self.baseurl = 'https://www.guazi.com'
        '''
        在進行介面測試的時候，我們會呼叫多個介面發出多個請求，在這些請求中有時候需要保持一些共用的資料，例如cookies資訊。
        requests庫的session物件能夠幫我們跨請求保持某些引數，也會在同一個session例項發出的所有請求之間保持cookies。
        '''
        self.s = requests.Session()
        self.s.headers = headers
        # 本地ip被封的話啟用該處ip代理池
        # self.s.proxies = proxies
        # 其中www代表瓜子二手車全國車源,如果只想爬某個城市的,如深圳的用sz替換www
        self.start_url = 'https://www.guazi.com/www/buy/'
        self.infonumber = 0     # 用來記錄爬取了多少條資訊用

    # get_page用來獲取url頁面
    def get_page(self, url):
        return pq(self.s.get(url).text)

    # page_url用來生成第n到第m頁的翻頁連結
    def page_url(self,n,m):
        page_start = n
        page_end = m
        # 新建空列表用來存翻頁連結
        page_url_list = []
        for i in range(page_start,page_end+1,1):
            base_url = 'https://www.guazi.com/www/buy/o{}/#bread'.format(i)
            page_url_list.append(base_url)

        return page_url_list

    # detail_url用來抓取詳情頁連結
    def detail_url(self, start_url):
        # 獲取star_url頁面
        content = self.get_page(start_url)
        # 解析頁面,獲取詳情頁連結content=pq(self.s.get(start_url).text)
        for chref in content('ul[@class="carlist clearfix js-top"]  > li > a').items():
            url = chref.attr.href
            detail_url = self.baseurl + url
            yield detail_url

    # carinfo用來抓取每輛車的所需資訊
    def carinfo(self, detail_url):
        content = self.get_page(detail_url)
        d = {}
        d['model'] = content('h2.titlebox').text().strip()                                          # 車型
        d['registertime'] = content('ul[@class="assort clearfix"] li[@class="one"] span').text()    # 上牌時間
        d['mileage'] = content('ul[@class="assort clearfix"] li[@class="two"] span').text()         # 表顯里程
        d['secprice'] = content('span[@class="pricestype"]').text()                                 # 報價
        d['newprice'] = content('span[@class="newcarprice"]').text()                                # 新車指導價(含稅)
        d['address'] = content('ul[@class="assort clearfix"]').find('li').\
                        eq(2).find('span').text()                                                   # 上牌地
        d['displacement'] = content('ul[@class="assort clearfix"]').\
                            find('li').eq(3).find('span').text()                                    # 排量
        return d


    def run(self,n,m):
        page_start = n
        page_end = m
        with open('guazidata{}to{}.txt'.format(page_start,page_end),'a',encoding='utf-8') as f:
            for pageurl in self.page_url(page_start,page_end):
                print(pageurl)
                print("讓俺歇歇10-15秒啦，太快會關小黑屋的！")
                time.sleep(random.randint(10, 15))
                for detail_url in self.detail_url(pageurl):
                    print(detail_url)
                    d = self.carinfo(detail_url)
                    f.write(d['model']+',')
                    f.write(d['registertime'] + ',')
                    f.write(d['mileage'] + ',')
                    f.write(d['secprice'] + ',')
                    f.write(d['newprice'] + ',')
                    f.write(d['address'] + ',')
                    f.write(d['displacement'] + '\n')
                    time.sleep(0.3)
                    self.infonumber += 1
                    print('爬了%d輛車,continue!'%self.infonumber)
                print('+'*10)


if __name__ == '__main__':
    gzcrawler = GuaziSpider()
    # 這兒改數字,例如:爬取第1頁到第100頁的資訊
    gzcrawler.run(1,100)

總結：

1、最開始沒加cookie，然後get不全網頁，然後不知道為何，本機chrome裡得到的cookie怎麼也不好使，這兒需要繼續研究，希望懂得給說道一下。當時對比檢視我要買車跟我要賣車獲得的js，一度以為這些頁面是js渲染防爬的，都想放棄了，幸好叫別人試了一下，才發現只要加cookie就好，我只能copy別人電腦的cookie

未加cookie，get不全資訊顯示是這樣子的：

2、爬取速度過快會被封IP，只能換代理IP解決；同時速度稍快，中間就會出現爬幾頁然後跳過幾頁不爬的情況，大概每7頁只能爬取一頁，而且40輛車還爬不完全，原因也想不明白。具體閾值沒測試，因為不急著用資料，就開了幾個終端使用幾個代理ip掛著慢點爬。

Python爬蟲之爬取瓜子二手車資訊- requests方法

Python爬蟲之爬取瓜子二手車資訊- requests方法

Python scrapy框架爬取瓜子二手車資訊資料

Python爬蟲之爬取煎蛋網妹子圖

python3 學習 3：python爬蟲之爬取動態載入的圖片，以百度圖片為例

Python爬蟲：爬取網站電影資訊

Python爬蟲之爬取各大幣交易網站公告——靜態網站.md

python學習（7）：python爬蟲之爬取動態載入的圖片，以百度圖片為例

Python爬蟲之爬取動態頁面資料

Python爬蟲之爬取知乎帖子並儲存到mysql（以及遇到問題和解決方法）

Python爬蟲之爬取內涵吧段子（urllib.request）

python --爬蟲基礎 --爬取今日頭條使用 requests 庫的基本操作, Ajax

python實戰之網路爬蟲（爬取網頁新聞資訊列表）

Python網路爬蟲之爬取淘寶網頁頁面 MOOC可以執行的程式碼

Python爬蟲專案--爬取某寶男裝資訊

python爬蟲3——爬取騰訊招聘全部招聘資訊

(轉)python爬蟲例項——爬取智聯招聘資訊

【Python爬蟲】爬取微信公眾號文章資訊準備工作

python初級實戰系列教程《一、爬蟲之爬取網頁、圖片、音視訊》

python爬蟲例項——爬取智聯招聘資訊

python爬蟲實戰爬取汽車之家上車型價格

Python爬蟲之爬取瓜子二手車資訊- requests方法

相關推薦