Python的爬蟲非常簡單，現在又有成熟的爬蟲框架scrapy。現在，我們來用scrapy爬取自己蝦米歌單上的歌曲。
通過這篇部落格，你將學到：

1. 基本的爬蟲設計
2. 模擬登陸
3. 維持登陸狀態
4. Xpath

（中的一點皮毛233）。本文預設讀者已經通過scrapy官方文件或中文版安裝好了，並試過了測試用例。
然後第一步建立專案：scrapy startproject xiami命令會在當前路徑下建立名為xiami的scrapy專案。

基本的爬蟲設計

從需求的角度出發，先想好我們要爬取的內容，簡單一點的話就爬取網頁的標題、使用者的名字、歌單的歌名。行文順序參照scrapy官方文件的習慣。

items

  先來修改items.py檔案。items是儲存資料的容器，它是scrapy框架自己的資料結構，與字典類似，都是鍵-值對的形式儲存資料。定義了items，我們就可以用scrapy的方式儲存資料到各種資料庫或者本地檔案。將來我們要把爬取到的歌單儲存到本地的json檔案中。

# -*- coding: utf-8 -*-

# Define here the models for your scraped items
# ...

import scrapy


class XiamiItem(scrapy.Item):
    # define the fields for your item here like:
 

    # name = scrapy.Field()
    pass

class XiamiItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    title = scrapy.Field()  # 網頁的標題
    name = scrapy.Field()  # 使用者的名字
    song = scrapy.Field()  # 歌單的歌名
    pass

這裡我們相當於只是建立了一個空的字典，然後定義了裡面的鍵值。下面我們需要定義爬蟲爬取資料，並把資料交付給items。

spider

當前專案下的spider資料夾下只有一個空的__init__.py檔案。該資料夾負責容納自定義的爬蟲模組。在spider資料夾下建立一個py檔案，就是一個爬蟲模組，當然它現在還沒有任何的功能。建立python file——xiami_spider.py，這就是我們用來爬取蝦米歌單的爬蟲了。然後定義它的基本元素

from scrapy.spiders import CrawlSpider, Rule

class XiamiSpider(CrawlSpider):
    name = "xiaoxia"  # 爬蟲名：小蝦
    allowed_domains = ["xiami.com"]
    start_urls = [
        "http://www.xiami.com"
    ]
    account_number = '9839****[email protected]'  # 換上你的賬號
    password = '123456'  # 換上你的密碼

    # 重寫了start_request方法，Spider類不會再自動呼叫該方法來為start_url中的url建立Request
    def start_requests(self):
        return [Request("https://login.xiami.com/member/login",
                        meta={'cookiejar': 1},
                        callback=self.post_login)]

• 在這個新建的類中，我們繼承的是CrawlSpider而不是普通的Spider，CrawlSpider是Spider的子類，所以它會有更多的功能，這樣可以少些很多程式碼。
• 定義了爬蟲的名字，最後在執行程式的時候，需要用到這個名字。
• 定義了爬取區域的大小。如果不講範圍限制在蝦米網站的網頁中，爬蟲如果不停地最終網頁中的新連結的話，可能會爬取到很多無關網站的無關內容
• 定義了初始的URL，對spider來說，爬取的迴圈類似下文:
  
  • 呼叫預設start_requensts()函式，以初始的URL初始化Request，並設定回撥函式。 當該request下載完畢並返回時，將生成response，並作為引數傳給該回調函式。spider中初始的request是通過呼叫 start_requests() 來獲取的。 start_requests() 讀取 start_urls 中的URL， 並以 parse 為回撥函式生成 Request 。
  • 在回撥函式內分析返回的(網頁)內容，返回 Item 物件或者 Request 或者一個包括二者的可迭代容器。 返回的Request物件之後會經過Scrapy處理，下載相應的內容，並呼叫設定的callback函式(函式可相同)。
  • 在回撥函式內，您可以使用 選擇器(Selectors) (您也可以使用BeautifulSoup, lxml 或者您想用的任何解析器) 來分析網頁內容，並根據分析的資料生成item。
  • 最後，由spider返回的item將被存到資料庫(由某些 Item Pipeline 處理)或使用 Feed exports 存入到檔案中。

# coding=utf-8
from scrapy.selector import Selector
from scrapy.http import Request, FormRequest, HtmlResponse
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor
from xiami.items import XiamiItem
from sys import argv


class XiamiSpider(CrawlSpider):
    print argv
    name = "xiaoxia"  # 爬蟲名：小蝦
    allowed_domains = ["xiami.com"]
    start_urls = [
        "http://www.xiami.com"
    ]
    account_number = '[email protected]'  # 換上你的賬號
    password = '159661312'  # 換上你的密碼
    headers = {
        "Accept": "application/json, text/javascript, */*; q=0.01",
        "Accept-Encoding": "gzip, deflate, br",
        "Accept-Language": "zh-CN,zh;q=0.8",
        "Connection": "keep-alive",
        "Content-Type": "application/x-www-form-urlencoded; charset=UTF-8",
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                      "(KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36",
        "Referer": "https://login.xiami.com/member/login?spm=a1z1s.6843761.226669498.1.2iL1jx"
    }
    '''
    "User-Agent":
            "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) "
            "Chrome/57.0.2987.133 Safari/537.36",
    '''
    rules = {
        Rule(LinkExtractor(allow=('/space/lib-song',)), callback='parse_page', follow=True),
    }

    # 重寫了start_request方法，Spider類不會再自動呼叫該方法來為start_url中的url建立Request
    def start_requests(self):
        return [Request("https://login.xiami.com/member/login",
                        meta={'cookiejar': 1},
                        callback=self.post_login)]

    # FormRequest
    def post_login(self, response):
        print 'Preparing login'
        # 下面這句話用於抓取請求頁面後返回頁面彙總的_xiamitoken欄位的文字，用於成功提交表單
        _xiamitoken = Selector(response).xpath('//input[@name="_xiamitoken"]/@value').extract_first()
        print '驗證資訊: ', _xiamitoken
        # FormRequest.from_response是Scrapy提供的一個函式，用於post表單
        # 登陸成功後，會呼叫after_login回撥函式
        return [FormRequest.from_response(response,
                                          meta={'cookiejar': response.meta['cookiejar']},
                                          headers=self.headers,
                                          formdata={
                                              'source': 'index_nav',
                                              '_xiamitoken': _xiamitoken,
                                              'email': self.account_number,
                                              'password': self.password
                                          },
                                          callback=self.after_login,
                                          dont_filter=True)]

    def after_login(self, response):
        print 'after login======='
        for url in self.start_urls:
            yield Request(url, meta={'cookiejar': response.meta['cookiejar']})  # 建立Request

    def _requests_to_follow(self, response):
        if not isinstance(response, HtmlResponse):
            return
        seen = set()
        for n, rule in enumerate(self._rules):
            links = [lnk for lnk in rule.link_extractor.extract_links(response)
                     if lnk not in seen]
            if links and rule.process_links:
                links = rule.process_links(links)
            for link in links:
                seen.add(link)
                r = Request(url=link.url, callback=self._response_downloaded)
                # 重寫
                r.meta.update(rule=n, link_text=link.text, cookiejar=response.meta['cookiejar'])
                yield rule.process_request(r)

    def parse_page(self, response):
        # print 'hh'
        mysong_list = Selector(response)
        songs = mysong_list.xpath('//td[@class="song_name"]/a/@title').extract()
        print songs[0]
        for song in songs:
            item = XiamiItem()
            item['title'] = 'xiami_music'
            item['name'] = self.account_number
            item['song'] = song
            yield item
        # print '---\n'
        # nexturl = mysong_list.xpath('//a[@class="p_redirect_l"]/@href').extract_first()
        # yield self.make_requests_from_url(nexturl)

# -*- coding: utf-8 -*-

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: http://doc.scrapy.org/en/latest/topics/item-pipeline.html
import json
import codecs
from scrapy.exceptions import DropItem


class XiamiPipeline(object):
    def __init__(self):
        self.song_seen = set()
        self.file = codecs.open('xiamisongs.jl', 'w', encoding='utf-8')

    def process_item(self, item, spider):
        """
        每個item pipeline元件都需要呼叫該方法，
        這個方法必須返回一個Item（或任何整合類）物件，
        或丟擲DropItem異常，
        被丟棄的item將不被後面的pipeline處理
        :param item:
        :param spider:
        :return:
        """
        # 過濾缺失資料
        # if True:
        #   return item
        # else:
        #   raise DropItem('reason')
        if spider.name == 'xiaoxia':
            if item['song'] in self.song_seen:
                raise DropItem('Duplicate song found: %s' % item['song'])
            else:
                self.song_seen.add(item['song'])
                '''儲存到json檔案(非必須)'''
                line = json.dumps(dict(item), ensure_ascii=False) + '\n'
                self.file.write(line)
                return item

    def close_spider(self, spider):
        print 'spider close'
        self.file.close()

# -*- coding: utf-8 -*-

# Scrapy settings for xiami project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
#     http://doc.scrapy.org/en/latest/topics/settings.html
#     http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html
#     http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html

BOT_NAME = 'xiami'

SPIDER_MODULES = ['xiami.spiders']
NEWSPIDER_MODULE = 'xiami.spiders'


# Crawl responsibly by identifying yourself (and your website) on the user-agent
# USER_AGENT = 'xiami (+http://www.xiami.com)'

# Obey robots.txt rules
ROBOTSTXT_OBEY = False

# Configure maximum concurrent requests performed by Scrapy (default: 16)
# #CONCURRENT_REQUESTS = 32

# Configure a delay for requests for the same website (default: 0)
# See http://scrapy.readthedocs.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
DOWNLOAD_DELAY = 0.25
# The download delay setting will honor only one of:
# #CONCURRENT_REQUESTS_PER_DOMAIN = 16
# #CONCURRENT_REQUESTS_PER_IP = 16

# Disable cookies (enabled by default)
COOKIES_ENABLED = True

# Disable Telnet Console (enabled by default)
# #TELNETCONSOLE_ENABLED = False

# Override the default request headers:
# #DEFAULT_REQUEST_HEADERS = {
#   'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
#   'Accept-Language': 'en',
# }

# Enable or disable spider middlewares
# See http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html
# #SPIDER_MIDDLEWARES = {
#    'xiami.middlewares.MyCustomSpiderMiddleware': 543,
# }

# Enable or disable downloader middlewares
# See http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html
# #DOWNLOADER_MIDDLEWARES = {
#    'xiami.middlewares.MyCustomDownloaderMiddleware': 543,
# }

# Enable or disable extensions
# See http://scrapy.readthedocs.org/en/latest/topics/extensions.html
# # EXTENSIONS = {
#    'scrapy.extensions.telnet.TelnetConsole': None,
# }

# Configure item pipelines
# See http://scrapy.readthedocs.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = {
   'xiami.pipelines.XiamiPipeline': 300,  # 0-1000表示執行順序
}

# Enable and configure the AutoThrottle extension (disabled by default)
# See http://doc.scrapy.org/en/latest/topics/autothrottle.html
# #AUTOTHROTTLE_ENABLED = True
# The initial download delay
# #AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
# #AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
# #AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
# #AUTOTHROTTLE_DEBUG = False

# Enable and configure HTTP caching (disabled by default)
# See http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
# #HTTPCACHE_ENABLED = True
# #HTTPCACHE_EXPIRATION_SECS = 0
# #HTTPCACHE_DIR = 'httpcache'
# #HTTPCACHE_IGNORE_HTTP_CODES = []
# #HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'

#!/usr/bin/python
# coding=utf-8
# 開始爬蟲的指令碼檔案

from scrapy.cmdline import execute
# execute('scrapy crawl xiaoxia'.split())
execute('scrapy crawl xiaoxia -o xiamisongs.jl'.split())

這是草稿，下週再修改完善