Python3 學習4：使用Beautiful Soup爬取小說
阿新 • • 發佈：2018-11-03
轉自：jack-Cui 老師的 http://blog.csdn.net/c406495762
執行平臺： Windows
Python版本： Python3.x
IDE： Sublime text3
一、Beautiful Soup簡介
簡單來說，Beautiful Soup是python的一個庫，最主要的功能是從網頁抓取資料。官方解釋如下：
-
Beautiful Soup提供一些簡單的、python式的函式用來處理導航、搜尋、修改分析樹等功能。它是一個工具箱，通過解析文件為使用者提供需要抓取的資料，因為簡單，所以不需要多少程式碼就可以寫出一個完整的應用程式。
-
Beautiful Soup自動將輸入文件轉換為Unicode編碼，輸出文件轉換為utf-8編碼。你不需要考慮編碼方式，除非文件沒有指定一個編碼方式，這時，Beautiful Soup就不能自動識別編碼方式了。然後，你僅僅需要說明一下原始編碼方式就可以了。
-
Beautiful Soup已成為和lxml、html6lib一樣出色的python直譯器，為使用者靈活地提供不同的解析策略或強勁的速度。
廢話不多說，直接開始動手吧！
二、實戰
1.背景介紹
小說網站-筆趣看：
URL： http://www.biqukan.com/
筆趣看是一個盜版小說網站，這裡有很多起點中文網的小說，該網站小說的更新速度稍滯後於起點中文網正版小說的更新速度。並且該網站只支援線上瀏覽，不支援小說打包下載。因此，本次實戰就是從該網站爬取並儲存一本名為《一念永恆》的小說，該小說是耳根正在連載中的一部玄幻小說。PS：本例項僅為交流學習，支援耳根大大，請上起點中文網訂閱。
2.Beautiful Soup安裝
我們我可以使用pip3或者easy_install來安裝，在cmd命令視窗中的安裝命令分別如下：
a)pip3安裝
<span style="color:#000000"><code>pip3 <span style="color:#000088">install</span> beautifulsoup4</code></span>- 1
b)easy_install安裝
<span style="color:#000000"><code>easy_install beautifulsoup4</code></span>- 1
3.預備知識
更為詳細內容，可參考官方文件：
URL：http://beautifulsoup.readthedocs.io/zh_CN/latest/
a)建立Beautiful Soup物件
<span style="color:#000000"><code>from bs4 import BeautifulSoup
#html為解析的頁面獲得html資訊,為方便講解，自己定義了一個html檔案
html = """
<span style="color:#006666"><<span style="color:#4f4f4f">html</span>></span>
<span style="color:#006666"><<span style="color:#4f4f4f">head</span>></span>
<span style="color:#006666"><<span style="color:#4f4f4f">title</span>></span>Jack_Cui<span style="color:#006666"></<span style="color:#4f4f4f">title</span>></span>
<span style="color:#006666"></<span style="color:#4f4f4f">head</span>></span>
<span style="color:#006666"><<span style="color:#4f4f4f">body</span>></span>
<span style="color:#006666"><<span style="color:#4f4f4f">p</span> <span style="color:#4f4f4f">class</span>=<span style="color:#009900">"title"</span> <span style="color:#4f4f4f">name</span>=<span style="color:#009900">"blog"</span>></span><span style="color:#006666"><<span style="color:#4f4f4f">b</span>></span>My Blog<span style="color:#006666"></<span style="color:#4f4f4f">b</span>></span><span style="color:#006666"></<span style="color:#4f4f4f">p</span>></span>
<span style="color:#006666"><<span style="color:#4f4f4f">li</span>></span><span style="color:#880000"><!--註釋--></span><span style="color:#006666"></<span style="color:#4f4f4f">li</span>></span>
<span style="color:#006666"><<span style="color:#4f4f4f">a</span> <span style="color:#4f4f4f">href</span>=<span style="color:#009900">"http://blog.csdn.net/c406495762/article/details/58716886"</span> <span style="color:#4f4f4f">class</span>=<span style="color:#009900">"sister"</span> <span style="color:#4f4f4f">id</span>=<span style="color:#009900">"link1"</span>></span>Python3網路爬蟲(一)：利用urllib進行簡單的網頁抓取<span style="color:#006666"></<span style="color:#4f4f4f">a</span>></span><span style="color:#006666"><<span style="color:#4f4f4f">br</span>/></span>
<span style="color:#006666"><<span style="color:#4f4f4f">a</span> <span style="color:#4f4f4f">href</span>=<span style="color:#009900">"http://blog.csdn.net/c406495762/article/details/59095864"</span> <span style="color:#4f4f4f">class</span>=<span style="color:#009900">"sister"</span> <span style="color:#4f4f4f">id</span>=<span style="color:#009900">"link2"</span>></span>Python3網路爬蟲(二)：利用urllib.urlopen傳送資料<span style="color:#006666"></<span style="color:#4f4f4f">a</span>></span><span style="color:#006666"><<span style="color:#4f4f4f">br</span>/></span>
<span style="color:#006666"><<span style="color:#4f4f4f">a</span> <span style="color:#4f4f4f">href</span>=<span style="color:#009900">"http://blog.csdn.net/c406495762/article/details/59488464"</span> <span style="color:#4f4f4f">class</span>=<span style="color:#009900">"sister"</span> <span style="color:#4f4f4f">id</span>=<span style="color:#009900">"link3"</span>></span>Python3網路爬蟲(三)：urllib.error異常<span style="color:#006666"></<span style="color:#4f4f4f">a</span>></span><span style="color:#006666"><<span style="color:#4f4f4f">br</span>/></span>
<span style="color:#006666"></<span style="color:#4f4f4f">body</span>></span>
<span style="color:#006666"></<span style="color:#4f4f4f">html</span>></span>
"""
#建立Beautiful Soup物件
soup = BeautifulSoup(html,'lxml')</code></span>如果將上述的html的資訊寫入一個html檔案，開啟效果是這樣的(<!–註釋–>為註釋內容，不會顯示)：
同樣，我們還可以使用本地HTML檔案來建立物件，程式碼如下：
<span style="color:#000000"><code>soup = BeautifulSoup(<span style="color:#4f4f4f">open</span>(test.html),<span style="color:#009900">'lxml'</span>)</code></span>- 1
使用如下程式碼格式化輸出：
<span style="color:#000000"><code>print(soup.prettify())</code></span>- 1
b)Beautiful Soup四大物件
Beautiful Soup將複雜HTML文件轉換成一個複雜的樹形結構,每個節點都是Python物件,所有物件可以歸納為4種:
- Tag
- NavigableString
- BeautifulSoup
- Comment
(1)Tag
Tag通俗點講就是HTML中的一個個標籤，例如
<span style="color:#000000"><code><span style="color:#006666"><<span style="color:#4f4f4f">title</span>></span>Jack_Cui<span style="color:#006666"></<span style="color:#4f4f4f">title</span>></span></code></span>- 1
上面的title就是HTML標籤，標籤加入裡面包括的內容就是Tag，下面我們來感受一下怎樣用 Beautiful Soup 來方便地獲取 Tags。
下面每一段程式碼中註釋部分即為執行結果：
<span style="color:#000000"><code>print(soup.title)
#<span style="color:#006666"><<span style="color:#4f4f4f">title</span>></span>Jack_Cui<span style="color:#006666"></<span style="color:#4f4f4f">title</span>></span>
print(soup.head)
#<span style="color:#006666"><<span style="color:#4f4f4f">head</span>></span> <span style="color:#006666"><<span style="color:#4f4f4f">title</span>></span>Jack_Cui<span style="color:#006666"></<span style="color:#4f4f4f">title</span>></span><span style="color:#006666"></<span style="color:#4f4f4f">head</span>></span>
print(soup.a)
#<span style="color:#006666"><<span style="color:#4f4f4f">a</span> <span style="color:#4f4f4f">class</span>=<span style="color:#009900">"sister"</span> <span style="color:#4f4f4f">href</span>=<span style="color:#009900">"http://blog.csdn.net/c406495762/article/details/58716886"</span> <span style="color:#4f4f4f">id</span>=<span style="color:#009900">"link1"</span>></span>Python3網路爬蟲(一)：利用urllib進行簡單的網頁抓取<span style="color:#006666"></<span style="color:#4f4f4f">a</span>></span>
print(soup.p)
#<span style="color:#006666"><<span style="color:#4f4f4f">p</span> <span style="color:#4f4f4f">class</span>=<span style="color:#009900">"title"</span> <span style="color:#4f4f4f">name</span>=<span style="color:#009900">"blog"</span>></span><span style="color:#006666"><<span style="color:#4f4f4f">b</span>></span>My Blog<span style="color:#006666"></<span style="color:#4f4f4f">b</span>></span><span style="color:#006666"></<span style="color:#4f4f4f">p</span>></span></code></span>我們可以利用 soup加標籤名輕鬆地獲取這些標籤的內容，是不是感覺比正則表示式方便多了？不過有一點是，它查詢的是在所有內容中的第一個符合要求的標籤，如果要查詢所有的標籤，我們在後面進行介紹。
我們也可驗證一下這些物件的型別：
<span style="color:#000000"><code><span style="color:#009900">print</span>(<span style="color:#000088">type</span>(<span style="color:#009900">soup</span>.<span style="color:#009900">title</span>))
<span style="color:#009900">#<class 'bs4.element.Tag'></span></code></span>- 1
- 2
對於Tag，有兩個重要的屬性：name和attrs
name：
<span style="color:#000000"><code>print(soup<span style="color:#009900">.name</span>)
print(soup<span style="color:#009900">.title</span><span style="color:#009900">.name</span>)
<span style="color:#009900">#[document]</span>
<span style="color:#009900">#title</span></code></span>soup 物件本身比較特殊，它的 name 即為 [document]，對於其他內部標籤，輸出的值便為標籤本身的名稱。
attrs：
<span style="color:#000000"><code>print(soup.a.attrs)
#{<span style="color:#009900">'class'</span><span style="color:#009900">:</span> [<span style="color:#009900">'sister'</span>], <span style="color:#009900">'href'</span><span style="color:#009900">:</span> <span style="color:#009900">'http://blog.csdn.net/c406495762/article/details/58716886'</span>, <span style="color:#009900">'id'</span><span style="color:#009900">:</span> <span style="color:#009900">'link1'</span>}</code></span>在這裡，我們把 a 標籤的所有屬性列印輸出了出來，得到的型別是一個字典。
如果我們想要單獨獲取某個屬性，可以這樣，例如我們獲取a標籤的class叫什麼，兩個等價的方法如下：
<span style="color:#000000"><code>print(soup.a[<span style="color:#009900">'class'</span>])
print(soup.a.<span style="color:#000088">get</span>(<span style="color:#009900">'class'</span>))
<span style="color:#009900">#['sister']</span>
<span style="color:#009900">#['sister']</span></code></span>(2)NavigableString
既然我們已經得到了標籤的內容，那麼問題來了，我們要想獲取標籤內部的文字怎麼辦呢？很簡單，用 .string 即可，例如
<span style="color:#000000"><code>print(soup.title.<span style="color:#000088">string</span>)
<span style="color:#009900">#Jack_Cui</span></code></span>(3)BeautifulSoup
BeautifulSoup 物件表示的是一個文件的全部內容.大部分時候,可以把它當作 Tag 物件，是一個特殊的 Tag，我們可以分別獲取它的型別，名稱，以及屬性：
<span style="color:#000000"><code><span style="color:#000088">print</span>(type(soup.name))
<span style="color:#000088">print</span>(soup.name)
<span style="color:#000088">print</span>(soup.attrs)
<span style="color:#880000">#<class 'str'></span>
<span style="color:#880000">#[document]</span>
<span style="color:#880000">#{}</span></code></span>(4)Comment
Comment物件是一個特殊型別的NavigableString物件，其實輸出的內容仍然不包括註釋符號，但是如果不好好處理它，可能會對我們的文字處理造成意想不到的麻煩。
<span style="color:#000000"><code>print(soup.li)
print(soup.li.string)
print(type(soup.li.string))
#<span style="color:#006666"><<span style="color:#4f4f4f">li</span>></span><span style="color:#880000"><!--註釋--></span><span style="color:#006666"></<span style="color:#4f4f4f">li</span>></span>
#註釋
#<span style="color:#006666"><<span style="color:#4f4f4f">class</span> '<span style="color:#4f4f4f">bs4.element.Comment</span>'></span></code></span>li標籤裡的內容實際上是註釋，但是如果我們利用 .string 來輸出它的內容，我們發現它已經把註釋符號去掉了，所以這可能會給我們帶來不必要的麻煩。
我們列印輸出下它的型別，發現它是一個 Comment 型別，所以，我們在使用前最好做一下判斷，判斷程式碼如下：
<span style="color:#000000"><code><span style="color:#4f4f4f">from</span> bs4 import <span style="color:#000088">element</span>
<span style="color:#000088">if</span> type(soup.li.<span style="color:#000088">string</span>) == <span style="color:#000088">element</span>.Comment:
print(soup.li.<span style="color:#000088">string</span>)</code></span>上面的程式碼中，我們首先判斷了它的型別，是否為 Comment 型別，然後再進行其他操作，如列印輸出。
c)遍歷文件數
(1)直接子節點(不包含孫節點)
contents：
tag的content屬性可以將tag的子節點以列表的方式輸出：
<span style="color:#000000"><code>print(soup.body.contents)
#['\n', <span style="color:#006666"><<span style="color:#4f4f4f">p</span> <span style="color:#4f4f4f">class</span>=<span style="color:#009900">"title"</span> <span style="color:#4f4f4f">name</span>=<span style="color:#009900">"blog"</span>></span><span style="color:#006666"><<span style="color:#4f4f4f">b</span>></span>My Blog<span style="color:#006666"></<span style="color:#4f4f4f">b</span>></span><span style="color:#006666"></<span style="color:#4f4f4f">p</span>></span>, '\n', <span style="color:#006666"><<span style="color:#4f4f4f">li</span>></span><span style="color:#880000"><!--註釋--></span><span style="color:#006666"></<span style="color:#4f4f4f">li</span>></span>, '\n', <span style="color:#006666"><<span style="color:#4f4f4f">a</span> <span style="color:#4f4f4f">class</span>=<span style="color:#009900">"sister"</span> <span style="color:#4f4f4f">href</span>=<span style="color:#009900">"http://blog.csdn.net/c406495762/article/details/58716886"</span> <span style="color:#4f4f4f">id</span>=<span style="color:#009900">"link1"</span>></span>Python3網路爬蟲(一)：利用urllib進行簡單的網頁抓取<span style="color:#006666"></<span style="color:#4f4f4f">a</span>></span>, <span style="color:#006666"><<span style="color:#4f4f4f">br</span>/></span>, '\n', <span style="color:#006666"><<span style="color:#4f4f4f">a</span> <span style="color:#4f4f4f">class</span>=<span style="color:#009900">"sister"</span> <span style="color:#4f4f4f">href</span>=<span style="color:#009900">"http://blog.csdn.net/c406495762/article/details/59095864"</span> <span style="color:#4f4f4f">id</span>=<span style="color:#009900">"link2"</span>></span>Python3網路爬蟲(二)：利#用urllib.urlopen傳送資料<span style="color:#006666"></<span style="color:#4f4f4f">a</span>></span>, <span style="color:#006666"><<span style="color:#4f4f4f">br</span>/></span>, '\n', <span style="color:#006666"><<span style="color:#4f4f4f">a</span> <span style="color:#4f4f4f">class</span>=<span style="color:#009900">"sister"</span> <span style="color:#4f4f4f">href</span>=<span style="color:#009900">"http://blog.csdn.net/c406495762/article/details/59488464"</span> <span style="color:#4f4f4f">id</span>=<span style="color:#009900">"link3"</span>></span>Python3網路爬蟲(三)：urllib.error異常<span style="color:#006666"></<span style="color:#4f4f4f">a</span>></span>, <span style="color:#006666"><<span style="color:#4f4f4f">br</span>/></span>, '\n']</code></span>輸出方式為列表，我們可以用列表索引來獲取它的某一個元素：
<span style="color:#000000"><code>print(soup.body.contents[1])
<span style="color:#006666"><<span style="color:#4f4f4f">p</span> <span style="color:#4f4f4f">class</span>=<span style="color:#009900">"title"</span> <span style="color:#4f4f4f">name</span>=<span style="color:#009900">"blog"</span>></span><span style="color:#006666"><<span style="color:#4f4f4f">b</span>></span>My Blog<span style="color:#006666"></<span style="color:#4f4f4f">b</span>></span><span style="color:#006666"></<span style="color:#4f4f4f">p</span>></span></code></span>children：
它返回的不是一個 list，不過我們可以通過遍歷獲取所有子節點，它是一個 list 生成器物件：
<span style="color:#000000"><code><span style="color:#000088">for</span> child <span style="color:#000088">in</span> soup.body.children:
<span style="color:#4f4f4f">print</span>(child)</code></span>結果如下圖所示：
(2)搜尋文件樹
find_all(name, attrs, recursive, text, limit, **kwargs)：
find_all() 方法搜尋當前tag的所有tag子節點,並判斷是否符合過濾器的條件。
1) name引數：
name 引數可以查詢所有名字為 name 的tag,字串物件會被自動忽略掉。
傳遞字元：
最簡單的過濾器是字串，在搜尋方法中傳入一個字串引數,Beautiful Soup會查詢與字串完整匹配的內容,下面的例子用於查詢文件中所有的<a>標籤：
<span style="color:#000000"><code>print(soup.find_all('a'))
#['\n', <span style="color:#006666"><<span style="color:#4f4f4f">p</span> <span style="color:#4f4f4f">class</span>=<span style="color:#009900">"title"</span> <span style="color:#4f4f4f">name</span>=<span style="color:#009900">"blog"</span>></span><span style="color:#006666"><<span style="color:#4f4f4f">b</span>></span>My Blog<span style="color:#006666"></<span style="color:#4f4f4f">b</span>></span><span style="color:#006666"></<span style="color:#4f4f4f">p</span>></span>, '\n', <span style="color:#006666"><<span style="color:#4f4f4f">li</span>></span><span style="color:#880000"><!--註釋--></span><span style="color:#006666"></<span style="color:#4f4f4f">li</span>></span>, '\n', <span style="color:#006666"><<span style="color:#4f4f4f">a</span> <span style="color:#4f4f4f">class</span>=<span style="color:#009900">"sister"</span> <span style="color:#4f4f4f">href</span>=<span style="color:#009900">"http://blog.csdn.net/c406495762/article/details/58716886"</span> <span style="color:#4f4f4f">id</span>=<span style="color:#009900">"link1"</span>></span>Python3網路爬蟲(一)：利用urllib進行簡單的網頁抓取<span style="color:#006666"></<span style="color:#4f4f4f">a</span>></span>, <span style="color:#006666"><<span style="color:#4f4f4f">br</span>/></span>, '\n', <span style="color:#006666"><<span style="color:#4f4f4f">a</span> <span style="color:#4f4f4f">class</span>=<span style="color:#009900">"sister"</span> <span style="color:#4f4f4f">href</span>=<span style="color:#009900">"http://blog.csdn.net/c406495762/article/details/59095864"</span> <span style="color:#4f4f4f">id</span>=<span style="color:#009900">"link2"</span>></span>Python3網路爬蟲(二)：利用urllib.urlopen傳送資料<span style="color:#006666"></<span style="color:#4f4f4f">a</span>></span>, <span style="color:#006666"><<span style="color:#4f4f4f">br</span>/></span>, '\n', <span style="color:#006666"><<span style="color:#4f4f4f">a</span> <span style="color:#4f4f4f">class</span>=<span style="color:#009900">"sister"</span> <span style="color:#4f4f4f">href</span>=<span style="color:#009900">"http://blog.csdn.net/c406495762/article/details/59488464"</span> <span style="color:#4f4f4f">id</span>=<span style="color:#009900">"link3"</span>></span>Python3網路爬蟲(三)：urllib.error異常<span style="color:#006666"></<span style="color:#4f4f4f">a</span>></span>, <span style="color:#006666"><<span style="color:#4f4f4f">br</span>/></span>, '\n']</code></span>傳遞正則表示式：
如果傳入正則表示式作為引數,Beautiful Soup會通過正則表示式的 match() 來匹配內容.下面例子中找出所有以b開頭的標籤,這表示<body>和<b>標籤都應該被找到
<span style="color:#000000"><code>import re
<span style="color:#000088">for</span> tag in soup.find_all(re.compile(<span style="color:#009900">"^b"</span>)):
print(tag.name)
<span style="color:#009900">#body</span>
<span style="color:#009900">#b</span>
<span style="color:#009900">#br</span>
<span style="color:#009900">#br</span>
<span style="color:#009900">#br</span></code></span>傳遞列表：
如果傳入列表引數，Beautiful Soup會將與列表中任一元素匹配的內容返回，下面程式碼找到文件中所有<title>標籤和<b>標籤：
<span style="color:#000000"><code>print(soup.find_all([<span style="color:#009900">'title'</span>,<span style="color:#009900">'b'</span>]))
#<span style="color:#008800">[<title>Jack_Cui</title>, <b>My Blog</b>]</span></code></span>傳遞True：
True 可以匹配任何值,下面程式碼查詢到所有的tag,但是不會返回字串節點：
<span style="color:#000000"><code><span style="color:#000088">for</span> tag <span style="color:#000088">in</span> soup.find_all(<span style="color:#000088">True</span>):
print(tag.name)</code></span>執行結果：
2)attrs引數
我們可以通過 find_all() 方法的 attrs 引數定義一個字典引數來搜尋包含特殊屬性的tag。
<span style="color:#000000"><code>print(soup.find_all(attrs={"class":"title"}))
#[<span style="color:#006666"><<span style="color:#4f4f4f">p</span> <span style="color:#4f4f4f">class</span>=<span style="color:#009900">"title"</span> <span style="color:#4f4f4f">name</span>=<span style="color:#009900">"blog"</span>></span><span style="color:#006666"><<span style="color:#4f4f4f">b</span>></span>My Blog<span style="color:#006666"></<span style="color:#4f4f4f">b</span>></span><span style="color:#006666"></<span style="color:#4f4f4f">p</span>></span>]</code></span>3)recursive引數
呼叫tag的 find_all() 方法時,Beautiful Soup會檢索當前tag的所有子孫節點,如果只想搜尋tag的直接子節點,可以使用引數 recursive=False。
4)text引數
通過 text 引數可以搜搜文件中的字串內容，與 name 引數的可選值一樣, text 引數接受字串 , 正則表示式 , 列表, True。
<span style="color:#000000"><code><span style="color:#000088">print</span>(soup.find_all(<span style="color:#000088">text</span>=<span style="color:#009900">"Python3網路爬蟲(三)：urllib.error異常"</span>))
#[<span style="color:#009900">'Python3網路爬蟲(三)：urllib.error異常'</span>]</code></span>5)limit引數
find_all() 方法返回全部的搜尋結構,如果文件樹很大那麼搜尋會很慢.如果我們不需要全部結果,可以使用 limit 引數限制返回結果的數量.效果與SQL中的limit關鍵字類似,當搜尋到的結果數量達到 limit 的限制時,就停止搜尋返回結果。
文件樹中有3個tag符合搜尋條件,但結果只返回了2個,因為我們限制了返回數量：
<span style="color:#000000"><code>print(soup.find_all(<span style="color:#009900">"a"</span>, limit=<span style="color:#006666">2</span>))
#[<<span style="color:#006666">a</span> class=<span style="color:#009900">"sister"</span> href=<span style="color:#009900">"http://blog.csdn.net/c406495762/article/details/58716886"</span> id=<span style="color:#009900">"link1"</span>>Python3網路爬蟲(一)：利用urllib進行簡單的網頁抓取</<span style="color:#006666">a</span>>, <<span style="color:#006666">a</span> class=<span style="color:#009900">"sister"</span> href=<span style="color:#009900">"http://blog.csdn.net/c406495762/article/details/59095864"</span> id=<span style="color:#009900">"link2"</span>>Python3網路爬蟲(二)：利用urllib.urlopen傳送資料</<span style="color:#006666">a</span>>]</code></span>6)kwargs引數
如果傳入 class 引數,Beautiful Soup 會搜尋每個 class 屬性為 title 的 tag 。kwargs 接收字串，正則表示式
<span style="color:#000000"><code>print(soup.find_all(class_="title"))
#[<span style="color:#006666"><<span style="color:#4f4f4f">p</span> <span style="color:#4f4f4f">class</span>=<span style="color:#009900">"title"</span> <span style="color:#4f4f4f">name</span>=<span style="color:#009900">"blog"</span>></span><span style="color:#006666"><<span style="color:#4f4f4f">b</span>></span>My Blog<span style="color:#006666"></<span style="color:#4f4f4f">b</span>></span><span style="color:#006666"></<span style="color:#4f4f4f">p</span>></span>]</code></span>4.小說內容爬取
掌握以上內容就可以進行本次實戰練習了
a)單章小說內容爬取
開啟《一念永恆》小說的第一章，進行審查元素分析。
URL：http://www.biqukan.com/1_1094/5403177.html
由審查結果可知，文章的內容存放在id為content，class為showtxt的div標籤中：
區域性放大：
因此我們，可以使用如下方法將本章小說內容爬取下來：
<span style="color:#000000"><code><span style="color:#880000"># -*- coding:UTF-8 -*-</span>
<span style="color:#4f4f4f">from</span> urllib import request
<span style="color:#4f4f4f">from</span> bs4 import BeautifulSoup
<span style="color:#000088">if</span> __name__ == <span style="color:#009900">"__main__"</span>:
download_url = <span style="color:#009900">'http://www.biqukan.com/1_1094/5403177.html'</span>
head = {}
head[<span style="color:#009900">'User-Agent'</span>] = <span style="color:#009900">'Mozilla/5.0 (Linux; Android 4.1.1; Nexus 7 Build/JRO03D) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.166 Safari/535.19'</span>
download_req = request.Request(url = download_url, headers = head)
download_response = request.urlopen(download_req)
download_html = download_response.<span style="color:#4f4f4f">read</span>().decode(<span style="color:#009900">'gbk'</span>,<span style="color:#009900">'ignore'</span>)
soup_texts = BeautifulSoup(download_html, <span style="color:#009900">'lxml'</span>)
texts = soup_texts.find_all(id = <span style="color:#009900">'content'</span>, class_ = <span style="color:#009900">'showtxt'</span>)
soup_text = BeautifulSoup(str(texts), <span style="color:#009900">'lxml'</span>)
<span style="color:#880000">#將\xa0無法解碼的字元刪除</span>
print(soup_text.div.<span style="color:#000088">text</span>.<span style="color:#4f4f4f">replace</span>(<span style="color:#009900">'\xa0'</span>,<span style="color:#009900">''</span>))</code></span>執行結果：
可以看到，我們已經順利爬取第一章內容，接下來就是如何爬取所有章的內容，爬取之前需要知道每個章節的地址。因此，我們需要審查《一念永恆》小說目錄頁的內容。
b)各章小說連結爬取
URL：http://www.biqukan.com/1_1094/
由審查結果可知，小說每章的連結放在了class為listmain的div標籤中。連結具體位置放在html->body->div->dd->dl->a的href屬性中，例如下圖的第759章的href屬性為/1_1094/14235101.html，那麼該章節的地址為：http://www.biqukan.com/1_1094/14235101.html
區域性放大：
因此，我們可以使用如下方法獲取正文所有章節的地址：
<span style="color:#000000"><code><span style="color:#880000"># -*- coding:UTF-8 -*-</span>
<span style="color:#000088">from</span> urllib <span style="color:#000088">import</span> request
<span style="color:#000088">from</span> bs4 <span style="color:#000088">import</span> BeautifulSoup
<span style="color:#000088">if</span> __name__ == <span style="color:#009900">"__main__"</span>:
target_url = <span style="color:#009900">'http://www.biqukan.com/1_1094/'</span>
head = {}
head[<span style="color:#009900">'User-Agent'</span>] = <span style="color:#009900">'Mozilla/5.0 (Linux; Android 4.1.1; Nexus 7 Build/JRO03D) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.166 Safari/535.19'</span>
target_req = request.Request(url = target_url, headers = head)
target_response = request.urlopen(target_req)
target_html = target_response.read().decode(<span style="color:#009900">'gbk'</span>,<span style="color:#009900">'ignore'</span>)
<span style="color:#880000">#建立BeautifulSoup物件</span>
listmain_soup = BeautifulSoup(target_html,<span style="color:#009900">'lxml'</span>)
<span style="color:#880000">#搜尋文件樹,找出div標籤中class為listmain的所有子標籤</span>
chapters = listmain_soup.find_all(<span style="color:#009900">'div'</span>,class_ = <span style="color:#009900">'listmain'</span>)
<span style="color:#880000">#使用查詢結果再建立一個BeautifulSoup物件,對其繼續進行解析</span>
download_soup = BeautifulSoup(str(chapters), <span style="color:#009900">'lxml'</span>)
<span style="color:#880000">#開始記錄內容標誌位,只要正文卷下面的連結,最新章節列表連結剔除</span>
begin_flag = <span style="color:#000088">False</span>
<span style="color:#880000">#遍歷dl標籤下所有子節點</span>
<span style="color:#000088">for</span> child <span style="color:#000088">in</span> download_soup.dl.children:
<span style="color:#880000">#濾除回車</span>
<span style="color:#000088">if</span> child != <span style="color:#009900">'\n'</span>:
<span style="color:#880000">#找到《一念永恆》正文卷,使能標誌位</span>
<span style="color:#000088">if</span> child.string == <span style="color:#009900">u"《一念永恆》正文卷"</span>:
begin_flag = <span style="color:#000088">True</span>
<span style="color:#880000">#爬取連結</span>
<span style="color:#000088">if</span> begin_flag == <span style="color:#000088">True</span> <span style="color:#000088">and</span> child.a != <span style="color:#000088">None</span>:
download_url = <span style="color:#009900">"http://www.biqukan.com"</span> + child.a.get(<span style="color:#009900">'href'</span>)
download_name = child.string
print(download_name + <span style="color:#009900">" : "</span> + download_url)</code></span>執行結果：
c)爬取所有章節內容，並儲存到檔案中
整合以上程式碼，並進行相應處理，編寫如下程式碼：
<span style="color:#000000"><code><span style="color:#880000"># -*- coding:UTF-8 -*-</span>
<span style="color:#4f4f4f">from</span> urllib import request
<span style="color:#4f4f4f">from</span> bs4 import BeautifulSoup
import re
import sys
<span style="color:#000088">if</span> __name__ == <span style="color:#009900">"__main__"</span>:
<span style="color:#880000">#建立txt檔案</span>
<span style="color:#4f4f4f">file</span> = <span style="color:#4f4f4f">open</span>(<span style="color:#009900">'一念永恆.txt'</span>, <span style="color:#009900">'w'</span>, encoding=<span style="color:#009900">'utf-8'</span>)
<span style="color:#880000">#一念永恆小說目錄地址</span>
target_url = <span style="color:#009900">'http://www.biqukan.com/1_1094/'</span>
<span style="color:#880000">#User-Agent</span>
head = {}
head[<span style="color:#009900">'User-Agent'</span>] = <span style="color:#009900">'Mozilla/5.0 (Linux; Android 4.1.1; Nexus 7 Build/JRO03D) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.166 Safari/535.19'</span>
target_req = request.Request(url = target_url, headers = head)
target_response = request.urlopen(target_req)
target_html = target_response.<span style="color:#4f4f4f">read</span>().decode(<span style="color:#009900">'gbk'</span>,<span style="color:#009900">'ignore'</span>)
<span style="color:#880000">#建立BeautifulSoup物件</span>
listmain_soup = BeautifulSoup(target_html,<span style="color:#009900">'lxml'</span>)
<span style="color:#880000">#搜尋文件樹,找出div標籤中class為listmain的所有子標籤</span>
chapters = listmain_soup.find_all(<span style="color:#009900">'div'</span>,class_ = <span style="color:#009900">'listmain'</span>)
<span style="color:#880000">#使用查詢結果再建立一個BeautifulSoup物件,對其繼續進行解析</span>
download_soup = BeautifulSoup(str(chapters), <span style="color:#009900">'lxml'</span>)
<span style="color:#880000">#計算章節個數</span>
numbers = (<span style="color:#4f4f4f">len</span>(download_soup.dl.contents) - <span style="color:#006666">1</span>) / <span style="color:#006666">2</span> - <span style="color:#006666">8</span>
index = <span style="color:#006666">1</span>
<span style="color:#880000">#開始記錄內容標誌位,只要正文卷下面的連結,最新章節列表連結剔除</span>
begin_flag = False
<span style="color:#880000">#遍歷dl標籤下所有子節點</span>
<span style="color:#000088">for</span> child in download_soup.dl.children:
<span style="color:#880000">#濾除回車</span>
<span style="color:#000088">if</span> child != <span style="color:#009900">'\n'</span>:
<span style="color:#880000">#找到《一念永恆》正文卷,使能標誌位</span>
<span style="color:#000088">if</span> child.<span style="color:#000088">string</span> == u<span style="color:#009900">"《一念永恆》正文卷"</span>:
begin_flag = True
<span style="color:#880000">#爬取連結並下載連結內容</span>
<span style="color:#000088">if</span> begin_flag == True and child.a != None:
download_url = <span style="color:#009900">"http://www.biqukan.com"</span> + child.a.<span style="color:#4f4f4f">get</span>(<span style="color:#009900">'href'</span>)
download_req = request.Request(url = download_url, headers = head)
download_response = request.urlopen(download_req)
download_html = download_response.<span style="color:#4f4f4f">read</span>().decode(<span style="color:#009900">'gbk'</span>,<span style="color:#009900">'ignore'</span>)
download_name = child.<span style="color:#000088">string</span>
soup_texts = BeautifulSoup(download_html, <span style="color:#009900">'lxml'</span>)
texts = soup_texts.find_all(id = <span style="color:#009900">'content'</span>, class_ = <span style="color:#009900">'showtxt'</span>)
soup_text = BeautifulSoup(str(texts), <span style="color:#009900">'lxml'</span>)
write_flag = True
<span style="color:#4f4f4f">file</span>.<span style="color:#4f4f4f">write</span>(download_name + <span style="color:#009900">'\n\n'</span>)
<span style="color:#880000">#將爬取內容寫入檔案</span>
<span style="color:#000088">for</span> <span style="color:#000088">each</span> in soup_text.div.<span style="color:#000088">text</span>.<span style="color:#4f4f4f">replace</span>(<span style="color:#009900">'\xa0'</span>,<span style="color:#009900">''</span>):
<span style="color:#000088">if</span> <span style="color:#000088">each</span> == <span style="color:#009900">'h'</span>:
write_flag = False
<span style="color:#000088">if</span> write_flag == True and <span style="color:#000088">each</span> != <span style="color:#009900">' '</span>:
<span style="color:#4f4f4f">file</span>.<span style="color:#4f4f4f">write</span>(<span style="color:#000088">each</span>)
<span style="color:#000088">if</span> write_flag == True and <span style="color:#000088">each</span> == <span style="color:#009900">'\r'</span>:
<span style="color:#4f4f4f">file</span>.<span style="color:#4f4f4f">write</span>(<span style="color:#009900">'\n'</span>)
<span style="color:#4f4f4f">file</span>.<span style="color:#4f4f4f">write</span>(<span style="color:#009900">'\n\n'</span>)
<span style="color:#880000">#列印爬取進度</span>
sys.<span style="color:#000088">stdout</span>.<span style="color:#4f4f4f">write</span>(<span style="color:#009900">"已下載:%.3f%%"</span> % float(index/numbers) + <span style="color:#009900">'\r'</span>)
sys.<span style="color:#000088">stdout</span>.flush()
index += <span style="color:#006666">1</span>
<span style="color:#4f4f4f">file</span>.<span style="color:#4f4f4f">close</span>()</code></span>程式碼略顯粗糙，執行效率不高，還有很多可以改進的地方，執行效果如下圖所示：
最終生成的txt檔案，如下圖所示：
生成的txt檔案，可以直接拷貝到手機中進行閱讀，手機閱讀軟體可以解析這樣排版的txt檔案。
PS：如果覺得本篇本章對您有所幫助，歡迎關注、評論、點贊，謝謝！
參考文章：
URL：http://cuiqingcai.com/1319.html
2017年5月6日更新：
對程式碼進行了更改：添加了對錯誤章節的處理，並剔除了不是正文的部分。支援《筆趣看》網站大部分的小說下載。
程式碼檢視：
Github程式碼連線