我們今天來用Python爬蟲爬取金庸所有的武俠小說，網址為：http://jinyong.zuopinj.com/，網頁如下：

　　Python代碼如下：

 1 # -*- coding: utf-8 -*-
 2 import urllib.request  
 3 from bs4 import BeautifulSoup
 4 
 5 #get book‘s chapter
 6 def get_chapter(url):
 7     html = urllib.request.urlopen(url)  
 8     content = html.read().decode(‘utf8
 
‘)
 9     html.close()
10     soup = BeautifulSoup(content, "lxml")
11     title = soup.find(‘h1‘).text    #chapter title
12     text = soup.find(‘div‘, id=‘htmlContent‘)    #chapter content
13     #processing the content to get nice style
14     content = text.get_text(‘\n‘,‘br/‘).replace(‘\n‘, ‘\n    
 
‘)
15     content = content.replace(‘　　‘, ‘\n　　‘)
16     return title, ‘    ‘+content
17         
18 def main():
19     books = [‘射雕英雄傳‘,‘天龍八部‘,‘鹿鼎記‘,‘神雕俠侶‘,‘笑傲江湖‘,‘碧血劍‘,‘倚天屠龍記‘,20              ‘飛狐外傳‘,‘書劍恩仇錄‘,‘連城訣‘,‘俠客行‘,‘越女劍‘,‘鴛鴦刀‘,‘白馬嘯西風‘,21              ‘雪山飛狐‘]
22     order = [1,2,3,4,5,6,7,8,10,11,12,14,15,13,9]  #
 
order of books to scrapy
23     #list to store each book‘s scrapying range
24     page_range = [1,43,94,145,185,225,248,289,309,329,341,362,363,364,375,385]
25 
26     for i,book in enumerate(books):
27         for num in range(page_range[i],page_range[i+1]):
28             url = "http://jinyong.zuopinj.com/%s/%s.html"%(order[i],num)
29             try:
30                 title, chapter = get_chapter(url)
31                 with open(‘E://%s.txt‘%book, ‘a‘, encoding=‘gb18030‘) as f:
32                     print(book+‘:‘+title+‘-->寫入成功！‘)
33                     f.write(title+‘\n\n\n‘)
34                     f.write(chapter+‘\n\n\n‘)
35             except Exception as e:
36                 print(e) 
37     print(‘全部寫入完畢!‘)
38 
39 main()

　　運行結果如下：

上面的運行結果“HTTP Error 404: Not Found”是因為這個網頁不存在，並不影響書本內容的完整性。我們可以去E盤查看文件是否下載成功：

·　　15本書都下載完畢了！整個過程才用了不到10分鐘！爬蟲的力量真是偉大啊~~

40行代碼爬取金庸所有武俠小說