1. 程式人生 > >Python爬蟲【解析庫之beautifulsoup】

Python爬蟲【解析庫之beautifulsoup】

close **kwargs contents pip and lac 代碼 ide num

解析庫的安裝

pip3 install beautifulsoup4

初始化 BeautifulSoup(str,"解析庫")

from bs4 import BeautifulSoup

html=‘‘‘
<div class="panel">
<div class="panel-heading">
<h4>Hello</h4>
</div>
<div class="panel-body">
<ul class="list" id="list-1">
<li class="element">Foo</li>

<li class="element">Bar</li>
<li class="element">Jay</li>
</ul>
<ul class="list list-small" id="list-2">
<li class="element">Foo</li>
<li class="element">Bar</li>
</ul>
</div>
</div>
‘‘‘

soup = BeautifulSoup(html,"lxml") # soup = BeautifulSoup(html,"html.parser")

標簽選擇器

選擇元素 soup.E

技術分享圖片
html = """
<html><head><title>The Dormouse‘s story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse‘s story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
""" from bs4 import BeautifulSoup soup = BeautifulSoup(html, lxml) print(soup.title) print(type(soup.title)) print(soup.head) print(soup.p) """ 打印結果: <title>The Dormouse‘s story</title> <class ‘bs4.element.Tag‘> <head><title>The Dormouse‘s story</title></head> <p class="title" name="dromouse"><b>The Dormouse‘s story</b></p> """
選擇元素

獲取名稱 soup.E.name

技術分享圖片
html = """
<html><head><title>The Dormouse‘s story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse‘s story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, lxml)
print(soup.title.name) # title
獲取名稱

獲取屬性 soup.E.attrs[ ] or soup.E[ ]

技術分享圖片
html = """
<html><head><title>The Dormouse‘s story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse‘s story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, lxml)
print(soup.p.attrs[name])
print(soup.p[name])
"""
dromouse
dromouse
"""
獲取屬性

獲取內容 soup.E.string

技術分享圖片
html = """
<html><head><title>The Dormouse‘s story</title></head>
<body>
<p clss="title" name="dromouse"><b>The Dormouse‘s story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, lxml)
print(soup.p.string)
"""
The Dormouse‘s story
"""
獲取內容

嵌套選擇 soup.E.E

技術分享圖片
html = """
<html><head><title>The Dormouse‘s story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse‘s story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, lxml)
print(soup.head.title.string)
"""
The Dormouse‘s story
"""
View Code

子節點 soup.E.contents

技術分享圖片
 1 html = """
 2 <html>
 3     <head>
 4         <title>The Dormouse‘s story</title>
 5     </head>
 6     <body>
 7         <p class="story">
 8             Once upon a time there were three little sisters; and their names were
 9             <a href="http://example.com/elsie" class="sister" id="link1">
10                 <span>Elsie</span>
11             </a>
12             <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> 
13             and
14             <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>
15             and they lived at the bottom of a well.
16         </p>
17         <p class="story">...</p>
18 """
19 from bs4 import BeautifulSoup
20 soup = BeautifulSoup(html, lxml)
21 print(soup.p.contents)
22 """
23 [‘\n            Once upon a time there were three little sisters; and their names were\n            ‘, <a class="sister" href="http://example.com/elsie" id="link1">
24 <span>Elsie</span>
25 </a>, ‘\n‘, <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>, ‘ \n            and\n            ‘, <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>, ‘\n            and they lived at the bottom of a well.\n        ‘]
26 
27 """
View Code

子節點 soup.E.children

技術分享圖片
html = """
<html>
    <head>
        <title>The Dormouse‘s story</title>
    </head>
    <body>
        <p class="story">
            Once upon a time there were three little sisters; and their names were
            <a href="http://example.com/elsie" class="sister" id="link1">
                <span>Elsie</span>
            </a>
            <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> 
            and
            <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>
            and they lived at the bottom of a well.
        </p>
        <p class="story">...</p>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, lxml)
print(soup.p.children)
for i, child in enumerate(soup.p.children):
    print(i, child)
"""
<list_iterator object at 0x00B116D0>
0 
            Once upon a time there were three little sisters; and their names were
            
1 <a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>
2 

3 <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>
4  
            and
            
5 <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>
6 
            and they lived at the bottom of a well.
"""
View Code

子孫節點 soup.E.descendants 包括標簽裏面的文本都屬於

技術分享圖片
html = """
<html>
    <head>
        <title>The Dormouse‘s story</title>
    </head>
    <body>
        <p class="story">
            Once upon a time there were three little sisters; and their names were
            <a href="http://example.com/elsie" class="sister" id="link1">
                <span>Elsie</span>
            </a>
            <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> 
            and
            <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>
            and they lived at the bottom of a well.
        </p>
        <p class="story">...</p>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, lxml)
print(soup.p.descendants)
for i, child in enumerate(soup.p.descendants):
    print(i, child)
"""
<generator object descendants at 0x03EBD420>
0 
            Once upon a time there were three little sisters; and their names were
            
1 <a class="sister" href="http://example.com/elsie" id="link1">
<span>Elsie</span>
</a>
2 

3 <span>Elsie</span>
4 Elsie
5 

6 

7 <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>
8 Lacie
9  
            and
            
10 <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>
11 Tillie
12 
            and they lived at the bottom of a well.
"""
View Code

父節點 soup.E.parent

技術分享圖片
html = """
<html>
    <head>
        <title>The Dormouse‘s story</title>
    </head>
    <body>
        <p class="story">
            Once upon a time there were three little sisters; and their names were
            <a href="http://example.com/elsie" class="sister" id="link1">
                <span>Elsie</span>
            </a>
            <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> 
            and
            <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>
            and they lived at the bottom of a well.
        </p>
        <p class="story">...</p>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, lxml)
print(soup.a.parent)
View Code

祖先節點 soup.E.parents

技術分享圖片
html = """
<html>
    <head>
        <title>The Dormouse‘s story</title>
    </head>
    <body>
        <p class="story">
            Once upon a time there were three little sisters; and their names were
            <a href="http://example.com/elsie" class="sister" id="link1">
                <span>Elsie</span>
            </a>
            <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> 
            and
            <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>
            and they lived at the bottom of a well.
        </p>
        <p class="story">...</p>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, lxml)
print(list(enumerate(soup.a.parents)))
View Code

兄弟節點 soup.E.next_siblings soup.E.previous_siblings

技術分享圖片
html = """
<html>
    <head>
        <title>The Dormouse‘s story</title>
    </head>
    <body>
        <p class="story">
            Once upon a time there were three little sisters; and their names were
            <a href="http://example.com/elsie" class="sister" id="link1">
                <span>Elsie</span>
            </a>
            <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> 
            and
            <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>
            and they lived at the bottom of a well.
        </p>
        <p class="story">...</p>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, lxml)
print(list(enumerate(soup.a.next_siblings)))
print(list(enumerate(soup.a.previous_siblings)))
View Code

標準選擇器

find_all( name , attrs , recursive , text , **kwargs )

可根據標簽名、屬性、內容查找文檔

標簽名獲取 soup.find_all(‘name‘)

技術分享圖片
html=‘‘‘
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
‘‘‘
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, lxml)
print(soup.find_all(ul))
View Code

屬性獲取 soup.find_all(attrs={})

技術分享圖片
html=‘‘‘
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1" name="elements">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
‘‘‘
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, lxml)
print(soup.find_all(attrs={id: list-1}))
print(soup.find_all(attrs={name: elements}))
View Code 技術分享圖片
html=‘‘‘
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
‘‘‘
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, lxml)
print(soup.find_all(id=list-1))
print(soup.find_all(class_=element))
第二種

文本內容獲取

技術分享圖片
html=‘‘‘
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
‘‘‘
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, lxml)
print(soup.find_all(text=Foo))
View Code

find( name , attrs , recursive , text , **kwargs )

find返回單個元素,find_all返回所有元素

html=‘‘‘
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
‘‘‘
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, lxml)
print(soup.find(ul))
print(type(soup.find(ul)))
print(soup.find(page))

find_parents() find_parent()

find_parents()返回所有祖先節點,find_parent()返回直接父節點。

find_next_siblings() find_next_sibling()

find_next_siblings()返回後面所有兄弟節點,find_next_sibling()返回後面第一個兄弟節點。

find_previous_siblings() find_previous_sibling()

find_previous_siblings()返回前面所有兄弟節點,find_previous_sibling()返回前面第一個兄弟節點。

find_all_next() find_next()

find_all_next()返回節點後所有符合條件的節點, find_next()返回第一個符合條件的節點

find_all_previous() 和 find_previous()

find_all_previous()返回節點後所有符合條件的節點, find_previous()返回第一個符合條件的節點

CSS選擇器

通過select()直接傳入CSS選擇器即可完成選擇 soup.select(css選擇器)

技術分享圖片
html=‘‘‘
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
‘‘‘
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, lxml)
print(soup.select(.panel .panel-heading))
print(soup.select(ul li))
print(soup.select(#list-2 .element))
print(type(soup.select(ul)[0]))
View Code 技術分享圖片
html=‘‘‘
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
‘‘‘
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, lxml)
for ul in soup.select(ul):
    print(ul.select(li))
View Code

獲取屬性 E.attrs[] or E[]

技術分享圖片
html=‘‘‘
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
‘‘‘
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, lxml)
for ul in soup.select(ul):
    print(ul[id])
    print(ul.attrs[id])
View Code

獲取內容 get_text()

技術分享圖片
html=‘‘‘
<div class="panel">
    <div class="panel-heading">
        <h4>Hello</h4>
    </div>
    <div class="panel-body">
        <ul class="list" id="list-1">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
            <li class="element">Jay</li>
        </ul>
        <ul class="list list-small" id="list-2">
            <li class="element">Foo</li>
            <li class="element">Bar</li>
        </ul>
    </div>
</div>
‘‘‘
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, lxml)
for li in soup.select(li):
    print(li.get_text())
View Code

總結

  • 推薦使用lxml解析庫,必要時使用html.parser
  • 標簽選擇篩選功能弱但是速度快
  • 建議使用find()、find_all() 查詢匹配單個結果或者多個結果
  • 如果對CSS選擇器熟悉建議使用select()
  • 記住常用的獲取屬性和文本值的方法

本文代碼皆來自崔慶才《Python3網絡爬蟲開發實戰》

Python爬蟲【解析庫之beautifulsoup】