python爬蟲知識點總結(七)PyQuery詳解
阿新 • • 發佈:2018-08-03
get 初始化 span 2個 查看 sel docs lin query
官方學習文檔:http://pyquery.readthedocs.io/en/latest/api.html
一、什麽是PyQuery?
答:強大有靈活的網頁解析庫,模仿jQuery實現。如果你覺得正則表達式寫起來太麻煩,如果你覺的BeautifulSoup語法太難記,如果你熟悉jQuery的語法,那麽PyQuery就是你的絕佳選擇。
二、安裝
pip3 install pyquery
三、初始化
1、字符串初始化
html = ‘‘‘ <div> <ul> <li class="item-0">first item</li> <li class="item-1"><a href="link2.html">second item</a></li> <li class="item-0 active"><a href="link3.html"><span class="blod">thrid item</span></a></li> <li class="item-1 active"><a href="link4.html">fourth item</a></li> <li class="item-0"><a href="link5.html">fifth item</a></li> </ul> </div> ‘‘‘ from pyquery import PyQuery as pq doc = pq(html) print(doc(‘li‘))
2、URL初始化
from pyquery import PyQuery as pq doc = pq(url="http://www.baidu.com") print(doc(‘head‘))
3、文本初始化
from pyquery import PyQuery as pq doc = pq(filename = ‘demo.html‘) print(doc(‘li‘))
四、基本CSS選擇器
html = ‘‘‘ <div id="container"> <ul class="list"> <li class="item-0">first item</li> <li class="item-1"><a href="link2.html">second item</a></li> <li class="item-0 active"><a href="link3.html"><span class="blod">thrid item</span></a></li> <li class="item-1 active"><a href="link4.html">fourth item</a></li> <li class="item-0"><a href="link5.html">fifth item</a></li> </ul> </div> ‘‘‘ from pyquery import PyQuery as pq doc = pq(html) print(doc(‘#container .list li‘))
1、查找元素
子元素
html = ‘‘‘ <div id="container"> <ul class="list"> <li class="item-0">first item</li> <li class="item-1"><a href="link2.html">second item</a></li> <li class="item-0 active"><a href="link3.html"><span class="blod">thrid item</span></a></li> <li class="item-1 active"><a href="link4.html">fourth item</a></li> <li class="item-0"><a href="link5.html">fifth item</a></li> </ul> </div> ‘‘‘ from pyquery import PyQuery as pq doc = pq(html) items = doc(‘.list‘) print(type(items)) print(items) lis = items.find(‘li‘) print(type(lis)) print(lis)
lis = items.children() print(type(lis)) print(lis)
lis = items.children(‘.active‘) print(lis)
父元素
html = ‘‘‘ <div id="container"> <ul class="list"> <li class="item-0">first item</li> <li class="item-1"><a href="link2.html">second item</a></li> <li class="item-0 active"><a href="link3.html"><span class="blod">thrid item</span></a></li> <li class="item-1 active"><a href="link4.html">fourth item</a></li> <li class="item-0"><a href="link5.html">fifth item</a></li> </ul> </div> ‘‘‘ from pyquery import PyQuery as pq doc = pq(html) items = doc(‘.list‘) container = items.parent() print(type(container)) print(container)
html = ‘‘‘ <div class="wrap"> <div id="container"> <ul class="list"> <li class="item-0">first item</li> <li class="item-1"><a href="link2.html">second item</a></li> <li class="item-0 active"><a href="link3.html"><span class="blod">thrid item</span></a></li> <li class="item-1 active"><a href="link4.html">fourth item</a></li> <li class="item-0"><a href="link5.html">fifth item</a></li> </ul> </div> </div> ‘‘‘ from pyquery import PyQuery as pq doc = pq(html) items = doc(‘.list‘) container = items.parents() print(type(container)) print(container)
parent = items.parents(‘.wrap‘) print(parents)
兄弟元素
html = ‘‘‘ <div class="wrap"> <div id="container"> <ul class="list"> <li class="item-0">first item</li> <li class="item-1"><a href="link2.html">second item</a></li> <li class="item-0 active"><a href="link3.html"><span class="blod">thrid item</span></a></li> <li class="item-1 active"><a href="link4.html">fourth item</a></li> <li class="item-0"><a href="link5.html">fifth item</a></li> </ul> </div> </div> ‘‘‘ from pyquery import PyQuery as pq doc = pq(html) li = doc(‘.list .item-0.active‘) print(li.siblings())
html = ‘‘‘ <div class="wrap"> <div id="container"> <ul class="list"> <li class="item-0">first item</li> <li class="item-1"><a href="link2.html">second item</a></li> <li class="item-0 active"><a href="link3.html"><span class="blod">thrid item</span></a></li> <li class="item-1 active"><a href="link4.html">fourth item</a></li> <li class="item-0"><a href="link5.html">fifth item</a></li> </ul> </div> </div> ‘‘‘ from pyquery import PyQuery as pq doc = pq(html) li = doc(‘.list .item-0.active‘) print(li.siblings(‘.active‘))
五、遍歷
1、單個元素
html = ‘‘‘ <div class="wrap"> <div id="container"> <ul class="list"> <li class="item-0">first item</li> <li class="item-1"><a href="link2.html">second item</a></li> <li class="item-0 active"><a href="link3.html"><span class="blod">thrid item</span></a></li> <li class="item-1 active"><a href="link4.html">fourth item</a></li> <li class="item-0"><a href="link5.html">fifth item</a></li> </ul> </div> </div> ‘‘‘ from pyquery import PyQuery as pq doc = pq(html) li = doc(‘.item-0.active‘) print(li)
html = ‘‘‘ <div class="wrap"> <div id="container"> <ul class="list"> <li class="item-0">first item</li> <li class="item-1"><a href="link2.html">second item</a></li> <li class="item-0 active"><a href="link3.html"><span class="blod">thrid item</span></a></li> <li class="item-1 active"><a href="link4.html">fourth item</a></li> <li class="item-0"><a href="link5.html">fifth item</a></li> </ul> </div> </div> ‘‘‘ from pyquery import PyQuery as pq doc = pq(html) lis = doc(‘li‘).items() print(type(lis)) for li in lis: print(li)
2、獲取信息
獲取屬性
html = ‘‘‘ <div class="wrap"> <div id="container"> <ul class="list"> <li class="item-0">first item</li> <li class="item-1"><a href="link2.html">second item</a></li> <li class="item-0 active"><a href="link3.html"><span class="blod">thrid item</span></a></li> <li class="item-1 active"><a href="link4.html">fourth item</a></li> <li class="item-0"><a href="link5.html">fifth item</a></li> </ul> </div> </div> ‘‘‘ from pyquery import PyQuery as pq doc = pq(html) a = doc(‘.item-0.active a‘) print(a) print(a.attr(‘href‘)) print(a.attr.href)
獲取文本
html = ‘‘‘ <div class="wrap"> <div id="container"> <ul class="list"> <li class="item-0">first item</li> <li class="item-1"><a href="link2.html">second item</a></li> <li class="item-0 active"><a href="link3.html"><span class="blod">thrid item</span></a></li> <li class="item-1 active"><a href="link4.html">fourth item</a></li> <li class="item-0"><a href="link5.html">fifth item</a></li> </ul> </div> </div> ‘‘‘ from pyquery import PyQuery as pq doc = pq(html) a = doc(‘.item-0.active a‘) print(a) print(a.attr(‘href‘)) print(a.text())
獲取HTML
html = ‘‘‘ <div class="wrap"> <div id="container"> <ul class="list"> <li class="item-0">first item</li> <li class="item-1"><a href="link2.html">second item</a></li> <li class="item-0 active"><a href="link3.html"><span class="blod">thrid item</span></a></li> <li class="item-1 active"><a href="link4.html">fourth item</a></li> <li class="item-0"><a href="link5.html">fifth item</a></li> </ul> </div> </div> ‘‘‘ from pyquery import PyQuery as pq doc = pq(html) li = doc(‘.item-0.active‘) print(li) print(li.html())
六、DOM操作
1、addClass\removeClass
html = ‘‘‘ <div class="wrap"> <div id="container"> <ul class="list"> <li class="item-0">first item</li> <li class="item-1"><a href="link2.html">second item</a></li> <li class="item-0 active"><a href="link3.html"><span class="blod">thrid item</span></a></li> <li class="item-1 active"><a href="link4.html">fourth item</a></li> <li class="item-0"><a href="link5.html">fifth item</a></li> </ul> </div> </div> ‘‘‘ from pyquery import PyQuery as pq doc = pq(html) li = doc(‘.item-0.active‘) print(li) li.removeClass(‘active‘) print(li) li.addClass(‘active‘) print(li)
2、attr、css
html = ‘‘‘ <div class="wrap"> <div id="container"> <ul class="list"> <li class="item-0">first item</li> <li class="item-1"><a href="link2.html">second item</a></li> <li class="item-0 active"><a href="link3.html"><span class="blod">thrid item</span></a></li> <li class="item-1 active"><a href="link4.html">fourth item</a></li> <li class="item-0"><a href="link5.html">fifth item</a></li> </ul> </div> </div> ‘‘‘ from pyquery import PyQuery as pq doc = pq(html) li = doc(‘.item-0.active‘) print(li) li.attr(‘name‘,‘link‘) print(li) li.css(‘font-size‘,‘14px‘) print(li)
3、remove
html = ‘‘‘ <div class="wrap"> <div id="container"> <ul class="list"> <li class="item-0">first item</li> <li class="item-1"><a href="link2.html">second item</a></li> <li class="item-0 active"><a href="link3.html"><span class="blod">thrid item</span></a></li> <li class="item-1 active"><a href="link4.html">fourth item</a></li> <li class="item-0"><a href="link5.html">fifth item</a></li> </ul> </div> </div> ‘‘‘ from pyquery import PyQuery as pq doc = pq(html) wrap = doc(‘.wrap‘) print(wrap.text()) wrap.find(‘p‘).remove() print(wrap.text())
其他DOM方法
http://pyquery.readthedocs.io/en/latest/api.html
七、偽類選擇器
html = ‘‘‘ <div class="wrap"> <div id="container"> <ul class="list"> <li class="item-0">first item</li> <li class="item-1"><a href="link2.html">second item</a></li> <li class="item-0 active"><a href="link3.html"><span class="blod">thrid item</span></a></li> <li class="item-1 active"><a href="link4.html">fourth item</a></li> <li class="item-0"><a href="link5.html">fifth item</a></li> </ul> </div> </div> ‘‘‘ from pyquery import PyQuery as pq doc = pq(html) li = doc("li:first-child") print(li) li = doc("li:last-child") print(li) # 標簽從0開始 li = doc("li:nth-child(2)") # ntj-child(2)獲取第2個標簽 print(li) li = doc("li:gt(2)") # gt-child(2)獲取比2大的標簽 print(li) li = doc("li:nth-child(2n)") # nth-child(2n)獲取偶數的標簽 print(li) li = doc("li:contains(second)") # contains(second)獲取包含second文本的標簽 print(li)
更多CSS選擇器可以查看 http://www.w3school.com.cn/css/index.html
官方文檔網站:http://pyquery.readthedocs.io
jQuery官方文檔:http://jquery.cuishifeng.cn/
python爬蟲知識點總結(七)PyQuery詳解