python一鍵電影搜索與下載
阿新 • • 發佈：2019-03-05
Coding sub ebr col 正常的 根據 rect finally web
代碼地址如下：
http://www.demodashi.com/demo/14313.html
python一鍵電影搜索與下載
概述
使用python搜索並爬取豆瓣電影信息,包括評分,主演,導演,類型,上映時間,電影簡介等信息,然後再從電影天堂搜索並爬取電影下載鏈接.
準備工作
安裝python3.6
略
安裝requests庫(用於請求靜態頁面)
pip install requests -i https://mirrors.ustc.edu.cn/pypi/web/simple安裝lxml庫(用於解析html文件)
pip install lxml -i https://mirrors.ustc.edu.cn/pypi/web/simple
本教程爬取的電影信息來自豆瓣電影,下載鏈接來自電影天堂
https://movie.douban.com/j/subject_suggest?q=電影名稱
http://s.ygdy8.com/plus/so.php?keytype=0&pagesize=10&searchtype=title&keyword=電影名稱頁面分析
豆瓣電影搜索
豆瓣電影搜索的鏈接如下:
https://movie.douban.com/j/subject_suggest?q=電影名稱只需要一個參數q,它的值是utf-8編碼的電影名稱,比如我們要搜索 星際穿越 相關信息, 其中 %e6%98%9f%e9%99%85%e7%a9%bf%e8%b6%8a 是 星際穿越 的url格式的utf-8編碼.:
https://movie.douban.com/j/subject_suggest?q=%e6%98%9f%e9%99%85%e7%a9%bf%e8%b6%8a服務器返回的搜索結果是一個json文件 subject_suggest.json ,如下:
[ { "episode" : "", "id" : "1889243", "img" : "https://img3.doubanio.com/view/photo/s_ratio_poster/public/p2206088801.jpg", "sub_title" : "Interstellar", "title" : "星際穿越", "type" : "movie", "url" : "https://movie.douban.com/subject/1889243/?suggest=%E6%98%9F%E9%99%85%E7%A9%BF%E8%B6%8A", "year" : "2014" }, { "episode" : "", "id" : "26263467", "img" : "https://img3.doubanio.com/view/photo/s_ratio_poster/public/p2445481443.jpg", "sub_title" : "The Science of Interstellar", "title" : "《星際穿越》中的科學", "type" : "movie", "url" : "https://movie.douban.com/subject/26263467/?suggest=%E6%98%9F%E9%99%85%E7%A9%BF%E8%B6%8A", "year" : "2014" }, { "episode" : "", "id" : "26255844", "img" : "https://img3.doubanio.com/view/photo/s_ratio_poster/public/p2519643575.jpg", "sub_title" : "Interstellar: Nolan's Odyssey", "title" : "星際穿越：諾蘭的奧德賽", "type" : "movie", "url" : "https://movie.douban.com/subject/26255844/?suggest=%E6%98%9F%E9%99%85%E7%A9%BF%E8%B6%8A", "year" : "2014" } ]
共搜索到了3個與 星際穿越 相關的結果,其中我們需要關註的有:
| key | 含義 |
|---|---|
| title | 標題 |
| sub_title | 子標題(英文標題) |
| url | 詳情鏈接 |
我們需要再次打開搜索結果中對應的電影詳情鏈接,獲取電影的評分,導演,主演,類型,上映時間,簡介,影評等信息.
比如我們打開搜索結果的第一項,結果如下:
我們打開它的源碼看看(按F12打開調試):
可以看到其head中的一個標簽 *** /html/head/script[@type="application/ld+json"] *** 中存放的是一個json文件,這個json中就包含了我們需要的所有電影信息,提取出來如下:
{
"@context": "http://schema.org",
"name": "星際穿越 Interstellar",
"url": "/subject/1889243/",
"image": "https://img3.doubanio.com/view/photo/s_ratio_poster/public/p2206088801.jpg",
"director":
[
{
"@type": "Person",
"url": "/celebrity/1054524/",
"name": "克裏斯托弗·諾蘭 Christopher Nolan"
}
]
,
"author":
[
{
"@type": "Person",
"url": "/celebrity/1275104/",
"name": "喬納森·諾蘭 Jonathan Nolan"
}
,
{
"@type": "Person",
"url": "/celebrity/1054524/",
"name": "克裏斯托弗·諾蘭 Christopher Nolan"
}
,
{
"@type": "Person",
"url": "/celebrity/1018568/",
"name": "基普·索恩 Kip Thorne"
}
]
,
"actor":
[
{
"@type": "Person",
"url": "/celebrity/1040511/",
"name": "馬修·麥康納 Matthew McConaughey"
}
,
{
"@type": "Person",
"url": "/celebrity/1048027/",
"name": "安妮·海瑟薇 Anne Hathaway"
}
,
{
"@type": "Person",
"url": "/celebrity/1000225/",
"name": "傑西卡·查斯坦 Jessica Chastain"
}
,
{
"@type": "Person",
"url": "/celebrity/1022593/",
"name": "卡西·阿弗萊克 Casey Affleck"
}
,
{
"@type": "Person",
"url": "/celebrity/1054509/",
"name": "邁克爾·凱恩 Michael Caine"
}
,
{
"@type": "Person",
"url": "/celebrity/1054443/",
"name": "馬特·達蒙 Matt Damon"
}
,
{
"@type": "Person",
"url": "/celebrity/1267954/",
"name": "麥肯吉·弗依 Mackenzie Foy"
}
,
{
"@type": "Person",
"url": "/celebrity/1325862/",
"name": "蒂莫西·柴勒梅德 Timothée Chalamet"
}
,
{
"@type": "Person",
"url": "/celebrity/1036407/",
"name": "艾倫·伯斯汀 Ellen Burstyn"
}
,
{
"@type": "Person",
"url": "/celebrity/1027824/",
"name": "約翰·利思戈 John Lithgow"
}
,
{
"@type": "Person",
"url": "/celebrity/1049518/",
"name": "韋斯·本特利 Wes Bentley"
}
,
{
"@type": "Person",
"url": "/celebrity/1201851/",
"name": "大衛·吉雅西 David Gyasi"
}
,
{
"@type": "Person",
"url": "/celebrity/1113911/",
"name": "比爾·歐文 Bill Irwin"
}
,
{
"@type": "Person",
"url": "/celebrity/1010536/",
"name": "托弗·戈瑞斯 Topher Grace"
}
,
{
"@type": "Person",
"url": "/celebrity/1313709/",
"name": "科萊特·沃夫 Collette Wolfe"
}
,
{
"@type": "Person",
"url": "/celebrity/1330971/",
"name": "弗朗西斯·X·麥卡蒂 Francis X. McCarthy"
}
,
{
"@type": "Person",
"url": "/celebrity/1344601/",
"name": "安德魯·博爾巴 Andrew Borba"
}
,
{
"@type": "Person",
"url": "/celebrity/1004844/",
"name": "喬什·斯圖沃特 Josh Stewart"
}
,
{
"@type": "Person",
"url": "/celebrity/1202795/",
"name": "萊雅·卡裏恩斯 Leah Cairns"
}
,
{
"@type": "Person",
"url": "/celebrity/1338863/",
"name": "利亞姆·迪金森 Liam Dickinson"
}
,
{
"@type": "Person",
"url": "/celebrity/1274631/",
"name": "傑夫·赫普內爾 Jeff Hephner"
}
,
{
"@type": "Person",
"url": "/celebrity/1045604/",
"name": "伊萊耶斯·加貝爾 Elyes Gabel"
}
,
{
"@type": "Person",
"url": "/celebrity/1018020/",
"name": "布魯克·史密斯 Brooke Smith"
}
,
{
"@type": "Person",
"url": "/celebrity/1000231/",
"name": "大衛·奧伊羅 David Oyelowo"
}
,
{
"@type": "Person",
"url": "/celebrity/1055380/",
"name": "威廉姆·德瓦內 William Devane"
}
,
{
"@type": "Person",
"url": "/celebrity/1366207/",
"name": "拉什·費加 Russ Fega"
}
,
{
"@type": "Person",
"url": "/celebrity/1344602/",
"name": "格裏芬·弗雷澤 Griffen Fraser"
}
,
{
"@type": "Person",
"url": "/celebrity/1344603/",
"name": "弗洛拉·諾蘭 Flora Nolan"
}
]
,
"datePublished": "2014-11-07",
"genre": ["\u5267\u60c5", "\u79d1\u5e7b", "\u5192\u9669"],
"duration": "PT2H49M",
"description": "近未來的地球黃沙遍野，小麥、秋葵等基礎農作物相繼因枯萎病滅絕，人類不再像從前那樣仰望星空，放縱想象力和靈感的迸發，而是每日在沙塵暴的肆虐下倒數著所剩不多的光景。在家務農的前NASA宇航員庫珀（馬修·麥...",
"@type": "Movie",
"aggregateRating": {
"@type": "AggregateRating",
"ratingCount": "638995",
"bestRating": "10",
"worstRating": "2",
"ratingValue": "9.2"
}
}| key | 含義 |
|---|---|
| name | 電影名稱 |
| director | 電影導演 |
| author | 主要演員 |
| datePublished | 上映時間 |
| genre | 電影類型 |
| description | 電影簡介 |
| aggregateRating[ratingValue] | 電影評分 |
電影天堂搜索
豆瓣電影搜索的鏈接如下:
http://s.ygdy8.com/plus/so.php?keytype=0&pagesize=10&searchtype=title&keyword=電影名稱只需要一個參數q,它的值是utf-8編碼的電影名稱,比如我們要搜索 星際穿越 相關信息. 其中 %D0%C7%BC%CA%B4%A9%D4%BD 是 星際穿越 的url格式的gb2312編碼::
http://s.ygdy8.com/plus/so.php?keytype=0&keyword=%D0%C7%BC%CA%B4%A9%D4%BD服務器返回的搜索結果是一個html頁面其中只有第一項是我們想要的結果,如下:
按F12打開調試可以看到,搜索結果列表包含在一個 class="co_content8" 的div標簽中.搜索結果的標題對應的鏈接就是電影詳情頁面,其中無用的廣告頁面的鏈接中包含 game ,我們可以據此過濾掉不想要的結果.
打開電影詳情頁面,可以看到下載鏈接包含在一個 bgcolor="#fdfddf" 的table中:
源碼詳解
使用requests下載靜態html頁面
該函數用於下載圖集列表頁面，這個頁面是靜態的，可以直接通過 requests.get(url) 函數抓取。但是有一點需要註意，為了把我們的爬蟲偽裝成正常的瀏覽器請求，避免我們的爬蟲被服務器禁止，我們需要給 requests 添加http請求頭，其中包含偽造的 User-Agent 瀏覽器標識
def download_page_html(url, sel=0):
phtml = None
page = None
try:
requests_header["Host"] = host_cookie[sel][0]
requests_header["Cookie"] = host_cookie[sel][1]
# 選擇一個隨機的User-Agent
requests_header["User-Agent"] = random.choice(user_agent_list)
# print(requests_header["User-Agent"])
# print(requests_header)
page = requests.get(url=url, headers=requests_header, timeout=15) # 請求指定的頁面
# print(page.encoding)
if page.encoding == "ISO-8859-1":
page.encoding = "gb2312" # 轉換頁面的編碼為gb2312(避免中文亂碼)
phtml = page.text # 提取請求結果中包含的html文本
# print("requests success")
except requests.exceptions.RequestException as e:
print("requests error:", e)
phtml = None
finally:
if page != None:
page.close()
return phtml從豆瓣電影上搜索並下載電影信息
該函數用於根據指定的電影名稱,從豆瓣電影服務器上搜索電影,解析搜索結果並顯示,然後根據輸入顯示指定搜索結果的詳情.
def movie_douban(mvsearch_name):
DOUBANMV_SEARCH_URL = "https://movie.douban.com/j/subject_suggest"
DOUBANMV_SEARCH_PAR = {"q": ""}
# mvsearch_name = "星際迷航"
if mvsearch_name == None:
return -1
DOUBANMV_SEARCH_PAR["q"] = mvsearch_name
# url參數編碼
mvsearch_par = parse.urlencode(DOUBANMV_SEARCH_PAR, encoding="utf-8")
# print(mvsearch_par)
mvsearch_url = "{0}?{1}".format(DOUBANMV_SEARCH_URL, mvsearch_par)
# print(mvsearch_url)
# 下載指定url
mvsearch_html = download_page_html(mvsearch_url, 2)
if mvsearch_html == None:
print("下載出錯,可能IP被服務器封禁,可稍後再試!")
return -1
# 解析下載的結果(json格式)
try:
mvsearch_json = json.loads(mvsearch_html)
except json.JSONDecodeError as e:
print("出現錯誤:", e)
return -1
if mvsearch_json == None or len(mvsearch_json) == 0:
print("解析出錯!")
return -1
# print(mvsearch_json)
# 輸出解析結果
print("共找到", len(mvsearch_json), "個關於", mvname, "的結果: ")
for i in range(len(mvsearch_json)):
print("\t", i+1, mvsearch_json[i]["title"],
"/", mvsearch_json[i]["sub_title"])
# 選擇需要查看的項
search_sel = input("請選擇需要查看的項:")
if search_sel.isdigit() != True:
print("輸入有誤!")
return -1
search_sel = int(search_sel)
if search_sel > len(mvsearch_json) or search_sel < 1:
print("輸入有誤!")
return -1
search_sel = search_sel - 1
# 獲取需要查看的項的url,下載需要查看的項
mvcontent_url = mvsearch_json[search_sel]["url"]
mvcontent_html = download_page_html(mvcontent_url, 2)
# 解析需要查看的項
doubanmv_etree_html = lxml.html.fromstring(mvcontent_html)
mvcontent_xpath = '/html/head//script[@type="application/ld+json"]/text()'
mvcontent_text = doubanmv_etree_html.xpath(mvcontent_xpath)
if mvcontent_text == None or len(mvcontent_text) == 0:
print("解析出錯")
return -1
mvcontent_text[0] = mvcontent_text[0].replace("\n", "") # 替換掉json字符串中的\n
try:
mvcontent_json = json.loads(mvcontent_text[0])
except json.JSONDecodeError as e:
print("解析出錯:", e)
return -1
if mvcontent_json == None or len(mvcontent_json) == 0:
print("解析出錯")
return -1
# 輸出電影詳情
print("\t電影名稱", mvcontent_json["name"])
# 合並顯示電影類型
mvcontent_genre = mvcontent_json["genre"]
mvcontent_genre_str = ""
for lst in mvcontent_genre:
mvcontent_genre_str += (lst + "/")
print("\t電影類型", mvcontent_genre_str)
print("\t上映時間", mvcontent_json["datePublished"])
print("\t豆瓣評分", mvcontent_json["aggregateRating"]["ratingValue"],
"(", mvcontent_json["aggregateRating"]["ratingCount"], ")")
print("\t電影導演", mvcontent_json["director"][0]["name"])
# 合並顯示電影主演(只顯示前5個)
mvcontent_actor = mvcontent_json["actor"]
mvcontent_actor_str = ""
mvcontent_actor_len = 0
for lst in mvcontent_actor:
mvcontent_actor_str += (lst["name"] + "/")
mvcontent_actor_len += 1
if mvcontent_actor_len > 5:
mvcontent_actor_str += "..."
break
print("\t電影主演", mvcontent_actor_str)
print("\t電影簡述", mvcontent_json["description"])
return 0從電影天堂上搜索並提取電影下載鏈接
該函數用於根據指定的電影名稱,從電影天堂服務器上搜索電影,解析搜索結果並顯示,然後根據輸入顯示指定搜索結果的下載鏈接.
def movie_tiantang(mvsearch_name):
MVSEARCH_URL = "http://s.ygdy8.com/plus/so.php"
MVSEARCH_PAR = {"kwtype": "0", "searchtype": "title",
"pagesize": "100", "keyword": ""}
MOVIE_URL = "http://www.ygdy8.com"
# mvsearch_name = "星球大戰"
# mvsearch_name = input("請輸入電影名稱(輸入\"exit\"退出):")
if mvsearch_name == None:
print("輸入有誤!")
return -1
# print("你輸入的電影名稱為:", mvsearch_name)
# 搜索電影
MVSEARCH_PAR["keyword"] = mvsearch_name
mvsearch_par = parse.urlencode(MVSEARCH_PAR, encoding="gb2312")
# print(mvsearch_par)
mvsearch_url = "{0}?{1}".format(MVSEARCH_URL, mvsearch_par)
# print(mvsearch_url)
mvsearch_html = download_page_html(mvsearch_url, 0)
if mvsearch_html == None:
print("下載出錯,可能IP被服務器封禁,可稍後再試!")
return -1
# print(mvsearch_html)
# 獲取搜索結果列表
etree_html = lxml.html.fromstring(mvsearch_html)
mvsearch_xpath = '//div[@class="co_content8"]/ul/tr/td/table[@width="100%"]'
mvsearch_list = etree_html.xpath(mvsearch_xpath)
# print(mvsearch_list)
if len(mvsearch_list) == 0:
print("未搜索到任何內容")
return -1
# print("共找到", len(mvsearch_list), "個關於", mvsearch_name, "的結果:")
mvcontent_url = []
mvcontent_title = []
# 提取搜索結果中的電影鏈接
mvsearch_list_len = len(mvsearch_list)
for idx in range(1, mvsearch_list_len+1):
# 提取鏈接
mv_title_url = etree_html.xpath(
mvsearch_xpath + '[{0}]//a[@href]/@href'.format(idx))
# print(mv_title_url)
if mv_title_url == None:
print("解析出錯!")
return -1
# 過濾掉遊戲
if mv_title_url[0].find("/html/game/") < 0:
mv_title_url = "{0}{1}".format(MOVIE_URL, mv_title_url[0])
mvcontent_url.insert(idx-1, mv_title_url)
# 提取標題
mv_title_str_lst = etree_html.xpath(
mvsearch_xpath + '[{0}]//a[@href]//text()'.format(idx))
if mv_title_str_lst == None:
print("解析出錯!")
return -1
mv_title_str = "".join(mv_title_str_lst)
mvcontent_title.insert(idx-1, mv_title_str)
# print("\t{0}, {1}, {2}".format(idx, mv_title_str, mv_title_url))
mvcontent_len = len(mvcontent_url)
if mvcontent_len == 0:
print("未搜索到有效結果!")
return -1
# print("其中", mvcontent_len, "個有效結果:")
print("共找到", mvcontent_len, "個關於", mvsearch_name, "的下載:")
for idx in range(mvcontent_len):
print("\t", idx+1, ", ",
mvcontent_title[idx], ", ", mvcontent_url[idx])
# 打開電影詳情頁面
mvcontent_sel = input("請選擇需要下載的項:")
if mvcontent_sel.isdigit() != True:
print("輸入有誤!")
return -1
mvcontent_sel = int(mvcontent_sel)
if mvcontent_sel > mvcontent_len or mvcontent_sel < 1:
print("輸入有誤!")
return -1
mvcontent_sel = mvcontent_sel - 1
# 下載電影詳情頁面
# print("即將下載: ", mvcontent_title[mvcontent_sel],
# ", " + mvcontent_url[mvcontent_sel])
mvcontent_html = download_page_html(mvcontent_url[mvcontent_sel], 1)
# print(mvcontent_html)
if mvcontent_html == None:
print("下載出錯,可能IP被服務器封禁,可稍後再試!")
return -1
# 提取電影下載鏈接
mvcontent_etree_html = lxml.html.fromstring(mvcontent_html)
# '//div[@id="Zoom"]/table/tr/td/table'
mvcontent_xpath = '//td[@bgcolor="#fdfddf"]'
mvcontent_dwurl_lst = []
mvcontent_urllst = mvcontent_etree_html.xpath(
mvcontent_xpath + "//a[@href]/text()")
if mvcontent_urllst == None:
print("解析出錯!")
return -1
for url in mvcontent_urllst:
mvcontent_dwurl_lst.append(url)
if mvcontent_dwurl_lst == None:
print("未找到下載鏈接!")
return -1
# print("共找到", len(mvcontent_dwurl_lst), "個下載鏈接:")
for dwurl in mvcontent_dwurl_lst:
print("\t", dwurl)
return 0程序運行方法
打開命令行,定位到源碼所在目錄,然後輸入 python py_movie.py ,回車運行
程序運行截圖
項目文件截圖
python一鍵電影搜索與下載
代碼地址如下：
http://www.demodashi.com/demo/14313.html
註：本文著作權歸作者，由demo大師代發，拒絕轉載，轉載需要作者授權
python一鍵電影搜索與下載