python3爬蟲入門程式

摘要： 適用於有且只有一點Python3和網頁基礎的朋友,大牛&路人請繞道 (本文很多廢話,第一次在網上長篇大論,所以激動的停不下來,如果有大佬路過,也希望不要直接繞道,煩請指點一二) 感謝部落格園給了我一個機會,我喜歡的id還沒有被搶注,真的是太可怕了 注:這是一段廢話,正文請直...

適用於有且只有一點Python3和網頁基礎的朋友,大牛&路人請繞道

(本文很多廢話,第一次在網上長篇大論,所以激動的停不下來,如果有大佬路過,也希望不要直接繞道,煩請指點一二)

感謝部落格園給了我一個機會,我喜歡的id還沒有被搶注,真的是太可怕了

注:這是一段廢話,正文請直接跳過這一段. 大二的時候因為愛好,自己學了點python(當初學主要是因為語法簡潔美觀,還沒有大括號,程式碼對齊?反正java程式碼也要對齊啊~),還好我學python的時候py3已經流行起來了,沒有學py2,不然又得好一陣折騰.

寫這個程式碼的背景:記得高二的時候逛騙子網站,出於好奇在網站上留下了同桌的手機號,結果這都過了大概大半年了吧,同桌還是接二連三的可以收到騙子的電話,但是原來的網址已經找不到了,於是就在百度隨便搜尋了關鍵詞"牛股"挨個檢視的,只找到了一個可以輸入手機號的,剩下的都是讓加微信的,本來想整整我們老師呢,想想還是免了,乾脆往他們資料庫填點東西玩吧,其實最終是否成功我也不能確定

• 操作環境
• win10 1803 64位
• Chrome 68.0.3440.106（正式版本） （64 位）
• pycharm-UI(pycharm專業版) 2018.2
• python-365
• 庫(非自帶庫用pip直接安裝就行):
  • pymysql :import pymysql
  • requests :import requests
  • json(自帶) :import json

  • Faker: :from faker import Faker

    首先選取目標

    目標網站是這個,url為: ofollow,noindex" target="_blank">http://gpyd.gp241.com/nyqpc/bd2.html?id=20110052 ,

    

    1.首先肯定是抓取一下post/get地址

    

    進入首頁後點擊"點選領取9月牛股"彈出對話方塊後,按F12彈出開發者工具

    

    在開發者工具中選中 "Network" ,隨後點選網頁中的點選領取,會看到 network 中多出來一條檔案資訊

    然後提取一下我們需要的資料放到pycharm中,並整理成這種json格式:

2.這樣我們就得到了這些資料:

url = r"https: // download.zslxt.com / tinterface.php"
headers = {
"Host": "download.zslxt.com",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36",
"Accept": "*/*",
"Accept-Language": "zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2",
"Accept-Encoding": "gzip, deflate, br",
"Referer": "http:/gpyd.gp241.com/nyqpc/bd2.html?id=20110052",
"Content-Type": "application/x-www-form-urlencoded; charset=UTF-8",
"Content-Length": "107",
"Origin": "http://gpyd.gp241.com",
"Connection": "keep-alive"
}
data = {
"bm": "gbk",
"gpdm": "",
"id": "20110052",
"phone": "15666668888",
"qudao": 98,
"remarks": "牛有圈百度2)"
}

這裡看data的引數也應該明白了,這裡就是我們剛才輸入的手機號了,別的程式碼可以不動,我剛才切換瀏覽器發現並沒有影響,不知道是怎麼來的,可能是跟百度推廣有關吧

然後,這樣之後就可以向網站傳送一條資料了

首先我們要使用requests庫,這裡就不介紹了,是一個可以用來請求get/post...還可以使用session保持登陸,用途很廣

這裡有一點需要注意,就是data資料不可以直接傳送,需要用json.dumps()方法轉為字串

requests可以返回請求資料,這段程式碼並沒有體現出來,但是請不要被誤導

import requests
import json

url = r"https: // download.zslxt.com / tinterface.php"
headers = {
"Host": "download.zslxt.com",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36",
"Accept": "*/*",
"Accept-Language": "zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2",
"Accept-Encoding": "gzip, deflate, br",
"Referer": "http:/gpyd.gp241.com/nyqpc/bd2.html?id=20110052",
"Content-Type": "application/x-www-form-urlencoded; charset=UTF-8",
"Content-Length": "107",
"Origin": "http://gpyd.gp241.com",
"Connection": "keep-alive"
}
data = {
"bm": "gbk",
"gpdm": "",
"id": "20110052",
"phone": "15666668888",
"qudao": 98,
"remarks": "牛有圈百度2)"
}

requests.post(url=url, headers=headers, data=json.dumps(data))

可是,總不能只發送一次吧

這裡先介紹一下python中最假的庫--Faker

import requests
import json
from faker import Faker

f = Faker(locale="zh-CN")


user_agent = f.user_agent()
phone = f.phone_number()
url = r"https: // download.zslxt.com / tinterface.php"
headers = {
"Host": "download.zslxt.com",
"User-Agent": user_agent,
"Accept": "*/*",
"Accept-Language": "zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2",
"Accept-Encoding": "gzip, deflate, br",
"Referer": "http:/gpyd.gp241.com/nyqpc/bd2.html?id=20110052",
"Content-Type": "application/x-www-form-urlencoded; charset=UTF-8",
"Content-Length": "107",
"Origin": "http://gpyd.gp241.com",
"Connection": "keep-alive"
}
data = {
"bm": "gbk",
"gpdm": "",
"id": "20110052",
"phone": phone,
"qudao": 98,
"remarks": "牛有圈百度2)"
}

req = requests.post(url=url, headers=headers, data=json.dumps(data))

然後就快要完成了,為了方便迴圈傳送資料,我們再把它整理成一段函式:

import requests
import json
from faker import Faker

f = Faker(locale="zh-CN")


def duang():
user_agent = f.user_agent()
phone = f.phone_number()
url = r"https: // download.zslxt.com / tinterface.php"
headers = {
"Host": "download.zslxt.com",
"User-Agent": user_agent,
"Accept": "*/*",
"Accept-Language": "zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2",
"Accept-Encoding": "gzip, deflate, br",
"Referer": "http:/gpyd.gp241.com/nyqpc/bd2.html?id=20110052",
"Content-Type": "application/x-www-form-urlencoded; charset=UTF-8",
"Content-Length": "107",
"Origin": "http://gpyd.gp241.com",
"Connection": "keep-alive"
}
data = {
"bm": "gbk",
"gpdm": "",
"id": "20110052",
"phone": phone,
"qudao": 98,
"remarks": "牛有圈百度2)"
}

req = requests.post(url=url, headers=headers, data=json.dumps(data))
return user_agent, phone, req

這樣我們就可以方便的進行呼叫了,寫個main函式來呼叫它

import requests
import json
from faker import Faker

f = Faker(locale="zh-CN")


def duang():
user_agent = f.user_agent()
phone = f.phone_number()
url = r"https: // download.zslxt.com / tinterface.php"
headers = {
"Host": "download.zslxt.com",
"User-Agent": user_agent,
"Accept": "*/*",
"Accept-Language": "zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2",
"Accept-Encoding": "gzip, deflate, br",
"Referer": "http:/gpyd.gp241.com/nyqpc/bd2.html?id=20110052",
"Content-Type": "application/x-www-form-urlencoded; charset=UTF-8",
"Content-Length": "107",
"Origin": "http://gpyd.gp241.com",
"Connection": "keep-alive"
}
data = {
"bm": "gbk",
"gpdm": "",
"id": "20110052",
"phone": phone,
"qudao": 98,
"remarks": "牛有圈百度2)"
}

req = requests.post(url=url, headers=headers, data=json.dumps(data))
return user_agent, phone, req


if __name__ == '__main__':
for i in range(100000):
user_agent, phone, req = duang()
print(i, '\t', phone, '\t', req.status_code, '\n', user_agent)

這裡就是輸出一下資訊啦,剛才出去吃飯的時候斷網了,只跑了3000多,這裡就不截圖了(如果真有用來練手的朋友可以嘗試自己完善一下程式碼,斷網後也可以等待並繼續執行)

附上全部程式碼(寫到mysql了):

import pymysql
import requests
import json
from faker import Faker

f = Faker(locale="zh-CN")


def duang():
user_agent = f.user_agent()
phone = f.phone_number()
url = r"https: // download.zslxt.com / tinterface.php"
headers = {
"Host": "download.zslxt.com",
"User-Agent": user_agent,
"Accept": "*/*",
"Accept-Language": "zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2",
"Accept-Encoding": "gzip, deflate, br",
"Referer": "http:/gpyd.gp241.com/nyqpc/bd2.html?id=20110052",
"Content-Type": "application/x-www-form-urlencoded; charset=UTF-8",
"Content-Length": "107",
"Origin": "http://gpyd.gp241.com",
"Connection": "keep-alive"
}
data = {
"bm": "gbk",
"gpdm": "",
"id": "20110052",
"phone": phone,
"qudao": 98,
"remarks": "牛有圈百度2)"
}

req = requests.post(url=url, headers=headers, data=json.dumps(data))
return user_agent, phone, req.status_code


if __name__ == '__main__':
for i in range(100000):
user_agent, phone, status_code = duang()
db = pymysql.connect("localhost", "root", "xiaoyan", "python")
cur = db.cursor()
cur.execute(f"INSERT INTO python1duang VALUES(default,'{user_agent}','{phone}','{status_code}')")
db.commit()
print(i, '\t', phone, '\t', status_code, '\n', user_agent)
db.close()