scrapy發送POST請求
阿新 • • 發佈:2018-06-30
self. parse requests 重寫 def AS color star xxxx
發送post請求
可以使用 yield scrapy.FormRequest(url, formdata, callback)
方法發送POST請求。
如果希望程序執行一開始就發送POST請求,可以重寫Spider類的start_requests(self)
方法,並且不再調用start_urls裏的url。
1 class mySpider(scrapy.Spider): 2 # start_urls = ["http://www.example.com/"] 3 4 def start_requests(self): 5 url = ‘http://www.renren.com/PLogin.do‘ 6 7 # FormRequest 是Scrapy發送POST請求的方法 8 yield scrapy.FormRequest( 9 url = url, 10 formdata = {"email" : "xxx", "password" : "xxx"}, 11 callback = self.parse_page 12 ) 13 def parse_page(self, response): 14 # do something
註意:表單數據的提交使用formdata,方法之間的傳參依然使用meta,例如:
class mySpider(scrapy.Spider): # start_urls = ["http://www.example.com/"] def start_requests(self): url = ‘http://www.renren.com/PLogin.do‘ # FormRequest 是Scrapy發送POST請求的方法 yield scrapy.FormRequest( url = url, formdata = {"email" : "xxx", "password" : "xxxxx"}, callback = self.parse_page, meta = {"key1":"value1","key2":"value2"} ) def parse_page(self, response): value1 = response.meta["key1"] # do something
scrapy發送POST請求