不使用brpop的時候其實也可以實現redis的訊息佇列,只是不是阻塞的,目前已知的問題長時間沒有任務的話,consumer會出現假死的狀態,使用redis3.0版本,聽說使用3.2以上的版本不會出現這種假死的問題,目前沒有測試:

 def parse_url(self):
        while True:
            url=self.redis.rpop(self.url_detail_list)    
            if not url:
                continue
            url='https://yaohuo.me/'+url
            headers = {
                'accept': 	"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
                'accept-encoding': "gzip, deflate, br",
                'accept-language': "zh-CN,zh;q=0.9",
                'cache-control': "no-cache",
                'pragma': "no-cache",
                'upgrade-insecure-requests': "1",
                'user-agent': "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.92 Safari/537.36",
            }
            try:
                response = self.s.get(url, headers=headers)
            except Exception:
                print traceback.format_exc()
                continue
 

使用 redis.rpop不會阻塞,這樣一直在死迴圈的找任務,佔用cpu,造成不必要的浪費,使用 redis.brpop在沒有任務的時候阻塞,設定timeout=0,即是有list中有任務來到的時候就會自動將任務pop出去,由consumer消費掉.修改後的程式碼:

 def parse_url(self):
        while True:
            url=self.redis.brpop(self.url_detail_list)[1]
            if not url:
                continue
            url='https://yaohuo.me/'+url
            headers = {
                'accept': 	"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
                'accept-encoding': "gzip, deflate, br",
                'accept-language': "zh-CN,zh;q=0.9",
                'cache-control': "no-cache",
                'pragma': "no-cache",
                'upgrade-insecure-requests': "1",
                'user-agent': "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.92 Safari/537.36",
            }
            try:
                response = self.s.get(url, headers=headers)
            except Exception:
                print traceback.format_exc()
                continue```