1. 程式人生 > >C# webrequest 抓取數據時,多個域Cookie的問題

C# webrequest 抓取數據時,多個域Cookie的問題

save copyto ews IT -c 如何 date ken 開發者工具

最近研究了下如何抓取為知筆記的內容,在抓取筆記裏的圖片內容時,老是提示403錯誤,用Chorme的開發者工具看了下:

技術分享圖片

這裏的Cookie來自兩個域,估計為知那邊是驗證了token(登錄後才能獲取到token)

下載圖片的代碼:

[csharp] view plain copy
  1. var path = "https://note.wiz.cn/" + str.TrimStart(‘/‘);
  2. var extension = Path.GetExtension(path);
  3. var filepath = AppPath.Combine("Images/" + DateTime.Now.Ticks + extension);
  4. const string userAgent ="Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.76 Safari/537.36";
  5. const string accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
  6. const string acceptLanguage = "zh-CN,zh;q=0.8";
  7. const string acceptEncoding = "gzip,deflate,sdch";
  8. var cookieContainer = new CookieContainer();
  9. var cookie = new Cookie
  10. {
  11. Name = "token".Trim(),
  12. Value = Token,
  13. Domain = ".wiz.cn".Trim() //設置cookie域
  14. };
  15. cookieContainer.Add(cookie);
  16. string[] cookiesArr = txtCookie.Text.Split(‘;‘);
  17. foreach (string s in cookiesArr)
  18. {
  19. string[] keyValuePair = s.Split(‘=‘);
  20. if (keyValuePair.Length > 1)
  21. {
  22. cookie = new Cookie
  23. {
  24. Name = keyValuePair[0].Trim(),
  25. Value = keyValuePair[1].Trim(),
  26. Domain = "note.wiz.cn" //設置cookie域
  27. };
  28. cookieContainer.Add(cookie);
  29. }
  30. }
  31. var newUri = new Uri(path);
  32. var webRequest = (HttpWebRequest)WebRequest.Create(newUri);
  33. webRequest.Timeout = 20000;
  34. //webRequest.CookieContainer = cookieContainer;
  35. webRequest.UserAgent = userAgent;
  36. webRequest.Accept = accept;
  37. webRequest.Headers["Accept-Language"] = acceptLanguage;
  38. webRequest.Headers["Accept-Charset"] = acceptEncoding;
  39. webRequest.Headers["Accept-Encoding"] = acceptEncoding;
  40. webRequest.KeepAlive = true;
  41. webRequest.Headers["Cache-Control"] = "no-cache";
  42. webRequest.Headers["Upgrade-Insecure-Requests"] = "1";
  43. webRequest.Headers["Pragma"] = "no-cache";
  44. webRequest.Headers["Cookie"] = "token=" + Token + ";" + txtCookie.Text.Trim();//todo: Cookie 要這樣賦值,不能用CookieContainer??
  45. webRequest.Referer = newUri.AbsoluteUri;
  46. HttpWebResponse rsp = (HttpWebResponse)webRequest.GetResponse();
  47. Stream stream = null;
  48. stream = rsp.GetResponseStream();
  49. Image.FromStream(stream).Save(filepath);
  50. // 釋放資源
  51. if (stream != null) stream.Close();
  52. if (rsp != null) rsp.Close();

奇怪的是:用 webRequest.CookieContainer = cookieContainer; 來跟cookie賦值,token參數總是賦不上,

後面改為:webRequest.Headers["Cookie"] = "token=" + Token + ";" + txtCookie.Text.Trim(); 就可以了,

CookieContainer 不是支持多個域的cookie嗎,難到跨域Cookie只能webRequest.Headers["Cookie"]這樣賦值嗎? 沒弄明白,有知道的童鞋不吝賜教。

C# webrequest 抓取數據時,多個域Cookie的問題