HttpClient 獲取 Cookie 的一次踩坑實錄

在使用HttpClient進行抓取一些網頁的時候，經常會保留從伺服器端發回的Cookie資訊，以便發起其他需要這些Cookie的請求。大多數情況下，我們使用內建的cookie策略，便能夠方便直接地獲取這些cookie。

@ Test
public void getCookie(){
    CloseableHttpClient httpClient = HttpClients.createDefault();
    HttpGet get=new HttpGet("http://www.baidu.com");
    HttpClientContext context = HttpClientContext.create();
    try {
        CloseableHttpResponse response = httpClient.execute(get, context);
        try{
            System.out.println(">>>>>>headers:");
            Arrays.stream(response.getAllHeaders()).forEach(System.out::println);
            System.out.println(">>>>>>cookies:");
            context.getCookieStore().getCookies().forEach(System.out::println);
        }
        finally {
            response.close();
        }
    } catch (IOException e) {
        e.printStackTrace();
    }finally {
        try {
            httpClient.close();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

1234567891011121314151617181920212223242526

@TestpublicvoidgetCookie(){CloseableHttpClient httpClient=HttpClients.createDefault();HttpGet get=newHttpGet("http://www.baidu.com");HttpClientContext context=HttpClientContext.create();try{CloseableHttpResponse response

=httpClient.execute(get,context);try{System.out.println(">>>>>>headers:");Arrays.stream(response.getAllHeaders()).forEach(System.out::println);System.out.println(">>>>>>cookies:");context.getCookieStore().getCookies().forEach(System.out::println);}finally{response

.close();}}catch(IOExceptione){e.printStackTrace();}finally{try{httpClient.close();}catch(IOExceptione){e.printStackTrace();}}}

列印結果

>>>>>>headers:
Server: bfe/1.0.8.18
Date: Tue, 12 Sep 2017 06:19:06 GMT
Content-Type: text/html
Last-Modified: Mon, 23 Jan 2017 13:28:24 GMT
Transfer-Encoding: chunked
Connection: Keep-Alive
Cache-Control: private, no-cache, no-store, proxy-revalidate, no-transform
Pragma: no-cache
Set-Cookie: BDORZ=27315; max-age=86400; domain=.baidu.com; path=/
>>>>>>cookies:
[version: 0][name: BDORZ][value: 27315][domain: baidu.com][path: /][expiry: null]

123456789101112

>>>>>>headers:Server:bfe/1.0.8.18Date:Tue,12Sep201706:19:06GMTContent-Type:text/htmlLast-Modified:Mon,23Jan201713:28:24GMTTransfer-Encoding:chunkedConnection:Keep-AliveCache-Control:private,no-cache,no-store,proxy-revalidate,no-transformPragma:no-cacheSet-Cookie:BDORZ=27315;max-age=86400;domain=.baidu.com;path=/>>>>>>cookies:[version:0][name:BDORZ][value:27315][domain:baidu.com][path:/][expiry:null]

但是也有一些網站返回的cookie並不一定完全符合規範，例如下面這個例子，從打印出的header中可以看到，這個cookie中的Expires屬性是時間戳形式，並不符合標準的時間格式，因此，httpclient對於cookie的處理失效，最終無法獲取到cookie，並且發出了一條警告資訊：“Invalid ‘expires’ attribute: 1505204523”

警告: Invalid cookie header: "Set-Cookie: yd_cookie=90236a64-8650-494b332a285dbd886e5981965fc4a93f023d; Expires=1505204523; Path=/; HttpOnly". Invalid 'expires' attribute: 1505204523
>>>>>>headers:
Date: Tue, 12 Sep 2017 06:22:03 GMT
Content-Type: text/html
Connection: keep-alive
Set-Cookie: yd_cookie=90236a64-8650-494b332a285dbd886e5981965fc4a93f023d; Expires=1505204523; Path=/; HttpOnly
Cache-Control: no-cache, no-store
Server: WAF/2.4-12.1
>>>>>>cookies:

123456789

警告:Invalid cookie header:"Set-Cookie: yd_cookie=90236a64-8650-494b332a285dbd886e5981965fc4a93f023d; Expires=1505204523; Path=/; HttpOnly".Invalid'expires'attribute:1505204523>>>>>>headers:Date:Tue,12Sep201706:22:03GMTContent-Type:text/htmlConnection:keep-aliveSet-Cookie:yd_cookie=90236a64-8650-494b332a285dbd886e5981965fc4a93f023d;Expires=1505204523;Path=/;HttpOnlyCache-Control:no-cache,no-storeServer:WAF/2.4-12.1>>>>>>cookies:

雖然我們可以利用header的資料，重新構造一個cookie出來，也有很多人確實也是這麼做的，但這種方法不夠優雅，那麼如何解決這個問題？網上相關的資料又很少，所以就只能先從官方文件入手。在官方文件3.4小節custom cookie policy中講到允許自定義的cookie策略，自定義的方法是實現CookieSpec介面，並通過CookieSpecProvider來完成在httpclient中的初始化和註冊策略例項的工作。好了，關鍵的線索在於CookieSpec介面，我們來看一下它的原始碼：

public interface CookieSpec {
……
    /**
      * Parse the {@ code "Set-Cookie"} Header into an array of Cookies.
      *
      * <p> This method will not perform the validation of the resultant<code>
      * {@link Cookie}s</p>
      *
      * @ see #validate
      *
      * @param header the {@ code Set-Cookie} received from the server
      * @param origin details of the cookie origin
      * @ return an array of {@ code Cookie}s parsed from the header
      * @throws MalformedCookieException if an exception occurs during parsing
      */
    List parse(Header header, CookieOrigin origin) throws MalformedCookieException;
……
}

123456789101112131415161718

publicinterfaceCookieSpec{……/** * Parse the {@ code "Set-Cookie"} Header into an array of Cookies. * * <p> This method will not perform the validation of the resultant<code> * {@link Cookie}s</p> * * @ see #validate * * @param header the {@ code Set-Cookie} received from the server * @param origin details of the cookie origin * @ return an array of {@ code Cookie}s parsed from the header * @throws MalformedCookieException if an exception occurs during parsing */List parse(Header header,CookieOrigin origin)throws MalformedCookieException;……}

在原始碼中我們發現了一個parse方法，看註釋就知道正是這個方法，將Set-Cookie的header資訊解析為Cookie物件，自然地再瞭解一下在httplcient中的預設實現DefaultCookieSpec，限於篇幅，原始碼就不貼了。在預設的實現中，DefaultCookieSpec主要的工作是判斷header中Cookie規範的型別，然後再呼叫具體的某一個實現。像上述這種Cookie，最終是交由NetscapeDraftSpec的例項來做解析，而在NetscapeDraftSpec的原始碼中，定義了預設的expires時間格式為“EEE, dd-MMM-yy HH:mm:ss z”

public class NetscapeDraftSpec extends CookieSpecBase {

    protected static final String EXPIRES_PATTERN = "EEE, dd-MMM-yy HH:mm:ss z";

    /** Default constructor */
    public NetscapeDraftSpec(final String[] datepatterns) {
        super(new BasicPathHandler(),
                new NetscapeDomainHandler(),
                new BasicSecureHandler(),
                new BasicCommentHandler(),
                new BasicExpiresHandler(
                        datepatterns != null ? datepatterns.clone() : new String[]{EXPIRES_PATTERN}));
    }

    NetscapeDraftSpec(final CommonCookieAttributeHandler... handlers) {
        super(handlers);
    }

    public NetscapeDraftSpec() {
        this((String[]) null);
    }
……
}

1234567891011121314151617181920212223

publicclassNetscapeDraftSpecextendsCookieSpecBase{protectedstaticfinalStringEXPIRES_PATTERN="EEE, dd-MMM-yy HH:mm:ss z";/** Default constructor */publicNetscapeDraftSpec(finalString[]datepatterns){super(newBasicPathHandler(),newNetscapeDomainHandler(),newBasicSecureHandler(),newBasicCommentHandler(),newBasicExpiresHandler(datepatterns!=null?datepatterns.clone():newString[]{EXPIRES_PATTERN}));}NetscapeDraftSpec(finalCommonCookieAttributeHandler...handlers){super(handlers);}publicNetscapeDraftSpec(){this((String[])null);}……}

到這裡已經比較清楚了，我們只需要將Cookie中expires的時間轉換為正確的格式，然後再送入預設的解析器就可以了。

解決方法：

自定義一個CookieSpec類，繼承DefaultCookieSpec
重寫parser方法
將Cookie中的expires轉換為正確的時間格式
呼叫預設的解析方法

實現如下（URL就不公開了，已經隱去）

public class TestHttpClient {
    
    String url = sth;

    class MyCookieSpec extends DefaultCookieSpec {
        @ Override
        public List parse(Header header, CookieOrigin cookieOrigin) throws MalformedCookieException {
            String value = header.getValue();
            String prefix = "Expires=";
            if (value.contains(prefix)) {
                String expires = value.substring(value.indexOf(prefix) + prefix.length());
                expires = expires.substring(0, expires.indexOf(";"));
                String date = DateUtils.formatDate(new Date(Long.parseLong(expires) * 1000L),"EEE, dd-MMM-yy HH:mm:ss z");
                value = value.replaceAll(prefix + "\\d{10};", prefix + date + ";");
            }
            header = new BasicHeader(header.getName(), value);
            return super.parse(header, cookieOrigin);
        }
    }

    @ Test
    public void getCookie() {

        CloseableHttpClient httpClient = HttpClients.createDefault();

        Registry cookieSpecProviderRegistry = RegistryBuilder.create()
                .register("myCookieSpec", context -> new MyCookieSpec()).build();//註冊自定義CookieSpec

        HttpClientContext context = HttpClientContext.create();
        context.setCookieSpecRegistry(cookieSpecProviderRegistry);

        HttpGet get = new HttpGet(url);
        get.setConfig(RequestConfig.custom().setCookieSpec("myCookieSpec").build());

        try {
            CloseableHttpResponse response = httpClient.execute(get, context);
            try{
                System.out.println(">>>>>>headers:");
                Arrays.stream(response.getAllHeaders()).forEach(System.out::println);
                System.out.println(">>>>>>cookies:");
                context.getCookieStore().getCookies().forEach(System.out::println);
            }
            finally {
                response.close();
            }
        } catch (IOException e) {
            e.printStackTrace();
        }finally {
            try {
                httpClient.close();
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
    }
}

1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556

publicclassTestHttpClient{Stringurl=sth;classMyCookieSpecextendsDefaultCookieSpec{@OverridepublicList parse(Header header,CookieOrigin cookieOrigin)throwsMalformedCookieException{Stringvalue=header.getValue();Stringprefix="Expires=";if(value.contains(prefix)){Stringexpires=value.substring(value.indexOf(prefix)+prefix.length());expires=expires.substring(0,expires.indexOf(";"));Stringdate=DateUtils.formatDate(newDate(Long.parseLong(expires)*1000L),"EEE, dd-MMM-yy HH:mm:ss z");value=value.replaceAll(prefix+"\\d{10};",prefix+date+";");}header=newBasicHeader(header.getName(),value);returnsuper.parse(header,cookieOrigin);}}@TestpublicvoidgetCookie(){CloseableHttpClient httpClient=HttpClients.createDefault();Registry cookieSpecProviderRegistry=RegistryBuilder.create().register("myCookieSpec",context->newMyCookieSpec()).build();//註冊自定義CookieSpecHttpClientContext context=HttpClientContext.create();context.setCookieSpecRegistry(cookieSpecProviderRegistry);HttpGet get=newHttpGet(url);get.setConfig(RequestConfig.custom().setCookieSpec("myCookieSpec").build());try{CloseableHttpResponse response=httpClient.execute(get,context);try{System.out.println(">>>>>>headers:");Arrays.stream(response.getAllHeaders()).forEach(System.out::println);System.out.println(">>>>>>cookies:");context.getCookieStore().getCookies().forEach(System.out::println);

HttpClient 獲取 Cookie 的一次踩坑實錄

HttpClient 獲取 Cookie 的一次踩坑實錄

記一次踩坑：使用Navicat連線Mysql8.0.11

mac sed 使用踩坑實錄

Spring Boot 開發系列一開發踩坑

JAVA實用案例之文件導出（JasperReport踩坑實錄）

ffmpeg 踩坑實錄（二）

android 人臉識別踩坑實錄

mysql獲取最後一次插入的id

Angular2，Springboot，Zuul，Shiro跨域CORS請求踩坑實錄

微信小程式踩坑實錄

記一次死坑，mongodb無法啟動報錯 reason: errno:111 Connection refused

Win10環境下安裝pytorch踩坑實錄

記一次填坑，在雲伺服器上安裝jupyter並遠端使用

MySql驅動8.0.12版本踩坑實錄

Docker踩坑實錄

如何在30秒內建構Spark環境--使用docker-compose 踩坑實錄

CentOS下Nvidia docker 2.0之安裝教程&踩坑實錄

CentOS 7下最新版Docker CE之無腦安裝教程&踩坑實錄

weex+vue2.x 踩坑實錄（不定期更新）

python xgboost踩坑實錄

HttpClient 獲取 Cookie 的一次踩坑實錄

相關推薦