urllib提供了較好的封裝，可以很方便的讀取http，ftp，file等協議資料，本篇只關注http。urllib的底層還是使用httplib模組，相比於httplib，urllib介面更加好用，功能更加強大。支援http代理，可從環境變數中獲取代理資訊，支援http basic auth，可自動處理302等。但也有不足，如不支援gzip等壓縮編碼，不支援摘要認證，NTML認證等。

urllib快速使用

urllib的urlopen方法很好用，代理使用如下：

import urllib
opener=urllib.urlopen('http://www.baidu.com/',proxies={'http':'http://root:
 
[email protected]:808'})
print opener.code

import urllib
def reporthook(bk,bs,total):
    print bk*bs,'b'
filename,message=urllib.urlretrieve("http://ww.baidu.com/",None,reporthook)
print message.getheader('Content-Length')
 

fp = self.open(url, data)
headers = fp.info()
if "content-length" in headers:
    size = int(headers["Content-Length"])

class Message:
    def __contains__(self, name):
        """Determine whether a message contains the named header."""
        return name.lower() in self.dict
        
    def __getitem__(self, name):
        """Get a specific header, as from a dictionary."""
        return self.dict[name.lower()]

urllib原始碼分析

1.總體流程

2.http basic auth（基本認證）

GET / HTTP/1.1
Host: 127.0.0.1

HTTP/1.1 401 Authorization Required
WWW-Authenticate: Basic realm="my site"

----------------------------------------------------------

GET / HTTP/1.1
Host: 127.0.0.1
Authorization: Basic cm9vdDpyb290

HTTP/1.1 200 OK

<Directory "c:/wamp/www/">
    AllowOverride authconfig
    Order Deny,Allow
    Deny from all
    Allow from 127.0.0.1
</Directory>

authname "my site"
authtype basic
authuserfile c:/wamp/www/user.txt
require valid-user

user_passwd, realhost = splituser(realhost) #realhost類似root:[email protected]
if user_passwd:
    import base64
    auth = base64.b64encode(user_passwd).strip()
    else:
        auth = None
    h = httplib.HTTP(host)
    if auth: h.putheader('Authorization', 'Basic %s' % auth)

3.代理

GET / HTTP/1.1
Host: www.baidu.com


HTTP/1.0 407 Unauthorized
Server: Proxy
Proxy-Authenticate: Basic realm="CCProxy Authorization"


GET / HTTP/1.1
Host: www.baidu.com
Proxy-Authorization: Basic cm9vdDpyb290Mg==

HTTP/1.1 200 OK

def __init__(self, proxies=None, **x509):
        if proxies is None:
            proxies = getproxies()
        assert hasattr(proxies, 'has_key'), "proxies must be a mapping"
        self.proxies = proxies

def getproxies_environment():
    proxies = {}
    for name, value in os.environ.items():
        name = name.lower()
        if value and name[-6:] == '_proxy':
            proxies[name[:-6]] = value
    return proxies

urllib提供的有用函式

1. quote(s, safe = '/')：   url編碼，將保留字元編碼為%20格式，主要用來編碼path前面資訊

fullurl = quote(fullurl, safe="%/:=&?~#+!$,;'@()*[]|")

增補：從python2.6版本才會主動編碼，2.5以及低版本不會

2. unquote(s):   quote的逆過程。

3. quote_plus(s, safe = '')：  query字串編碼，與quote不同點就是將“空格”用“+"表示，主要用於urlencode函式中。

4. unquote_plus(s)：   quote_plus的逆過程。

5.urlencode(query,doseq=0) :   將2元素的序列或字典編碼成查詢字串，doseq允許第二元素也可以為序列，query字串都是用quote_plus編碼。

注意：這裡有必要對quote,quote_plus和urlencode做個說明

quote用於url的編碼，而quote_plus用於post的data編碼，所以而urlencode內部是用quote_plus實現的，所以urlencode一般只用做post data使用，get時不用。

官方文件說明：

urllib.unquote(string)
Replace %xx escapes by their single-character equivalent.

Example: unquote('/%7Econnolly/') yields '/~connolly/'.

urllib.unquote_plus(string)
Like unquote(), but also replaces plus signs by spaces, as required for unquoting HTML form values.

urllib.urlencode(query[, doseq])
Convert a mapping object or a sequence of two-element tuples to a “percent-encoded” string, suitable to pass to urlopen() above as the optional data argument. This is useful to pass a dictionary of form fields to a POST request. The resulting string is a series of key=value pairs separated by '&' characters, where both key and value are quoted using quote_plus() above. When a sequence of two-element tuples is used as the query argument, the first element of each tuple is a key and the second is a value. The value element in itself can be a sequence and in that case, if the optional parameter doseq is evaluates to True, individual key=value pairs separated by '&' are generated for each element of the value sequence for the key. The order of parameters in the encoded string will match the order of parameter tuples in the sequence. The urlparse module provides the functions parse_qs() and parse_qsl() which are used to parse query strings into Python data structures.

6 .url2pathname：   將file協議url轉換為本地路徑

     ：

        

_typeprog = None
def splittype(url):
    """splittype('type:opaquestring') --> 'type', 'opaquestring'."""
    global _typeprog
    if _typeprog is None:
        import re
        _typeprog = re.compile('^([^/:]+):')

    match = _typeprog.match(url)
    if match:
        scheme = match.group(1)
        return scheme.lower(), url[len(scheme) + 1:]
    return None, url