1. 程式人生 > >Node.js Zlib模組

Node.js Zlib模組

  • Zlib
    • Compressing HTTP requests and responses
    • Memory Usage Tuning
    • Flushing
    • Constants
    • Class Options
    • Class: zlib.Deflate
    • Class: zlib.DeflateRaw
    • Class: zlib.Gunzip
    • Class: zlib.Gzip
    • Class: zlib.Inflate
    • Class: zlib.InflateRaw
    • Class: zlib.Unzip
    • Class: zlib.Zlib
      • zlib.flush([kind], callback)
      • zlib.params(level, strategy, callback)
      • zlib.reset()
    • zlib.constants
    • zlib.createDeflate([options])
    • zlib.createDeflateRaw([options])
    • zlib.createGunzip([options])
    • zlib.createGzip([options])
    • zlib.createInflate([options])
    • zlib.createInflateRaw([options])
    • zlib.createUnzip([options])
    • Convenience Methods
      • zlib.deflate(buf[, options], callback)
      • zlib.deflateSync(buf[, options])
      • zlib.deflateRaw(buf[, options], callback)
      • zlib.deflateRawSync(buf[, options])
      • zlib.gunzip(buf[, options], callback)
      • zlib.gunzipSync(buf[, options])
      • zlib.gzip(buf[, options], callback)
      • zlib.gzipSync(buf[, options])
      • zlib.inflate(buf[, options], callback)
      • zlib.inflateSync(buf[, options])
      • zlib.inflateRaw(buf[, options], callback)
      • zlib.inflateRawSync(buf[, options])
      • zlib.unzip(buf[, options], callback)
      • zlib.unzipSync(buf[, options])

The zlib module provides compression functionality implemented using Gzip and Deflate/Inflate. It can be accessed using:

zlib模組提供了用Gzip和Deflate/Inflate實現的壓縮功能。它可以通過如下方式訪問:

const zlib = require('zlib');

Compressing or decompressing a stream (such as a file) can be accomplished by piping the source stream data through a zlib stream into a destination stream:

壓縮或解壓一個流(諸如一個檔案)可以通過管道將源資料流通過一個zlib流轉化為目標流。

const gzip = zlib.createGzip();
const fs = require('fs');
const rs = fs.createReadStream('input.txt');
const ws = fs.createWriteStream('input.txt.gz');
rs.pipe(gzip).pipe(ws);

這裡寫圖片描述

It is also possible to compress or decompress data in a single step:

它也可以在一步裡面壓縮或解壓資料:

const zlib = require('zlib');
const input = '.................................';
zlib.deflate(input, (err, buffer) => {
    if(err) {
        throw err;
    }
    console.log(buffer.toString('base64'));
    //輸出 eJzT0yMAAGTvBe8=
});

zlib.deflateSync(input).toString('base64');
//返回 'eJzT0yMAAGTvBe8='

const buffer = Buffer.from('eJzT0yMAAGTvBe8=', 'base64');
zlib.unzip(buffer, (err, buffer) => {
    if(err) {
        throw err;
    }
    console.log(buffer.toString());
    //輸出 .................................
});

zlib.unzipSync(buffer).toString();
//返回 '................................'

Compressing HTTP requests and responses

The zlib module can be used to implement support for the gzip and deflate content-encoding mechanisms defined by HTTP.

The HTTP Accept-Encoding header is used within an http request to identify the compression encodings accepted by the client. The Content-Encoding header is used to identify the compression encodings actually applied to a message.

Note: the examples given below are drastically simplified to show the basic concept. Using zlib encoding can be expensive, and the results ought to be cached. See Memory Usage Tuning for more information on the speed/memory/compression tradeoffs involved in zlib usage.

Running a gzip operation on every request is quite expensive. It would be much more efficient to cache the compressed buffer.

zlib模組可以用來實現對HTTP定義的gzip和deflate內容編碼機制的支援。

HTTP支援編碼型別(accept-Encoding)首部用在http請求中,來定義客戶端支援的壓縮編碼型別。內容編碼(content-encoding)頭部用來定義訊息實際應用的壓縮編碼型別。

注意: 下面的例子已經被大大的簡化了,用來展示最基本的概念。使用zlib編碼的代價是非常昂貴的,並且它的結果應該(ought to)被快取。詳細資訊可以檢視記憶體使用調整章節(Memory Usage Tuning),以在 速度/記憶體/壓縮 中權衡利弊,來決定是否使用zlib。

每一次請求的時候都執行一遍gzip操作的代價是非常昂貴的。因此快取壓縮的資料是非常有意義的。

const zlib = require('zlib');
const http = require('http');
const fs = require('fs');

//client request example
const request = http.get( { host: 'example.com', 
                            path: '/',
                            port: 80,
                            headers: { 'Accept-Encoding': 'gzip,deflate' } } );
request.on('response', (response) => {
    var output = fs.createWriteStream('example.html');
    var contentEncoding = response.headers['content-encoding'];
    console.log('response.headers[\'content-encoding\']: ' + contentEncoding);
    switch(contentEncoding) {
        case 'gzip':
            response.pipe(zlib.createGunzip()).pipe(output);
            break;
        case 'deflate':
            response.pipe(zlib.createInflate(zlib.createInflate())).pipe(output);
            break;
        default:
            response.pipe(output);
            break;
    }
});

//server example
http.createServer((request, response) => {
    var rawStream = fs.createReadStream('example.html');
    var acceptEncoding = request.headers['accept-encoding'];
    console.log('response.headers[\'accept-encoding\']: ' + acceptEncoding);
    if(!acceptEncoding) {
        acceptEncoding = '';
    }
    if(acceptEncoding.match(/\bgzip\b/)) {
        response.writeHead(200, {'content-encoding': 'gzip'});
        rawStream.pipe(zlib.createGzip()).pipe(response);
    } else if(acceptEncoding.match(/\bdeflate\b/)) {
        response.writeHead(200, {'content-encoding': 'deflate'});
        rawStream.pipe(zlib.createDeflate).pipe(response);
    } else {
        response.writeHead(200, {});
        rawStream.pipe(response);
    }
}).listen(8080);

這裡寫圖片描述

注意:在get請求中JSON物件headers中的鍵值對無論大小寫均是合法的,因為伺服器在將其轉化成物件時,會自動將這些鍵值對全部轉為小寫。因此在從response.headers物件中去取屬性的時候,只能用小寫的鍵名去取。

By default, the zlib methods with throw an error when decompressing truncated data. However, if it is known that the data is incomplete, or the desire is to inspect only the beginning of a compressed file, it is possible to suppress the default error handling by changing the flushing method that is used to compressed the last chunk of input data:

預設的情況是:當zlib函式壓縮的資料被截斷的時候,將會丟擲一個異常。然而,如果眾所周知該資料是不完整的,或者期望只在壓縮檔案開始部分檢查,可以通過改變沖刷函式來阻止預設的錯誤處理操作,該方法經常用在壓縮最後一塊輸入資料的時候。

const zlib = require('zlib');
// This is a truncated version of the buffer from the above examples
//完整的應該是 'eJzT0yMAAGTvBe8='
const buffer = Buffer.from('eJzT0yMA', 'base64');

zlib.unzip(buffer, { finishFlush: zlib.Z_SYNC_FLUSH }, (err, buffer) => {
  if (!err) {
    console.log(buffer.toString());
  } else {
    // handle error
    console.error('error occurs');
  }
});
//輸出 ..

zlib.unzip(buffer, (err, buffer) => {
  if (!err) {
    console.log(buffer.toString());
  } else {
    // handle error
    console.error('error occurs');
  }
});
//輸出 error occurs

This will not change the behavior in other error-throwing situations, e.g. when the input data has an invalid format. Using this method, it will not be possible to determine whether the input ended prematurely or lacks the integrity checks, making it necessary to manually check that the decompressed result is valid.

該方法不會改變其他錯誤被丟擲時的行為,舉個例子,但輸入資料是非法的格式的時候,使用該方法,它將無法判斷出輸入的資料已經結束了還是缺乏完整性校驗,因此手動校驗壓縮結果的完整性是非常必要的。

Memory Usage Tuning

記憶體使用調整

From zlib/zconf.h, modified to node.js’s usage:

The memory requirements for deflate are (in bytes):
在zlib/zconf.h中,可以修改node.js的記憶體使用量:
deflate操作的記憶體需求量為(單位:位元組):

(1 << (windowBits+2)) +  (1 << (memLevel+9))

That is: 128K for windowBits=15 + 128K for memLevel = 8 (default values) plus a few kilobytes for small objects.

For example, to reduce the default memory requirements from 256K to 128K, the options should be set to:

( 1 << ( 15+2 ) + ( 1 << ( 8 + 9 ) ) = 128KB,預設值windowBits=15,memLevel=8,另外加上一些很小的物件的極少的幾千位元組的空間。

舉個例子,將預設的記憶體使用量從256KB減少到128KB,其選項可以被這樣設定:

{ windowBits: 14, memLevel: 7 }

This will, however, generally degrade compression.

The memory requirements for inflate are (in bytes)

然而這樣通常需要壓縮降級。

inflate操作的記憶體需求量為(單位:位元組)

1 << windowBits

That is, 32K for windowBits=15 (default value) plus a few kilobytes for small objects.

This is in addition to a single internal output slab buffer of size chunkSize, which defaults to 16K.

1 << 15 = 32KB,預設值windowBits=15,另外加上一些很小的物件的極少的幾千位元組的空間。

除此之外,還有一個單獨的比較大的內部輸出緩衝區chunkSize,它的預設值是16KB。

The speed of zlib compression is affected most dramatically by the level setting. A higher level will result in better compression, but will take longer to complete. A lower level will result in less compression, but will be much faster.

In general, greater memory usage options will mean that Node.js has to make fewer calls to zlib because it will be able to process more data on each write operation. So, this is another factor that affects the speed, at the cost of memory usage.

等級的設定對zlib壓縮速度的影響是非常引人注目的。較大的壓縮率會造成較大的壓縮量,但是將會花掉很長的時間去完成該操作。較小的壓縮率會生成較小的壓縮量,但是操作會更快。

通常,設定更大的記憶體量選項意味著Node.js很少會使用zlib,因為它可以在每一次的寫操作中處理更多的資料。同樣,另外一個影響速度的因素也是記憶體的使用量。

Flushing

沖刷操作

Calling .flush() on a compression stream will make zlib return as much output as currently possible. This may come at the cost of degraded compression quality, but can be useful when data needs to be available as soon as possible.

In the following example, flush() is used to write a compressed partial HTTP response to the client:

在壓縮流中呼叫flush()方法將會使得zlib當前返回近可能多的輸出資料。這將會以降低壓縮品質為代價,但是當資料儘快要用的時候該方法是非常有效的。

從下面的例子可以看出,flush()函式可以用來寫一個壓縮的部分的HTTP響應到客戶端(開啟瀏覽器也可以正常接收資料,但中文是亂碼)

const zlib = require('zlib');
const http = require('http');
const process = require('process');
http.createServer((request, response) => {
    //For the sake of simplicity, the Accept-Encoding checks are omitted.
    //因為考慮到簡單的緣故,因此省略Accept-Encoding的校驗
    response.writeHead(200, {'content-encoding': 'gzip'});
    const gzip = zlib.createGzip();
    gzip.pipe(response);
    setInterval(()=>{
        gzip.write(`The current time is ${Date()}\n`, ()=>{
            //The data has been passed to zlib, but the compression algorithm may have decided to buffer the data for more efficient compression. Calling .flush() will make the data avaliable as soon as the client is ready to receive it.
            //資料已經被轉化為zlib,但是壓縮演算法可能會將資料快取起來使得壓縮效率更高。呼叫.flush()方法會使得資料立馬被壓縮,一旦客戶端準備好就可以隨時接收資料。
            gzip.flush();
            console.log('writen');
        });
    }, 1000);
}).listen(8080);

const request = http.get( { host:'127.0.0.1',
                            path:'/',
                            port:8080,
                            headers: {'Accept-Encoding':'gzip'} } );
request.on('response', (response)=>{
    console.log('response');
    //response事件只會被觸發一次,由於管道已經建立,因此伺服器端的輸出會源源不斷地流向該管道,而不用反覆觸發response事件
    if(response.headers['content-encoding'] === 'gzip') {
        response.pipe(zlib.createGunzip()).pipe(process.stdout);
    } else {
        console.log('response.headers[\'content-encoding\']:', response.headers['content-encoding']);
    }
});

Constants

常量

Added in: v0.5.8

All of the constants defined in zlib.h are also defined on require(‘zlib’). In the normal course of operations, it will not be necessary to use these constants. They are documented so that their presence is not surprising. This section is taken almost directly from the zlib documentation. See http://zlib.net/manual.html#Constants for more details.

所有的常量都定義在zlib.h檔案中,也定義在require(‘zlib’)物件中。在一個正常的操作中,是不需要使用這些常量的。它們被記錄在案,因此他們的出現也不足為奇。這個章節大部分都是直接從zlib文件中搬過來的。詳細資訊請參閱 http://zlib.net/manual.html#Constants

Allowed flush values.

允許沖刷的值

  • zlib.Z_NO_FLUSH
  • zlib.Z_PARTIAL_FLUSH
  • zlib.Z_SYNC_FLUSH
  • zlib.Z_FULL_FLUSH
  • zlib.Z_FINISH
  • zlib.Z_BLOCK
  • zlib.Z_TREES

Return codes for the compression/decompression functions.

壓縮和解壓縮函式的返回碼。

Negative values are errors, positive values are used for special but normal events.

負值表示錯誤,正值用來表示一些特殊的但是正常的事件。

  • zlib.Z_OK
  • zlib.Z_STREAM_END
  • zlib.Z_NEED_DICT
  • zlib.Z_ERRNO
  • zlib.Z_STREAM_ERROR
  • zlib.Z_DATA_ERROR
  • zlib.Z_MEM_ERROR
  • zlib.Z_BUF_ERROR
  • zlib.Z_VERSION_ERROR

Compression levels.

壓縮等級

  • zlib.Z_NO_COMPRESSION
  • zlib.Z_BEST_SPEED(需要的話)
  • zlib.Z_BEST_COMPRESSION
  • zlib.Z_DEFAULT_COMPRESSION

Compression strategy.

壓縮策略

  • zlib.Z_FILTERED
  • zlib.Z_HUFFMAN_ONLY(哈弗曼)
  • zlib.Z_RLE(run-length encoding遊程編碼,或行程長度編碼,或變動長度編碼法,在控制論中對於二值影象而言是一種編碼方法)
  • zlib.Z_FIXED
  • zlib.Z_DEFAULT_STRATEGY

The deflate compression method (the only one supported in this version).

deflate壓縮方法(只在該版本被支援)

  • zlib.Z_DEFLATED

For initializing zalloc, zfree, opaque.

用來透明的初始化、釋放資源。

  • zlib.Z_NULL

Class Options

Added in: v0.11.1

Each class takes an options object. All options are optional.

每一個類都有一個引數物件。所有的引數都是可選的。

Note that some options are only relevant when compressing, and are ignored by the decompression classes.

注意:一些選項只與壓縮有關,對於解壓的類可以忽略。

  • flush (default: zlib.Z_NO_FLUSH)
  • finishFlush (default: zlib.Z_FINISH)
  • chunkSize (default: 16*1024)
  • windowBits
  • level (compression only)
  • memLevel (compression only)
  • strategy (compression only)
  • dictionary (deflate/inflate only, empty dictionary by default)

See the description of deflateInit2 and inflateInit2 at http://zlib.net/manual.html#Advanced for more information on these.

Class: zlib.Deflate

Added in: v0.5.8
Compress data using deflate.

使用deflate壓縮資料。

Class: zlib.DeflateRaw

Added in: v0.5.8
Compress data using deflate, and do not append a zlib header.

使用dflate壓縮資料,並且不新增zlib頭部。

Class: zlib.Gunzip

Added in: v0.5.8
Decompress a gzip stream.

解壓一個gzip流。

Class: zlib.Gzip

Added in: v0.5.8
Compress data using gzip.

使用gzip壓縮資料。

Class: zlib.Inflate

Added in: v0.5.8
Decompress a deflate stream.

解壓deflate流。

Class: zlib.InflateRaw

Added in: v0.5.8
Decompress a raw deflate stream.

解壓一個原始(不帶zlib頭部)的deflate流。

Class: zlib.Unzip

Added in: v0.5.8
Decompress either a Gzip- or Deflate-compressed stream by auto-detecting the header.

通過自動檢測頭部,解壓Gzip或Deflate壓縮流。

Class: zlib.Zlib

Added in: v0.5.8
Not exported by the zlib module. It is documented here because it is the base class of the compressor/decompressor classes.

不是由zlib模組匯出的。它之所以被歸檔在這裡是因為它是壓縮器/解壓器的基類。

zlib.flush([kind], callback)

Added in: v0.5.8
kind defaults to zlib.Z_FULL_FLUSH.

Flush pending data. Don’t call this frivolously, premature flushes negatively impact the effectiveness of the compression algorithm.

Calling this only flushes data from the internal zlib state, and does not perform flushing of any kind on the streams level. Rather, it behaves like a normal call to .write(), i.e. it will be queued up behind other pending writes and will only produce output when data is being read from the stream.

kind引數預設是zlib.Z_FULL_FLUSH

沖刷待處理的資料。不要愚昧地呼叫它,過早的沖刷會對壓縮演算法的有效性產生負面的影響。

呼叫它只會在zlib的內部狀態中被沖刷,並不會再流級別上執行沖刷。相反的,它的行為就像一個普通的寫呼叫。它將會排在其它待寫的操作後面,並且只有當資料正在被讀的時候才會產生輸出。

zlib.params(level, strategy, callback)

Added in: v0.11.4
Dynamically update the compression level and compression strategy. Only applicable to deflate algorithm.

動態更新壓縮級別和壓縮策略。只適用於deflate(壓縮)演算法。

zlib.reset()

Added in: v0.7.0
Reset the compressor/decompressor to factory defaults. Only applicable to the inflate and deflate algorithms.

重置壓縮器/解壓器為出廠設定。只有inflate(解壓)和deflate(壓縮)演算法適用該函式。

zlib.constants

Provides an object enumerating Zlib-related constants.

返回一個枚舉了Zlib相關常量的物件。

zlib.createDeflate(options)

Added in: v0.5.8
Returns a new Deflate object with an options.

返回一個新的帶引數的Deflate物件(用於壓縮)。

zlib.createDeflateRaw(options)

Added in: v0.5.8
Returns a new DeflateRaw object with an options.

返回一個新的帶引數的DeflateRaw物件(生成的流不使用zlib頭部,用於壓縮)。

zlib.createGunzip(options)

Added in: v0.5.8
Returns a new Gunzip object with an options.

返回一個新的帶引數的Gunzip物件(用於解壓)。

zlib.createGzip(options)

Added in: v0.5.8
Returns a new Gzip object with an options.

返回一個新的帶引數的Gzip物件(用於壓縮)。

zlib.createInflate(options)

Added in: v0.5.8
Returns a new Inflate object with an options.

返回一個新的帶引數的Infalte物件(用於解壓)。

zlib.createInflateRaw(options)

Added in: v0.5.8
Returns a new InflateRaw object with an options.

返回一個新的帶引數的InflateRaw物件(生成的流不使用zlib頭部,用於解壓)。

zlib.createUnzip(options)

Added in: v0.5.8
Returns a new Unzip object with an options.

返回一個新的帶引數的Unzip物件(用於解壓)。

Convenience Methods

便利的方法

All of these take a Buffer or string as the first argument, an optional second argument to supply options to the zlib classes and will call the supplied callback with callback(error, result).

Every method has a *Sync counterpart, which accept the same arguments, but without a callback.

所有的這些函式的第一個引數都是位元組陣列或者字串
xxx
並且會呼叫提供的回撥函式callback(error, result)。

每一個方法都有一個同步的副本,它支援的引數除回撥函式外均相同。

zlib.deflate(buf[, options], callback)

Added in: v0.6.0

使用deflate(非同步)壓縮一個位元組陣列或字串。

zlib.deflateSync(buf[, options])

Added in: v0.11.12
Compress a Buffer or string with Deflate.

使用deflate(同步)壓縮一個位元組陣列或字串。

zlib.deflateRaw(buf[, options], callback)

Added in: v0.6.0

使用deflate(非同步,不帶zlib頭部)壓縮一個位元組陣列或字串。

zlib.deflateRawSync(buf[, options])

Added in: v0.11.12
Compress a Buffer or string with DeflateRaw.

使用deflate(同步,不帶zlib頭部)壓縮一個位元組陣列或字串。

zlib.gunzip(buf[, options], callback)

Added in: v0.6.0

使用gzip(非同步)解壓一個位元組陣列或字串。

zlib.gunzipSync(buf[, options])

Added in: v0.11.12
Decompress a Buffer or string with Gunzip.

使用gzip(同步)解壓一個位元組陣列或字串。

zlib.gzip(buf[, options], callback)

Added in: v0.6.0

使用gzip(非同步)壓縮一個位元組陣列或字串。

zlib.gzipSync(buf[, options])

Added in: v0.11.12
Compress a Buffer or string with Gzip.

使用gzip(同步)壓縮一個位元組陣列或字串。

zlib.inflate(buf[, options], callback)

Added in: v0.6.0

使用inflate(非同步)解壓一個位元組陣列或字串。

zlib.inflateSync(buf[, options])

Added in: v0.11.12
Decompress a Buffer or string with Inflate.

使用inflate(同步)解壓一個位元組陣列或字串。

zlib.inflateRaw(buf[, options], callback)

Added in: v0.6.0

使用inflateRaw(非同步,不帶zlib頭部)解壓一個位元組陣列或字串。

zlib.inflateRawSync(buf[, options])

Added in: v0.11.12
Decompress a Buffer or string with InflateRaw.

使用inflateRaw(同步,不帶zlib頭部)解壓一個位元組陣列或字串。

zlib.unzip(buf[, options], callback)

Added in: v0.6.0

使用unzip(非同步)解壓一個位元組陣列或字串。

zlib.unzipSync(buf[, options])

Added in: v0.11.12
Decompress a Buffer or string with Unzip.

使用unzip(同步)解壓一個位元組陣列或字串。