Linux 緩存服務varnish
阿新 • • 發佈：2018-06-07
linux varnishVarnish是一款高性能的開源HTTP加速器,Squid服務也是，兩者的關系就像apache跟nginx，Varnish更年輕輕量,Squid更成熟穩重。
Varnish4.0工作工作流程如下圖
各個點說明
vcl_recv：接收請求
vcl_pass, 跳過緩存
vcl_hit, 緩存命中
vcl_miss, 緩存丟失
vcl_pipe, 非http
vcl_purge, 手動清楚緩存為過期
vcl_synth, 手動清楚緩存過期同步
vcl_deliver 發送給客戶端
每個節點上的變量作用域如下圖
Varnish4.0工作工作流程如下圖
各個點說明
vcl_recv：接收請求
vcl_pass, 跳過緩存
vcl_hit, 緩存命中
vcl_miss, 緩存丟失
vcl_pipe, 非http
vcl_purge, 手動清楚緩存為過期
vcl_synth, 手動清楚緩存過期同步
vcl_deliver 發送給客戶端
每個節點上的變量作用域如下圖
內建變量：
1、req.*：request，表示由客戶端發來的請求報文相關；
req.http.User-Agent, req.http.Referer, ...2、bereq.*：由varnish發往BE主機的httpd請求相關；
3、 beresp.*：由BE主機響應給varnish的響應報文相關；
4、 resp.*：由varnish響應給client相關；
5、obj.*：存儲在緩存空間中的緩存對象的屬性；只讀；
常用變量：
bereq.http.HEADERS
bereq.request：請求方法；
bereq.url：請求的url；
bereq.proto：請求的協議版本；
bereq.backend：指明要調用的後端主機；
req.http.Cookie：客戶端的請求報文中Cookie首部的值；
req.http.User-Agent ~ "chrome"//-------------------------------------
beresp.http.HEADERS
beresp.status：響應的狀態碼；
reresp.proto：協議版本；
beresp.backend.name：BE主機的主機名；
beresp.ttl：BE主機響應的內容的余下的可緩存時長；
//-------------------------------------
obj.hits：此對象從緩存中命中的次數；
obj.ttl：對象的ttl值
//-------------------------------------
server.ip
server.hostname
//-------------------------------------client.ip
Varnish的特長體現在緩存命中vcl_hit，以及緩存清理vcl_purge方便。
簡單使用說明
1、 安裝
[root@node1 ~]# yum install varnish2、 查看
[root@node1 ~]# rpm -ql varnish
/etc/varnish #配置文件目錄
/etc/varnish/default.vcl #配置各Child/Cache線程的緩存策略；
/etc/varnish/varnish.params # 配置varnish服務進程的工作特性，例如監聽的地址和端口，緩存機制；
/run/varnish.pid
/usr/bin/varnishadm #客戶端腳本
/usr/bin/varnishtest #測試工具程序：
#Shared Memory Log交互工具：
/usr/bin/varnishhist
/usr/bin/varnishlog
/usr/bin/varnishncsa
/usr/bin/varnishstat
/usr/bin/varnishtop
/usr/sbin/varnishd #主程序
/usr/sbin/varnish_reload_vcl #編譯配置文件
#啟動腳本
/usr/lib/systemd/system/varnish.service
/usr/lib/systemd/system/varnishlog.service #持久化日誌 二選一
/usr/lib/systemd/system/varnishncsa.service #持久化日誌 二選一3、配置文件
[root@node1 varnish]# vim /etc/varnish/varnish.params
RELOAD_VCL=1 #啟動程序後能否reload配置文件
VARNISH_VCL_CONF=/etc/varnish/default.vcl #
VARNISH_LISTEN_PORT=80 #默認6081 端口 如果直接當web服務器用改80
# Admin interface listen address and port 管理端口地址
VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1
VARNISH_ADMIN_LISTEN_PORT=6082
VARNISH_SECRET_FILE=/etc/varnish/secret #密鑰
#VARNISH_STORAGE="malloc,256M" #緩存
VARNISH_STORAGE="file,/www/data/varnish,1g"
VARNISH_USER=varnish
VARNISH_GROUP=varnish[root@node1 data]# mkdir varnish/cache
[root@node1 data]# chown -R varnish.varnish varnish4、啟動
[root@node1 varnish]# systemctl start varnish
[root@node1 varnish]# ss -lntup|grep varnish
tcp LISTEN 0 128 *:80 *:* users:(("varnishd",pid=1765,fd=7))
tcp LISTEN 0 10 127.0.0.1:6082 *:* users:(("varnishd",pid=1764,fd=6))
tcp LISTEN 0 128 :::80 :::* users:(("varnishd",pid=1765,fd=8))5、緩存策略後臺配置
[root@node1 varnish]# vim /etc/varnish/default.vcl
backend default {
.host = "192.168.1.202";
.port = "80";
}6、重載
Loading vcl from /etc/varnish/default.vcl
Current running config name is
Using new config name reload_2018-06-06T09:17:17
VCL compiled.
VCL ‘reload_2018-06-06T09:17:17‘ now active
available 0 boot
active 0 reload_2018-06-06T09:17:17
Done7、admin管理
[root@node1 varnish]# varnishadm -h
varnishadm: invalid option -- ‘h‘
usage: varnishadm [-n ident] [-t timeout] [-S secretfile] -T [address]:port command [...]
-n is mutually exlusive with -S and -T
[root@node1 varnish]# varnishadm -S /etc/varnish/secret -T 127.0.0.1:6082
200
-----------------------------
Varnish Cache CLI 1.0
-----------------------------
Linux,3.10.0-693.el7.x86_64,x86_64,-sfile,-smalloc,-hcritbit
varnish-4.0.5 revision 07eff4c29
Type ‘help‘ for command list.
Type ‘quit‘ to close CLI session.
varnish> vcl.list #配置文件版本
200
available 0 boot
active 0 reload_2018-06-06T09:17:17
#切換vcl配置 每次配置文件加載都會保存下來，在這裏可以切換
vcl.use boot
200
VCL ‘boot‘ now active
vcl.use reload_2018-06-06T09:17:17
200
VCL ‘reload_2018-06-06T09:17:17‘ now active
#查看默認配置選項
vcl.show -v reload_2018-06-06T09:17:17
#線程數量
param.show thread_pools
200
thread_pools
Value is: 2 [pools] (default)
Default is: 2
Minimum is: 1
param.show thread_pool_max #min
200
thread_pool_max
Value is: 5000 [threads] (default)
Default is: 5000
Minimum is: 100
#設置參數
varnish> param.set thread_pools 4
200
#存儲查看
varnish> storage.list
200
Storage devices:
storage.Transient = malloc
storage.s0 = file
#後端查看
varnish> backend.list
200
Backend name Refs Admin Probe
default(127.0.0.1,,8080) 1 probe Healthy (no probe)
default(192.168.1.202,,80) 1 probe Healthy (no probe)
#--------------------------------------------------------------------------
#非交互式
[root@node1 varnish]# varnishadm -S /etc/varnish/secret -T 127.0.0.1:6082 status
Child in state running規則編寫
默認規則查看,很重要，因為varnish先讀取客戶端配置，在讀取默認規則
[root@node1 varnish]# varnishadm -S /etc/varnish/secret -T 127.0.0.1:6082 vcl.list
available 0 boot
available 0 reload_2018-06-06T09:17:17
available 0 reload_2018-06-06T10:43:58
available 0 reload_2018-06-06T11:08:20
available 0 reload_2018-06-06T12:52:47
available 0 reload_2018-06-06T13:04:19
active 0 reload_2018-06-06T13:06:08
[root@node1 varnish]# varnishadm -S /etc/varnish/secret -T 127.0.0.1:6082 vcl.show -v reload_2018-06-06T13:06:081、不檢查緩存
sub vcl_recv {
if (req.url ~ "(?i)^/(login|admin)") {
return(pass);
}
}
2、合成錯誤碼
sub vcl_recv {
if (req.http.User-Agent ~ "(?i)curl"){
return(synth(403));
}
}
marvindeMacBook-Pro:~ marvin$ curl http://192.168.1.200/admin.php
<!DOCTYPE html>
<html>
<head>
<title>403 Forbidden</title>
</head>
<body>
<h1>Error 403 Forbidden</h1>
<p>Forbidden</p>
<h3>Guru Meditation:</h3>
<p>XID: 196684</p>
<hr>
<p>Varnish cache server</p>
</body>
</html>3、默認vcl_recv策略
sub vcl_recv {
if (req.method != "GET" &&
req.method != "HEAD" &&
req.method != "PUT" &&
req.method != "POST" &&
req.method != "TRACE" &&
req.method != "OPTIONS" &&
req.method != "DELETE") {
/* Non-RFC2616 or CONNECT which is weird. */
return (pipe);
}
if (req.method != "GET" && req.method != "HEAD") {
/* We only deal with GET and HEAD by default */
return (pass);
}
if (req.http.Authorization || req.http.Cookie) {
/* Not cacheable by default */
return (pass);
}
return (hash);
}4、指定資源去除cookie做緩存
sub vcl_recv {
if (req.url ~ "(?i)\.(jpg|jpge)$" ){
unset req.http.Cookie;
return(hash);
}
}
sub vcl_backend_response {
if (beresp.http.cache-control !~ "s-maxage") {
if (bereq.url ~ "(?i)\.(jpg|jpeg|png|gif|css|js)$") {
unset beresp.http.Set-Cookie;
set beresp.ttl = 3600s;
}
}
} 
5、客戶端ip
vcl_recv： 如果多級代理需要加 ，默認是有X-Forwarded-For
if (req.http.X-Forwarded-For){
set req.http.X-Forwarded-For = req.http.X-Forwarded-For + "," + client.ip;
}else {
set req.http.X-Forwarded-For = client.ip;
}
6、緩存修剪(實用)
#白名單設置 掩碼寫在外面
acl purgers{
# "192.168.1.104";
"127.0.0.1"/8;
}
sub vcl_recv {
if(req.method == "PURGE"){
if (!client.ip ~ purgers){
return(synth(403));
}
return(purge);
}
}
#默認規則裏面的行為
sub vcl_purge {
return (synth(200, "Purged"));
} 
以上三次請求，第一次刪除緩存，第二次沒有命中（添加緩存），第三次命中緩存
在交互模式下(用於臨時清理) 這種方式很有用
ban req.url ~ ^/javascript #清空所有/javascript 開頭的
ban req.url ~ ^/ #清空所有
#配置文件清空
if (req.method == "BAN") {
ban("req.http.host == " + req.http.host + " && req.url == " + req.url);
# Throw a synthetic page so the request won‘t go to the backend.
return(synth(200, "Ban added"));
}7、綁定多個後臺
import directors;
#默認是backend default
backend websrv1 {
.host = "192.168.1.201";
.port = "80";
}
backend websrv2 {
.host = "192.168.1.202";
.port = "80";
}
sub vcl_init {
#輪詢
new websrvs = directors.round_robin();
websrvs.add_backend(websrv1);
websrvs.add_backend(websrv2);
#權重
new websrvs3 = directors.random();
websrvs3.add_backend(websrv1,1);
websrvs3.add_backend(websrv2,2);
}
#可以區分出來 動態頁面跟靜態頁面
sub vcl_recv {
if (req.url ~ "(?i)\.php$") {
set req.backend_hint = websrv1;
} else {
set req.backend_hint = websrvs.backend();
}
}
#---------------------------------------------------------
#會話綁定
sub vcl_init {
new h = directors.hash();
h.add_backend(one, 1); // backend ‘one‘ with weight ‘1‘
h.add_backend(two, 1); // backend ‘two‘ with weight ‘1‘
}
sub vcl_recv {
// pick a backend based on the cookie header of the client
set req.backend_hint = h.backend(req.http.cookie);
}8、健康狀態監測
probe www_probe {
.url = "/index.html";
.timeout = 1s;
.interval = 4s;
.window = 5; #探測5次 成功3次算在線
.threshold = 3;
.expected_response：期望的響應碼，默認為200；
}
backend websrv1 {
.host = "192.168.1.201";
.port = "80";
.probe = www_probe;
}
backend websrv2 {
.host = "192.168.1.202";
.port = "80";
.probe = www_probe;
}
[root@node1 varnish]# varnishadm -S /etc/varnish/secret -T 127.0.0.1:6082 backend.list
Backend name Refs Admin Probe
default(192.168.1.201,,80) 9 probe Healthy (no probe)
websrv1(192.168.1.201,,80) 5 probe Healthy 5/5
websrv2(192.168.1.202,,80) 5 probe Healthy 5/5
在交互模式下手動設置下線
backend.set_health websrv Sick
backend.set_health websrv Healthy服務進程配置
DAEMON_OPTS="-p thread_pool=3 thread_pool_min=5 -p thread_pool_max=500 -p thread_pool_timeout=300"
thread_pool 2 線程池 小於等於cpu
thread_pool_min 100 每個線程池最小幾個線程 默認100
thread_pool_max 5000 最大幾個線程 默認5000
thread_pool_timeout 300.000 空閑線程的最長時常 超過thread_pool_min個數後 空閑的超過這個時間就會被殺死
thread_queue_limit 20 線程池隊列長度默認20 可以增加
thread_pool_add_delay 0.000 [seconds] 增加不延遲
thread_pool_destroy_delay 1 second 殺死延遲一秒
thread_pool_fail_delay 0.200 [seconds] 創建失敗時候，間隔多少時間在重新創建
client：
send_timeout 600.000 發送響應包給客戶端
timeout_idle 5 保持連接超時空閑時常 可調增加
timeout_req 2 接收客戶端請求報文首部 超過2秒放棄
cli_timeout admin 管理工具請求超時時常 backend BE_NAME {
...
.connect_timeout = 3.5s;
.first_byte_timeout = 60s;
.between_bytes_timeout = 60s;
}log
1、varnishstat - Varnish Cache statistics
-1
-1 -f FILED_NAME
-l：可用於-f選項指定的字段名稱列表；
MAIN.cache_hit
MAIN.cache_miss
# varnishstat -1 -f MAIN.cache_hit -f MAIN.cache_miss
# varnishstat -l -f MAIN -f MEMPOOL
[root@node1 varnish]# varnishstat -1 #一次顯示所有
[root@node1 varnish]# varnishstat -1 -f MAIN.cache_hit -f MAIN.cache_miss
MAIN.cache_hit 63 0.00 Cache hits
MAIN.cache_miss 18 0.00 Cache misses2、varnishtop - Varnish log entry ranking
-1 Instead of a continously updated display, print the statistics once and exit.
-i taglist，可以同時使用多個-i選項，也可以一個選項跟上多個標簽；
-I <[taglist:]regex>
-x taglist：排除列表
-X <[taglist:]regex>
[root@node1 varnish]# varnishtop -i RespStatus #壓測 狀態碼3、日誌文件，一般不開啟，一級代理采用
[root@node1 varnish]# systemctl start varnishncsa
[root@node1 varnish]# tail /var/log/varnish/varnishncsa.log
192.168.1.104 - - [06/Jun/2018:20:54:42 +0800] "GET http://192.168.1.200/robots.txt HTTP/1.1" 404 571 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36"Linux 緩存服務varnish