k8s, etcd 多節點叢集部署問題排查記錄
阿新 • • 發佈：2018-12-16
目錄
文章目錄
- 目錄
- 部署環境
- 1. etcd 叢集啟動失敗
- 2. etcd 健康狀態檢查失敗
- 3. kube-apiserver 啟動失敗
- 4. kubelet 啟動失敗
- 5. Approved CSR 後獲取 nodes 失敗
- 6. 訪問 pod app 失敗
部署環境
雙節點 IP 配置：
# cat /etc/hosts 192.168.1.5 vmnote0 192.168.1.12 vmnote1
部署文件：https://jimmysong.io/kubernetes-handbook/practice/install-kubernetes-on-centos.html
1. etcd 叢集啟動失敗
etcd 雙節點配置之一：
[[email protected] ~]# cat /usr/lib/systemd/system/etcd.service [Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target Documentation=https://github.com/coreos [Service] Type=notify WorkingDirectory=/var/lib/etcd/ ExecStart=/usr/local/bin/etcd \ --name vmnode0 \ --cert-file=/etc/kubernetes/ssl/kubernetes.pem \ --key-file=/etc/kubernetes/ssl/kubernetes-key.pem \ --peer-cert-file=/etc/kubernetes/ssl/kubernetes.pem \ --peer-key-file=/etc/kubernetes/ssl/kubernetes-key.pem \ --trusted-ca-file=/etc/kubernetes/ssl/ca.pem \ --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \ --initial-advertise-peer-urls https://192.168.1.5:2380 \ --listen-peer-urls https://192.168.1.5:2380 \ --listen-client-urls https://192.168.1.5:2379,http://127.0.0.1:2379 \ --advertise-client-urls https://192.168.1.5:2379 \ --initial-cluster-token etcd-cluster-1 \ --initial-cluster vmnode0=https://192.168.1.5:2380,vmnode1=https://192.168.1.12:2380 \ --initial-cluster-state new \ --data-dir=/var/lib/etcd Restart=on-failure RestartSec=5 LimitNOFILE=65536 [Install] WantedBy=multi-user.target
etcd 雙節點配置之二：
[Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target Documentation=https://github.com/coreos [Service] Type=notify WorkingDirectory=/var/lib/etcd/ ExecStart=/usr/local/bin/etcd \ --name vmnode1 \ --cert-file=/etc/kubernetes/ssl/kubernetes.pem \ --key-file=/etc/kubernetes/ssl/kubernetes-key.pem \ --peer-cert-file=/etc/kubernetes/ssl/kubernetes.pem \ --peer-key-file=/etc/kubernetes/ssl/kubernetes-key.pem \ --trusted-ca-file=/etc/kubernetes/ssl/ca.pem \ --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \ --initial-advertise-peer-urls https://192.168.1.12:2380 \ --listen-peer-urls https://192.168.1.12:2380 \ --listen-client-urls https://192.168.1.12:2379,http://127.0.0.1:2379 \ --advertise-client-urls https://192.168.1.12:2379 \ --initial-cluster-token etcd-cluster-1 \ --initial-cluster vmnode0=https://192.168.1.5:2380,vmnode1=https://192.168.1.12:2380 \ --initial-cluster-state new \ --data-dir=/var/lib/etcd Restart=on-failure RestartSec=5 LimitNOFILE=65536 [Install] WantedBy=multi-user.target
問題：
Dec 14 12:30:30 k8s-node2.localdomain etcd[2560]: warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
Dec 14 12:30:30 k8s-node2.localdomain etcd[2560]: member 218d8bfb33a29c6 has already been bootstrapped
Dec 14 12:30:30 k8s-node2.localdomain systemd[1]: etcd.service: main process exited, code=exited, status=1/FAILURE
Dec 14 12:30:30 k8s-node2.localdomain systemd[1]: Failed to start Etcd Server.
解決
主要問題在 member 218d8bfb33a29c6 has already been bootstrapped，原因：
One of the member was bootstrapped via discovery service. You must remove the previous data-dir to clean up the member information. Or the member will ignore the new configuration and start with the old configuration. That is why you see the mismatch.
2. etcd 健康狀態檢查失敗
[[email protected] ~]# etcdctl \
> --ca-file=/etc/kubernetes/ssl/ca.pem \
> --cert-file=/etc/kubernetes/ssl/kubernetes.pem \
> --key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
> cluster-health
2018-12-14 13:13:15.280712 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
2018-12-14 13:13:15.281964 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
failed to check the health of member 218d8bfb33a29c6 on https://192.168.1.12:2379: Get https://192.168.1.12:2379/health: net/http: TLS handshake timeout
member 218d8bfb33a29c6 is unreachable: [https://192.168.1.12:2379] are all unreachable
failed to check the health of member 499bc1bc6765950c on https://192.168.1.5:2379: Get https://192.168.1.5:2379/health: net/http: TLS handshake timeout
member 499bc1bc6765950c is unreachable: [https://192.168.1.5:2379] are all unreachable
cluster is unhealthy
解決
主要原因：https://192.168.1.12:2379/health: net/http: TLS handshake timeout，等待超時，一般來說可能是因為設定了 http_proxy/https_proxy 做代理翻牆導致不能訪問自身和內網。 所以配置 no_proxy：
PROXY_HOST=127.0.0.1
export all_proxy=http://$PROXY_HOST:8118
export ftp_proxy=http://$PROXY_HOST:8118
export http_proxy=http://$PROXY_HOST:8118
export https_proxy=http://$PROXY_HOST:8118
export no_proxy='localhost,192.168.1.5,192.168.1.12'
3. kube-apiserver 啟動失敗
[[email protected] kubernetes]# systemctl status kube-apiserver
● kube-apiserver.service - Kubernetes API Service
Loaded: loaded (/usr/lib/systemd/system/kube-apiserver.service; enabled; vendor preset: disabled)
Active: failed (Result: start-limit) since Sat 2018-12-15 03:07:12 UTC; 4s ago
Docs: https://github.com/GoogleCloudPlatform/kubernetes
Process: 3659 ExecStart=/usr/local/bin/kube-apiserver $KUBE_LOGTOSTDERR $KUBE_LOG_LEVEL $KUBE_ETCD_SERVERS $KUBE_API_ADDRESS $KUBE_API_PORT $KUBELET_PORT $KUBE_ALLOW_PRIV $KUBE_SERVICE_ADDRESSES $KUBE_ADMISSION_CONTROL $KUBE_API_ARGS (code=exited, status=1/FAILURE)
Main PID: 3659 (code=exited, status=1/FAILURE)
Dec 15 03:07:11 k8s-master.localdomain systemd[1]: kube-apiserver.service: main process exited, code=exited, status=1/FAILURE
Dec 15 03:07:11 k8s-master.localdomain systemd[1]: Failed to start Kubernetes API Service.
Dec 15 03:07:11 k8s-master.localdomain systemd[1]: Unit kube-apiserver.service entered failed state.
Dec 15 03:07:11 k8s-master.localdomain systemd[1]: kube-apiserver.service failed.
Dec 15 03:07:12 k8s-master.localdomain systemd[1]: kube-apiserver.service holdoff time over, scheduling restart.
Dec 15 03:07:12 k8s-master.localdomain systemd[1]: start request repeated too quickly for kube-apiserver.service
Dec 15 03:07:12 k8s-master.localdomain systemd[1]: Failed to start Kubernetes API Service.
Dec 15 03:07:12 k8s-master.localdomain systemd[1]: Unit kube-apiserver.service entered failed state.
Dec 15 03:07:12 k8s-master.localdomain systemd[1]: kube-apiserver.service failed.
看不出什麼問題，回過頭來看 service 的 systemd 配置：
[Unit]
Description=Kubernetes API Service
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
After=etcd.service
[Service]
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/apiserver
ExecStart=/usr/local/bin/kube-apiserver \
$KUBE_LOGTOSTDERR \
$KUBE_LOG_LEVEL \
$KUBE_ETCD_SERVERS \
$KUBE_API_ADDRESS \
$KUBE_API_PORT \
$KUBELET_PORT \
$KUBE_ALLOW_PRIV \
$KUBE_SERVICE_ADDRESSES \
$KUBE_ADMISSION_CONTROL \
$KUBE_API_ARGS
Restart=on-failure
Type=notify
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
手動拼裝出 ExecStart 來執行：
[[email protected] ~]# /usr/local/bin/kube-apiserver --logtostderr=true --v=0 --advertise-address=192.168.1.5 --bind-address=192.168.1.5 --insecure-bind-address=192.168.1.5 --insecure-port=8080 --insecure-bind-address=127.0.0.1 --etcd-servers=https://192.168.1.5:2379,https://192.168.1.12:2379 --port=8080 --kubelet-port=10250 --allow-privileged=true --service-cluster-ip-range=10.254.0.0/16 --admission-control=ServiceAccount,NamespaceLifecycle,NamespaceExists,LimitRanger,ResourceQuota --authorization-mode=RBAC --runtime-config=rbac.authorization.k8s.io/v1beta1 --kubelet-https=true --experimental-bootstrap-token-auth --token-auth-file=/etc/kubernetes/token.csv --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem --tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem --client-ca-file=/etc/kubernetes/ssl/ca.pem --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem --etcd-cafile=/etc/kubernetes/ssl/ca.pem --etcd-certfile=/etc/kubernetes/ssl/kubernetes.pem --etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem --enable-swagger-ui=true --apiserver-count=3 --audit-log-maxage=30 --audit-log-maxbackup=3 --audit-log-maxsize=100 --audit-log-path=/var/lib/audit.log --event-ttl=1h
/usr/local/bin/kube-apiserver: line 30: /lib/lsb/init-functions: No such file or directory
問題出現了：/usr/local/bin/kube-apiserver: line 30: /lib/lsb/init-functions: No such file or directory。
安裝 redhat-lsb 包：
yum install redhat-lsb -y
執行還是錯誤：
[[email protected] ~]# /usr/local/bin/kube-apiserver --logtostderr=true --v=0 --advertise-address=192.168.1.5 --bind-address=192.168.1.5 --insecure-bind-address=192.168.1.5 --insecure-port=8080 --insecure-bind-address=127.0.0.1 --etcd-servers=https://192.168.1.5:2379,https://192.168.1.12:2379 --port=8080 --kubelet-port=10250 --allow-privileged=true --service-cluster-ip-range=10.254.0.0/16 --admission-control=ServiceAccount,NamespaceLifecycle,NamespaceExists,LimitRanger,ResourceQuota --authorization-mode=RBAC --runtime-config=rbac.authorization.k8s.io/v1beta1 --kubelet-https=true --experimental-bootstrap-token-auth --token-auth-file=/etc/kubernetes/token.csv --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem --tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem --client-ca-file=/etc/kubernetes/ssl/ca.pem --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem --etcd-cafile=/etc/kubernetes/ssl/ca.pem --etcd-certfile=/etc/kubernetes/ssl/kubernetes.pem --etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem --enable-swagger-ui=true --apiserver-count=3 --audit-log-maxage=30 --audit-log-maxbackup=3 --audit-log-maxsize=100 --audit-log-path=/var/lib/audit.log --event-ttl=1h
/opt/bin/kube-apiserver not present or not executable [FAILED]
還是錯誤，這裡進入了一個無限迴圈的大坑。我在 CentOS 上拉了一個 Ubuntu 的 kube-apiserver 執行檔案。所以很多依賴程式並不存在，最後決定重新拉取執行檔案。
解決
$ wget https://dl.k8s.io/v1.6.0/kubernetes-server-linux-amd64.tar.gz
$ tar -xzvf kubernetes-server-linux-amd64.tar.gz
$ cd kubernetes
$ tar -xzvf kubernetes-src.tar.gz
$ cp -r server/bin/{kube-apiserver,kube-controller-manager,kube-scheduler,kubectl,kube-proxy,kubelet} /usr/local/bin/
4. kubelet 啟動失敗
[[email protected] opt]# systemctl status kubelet
● kubelet.service - Kubernetes Kubelet Server
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Active: failed (Result: start-limit) since Sat 2018-12-15 08:09:53 UTC; 1s ago
Docs: https://github.com/GoogleCloudPlatform/kubernetes
Process: 12585 ExecStart=/usr/local/bin/kubelet $KUBE_LOGTOSTDERR $KUBE_LOG_LEVEL $KUBELET_API_SERVER $KUBELET_ADDRESS $KUBELET_PORT $KUBELET_HOSTNAME $KUBE_ALLOW_PRIV $KUBELET_POD_INFRA_CONTAINER $KUBELET_ARGS (code=exited, status=200/CHDIR)
Main PID: 12585 (code=exited, status=200/CHDIR)
Dec 15 08:09:53 k8s-master.localdomain systemd[1]: Unit kubelet.service entered failed state.
Dec 15 08:09:53 k8s-master.localdomain systemd[1]: kubelet.service failed.
Dec 15 08:09:53 k8s-master.localdomain systemd[1]: kubelet.service holdoff time over, scheduling restart.
Dec 15 08:09:53 k8s-master.localdomain systemd[1]: Stopped Kubernetes Kubelet Server.
Dec 15 08:09:53 k8s-master.localdomain systemd[1]: start request repeated too quickly for kubelet.service
Dec 15 08:09:53 k8s-master.localdomain systemd[1]: Failed to start Kubernetes Kubelet Server.
Dec 15 08:09:53 k8s-master.localdomain systemd[1]: Unit kubelet.service entered failed state.
Dec 15 08:09:53 k8s-master.localdomain systemd[1]: kubelet.service failed.
監控日誌輸出：
Dec 15 08:25:30 k8s-master.localdomain systemd[1]: Started Kubernetes Kubelet Server.
Dec 15 08:25:30 k8s-master.localdomain systemd[1]: kubelet.service: main process exited, code=exited, status=200/CHDIR
Dec 15 08:25:30 k8s-master.localdomain systemd[1]: Unit kubelet.service entered failed state.
Dec 15 08:25:30 k8s-master.localdomain systemd[1]: kubelet.service failed.
Dec 15 08:25:30 k8s-master.localdomain systemd[1]: kubelet.service holdoff time over, scheduling restart.
Dec 15 08:25:30 k8s-master.localdomain systemd[1]: Stopped Kubernetes Kubelet Server.
Dec 15 08:25:30 k8s-master.localdomain systemd[1]: Started Kubernetes Kubelet Server.
Dec 15 08:25:30 k8s-master.localdomain systemd[13491]: Failed at step CHDIR spawning /usr/local/bin/kubelet: No such file or directory
Dec 15 08:25:30 k8s-master.localdomain systemd[1]: kubelet.service: main process exited, code=exited, status=200/CHDIR
Dec 15 08:25:30 k8s-master.localdomain systemd[1]: Unit kubelet.service entered failed state.
Dec 15 08:25:30 k8s-master.localdomain systemd[1]: kubelet.service failed.
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: kubelet.service holdoff time over, scheduling restart.
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: Stopped Kubernetes Kubelet Server.
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: Started Kubernetes Kubelet Server.
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: kubelet.service: main process exited, code=exited, status=200/CHDIR
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: Unit kubelet.service entered failed state.
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: kubelet.service failed.
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: kubelet.service holdoff time over, scheduling restart.
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: Stopped Kubernetes Kubelet Server.
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: Started Kubernetes Kubelet Server.
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: kubelet.service: main process exited, code=exited, status=200/CHDIR
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: Unit kubelet.service entered failed state.
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: kubelet.service failed.
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: kubelet.service holdoff time over, scheduling restart.
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: Stopped Kubernetes Kubelet Server.
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: Started Kubernetes Kubelet Server.
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: kubelet.service: main process exited, code=exited, status=200/CHDIR
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: Unit kubelet.service entered failed state.
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: kubelet.service failed.
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: kubelet.service holdoff time over, scheduling restart.
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: Stopped Kubernetes Kubelet Server.
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: start request repeated too quickly for kubelet.service
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: Failed to start Kubernetes Kubelet Server.
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: Unit kubelet.service entered failed state.
Dec 15 08:25:31 k8s-master.localdomain systemd[1]: kubelet.service failed.
Failed at step CHDIR spawning /usr/local/bin/kubelet: No such file or directory 非常可以，檢視這個檔案的路徑是否正確。奇怪的是這個檔案路徑是正確的，於是繼續看 systemd 配置檔案：
[Unit]
Description=Kubernetes Kubelet Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service
[Service]
WorkingDirectory=/var/lib/kubelet
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/kubelet
ExecStart=/usr/local/bin/kubelet \
$KUBE_LOGTOSTDERR \
$KUBE_LOG_LEVEL \
$KUBELET_API_SERVER \
$KUBELET_ADDRESS \
$KUBELET_PORT \
$KUBELET_HOSTNAME \
$KUBE_ALLOW_PRIV \
$KUBELET_POD_INFRA_CONTAINER \
$KUBELET_ARGS
Restart=on-failure
[Install]
WantedBy=multi-user.target
檔案路徑還有 /var/lib/kubelet，檢視一番，的確沒有。
解決
mkdir /var/lib/kubelet
5. Approved CSR 後獲取 nodes 失敗
[[email protected] ~]# kubectl get csr
NAME AGE REQUESTOR CONDITION
csr-bv37w 19m kubelet-bootstrap Approved
csr-bwlxd 1m kubelet-bootstrap Approved
csr-f8w38 33m kubelet-bootstrap Approved
csr-g8927 47m kubelet-bootstrap Approved
csr-h7wph 4m kubelet-bootstrap Approved
csr-hpl81 50m kubelet-bootstrap Approved
csr-qxxsh 40m kubelet-bootstrap Approved
csr-r7vzl 51m kubelet-bootstrap Approved
csr-w1ccb 21m kubelet-bootstrap Approved
[[email protected] ~]# kubectl get nodes
No resources found.
檢視日誌：
[[email protected] ~]# journalctl -xe -u kube* | grep error
Dec 15 08:50:15 k8s-master.localdomain kube-scheduler[14917]: E1215 08:50:15.200937 14917 leaderelection.go:229] error retrieving resource lock kube-system/kube-scheduler: Get http://192.168.1.5:8080/api/v1/namespaces/kube-system/endpoints/kube-scheduler: dial tcp 192.168.1.5:8080: getsockopt: connection refused
kube-controller-manager 訪問 kube-apiserver 的 192.168.1.5:8080 失敗了，那麼檢視一下埠資訊：
[[email protected] ~]# netstat -lpntu
...
tcp 0 0 127.0.0.1:8080 0.0.0.0:* LISTEN 14885/kube-apiserve
8080 埠繫結的 ip 是 127.0.0.1 而不是 192.168.1.5，可能是配置檔案有問題？檢視一下：
KUBE_API_ADDRESS="--advertise-address=192.168.1.5 --bind-address=192.168.1.5 --insecure-bind-address=192.168.1.5 --insecure-port=8080 --insecure-bind-address=127.0.0.1"
的確繫結到了 127.0.0.1。
解決
修改 /etc/kubernetes/apiserver 配置檔案：
KUBE_API_ADDRESS="--advertise-address=192.168.1.5 --bind-address=192.168.1.5 --insecure-bind-address=192.168.1.5 --insecure-port=8080 --insecure-bind-address=192.168.1.5"
重啟，解決。
[[email protected] kubernetes]# kubectl get csr
NAME AGE REQUESTOR CONDITION
csr-96lj4 37s kubelet-bootstrap Approved,Issued
csr-bv37w 23m kubelet-bootstrap Approved,Issued
csr-bwlxd 5m kubelet-bootstrap Approved,Issued
csr-dpqgm 37s kubelet-bootstrap Approved,Issued
csr-f8w38 37m kubelet-bootstrap Approved,Issued
csr-g8927 52m kubelet-bootstrap Approved,Issued
csr-h7wph 9m kubelet-bootstrap Approved,Issued
csr-hpl81 55m kubelet-bootstrap Approved,Issued
csr-qxxsh 44m kubelet-bootstrap Approved,Issued
csr-r7vzl 56m kubelet-bootstrap Approved,Issued
csr-w1ccb 26m kubelet-bootstrap Approved,Issued
[[email protected] kubernetes]# kubectl get nodes
NAME STATUS AGE VERSION
192.168.1.12 NotReady 10s v1.6.0
192.168.1.5 Ready 7s v1.6.0
6. 訪問 pod app 失敗
[email protected] ~]# kubectl run nginx --replicas=2 --labels="run=load-balancer-example" --image=nginx --port=80
deployment "nginx" created
[[email protected] ~]# kubectl expose deployment nginx --type=NodePort --name=example-service
service "example-service" exposed
[[email protected] ~]# kubectl get deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
nginx 2 2 2 0 27s
[[email protected] ~]# kubectl describe svc example-service
Name: example-service
Namespace: default
Labels: run=load-balancer-example
Annotations: <none>
Selector: run=load-balancer-example
Type: NodePort
IP: 10.254.109.60
Port: <unset> 80/TCP
NodePort: <unset> 30019/TCP
Endpoints: 172.17.0.2:80
Session Affinity: None
Events: <none>
[[email protected] ~]# curl "10.254.109.60:80"
^C
[[email protected] ~]# curl "172.17.0.2:80"
^C
解決
第一反應，可能還是 no_proxy 代理的問題，試一試。
export no_proxy='localhost,192.168.1.5,192.168.1.12,10.254.109.60,172.17.0.2'
再試一次就可以了。
[[email protected] kubernetes]# source /etc/profile
[[email protected] kubernetes]# curl "10.254.109.60:80"
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
[[email protected] ~]# curl "172.17.0.2:80"
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>