為了將Kubeflow（https://github.com/kubeflow/kubeflow）/Kubernetes等映象放到本地叢集部署或者更新，需要一系列的操作。如果叢集的多個節點同時訪問外部映象服務，將帶來較大的併發網路流量，不僅速度慢、而且費用增加。因此我將其分為兩個階段來進行，第一階段將映象下載到本地，第二階段各個節點從本地檔案系統或映象服務來獲取映象的拷貝。

1、從 gcr到本地儲存

這一階段其實也分為兩個步驟。

首先，從能夠訪問到gcr的站點（https://www.katacoda.com）下載。如：

echo ""
echo "================================================================="
echo "pull kubeflow images for system from gcr.io and hub.docker.com..."
echo "This tools created by openthings, NO WARANTY. 2018.07.10."
echo "================================================================="

echo ""
echo "1. centraldashboard"
docker pull gcr.io/kubeflow-images-public/centraldashboard:v0.2.1

echo ""
echo "2. jupyterhub-k8s"
docker pull gcr.io/kubeflow/jupyterhub-k8s:v20180531-3bb991b1

echo ""
echo "3. tf_operator"
docker pull gcr.io/kubeflow-images-public/tf_operator:v0.2.0 

echo ""
echo "4. ambassador"
docker pull quay.io/datawire/ambassador:0.30.1

echo ""
echo "5. redis"
docker pull redis:4.0.1 

echo ""
echo "6. seldonio/cluster-manager"
docker pull seldonio/cluster-manager:0.1.6

echo ""
echo "Finished."
echo ""

 

然後將映象推送到國內站點（如阿里雲-http://registry.cn-hangzhou.aliyuncs.com）。如

echo ""
echo "================================================================="
echo "Push kubeflow images for system to aliyun.com ..."
echo "This tools created by openthings, NO WARANTY. 2018.07.10."
echo "================================================================="

MY_REGISTRY=registry.cn-hangzhou.aliyuncs.com/openthings

echo ""
echo "1. centraldashboard"
docker tag gcr.io/kubeflow-images-public/centraldashboard:v0.2.1 ${MY_REGISTRY}/kubeflow-images-public-centraldashboard:v0.2.1
docker push ${MY_REGISTRY}/kubeflow-images-public-centraldashboard:v0.2.1

echo ""
echo "2. jupyterhub-k8s"
docker tag gcr.io/kubeflow/jupyterhub-k8s:v20180531-3bb991b1 ${MY_REGISTRY}/kubeflow-jupyterhub-k8s:v20180531-3bb991b1
docker push ${MY_REGISTRY}/kubeflow-jupyterhub-k8s:v20180531-3bb991b1

echo ""
echo "3. tf_operator"
docker tag gcr.io/kubeflow-images-public/tf_operator:v0.2.0 ${MY_REGISTRY}/kubeflow-images-public-tf_operator:v0.2.0
docker push ${MY_REGISTRY}/kubeflow-images-public-tf_operator:v0.2.0

echo ""
echo "4. ambassador"
docker tag quay.io/datawire/ambassador:0.30.1 ${MY_REGISTRY}/quay-io-datawire-ambassador:0.30.1
docker push ${MY_REGISTRY}/quay-io-datawire-ambassador:0.30.1

echo ""
echo "5. redis"
docker tag redis:4.0.1 ${MY_REGISTRY}/redis:4.0.1
docker push ${MY_REGISTRY}/redis:4.0.1

echo ""
echo "6. seldonio/cluster-manager"
docker tag seldonio/cluster-manager:0.1.6 ${MY_REGISTRY}/seldonio-cluster-manager:0.1.6
docker push ${MY_REGISTRY}/seldonio-cluster-manager:0.1.6

echo ""
echo "Finished."
echo ""
 

然後，可以從阿里雲下載到本地，恢復為原始的名稱：

echo ""
echo "================================================================="
echo "Pull kubeflow images for system from aliyun.com ..."
echo "This tools created by openthings, NO WARANTY. 2018.11.28."
echo "================================================================="

MY_REGISTRY=registry.cn-hangzhou.aliyuncs.com/openthings

echo ""
echo "1. centraldashboard"
docker pull ${MY_REGISTRY}/kubeflow-images-public-centraldashboard:v0.2.1
docker tag ${MY_REGISTRY}/kubeflow-images-public-centraldashboard:v0.2.1 gcr.io/kubeflow-images-public/centraldashboard:v0.2.1 

echo ""
echo "2. jupyterhub-k8s"
docker pull ${MY_REGISTRY}/kubeflow-jupyterhub-k8s:v20180531-3bb991b1
docker tag ${MY_REGISTRY}/kubeflow-jupyterhub-k8s:v20180531-3bb991b1 gcr.io/kubeflow/jupyterhub-k8s:v20180531-3bb991b1

echo ""
echo "3. tf_operator"
docker pull ${MY_REGISTRY}/kubeflow-images-public-tf_operator:v0.2.0
docker tag ${MY_REGISTRY}/kubeflow-images-public-tf_operator:v0.2.0 gcr.io/kubeflow-images-public/tf_operator:v0.2.0

echo ""
echo "4. ambassador"
docker pull ${MY_REGISTRY}/quay-io-datawire-ambassador:0.30.1
docker tag ${MY_REGISTRY}/quay-io-datawire-ambassador:0.30.1 quay.io/datawire/ambassador:0.30.1

echo ""
echo "5. redis"
docker pull ${MY_REGISTRY}/redis:4.0.1
docker tag ${MY_REGISTRY}/redis:4.0.1 redis:4.0.1

echo ""
echo "6. seldonio/cluster-manager"
docker pull ${MY_REGISTRY}/seldonio-cluster-manager:0.1.6
docker tag ${MY_REGISTRY}/seldonio-cluster-manager:0.1.6 seldonio/cluster-manager:0.1.6

echo ""
echo "Finished."
echo ""
 

從阿里雲下載到本地後，可以推送到本地映象服務（如Harbor）或者打包為*.tar檔案。

2、從本地儲存到叢集部署

從本地Harbor中安裝，使用docker tag將映象改名後，就可以使用了。可以參考上面的從阿里雲下載的方法。

打包為*.tar檔案，參見：

echo "==================================================================="
echo "Save Kubeflow images to tar."
echo "This tool created by https://my.oschina.net/u/2306127"
echo "Please visit https://github.com/openthings/kubernetes-tools"

echo "###################################################################"
echo "Kubeflow 0.3.3 ML system images."
echo "-------------------------------------------------------------------"

echo "A1.>> centraldashboard"
docker save gcr.io/kubeflow-images-public/centraldashboard:v0.2.1 -o A1-kubeflow-centraldashboard-v0.2.1.tar
echo ""

echo "A2.>> jupyterhub-k8s"
docker save gcr.io/kubeflow/jupyterhub-k8s:v20180531-3bb991b1 -o A2-kubeflow-jupyterhub-k8s-v20180531-3bb991b1.tar
echo ""

echo "A3.>> tf_operator"
docker save gcr.io/kubeflow-images-public/tf_operator:v0.2.0 -o A3-kubeflow-tf_operator-v0.2.0.tar
echo ""

echo "A4.>> ambassador"
docker save quay.io/datawire/ambassador:0.30.1 -o A4-kubeflow-ambassador-0.30.1.tar
echo ""

echo "A5.>> redis"
docker save redis:4.0.1  -o A5-kubeflow-redis-4.0.1.tar
echo ""

echo "A6.>> seldonio/cluster-manager"
docker save seldonio/cluster-manager:0.1.6 -o A6-kubeflow-seldonio-cluster-manager-0.1.6.tar
echo ""

echo "=================================================================="
echo "Kubeflow worker engine images......"
echo "B1.>> Tensorflow notebook CPU"
docker save gcr.io/kubeflow-images-public/tensorflow-1.12.0-notebook-cpu:v-base-76107ff-897 -o A6-kubeflow-tensorflow-1.12.0-notebook-cpu-v-base-76107ff-897.tar
echo ""

echo "B2.>> Tensorflow notebook GPU"
docker save gcr.io/kubeflow-images-public/tensorflow-1.12.0-notebook-gpu:v-base-76107ff-897 -o A6-kubeflow-tensorflow-1.12.0-notebook-gpu-v-base-76107ff-897.tar
echo ""

echo "==================================================================="
echo "Save Kubeflow images Finished."
echo "This tool created by https://my.oschina.net/u/2306127"
echo "Please visit https://github.com/openthings/kubernetes-tools"
echo "==================================================================="
echo ""

將所有的映象壓縮為一個zip包，然後上傳到工作節點：

echo "Uploading 10.1.1.202"
sshpass -p xxxx scp kf-images-0.3.3.zip [email protected]:/home/supermap/

echo "Uploading 10.1.1.203"
sshpass -p xxxx scp kf-images-0.3.3.zip [email protected]:/home/supermap/

echo "Uploading 10.1.1.142"
sshpass -p xxxx scp kf-images-0.3.3.zip [email protected]:/home/supermap/

echo "Uploading 10.1.1.193"
sshpass -p xxxx scp kf-images-0.3.3.zip [email protected]:/home/supermap/

echo "Uploading 10.1.1.234"
sshpass -p xxxx scp kf-images-0.3.3.zip [email protected]:/home/supermap/

echo "Uploading 10.1.1.205"
sshpass -p xxxx scp kf-images-0.3.3.zip [email protected]:/home/supermap/

echo "Uploading 10.1.1.112"
sshpass -p xxxx scp kf-images-0.3.3.zip [email protected]:/home/supermap/

echo "Upload kf-images-0.3.3.zip Finished."

然後，在各個節點上恢復到Docker的原始映象名稱。如下：

echo "==================================================================="
echo "Load Kubeflow images from tar."
echo "This tool created by https://my.oschina.net/u/2306127"
echo "Please visit https://github.com/openthings/kubernetes-tools"

echo "###################################################################"
echo "Kubernetes core system images."
echo "-------------------------------------------------------------------"

echo "A1<< centraldashboard"
sudo docker load -i A1-kubeflow-centraldashboard-v0.2.1.tar
echo ""

echo "A2<< jupyterhub-k8s"
sudo docker load -i A2-kubeflow-jupyterhub-k8s-v20180531-3bb991b1.tar
echo ""

echo "A3<< tf_operator"
sudo docker load -i A3-kubeflow-tf_operator-v0.2.0.tar
echo ""

echo "A4<< ambassador"
sudo docker load -i A4-kubeflow-ambassador-0.30.1.tar
echo ""

echo "A5<< redis"
sudo docker load -i A5-kubeflow-redis-4.0.1.tar
echo ""

echo "A6<< seldonio/cluster-manager"
sudo docker load -i A6-kubeflow-seldonio-cluster-manager-0.1.6.tar
echo ""

echo "=================================================================="
echo "Kubeflow worker engine images......"
echo "B1<< Tensorflow notebook CPU"
sudo docker load -i A6-kubeflow-tensorflow-1.12.0-notebook-cpu-v-base-76107ff-897.tar
echo ""

echo "B2<< Tensorflow notebook GPU"
sudo docker load -i A6-kubeflow-tensorflow-1.12.0-notebook-gpu-v-base-76107ff-897.tar
echo ""

echo "==================================================================="
echo "Load Kubeflow images Finished."
echo "This tool created by https://my.oschina.net/u/2306127"
echo "Please visit https://github.com/openthings/kubernetes-tools"
echo "==================================================================="

在每一個節點執行上面的指令碼，也可以使用 ansible來遠端批量執行。

ansible all -i hosts_ansible -m shell -a "unzip -u /home/supermap/kf-images-0.3.3.zip && cd /home/supermap/kf-images-0.3.3 && ./kf-images-load.sh" --ask-sudo-pass --become --become-method=sudo

上面的hosts_ansible為ansible的hosts列表檔案（請參考 Ansible快速開始-指揮叢集 ）。

上面的這個過程也適用於Kubernetes本身映象的下載和更新。更多參考：

• Kubeflow映象的快速下載（V0.3.3）
• Kubernetes 1.12.3快速升級

檢視映象是否有新的版本：

• https://console.cloud.google.com/gcr/images/kubeflow-images-public