【原創】大數據基礎之Mesos（1）簡介、安裝、使用
Mesos 1.7.1
官方：http://mesos.apache.org/
一 簡介
Program against your datacenter like it’s a single pool of resources
mesos使數據中心（龐大的服務器集群）看起來像是一個資源（CPU、內存、存儲、網絡帶寬、端口等）池；
Apache Mesos abstracts CPU, memory, storage, and other compute resources away from machines (physical or virtual), enabling fault-tolerant and elastic distributed systems to easily be built and run effectively.
mesos對服務器（物理機或者虛擬機）的資源（CPU、內存、存儲、網絡帶寬、端口等）進行抽象；
What is Mesos? A distributed systems kernel
Mesos is built using the same principles as the Linux kernel, only at a different level of abstraction. The Mesos kernel runs on every machine and provides applications (e.g., Hadoop, Spark, Kafka, Elasticsearch) with API’s for resource management and scheduling across entire datacenter and cloud environments.
mesos使用linux內核相同的原則進行設計和構建，mesos運行在每臺服務器上，使得各種應用（hadoop、spark、kafka、es等）可以通過api的方式進行資源管理和調度，無論是在數據中心或者雲上；
The above figure shows the main components of Mesos. Mesos consists of a master daemon that manages agent daemons running on each cluster node, and Mesos frameworks that run tasks on these agents.
mesos由master和agent組成，其中agent運行在集群中的每臺服務器上；mesos framework（應用，比如hadoop、spark等）會在agent上運行task；
角色
master & agent
The master enables fine-grained sharing of resources (CPU, RAM, …) across frameworks by making them resource offers. Each resource offer contains a list of <agent ID, resource1: amount1, resource2: amount2, ...>. The master decides how many resources to offer to each framework according to a given organizational policy, such as fair sharing or strict priority. To support a diverse set of policies, the master employs a modular architecture that makes it easy to add new allocation modules via a plugin mechanism.
master通過resouce offer的方式來實現細粒度的框架間資源共享；每一個resource offer包含一個列表，每個元素看起來是<agent ID, resource1: amount1, resource2: amount2, ...>，即某個agent的各項資源的剩余情況；master根據給定的分組策略（比如fair sharing或strict priority）來決定為每個framework提供多少資源；master還提供插件機制來滿足定制化需求；
概念
1 framework（scheduler & executor）
A framework running on top of Mesos consists of two components: a scheduler that registers with the master to be offered resources, and an executor process that is launched on agent nodes to run the framework’s tasks. While the master determines how many resources are offered to each framework, the frameworks‘ schedulers select which of the offered resources to use. When a framework accepts offered resources, it passes to Mesos a description of the tasks it wants to run on them. In turn, Mesos launches the tasks on the corresponding agents.
一個framework運行在mesos之上，有兩部分組成：1）scheduler向mesos master註冊並接收resource offer；2）executor在mesos agent上啟動來運行task；
當scheduler收到resource offer後可以選擇使用其中哪些資源（也可以拒絕），如果一些資源被確認使用，scheduler會給mesos發送希望在這些資源上運行的task描述，然後mesos就會在對應的agent上啟動task；
2 resource offer
二 安裝
rpm安裝
# yum install libevent libevent-devel cyrus-sasl cyrus-sasl-devel cyrus-sasl-md5 subversion-devel
$ wget http://repos.mesosphere.com/el/7/x86_64/RPMS/mesos-1.7.1-2.0.1.el7.x86_64.rpm
$ rpm -ivh mesos-1.7.1-2.0.1.el7.x86_64.rpm
手工編譯安裝
1 環境準備
$ wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo -O /etc/yum.repos.d/epel-apache-maven.repo
$ yum install -y epel-release
$ bash -c ‘cat > /etc/yum.repos.d/wandisco-svn.repo <<EOF
[WANdiscoSVN]
name=WANdisco SVN Repo 1.9
enabled=1
baseurl=http://opensource.wandisco.com/centos/7/svn-1.9/RPMS/\$basearch/
gpgcheck=1
gpgkey=http://opensource.wandisco.com/RPM-GPG-KEY-WANdisco
EOF‘
$ yum update systemd
$ yum groupinstall -y "Development Tools"
$ yum install -y apache-maven python-devel python-six python-virtualenv java-1.8.0-openjdk-devel zlib-devel libcurl-devel openssl-devel cyrus-sasl-devel cyrus-sasl-md5 apr-devel subversion-devel apr-util-devel
以上為centos7的準備過程，其他系統詳見編譯過程參考
2 下載編譯安裝
$ wget http://www.apache.org/dist/mesos/1.7.1/mesos-1.7.1.tar.gz
$ tar -zxf mesos-1.7.1.tar.gz
$ cd mesos-1.7.1
$ mkdir build
$ cd build
$ ../configure
$ make
$ make check
$ make install
啟動
1 單機啟動
$ ./bin/mesos-master.sh --ip=127.0.0.1 --work_dir=/var/lib/mesos
$ ./bin/mesos-agent.sh --master=127.0.0.1:5050 --work_dir=/var/lib/mesos
訪問 http://127.0.0.1:5050
2 集群啟動
$ ./bin/mesos-master.sh --work_dir=/var/lib/mesos --zk=zk://host1:port1,host2:port2,.../path
$ ./bin/mesos-agent.sh --work_dir=/var/lib/mesos --master=zk://host1:port1,host2:port2,.../path
如果是rpm安裝，可以修改配置文件
$ vi /etc/mesos/zk
然後通過service啟動
$ service mesos-master start
$ service mesos-slave start
其他配置修改
$ vi /etc/mesos-master/quorum
$ vi /etc/mesos-master/work_dir
$ vi /etc/mesos-slave/work_dir
編譯過程參考：
http://mesos.apache.org/documentation/latest/building/
啟動參數參考：
http://mesos.apache.org/documentation/latest/configuration/master-and-agent/
http://mesos.apache.org/documentation/latest/configuration/master/
http://mesos.apache.org/documentation/latest/configuration/agent/
附：mesos編譯過程中 ../configure 這一步有可能報錯
checking for python... /data/anaconda2/bin/python
checking for python version... 2.7
checking for python platform... linux2
checking for python script directory... ${prefix}/lib/python2.7/site-packages
checking for python extension module directory... ${exec_prefix}/lib/python2.7/site-packages
checking for python2.7... (cached) /data/anaconda2/bin/python
checking for a version of Python >= ‘2.1.0‘... yes
checking for a version of Python >= ‘2.6‘... yes
checking for the distutils Python package... yes
checking for Python include path... -I/data/anaconda2/include/python2.7
checking for Python library path... -L/data/anaconda2/lib -lpython2.7
checking for Python site-packages path... /data/anaconda2/lib/python2.7/site-packages
checking python extra libraries... -lpthread -ldl -lutil -lm
checking python extra linking flags... -Xlinker -export-dynamic
checking consistency of all components of python development environment... no
configure: error: in `/data/mesos-1.7.0/build‘:
configure: error:
Could not link test program to Python. Maybe the main Python library has been
installed in some non-standard library path. If so, pass it to configure,
via the LDFLAGS environment variable.
Example: ./configure LDFLAGS="-L/usr/non-standard-path/python/lib"
============================================================================
ERROR!
You probably have to install the development version of the Python package
for your distribution. The exact name of this package varies among them.
============================================================================See `config.log‘ for more details
這時優先看有沒有安裝python-devel，安裝之後重試configure
$ yum install python-devel
如果已經安裝過python-devel還是報相同的錯，極有可能是存在多個python版本導致的（比如系統自帶一個python，anaconda自帶一個python），這時通過conda安裝python-devel後還是報相同的錯
$ conda create --name dev python=[version here]
可以通過恢復系統默認python來暫時解決
$ vi ~/.bashrc
安裝anaconda後默認會修改當前用戶的.bashrc，將以下部分註釋即可
# added by Anaconda2 2018.12 installer
# >>> conda init >>>
...
# <<< conda init <<<
三 使用
配置文件
~/.mesos/config.toml
命令
$ mesos help
【原創】大數據基礎之Mesos（1）簡介、安裝、使用