centos環境的python、scrapy部署
1 操作系統
Centos 6.5 64bit 或以上版本
2 軟件環境
提示:(1)用什麽軟件;(2)運行環境(3)開發包。
- Python:Python-2.7.6.tgz
- Scrapy:Scrapy-1.0.3.tar.gz
- Selenium:selenium (2.48.0)
- Phantomjs:phantomjs-1.9.8-linux-x86_64.tar.bz2
- MySQL-python:MySQL-python-1.2.5.zip
- MySQL:MySQL-5.6.27-1.linux_glibc2.5.x86_64.rpm-bundle.tar
3 安裝步驟
1. 安裝依賴庫
yum install zlib-devel bzip2-devel ncurses-devel sqlite-devel readline-devel tk-devel mysql-devel gcc libxslt-devel libxml2-devel libffi-devel openssl-devel python-devel
# yum install zlib-devel
# yum install bzip2-devel
# yum install openssl-devel
# yum install ncurses-devel
# yum install sqlite-devel
# yum install readline-devel
# yum install tk-devel
# yum install mysql-devel
# yum install gcc
# yum install make (centos可能已自帶,不用安裝)
安裝Scrapy必裝:
# yum install libxslt-devel
# yum install libxml2-devel
# yum install libffi-devel
2. 升級Python
(下載地址:https://www.python.org/ftp/python/2.7.6/Python-2.7.6.tgz)
Centos6.5系統自帶python2.6.6,需要升級到Python2.7
安裝庫文件:yum install zlib-devel openssl-devel sqlite-devel
下載:wget https://www.python.org/ftp/python/2.7.6/Python-2.7.6.tgz
解壓:tar zxvf Python-2.7.6.tgz
安裝:
cd Python-2.7.6
./configure --prefix=/usr/local/python
make
make install
創建軟鏈接:ln -sf /usr/local/python/bin/python2.7 /usr/bin/python
修正yum:修改文件/usr/bin/yum的第一行(#!/usr/bin/python改為#!/usr/bin/python2.6)
3. 安裝setuptools和pip
使用pip安裝python模塊時出現報錯,建議先更新setuptools和pip來排除。
下載地址:https://pypi.python.org/packages/source/s/setuptools/setuptools-18.4.tar.gz
解壓後安裝:python setup.py install(或pip install --upgrade setuptools)
下載地址:https://pypi.python.org/packages/source/p/pip/pip-7.1.2.tar.gz
解壓後安裝:python setup.py install(或pip install --upgrade pip)
4. 安裝scrapy+selenium+phantomjs
下載地址:https://pypi.python.org/packages/source/S/Scrapy/Scrapy-1.0.3.tar.gz
解壓後安裝:python setup.py install
(也可使用pip install scrapy
命令安裝)
註意:使用pip install scrapy
安裝時可能由於網絡異常或下載安裝其他依賴庫時報錯,可以單獨下載依賴庫進行安裝。
selenium+phantomjs按需安裝,如果使用了phantomjs類瀏覽器來解析js則安裝:
下載地址:https://pypi.python.org/packages/source/s/selenium/selenium-2.48.0.tar.gz
解壓後安裝:python setup.py install
(也可使用pip install selenium
命令安裝)
下載地址:
https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-1.9.8-linux-i686.tar.bz2
wget https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-1.9.8-linux-i686.tar.bz2
tar jxvf phantomjs-1.9.8-linux-i686.tar.bz2
mv phantomjs-1.9.8-linux-i686 /usr/local/src/phantomjs
ln -sf /usr/local/src/phantomjs/bin/phantomjs /usr/local/bin/phantomjs
chmod a+x phantomjs
5. 安裝MySQL-python
Python使用MySQL數據庫,需要安裝該模塊。
下載地址:
https://pypi.python.org/packages/source/M/MySQL-python/MySQL-python-1.2.5.zip
解壓後安裝:python setup.py install
pip install mysql-python
6. 安裝MySQL
下載地址:
http://dev.mysql.com/get/Downloads/MySQL-5.6/MySQL-5.6.27-1.linux_glibc2.5.x86_64.rpm-bundle.tar
解壓後安裝使用rpm命令安裝
rpm MySQL-server-5.6.27-1.linux_glibc2.5.x86_64.rpm
rpm MySQL-client-5.6.27-1.linux_glibc2.5.x86_64.rpm
rpm MySQL-devel-5.6.27-1.linux_glibc2.5.x86_64.rpm
rpm MySQL-shared-5.6.27-1.linux_glibc2.5.x86_64.rpm
4 部署註意點
1、不要忘記修改環境變量
2、安裝MySQL後需要修改默認字符集為utf8
centos環境的python、scrapy部署