當伺服器在做資料遷移的時候，老是要安裝相關的scrapy軟體，今天來整理一下。

安裝依賴

yum update -y
yum groupinstall 'Development Tools'
yum install gcc gcc-devel -y
yum install libxml2 libxml2-devel -y
yum install libxslt libxslt-devel -y
yum install openssl openssl-devel -y
yum install libffi libffi-devel -y
yum install sqlite sqlite-devel
 
 -y

安裝python3

tar -zxvf Python-3.6.5.tgz
cd Python-3.6.5
./configure --prefix=/usr/local/python3
make && make install

新增軟連線

ln -s /usr/local/python3/bin/python3 python3
ln -s /usr/local/python3/bin/pip3 pip3

這裡筆者習慣使用pip3而不是pip

檢查安裝

python3

Python 3.6.0 (default, Jun  1 2018, 23:15:53)                                                                                                             
[GCC 4.8
 
.5 20150623 (Red Hat 4.8.5-11)] on linux                                                                                                          
Type "help", "copyright", "credits" or "license" for more information.                                                                                    
>>>  
 

安裝Twisted

wget https://twistedmatrix.com/Releases/Twisted/18.4/Twisted-18.4.0.tar.bz2
tar jxvf Twisted-18.4.0.tar.bz2
cd Twisted-18.4.0
python setup.py install

安裝scrapy

pip3 install lxml
pip3 install scrapy
# 建立軟連線
ln -s /usr/local/python3/bin/scrapy /usr/bin/scrapy

檢查安裝

scrapy -v

Scrapy 1.5.0 - no active project                                                                                                                          

Usage:                                                                                                                                                    
  scrapy <command> [options] [args]                                                                                                                       

Available commands:                                                                                                                                       
  bench         Run quick benchmark test                                                                                                                  
  fetch         Fetch a URL using the Scrapy downloader                                                                                                   
  genspider     Generate new spider using pre-defined templates                                                                                           
  runspider     Run a self-contained spider (without creating a project)                                                                                  
  settings      Get settings values                                                                                                                       
  shell         Interactive scraping console                                                                                                              
  startproject  Create new project                                                                                                                        
  version       Print Scrapy version                                                                                                                      
  view          Open URL in browser, as seen by Scrapy                                                                                                    

  [ more ]      More commands available when run from project directory

常用庫安裝

pip3 install requests
pip3 install selenium
pip3 install beautifulsoup4
pip3 install pyquery
pip3 install pymongo
pip3 install redis
pip3 install flask
pip3 install django
pip3 install jupyter