【Tools】Anaconda Operaction

專為資料科學和機器學習工作流程而設計，是一個開源包管理器，環境管理器，以及Python和R程式語言的分發。它通常用於大規模資料處理，科學計算和預測分析。pip install xxx ,在特定環境裡使用pip，下載的包會存在特定環境的目錄裡面，例如:D:\Anaconda3\envs\nlp\Lib\site-packages\fasttext； conda install xxx ，不管在什麼環境下載的包，都統一放在一個目錄裡面：D:\Anaconda3\pkgs\fasttext， 如果另一個環境也需要這個包，不需要重新下載；

Anaconda:

• 新增conda 映象源:

conda config --add channels http://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/

conda config --add channels http://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/

conda config --add channels http://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/Paddle/

conda config --set show_channel_urls yes

#-------

conda config --add channels conda-forge   #新增conda 源

#檢視所有的 conda .condarc 配置檔案內容

conda config --get

• 要充分利用conda 包的好處，遇到需要安裝什麼包的優選conda：https://anaconda.org/wdroz/openpose-gpu
• 通過原始碼編譯的是在其他已經都編譯的沒有的狀況下，或者需要修改原始碼
• 設定不進入conda環境，新版的anaconda（python3.7）安裝後自動進入虛擬環境base，執行命令：

​ conda config --set auto_activate_base false

conda package

• A conda package is a compressed tarball file (.tar.bz2) or .conda file that contains:

  • system-level libraries.
  • Python or other modules.
  • executable programs and other components.
  • metadata under the info/ directory.
  • a collection of files that are installed directly into an install prefix.

• package structure:

  ├── bin            #relevant binaries for package
  
  │   └── pyflakes
  
  ├── info     # package metadata
  
  │   ├── LICENSE.txt
  
  │   ├── files
  
  │   ├── index.json
  
  │   ├── paths.json
  
  │   └── recipe
  
  └── lib       # relevant library files
  
      └── python3.5
  
  

  • metadata contain dependencies to system core, low-level libraries and can contain links to software files that are automatically downloaded when executed.

  • index.json

    • metadata about the package including platform, version, dependencies, and build info

  {
  
    "arch": "x86_64",
  
    "build": "py37hfa4b5c9_1",
  
    "build_number": 1,
  
    "depends": [
  
      "depend > 1.1.1"
  
    ],
  
    "license": "BSD 3-Clause",
  
    "name": "fun-packge",
  
    "platform": "linux",
  
    "subdir": "linux-64",
  
    "timestamp": 1535416612069,
  
    "version": "0.0.0"
  
  }
  
  

  • paths.json
    • a list of files in the package, along with their associated SHA-256, size in bytes, and the type of path (eg. hardlink vs. softlink)

  {
  
    "paths": [
  
      {
  
        "_path": "lib/python3.7/site-packages/fun-packge/__init__.py",
  
        "path_type": "hardlink",
  
        "sha256": "76f3b6e34feeb651aff33ca59e0279c4eadce5a50c6ad93b961c846f7ba717e9",
  
        "size_in_bytes": 2067
  
      },
  
      {
  
        "_path": "lib/python3.7/site-packages/fun-packge/__config__.py",
  
        "path_type": "hardlink",
  
        "sha256": "348e3602616c1fe4c84502b1d8cf97c740d886002c78edab176759610d287f06",
  
        "size_in_bytes": 87519
  
      },
  
      ...
  
  }
  
  

Conda Dir

• root_dir:
• pkgs: contains decompressed packages, ready to be linked in conda environments. Each package resides in a subdirectory corresponding to its canonical name.
• envs: conda environments
  • bin
  • include
  • share

.condarc

• Where conda looks for packages.
• how conda uses a proxy server.
• Where conda lists known environments.
• Whether to update the Bash prompt with the currently activated environment name.
• Whether user-built packages should be uploaded to Anaconda.org.
• What default packages or features to include in new environments.

Conda 下載流程

Conda follows these steps when installing a package:

1. Downloading and processing index metadata.
2. Reducing the index.
3. Expressing the package data and constraints as a SAT problem.
4. Running the solver.
5. Downloading and extracting packages.
6. Verifying package contents.
7. Linking packages from package cache into environments.

包首先現在到這個目錄下

環境打包

#  1.建立環境的快照或備份

conda create --name snapshot --clone myenv

#  2. 複製環境  相同作業系統

# 生成 spec list 檔案

conda list --explicit > spec-list.txt

pip freeze > requirements.txt

# 建立環境

conda create --name myname --file spec-list.txt

#  3. 生成environment.yml 跨平臺作業系統復現環境

# 匯出environment.yml 檔案， 包括 pip 安裝軟體

conda env export > environment.yml

#建立環境

conda env create -f environment.yml

pip install -r requirements.txt

#  4.Pack 打包，包括該環境中安裝的軟體包的所有二進位制檔案， conda-pack 指定平臺和作業系統，目標計算機必須集有與源計算機相同的平臺和作業系統

# 安裝

conda install -c conda-forge conda-pack

# 打包一個環境

# pack environment my_env into my_env.tar.gz

conda pack -n my_env

# pack environment my_env into out_name.tar.gz

conda pack -n my_env -o out_name.tar.gz

# pack environment located at an explicit path into my_env.tar.gz

conda pack -p /explicit/path/to/my_env

# 建立環境

mkdir -p my_env

tar -xzvf my_env.tar.gz -C my_env

# Use Python without activating or fixing the prefixes. Most Python

# libraries will work fine, but things that require prefix cleanups

# will fail.

./my_env/bin/python

# Activate the environment. This adds `my_env/bin` to your path

source my_env/bin/activate

# Run Python from in the environment

(my_env) $ python

# Cleanup prefixes from in the active environment.

# Note that this command can also be run without activating the environment

# as long as some version of Python is already installed on the machine.

(my_env) $ conda-unpack

包管理

#方式一

#	資料夾那放到對應python 環境目錄下lib->sitepackages然後import 就行。例如 winguiauto.py 檔案放在Anaconda3\envs\pytorch-openpose\Lib\site-packages， 直接import winguiauto ，可能需要重啟下

#Tips  python 目錄(pytorch-openpose) >where python

	#C:\Users\dell\Anaconda3\envs\pytorch-openpose\python.exe

	#C:\Users\dell\Anaconda3\python.exe	#C:\Users\dell\AppData\Local\Microsoft\WindowsApps\python.exe

#方式二： 批量包下載

# 新建 .txt 檔案 內容（h5requipy，numpy，tensorflow==2.0.0-alpha0）單獨成行，

pip install -r requirement.txt #進行下載  -i https://pypi.tuna.tsinghua.edu.cn/simple/

#方式三： 直接 conda install / pip install 進行下載

#方式四： 離線下載

#	直接下載的whl檔案，通過pip install （路徑+whl檔名）

#	可以下載到本地 anaconda\install\Lib\site-packages路徑下，或者線上下載安裝

#檢視版本

conda --version

conda update conda

#環境管理

conda create --name pysot python=3.7

pip install -r requirements.txt

conda info --envs        #conda env list

conda activate pysot

conda deactivate pysot

#根據environment.yml 建立環境

conda env create -f environment.yml  #yml 第一行是環境名稱

conda remove --name ENVNAME --all

#環境遷移

#包管理

conda search package

conda install package

conda uninstall package --name env

conda install scipy=0.15.0  #下載指定版本

conda list  # list package

conda list -n env  #檢視一個環境中所有的包

#刪除包

conda remove -n myenv scipy

#install with channel

conda install scipy --channel conda-forge

#本地下載   本地下載不會解決包依賴問題

conda install /packages-path/packages-filename.tar

conda config --set env_prompt '({name})'

檢視包依賴

• The tarball is unarchived in the pkgs directory
• Files are hard-linked to the install path
• Shebang lines and other instance of a place-holder prefix are replaced with the install prefix
• The metadata is updated, so that conda knows that it is installed
• the process of command conda create -n py3k python=3.7
• Environments are simple: just link the package to a different directory
• Hand-links are very cheap, and very fast
• Conda environments are completely independent installations of everything
  • No fiddling with pythonPath or symlinking site-package
• Activating an environment just means changing your path so that its bin/ or scripts/ comes first

To find what packages are depending on a specific package in your environment, there is not one specific conda command. It requires a series of steps:

1. List the dependencies that a specific package requires to run: conda info package_name
2. Find your installation’s package cache directory: conda info
3. Find package dependencies. By default, Anaconda/Miniconda stores packages in ~/anaconda/pkgs/ (or ~/opt/pkgs/ on macOS Catalina). Each package has an index.json file which lists the package’s dependencies. This file resides in ~anaconda/pkgs/package_name/info/index.json.
4. Now you can find what packages depend on a specific package. Use grep to search all index.json files as follows: grep package_name ~/anaconda/pkgs/*/info/index.json

包匯入原理:

• 怎麼載入？——載入方式和用法

  1. import package 讀這個包的__init__.py，也就是說匯入包的本質是執行包下面的__init__.py檔案，執行結束後會包名的目錄下生成一個"pycache / init.cpython-36.pyc" 檔案
  2. import module 讀整個模組的內容
  3. import package1.package2.package3.module4　package讀取__init__.py，module讀取整個模組內容，按順序讀注意：因為package是讀取__init__.py，所以呼叫的時候必須在__init__.py有引用的東西才能呼叫，否則會報錯。
  4. 從指定路徑匯入:

  import sys
  
  sys.path.append('/path/to/folder/containing/my_package')  #注意是要匯入包的上級目錄
  
  import my_package
  
  

• 案例

├── sub_ui.py├── main.py├── test/│   └── test1.py└── model/    └── model1.py#-----  同級目錄匯入 main.py匯入sub_ui.pyimport sub_ui  或from sub_ui import *#-----  子目錄，main.py匯入model/model1.py需要在model/下建立__init__.py空檔案，讓編譯器認為這是一個模組。    import model.model1或    from model.model1 import *#----   跨目錄匯入 跨目錄，model1.py匯入test/test1.py需要在test1/下建立__init__.py空檔案import sys sys.path.append("..") import test.test1 或者 import sys sys.path.append("..") import test.test1 import *

• 　從哪裡匯入:python中，每個.py檔案被稱之為模組，每個具有__init__.py檔案的目錄被稱為包。只要模組或者包所在的目錄在sys.path中，就可以使用import 模組或import 包來使用.對於python來說，所有被載入到記憶體的模組都是放在sys.modules裡面，所以執行import時會首先去該列表中查詢是否已新增。如果已經在sys.modules中，那就簡單了，只需要將該module的name新增到我們正在呼叫該module的本地空間中。如果還沒有放在sys.modules中，就需要在sys.path所有路徑的目錄中去按順序查詢該模組的檔案，這些檔案一般字尾為".py"、".pyo"、".pyc"、".pyd"、".dll"，找到這些模組後就可以將這些模組新增到sys.modules中，再將module name匯入到本地。

>>> import sys>>> sys.modules{'copy_reg': <module 'copy_reg' from '/usr/lib/python2.7/copy_reg.pyc'>, 'sre_compile': <module 'sre_compile' from '/usr/lib/python2.7/sre_compile.pyc'>, '_sre': <module '_sre' (built-in)>, 'encodings': <module 'encodings' from '/usr/lib/python2.7/encodings/__init__.pyc'>, 'site': <module 'site' from '/usr/lib/python2.7/site.pyc'>, '__builtin__': <module '__builtin__' (built-in)>, 'sysconfig': <module 'sysconfig' from '/usr/lib/python2.7/sysconfig.pyc'>, '__main__': <module '__main__' (built-in)>, 'encodings.encodings': None, 'abc': <module 'abc' from '/usr/lib/python2.7/abc.pyc'>, 'posixpath': <module 'posixpath' from '/usr/lib/python2.7/posixpath.pyc'>, '_weakrefset': <module '_weakrefset' from '/usr/lib/python2.7/_weakrefset.pyc'>, 'errno': <module 'errno' (built-in)>, 'encodings.codecs': None, 'sre_constants': <module 'sre_constants' from '/usr/lib/python2.7/sre_constants.pyc'>, 're': <module 're' from '/usr/lib/python2.7/re.pyc'>, '_abcoll': <module '_abcoll' from '/usr/lib/python2.7/_abcoll.pyc'>, 'types': <module 'types' from '/usr/lib/python2.7/types.pyc'>, '_codecs': <module '_codecs' (built-in)>, 'encodings.__builtin__': None, '_warnings': <module '_warnings' (built-in)>, 'genericpath': <module 'genericpath' from '/usr/lib/python2.7/genericpath.pyc'>, 'stat': <module 'stat' from '/usr/lib/python2.7/stat.pyc'>, 'zipimport': <module 'zipimport' (built-in)>, '_sysconfigdata': <module '_sysconfigdata' from '/usr/lib/python2.7/_sysconfigdata.pyc'>, 'mpl_toolkits': <module 'mpl_toolkits' (built-in)>, 'warnings': <module 'warnings' from '/usr/lib/python2.7/warnings.pyc'>, 'UserDict': <module 'UserDict' from '/usr/lib/python2.7/UserDict.pyc'>, 'encodings.utf_8': <module 'encodings.utf_8' from '/usr/lib/python2.7/encodings/utf_8.pyc'>, 'sys': <module 'sys' (built-in)>, 'codecs': <module 'codecs' from '/usr/lib/python2.7/codecs.pyc'>, 'readline': <module 'readline' from '/usr/lib/python2.7/lib-dynload/readline.x86_64-linux-gnu.so'>, '_sysconfigdata_nd': <module '_sysconfigdata_nd' from '/usr/lib/python2.7/plat-x86_64-linux-gnu/_sysconfigdata_nd.pyc'>, 'os.path': <module 'posixpath' from '/usr/lib/python2.7/posixpath.pyc'>, '_locale': <module '_locale' (built-in)>, 'sitecustomize': <module 'sitecustomize' from '/usr/lib/python2.7/sitecustomize.pyc'>, 'signal': <module 'signal' (built-in)>, 'traceback': <module 'traceback' from '/usr/lib/python2.7/traceback.pyc'>, 'linecache': <module 'linecache' from '/usr/lib/python2.7/linecache.pyc'>, 'posix': <module 'posix' (built-in)>, 'encodings.aliases': <module 'encodings.aliases' from '/usr/lib/python2.7/encodings/aliases.pyc'>, 'exceptions': <module 'exceptions' (built-in)>, 'sre_parse': <module 'sre_parse' from '/usr/lib/python2.7/sre_parse.pyc'>, 'os': <module 'os' from '/usr/lib/python2.7/os.pyc'>, '_weakref': <module '_weakref' (built-in)>}>>> sys.path['', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/home/ldd/.local/lib/python2.7/site-packages', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/gtk-2.0']

Error record:

• Missing dependencies for SOCKS support. 解決方法： unset all_proxy

linux 安裝Anaconda

---

#本地安裝pip install ~/Dowloads/torch-0.2.0.post3-cp35-cp35m-manylinux1_x86_64.whl#conda 安裝本地包conda install --use-local pytorch-0.4.0-py35_cuda8.0.61_cudnn7.1.2_1.tar.bz2#change the channelconda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/conda config --set show_channel_urls yes#下載安裝curl -O https://repo.anaconda.com/archive/Anaconda3-5.1.0-Linux-x86_64.sh#通過SHA-256校驗和通過加密雜湊驗證來驗證安裝程式的資料完整性sha256sum Anaconda3-5.1.0-Linux-x86_64.sh#安裝bash Anaconda3-5.1.0-Linux-x86_64.sh#啟用環境，否則無法找到conda命令source ~/.bashrcconda info#用來檢視conda 內部安裝環境版本等資訊。#驗證安裝是否正確 羅列conda 安裝所有軟體conda list#建立 一個環境my_env  Python版本為3conda create --name my_env python=3   #使用 sudo 命令建立，否則無法在環境中註冊#在環境my_env35中安裝numpy包 conda install --name my_env35 numpy#啟用環境 ，這樣會有個字首#更新 conda使用程式conda update conda#更新 或解除安裝 anaconda 安裝包conda remove -n envname --all   #delete all the package and environmentconda update anaconda conda install anaconda-cleanrm -rf anacondasudo gedit ~/.bashrc #刪除環境變數source ~/.bashrc #使環境變數生效

Pycharm ：

#PyCharm：https://www.jetbrains.com/pycharm/download/#section=linuxcd downloads/pycharm-2018.1/binbash pycharm.sh#進入file setting 設定選擇編譯器

Win環境配置

F:\Anaconda3F:\Anaconda3\ScriptsF:\Anaconda3\Library\bin#shell 中加入這三個目錄，才可以在cmd 中使用conda

Jupyter使用

若想要關閉已經開啟的終端和“ipynb”格式的筆記本，僅僅關閉其頁面是無法徹底退出程式的，需要在Running頁面點選其對應的“Shutdown”。

• 安裝： conda install jupyter notebook

• 修改預設的主目錄：jupyter notebook --generate-config

• 啟動目錄： jupyter notebook， http://localhost:8888； 可以對埠進行對映處理

  • 指定埠號來啟動： jupyter notebook --port <port_number>
  • 啟動服務但不開啟網頁： jupyter notebook --no-browser
  • 在conda 虛擬環境中執行啟動 jupyter 命令，kernel 會自動關聯所有的conda環境；

• 幫助命令: jupyter notebook --help

• 將你conda建立的環境與Jupyter Notebook相關聯： conda install nb_conda

• 載入本地python檔案：%load Python檔案的絕對路徑

• 直接執行本地python檔案：%run Python檔案的絕對路徑； !python3 Python檔案的絕對路徑； !shell命令；

• 學習連結： https://www.jianshu.com/p/91365f343585#conda

• ModuleNotFoundError: No module named 'jupyter_nbextensions_configurator'

  • pip install jupyter_nbextensions_configurator
    
    

• Error executing Jupyter command 'notebook': [Errno 2] No such file or directory

  • jupyter notebook --port=11220 --allow-root #不適用sudo 許可權就可以執行了

• Jupyter 常用快捷鍵

  ctrl + / #進行註釋

  cell操作
  
  Ctrl+enter 執行本cell

shift+enter 執行本cell且 向下建立一個新cell

  A 向上建立一個cell

B 向下建立一個cell

d d 刪除cell

O 收起output 或者開啟output

  在cell裡面的快捷鍵
  
  Esc+m m 把cell切換至markdown模式
  
  Esc+y y 把cell切換至code模式
  
  Esc+l l 顯示行數
  
  Esc + F 在程式碼中查詢、替換，忽略輸出。
  
  Esc + O 在 cell 和輸出結果間切換。

  複合快捷鍵
  
  shift 選中多個cell
  
  然後執行 shift+M 合併選中cell

Error Record

.1. 代理問題

#Anaconda 使用conda指令出現ValueError: Unable to determine SOCKS version from socks://127.0.0.1:1080/ 解決辦法printenv | grep all_proxy  #檢視代理unset all_proxy && unset ALL_PROXY   #export all_proxy=""  export all_proxy="socks5://127.0.0.1:1080"export http_proxyexport https_proxyunset http_proxyunset https_proxyconda install pysocks     #下次建立環境是同時下載 pip和pysocks 包source ~/.bashrc

每次在虛擬環境內部使用pip的時候都需要獲取虛擬環境內部的 pip命令路徑：/Users/stefanannihilater/anaconda3/envs/setests3/bin/pip，顯然這樣非常的繁瑣。

如果想在虛擬環境內部使用簡單的pip install package呼叫虛擬環境內部的pip命令的話，只需要我們在建立虛擬環境的時候指定pip只對虛擬環境生效，而不影響全域性庫：

conda create -n 虛擬環境名 pip pysocks

• pip：是為了指定pip命令只對當前虛擬環境生效
• pysocks：是pip命令的依賴，如果不寫，在虛擬環境內使用 pip命令的時候會出現Missing dependencies for SOCKS support.的報錯。

下次可以把包下載到本地，然後區域網上傳伺服器，減少伺服器流量開銷；

• 錯誤記錄SSL

  • 方式一：環境變數是否配好：

  D:\Anaconda3; D:\Anaconda3\Scripts; D:\Anaconda3\Library\bin
  
  

  • 方式二： 映象網址是否可以訪問
  • 方式三： 安裝openssl 工具：地址

.2. cudatoolkit=11.1. 下載失敗

conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/conda config --set show_channel_urls yesconda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch/conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/conda install pytorch torchvision torchaudio cudatoolkit=11.1

.3. torchvision 庫無法使用

Download error with CIFAR10/Mnist based on torchvision.datasets

• uninstall and reinstall torchvision

conda unstall torchvisionconda install pytorch torchvision torchaudio cudatoolkit=11.1

.4. pip 代理

.5. GPU資源問題

RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED；

• GPU被沾滿了，沒有分配資源；