Setting up fastai on Google Cloud Platform

I recently followed along a well-written medium story about setting up Google Cloud machine for fast.ai. While it is a very detailed and nice guide, it worked for me once but it does not work anymore for me so I think the script used is broken and needs update. So I will try not to repeat what’s already been said, but will try to update or suggest slightly different ways of doing things. So I suggest first reading the original guide and then come back here to find out what I suggest differently.

At the point in the guide you have a compute engine running and supposed to run the following after connecting with ssh,

curl https://raw.githubusercontent.com/howkhang/fastai-v2-setup/master/setup.sh | bash

Please don’t run the above and continue reading. Currently starting jupyter notebook after running above script fails for me (let me know if you succeeds in working though). So I updated the script. Run the following in the terminal once you ssh into your GCP machine.

curl https://raw.githubusercontent.com/wontheone1/fast-ai-setup-on-gcp/e28519a5b592dc942879d3dc2a1851aac806c227/setup.sh | bash -x

And then ssh connection will drop automatically as the script runs sudo reboot. Now the setup is finished, you can ssh into it again and start notebook by jupyter notebook. Open another terminal and run

gcloud compute ssh [email protected] --ssh-flag="-L" --ssh-flag="8888:localhost:8888"

Then you can use the notebook on your localhost:8888 on your browser. Easy! isn’t it? If you only care about setting up and start using jupyter notebook then there’s nothing more for you to read but if you are interested then I will explain what I changed from original script and why.

The first difference I have from original script is following.

Original:

echo "c.NotebookApp.ip = '*'" >> ~/.jupyter/jupyter_notebook_config.py

Mine:

echo "c.NotebookApp.ip = '0.0.0.0'" >> ~/.jupyter/jupyter_notebook_config.py

That line of original script was the reason why install was failing for me and I learned from fastai forum that by changing * to 0.0.0.0 notebook will work again. And then the other change I made to original script is the next part

wget https://repo.continuum.io/archive/Anaconda3-5.0.1-Linux-x86_64.shbash Anaconda3-5.0.1-Linux-x86_64.sh -b

Since the latest version of Anaconda I can find is 5.3.0 and I wanted to delete the installation file once I finished installing Anaconda I changed the original script to following.

# INSTALL ANACONDA 5.3.0 

ANACONDA=Anaconda3-5.3.0-Linux-x86_64.sh 

Also some trivial things I changed from original script is that a) my script does NOT download the dogs and cats dataset unlike the original one. That’s because I thought not everyone who sets up fastai on GCP wants to do dogs and cats problem so I wanted this script to serve more general population.

Next part I replaced is the following,

echo 'export PATH=~/anaconda3/bin:$PATH' >> ~/.bashrcexport PATH=~/anaconda3/bin:$PATHsource ~/.bashrc

and my script is

sudo ln -s /home/$USER/anaconda3/etc/profile.d/conda.sh /etc/profile.d/conda.sh

The reason for this change is adding Anaconda bin directory to PATH env variable is not a recommended way of making conda command available. Refer this Github issue and change log in conda repository.

Most of other parts remain the same as in original script but small trivial things I added are

# enable nvcc --version commandsudo apt install nvidia-cuda-toolkit -y

# Kaggle clipip install --upgrade pippip install kaggle

These are just the things I find myself keep using when I setup a machine for machine learning and I think most of other people will find it useful as well. Last but not least, I use bash’s -x option to printout the all the commands in the script. I found it useful when modifying the setup script. I recommend you to also keep it that way since this kinda of script can be broken in the future since it depends too much on external world. For example, links can become unavailable or fastai’s future version may require different kinds of setup. So in case it breaks in the future, using -x option will be useful in debugging the script and updating it.

In case you want to download the cats and dogs data set like the original, you can run the following on your terminal. Doing so will also create a symlink for ~/data in ~/fastai/courses/dl1.

cd ~mkdir datacd datawget http://files.fast.ai/data/dogscats.zipunzip -q dogscats.zipcd ~/fastai/courses/dl1/ln -s ~/data ./

That’s all for explaining my script. I also want to talk about small tip on GCP. I needed to create and delete VM instance multiple times to test my script. And you could be creating / deleting VM multiple times for whatever reasons as well. What I recommend in those situation is not just create VM every time you have to. Because whenever you do it, you have to choose OS, number of CPU, memory, storage, network tag etc. Instead of doing that, create an Instance templates once and whenever you need to create a VM you can create it from the template so no need to choose all the options every time you do so.

To create an instance template, go to ‘Compute Engine > Instance templates’. Click ‘CREATE INSTANCE TEMPLATE’. Choose everything as you would when you create a single VM instance. Click the name of the template you created, Click ‘CREATE VM’. Decide name and click ‘CREATE’. That is it for creating instances from template.

I hope this was helpful to you, share with me if you have a better setup process.