Urika Training

From Research Computing @ TAMUQ
Jump to navigation Jump to search

Venue and Timings

Conference Room # 310, TAMUQ Building
Timings: 0830-0430pm
First Coffee Break at: 10am - 1015am
Lunch Break: 12pm to 01pm
Afternoon Coffee Break: 230pm - 0245pm

Path to shared files



#SBATCH --account=ut
#SBATCH --reservation=ut
#SBATCH --partition=l_long
#SBATCH --qos=lcustom3

How to request allocation?

salloc -N 4 --account=ut --reservation=ut --partition=l_long --qos=lcustom3 start_analytics

Create tunnel for jupyter notebooks

Instructions for setting up tunneling on raad2 for Jupyter notebooks

  1. Create a directory called "notebooks" (no quotes) in your home directory. Copy all the files from /lustre/share/urika_training/uxc_1.1/notebooks to this directory. Also, copy the file alldemo.yml from /lustre/share/urika_training/uxc_1.1/Anaconda to this directory.
  2. Create tunnel from laptop through raad2 elogin node to raad2-login1 OR raad2-login2
  3. Choose any 3 random number (port_num_1, port_num_2, port_num_3) b/w [10000 - 50000], these will be used as port numbers for tunneling.
  4. From a terminal session on your laptop:
(use your own login instead of jamaltb30!)
laptop$ ssh -L <port_num_1>:raad2-login2:22 -N -f <user_name>@raad2.qatar.tamu.edu
  1. This will return you to your laptop prompt, leaving a tunnel active. Then log directly into the login2 node:
laptop$ ssh -l jamaltb30 -L <port_num_2>:localhost:<port_num_3> -p <port_num_1> localhost
  1. Load the analytics module
module load analytics
  1. Start the analytics package on login2 with the -d flag
jamaltb30@raad2-login2:~> start_analytics -d
  1. Create a conda environment to run the demos. You will only need to do this once.
jamaltb30@raad2-login2:~> conda env create -f alldemo.yml
jamaltb30@raad2-login2:~> exit
  1. Start up the analytics cluster, specifying the communications ports and conda environment
jamaltb30@raad2-login2:~> salloc -N 4 -p l_long --qos=lcustom3 --account=ut --reservation=ut start_analytics --login-port <port_num_3> --ui-port <port_num_2> --dask-env alldemo
  1. export the python environment to all the compute nodes in the cluster
a_student@nid00032:~> export PYSPARK_PYTHON=$(which python)
  1. Run the jupyter notebook
jupyter notebook --port <port_num_2>

After some messages, it will give you a string with a passcode that you can paste into the address window of your local browser on your laptop.
The jupyter notebook should start up!