Interactive Apps

Click on OOD Dashboard's Interactive Apps dropdown menu will display a number of browser-based GUI and server-dependent apps. This is an alternative to the traditional X11 GUI forwarding via external terminal.

*At this moment, the Desktop app is not yet setup, and a number of GUI-based apps (ANSYS, MatLab, and COMSOL).

  • Abaqus/CAE
  • VSCode Server
  • Jupyter and Spark
  • Jupyter Notebook
  • Jupyter for Tensorflow
  • RStudio Server
  • TensorBoard

The interactive apps will have an form for users to specify resource requirement in a manner similar to the well known JupyerHub interface for Palmett (*JupyterHub will eventually be retired). In the remainder of this section, a brief introduction to each interactive app will be provided, and any special consideration, if necessary, will be provided.

ABAQUS/CAE

ABAQUS is a Finite Element Analysis software used for engineering simulations. Currently, ABAQUS versions 6.10, 6.13, 6.14 are available on Palmetto cluster as modules.

$ module avail abaqus

abaqus/6.10 abaqus/6.13 abaqus/6.14

To see license usage of ABAQUS-related packages, you can use the lmstat command:

/software/USR_LOCAL/flexlm/lmstat -a -c /software/USR_LOCAL/flexlm/licenses/abaqus.dat

Running ABAQUS interactive viewer

To run the interactive viewer, you must log in with X11 tunneling enabled, and then ask for an interactive session

VSCode

VSCode is a text editor and debugger for working with many different language. VSCode comes with github integration by default and allows for language specific syntax highlighting and autocomplete with IntelliSense.

Currently, VSCode Server version 3.10.2 is available on the Palmetto cluster.

Starting a VSCode Server

Start by heading to the Open OnDemand website where on the nav bar you can find the interactive apps.

Select Code Server (VSCode)

Once you have selected to create a VSCode server you will be prompted with a GUI where you can choose what hardware and software your server will have.

Normally, the number of resource chunks should be 1 unless you are working with a clustered computing software.

Once you launch your server you will be taken to your current list of interactive sessions. Your job will be queued until resources are available to handle your request.

Once your resources are available you will be able to connect to your VSCode server with a button that says Connect to VS Code.

If you get stuck in the queuing phase, try lowering the resources your server is using or making sure interconnect = any in the server request options.

VSCode Dashboard

Once you have connect to your VSCode server you will be sent to the VSCode dashboard. Here you can choose a directory to start working in by clicking open folder then specifying the directory.

Additionally in the VSCode server you have access to the terminal. Clicking on the context menu on the top leftmost position of the screen will bring up a menu from there you are able to open up a terminal by following the path Terminal->New Terminal.

Once you are in your terminal you can run any of the software that you loaded while requesting the server.

$ module avail
Currently Loaded Modules:
1) anaconda3/2019.10-gcc/8.3.1   2) libxml2/2.9.10-gcc/8.3.1   3) openmpi/3.1.4-gcc/8.3.1-ucx 

Utilizing VSCode features

The leftmost area of the screen has a few icons where you can use some of the features of the VSCode text editor. In order of appearence these features are :
1) Directory browser
2) Find and replace tool
3) Github integration
4) Debugger
5) Extension marketplace

The first item is the directory browser this is where you can create, edit, or delete files.

The second item is a find and replace tool that works throughout your entire working directory. We can see here that searching for return showed up in 3 different files in our current working directory.

The third item allows VSCode to integrate with github, intially you can start a repository within your current working directory or if you are already in a git repository you can see file changes made and perform git operations.
Here we can see that the file readme.txt has been added and Test.c has been modified.

The fourth item is the built in debugger. The debugger requires extensions to use and compatibility with Palmetto's VSCode server is not guaranteed.

The fifth item is the extension marketplace where you can install community made extensions to modify your version of VSCode.These extensions differ from the desktop version of VSCode as some are closed source. The extensions will persist throughout vscode server sessions as they are stored in your home directory at ~/.local/share/code-server/extensions

You can see further documentation and VSCode server specific commands at the code server documentation website

JupyterLab

JupyterLab on the Palmetto Cluster

OpenOD allows users to interact with the Palmetto cluster from their web browsers using the Jupyter Lab interface, and to prototype and develop code in Python, R, MATLAB and several other languages.

Launching a Jupyter Server on Palmetto

  1. Go to the OpenOD website.
  2. Log in with your Palmetto user ID and password:
  3. Once you are logged in, click on Interactive Apps on the top navigation bar.
  4. Click on the Jupyter Notbook link.

  1. Select the resources (CPU cores, memory, walltime, etc.,) required for your session.

  1. Once you launch your server you will be taken to your current list of interactive sessions. Your job will be queued until resources are available to handle your request.

  1. Once your server has had it's resources allocated you can connect to your server with a button that says Connect to Jupyter, your browser will then be sent to the JupyterLab dashboard.

JupyterLab dashboard

After your Jupyter server successfully launched on Palmetto, your browser will present you with a JupyterLab dashboard.

The dashboard is the primary interface of the notebook server. We are using the JupyterLab version of the dashboard to allow users more control over the notebook workspace inside the browser.

Files and folders

  • The files and folders are shown on the left vertical partition of the JupyterLab dashboard. Right-click on a file or folder will give you a number of operational options, as shown in the figure below.

  • To upload small files from your computer to Palmetto via the dashboard, click on the bold up arrow at the top of the file/folder areas (the red circle).

Launchers

  • Launchers are tabs internal to the browser presenting the JupyterLab dashboard. Launcher resperents a significant change to the traditional one-browser-tab-per-notebook/terminal presentation of Jupyter Notebook.
  • On Palmetto, there are three types of Launchers: Notebook, Console, and Other.
  • Other includes browser-based terminal, text editors, and markdown editor.

  • The Notebook launcher will list all available Jupyter notebook kernels that your account has.
  • Click on a kernel will launch the corresponding notebook in the current launcher tab.

  • The Console launcher will launch a command-based interface for the available Jupyter notebook kernels that your account has (note the similar list as the Notebook launcher).

  • An example of a terminal launcher tab is shown below, from the Other launcher.

The most significant impact of the JupyterLab dashboard is its ability to rearrange multiple launcher tabs within the same browser, similar to a typical programming IDE. By dragging the tabs around, you can have a versatile work environment.

Adding new notebook kernels

In addition to the default kernels provided (Python, R, and MATLAB), it is possible to create your own kernels. To install kernels for other languages, see the setup instructions for the language here.

For custom Python kernels, we recommend using Conda environments, and ipykernel to generate a kernel from a Conda environment:

` $ conda create --name myenv python=2.7 $ source activate myenv $ conda install jupyter $ python -m ipykernel install --user --name python_custom --display-name "My Python"

Resources and other readings

A collection of Jupyter notebooks created for research and teaching across several fields in scientific computing, data analysis and related fields is available here:

https://github.com/ipython/ipython/wiki/A-gallery-of-interesting-IPython-Notebooks

Jupyter and IPython documentation

The official Jupyter Notebook documentation:

https://jupyter-notebook.readthedocs.io/en/latest/

The documentation for IPython, the kernel used to create Python-based notebooks:

http://ipython.readthedocs.io/en/stable/index.html

For a list of packages useful for scientific computing and data analysis (and links to the relevant documentation), see here:

https://www.scipy.org/about.html

Spark + Jupyter

OpenOD has the ability to use the Python Spark library by using pre-configured JupyterLab sessions. Currently, Spark versions 2.4.8 and 3.1.2 are available on Palmetto

Spark is a high-level cluster computing system that has bindings for Python. It can be used to write programs that utilize parallelization for faster computation when working with big data or machine learning.

Launching a Jupyter Lab Server with spark on Palmetto

  1. Go to the OpenOD website.
  2. Log in with your Palmetto user ID and password:
  3. Once you are logged in, click on Interactive Apps on the top navigation bar.
  4. Click on the Jupyter + Spark link.

  5. Once you have arrived at the server configuration page, select the resources (CPU cores, memory, walltime, etc.,) required for your session. When using Spark you usually want more than 1 resource chunk. Here you can also choose what version of spark you would like to use.

  6. Once you launch your server you will be taken to your current list of interactive sessions. Your job will be queued until resources are available to handle your request.

  7. Once your server has had it's resources allocated you can connect to your server with a button that says Connect to Jupyter, your browser will then be sent to the JupyterLab dashboard.

Running Python with spark

When arriving to the Jupyter Lab dashboard you can either open an existing notebook using the file browser or create a new one with the launcher.

The launcher can be accessed by clicking the plus sign (+) at the top left of the dash board

From the launcher you can start a notebook or a Python terminal using the spark kernel. The Python kernel with spark will be named PySpark.

After clicking these you will be put into a Notebook or Console that is running the python + Spark kernel.

You can also change the kernel of an existing notebook to the Spark kernel by clicking on the kernel name at the top right of the notebook and selecting PySpark on the dropdown menu for kernels.

You can get started with programming with Python + Spark at the offical [Spark docs](https://spark.apache.org/docs/0.9.1/python-programming-guide.html

Tensorflow + Jupyter

This page explains how to install the TensorFlow package for use with GPUs on the cluster, and how to use it with OpenOD via Jupyter Lab.

This guide is created primarily for TensorFlow 2+. This version of TensorFlow requires AVX2 support from CPU, which is not available on the older nodes. Currently Palmetto nodes from Phase 12 and up support AVX2.

If you are using codes built using TensorFlow 1, please refer to this migration documentation to help with your code.

Installing TensorFlow GPU node

1) Request an interactive session on a GPU node. For example:

$ qsub -I -l select=1:ncpus=24:mem=125gb:ngpus=2:gpu_model=k40:interconnect=10ge,walltime=72:00:00

2) Load the Anaconda module:

$ module load anaconda3/2019.10-gcc/8.3.1 cuda/11.0.3-gcc/7.5.0 cudnn/8.0.0.180-11.0-linux-x64-gcc/7.5.0

3) Create a directory to store the Python virtual environment packages:

$ mkdir -p ~/software/venv
$ python3 -m venv --system-site-packages ~/software/venv/tf_gpu

4) Activate the virtual environment:

$ source ~/software/venv/tf_gpu/bin/activate

5) Install TensorFlow:

$ pip install --upgrade pip
$ pip install tensorflow==2.4

This will automatically install some packages that are required for Tensorflow, like SciPy or NumPy. If you need additional packages (for example, Pandas), you can type

$ pip install pandas

6) Install TensorFlow Jupyter Kernel and add the required libraries to the jhubrc file. More info on the jhubrc file can be found here.

$ python3 -m ipykernel install --user --name tf_gpu --display-name TensorflowGPU
$ echo "module load anaconda3/2019.10-gcc/8.3.1 cuda/11.0.3-gcc/7.5.0 cudnn/8.0.0.180-11.0-linux-x64-gcc/7.5.0" >> ~/.jhubrc

Installing TensorFlow for non-GPU node

1) Request an interactive session without GPU specification. For example:

$ qsub -I -l select=1:ncpus=24:mem=125gb:interconnect=10ge,walltime=72:00:00

2) Load the required modules

$ module load cuda/10.2.89-gcc/8.3.1

3) Create a directory to store the Python virtual environment packages:

$ mkdir -p ~/software/venv
$ python3 -m venv --system-site-packages ~/software/venv/tf_cpu

4) Activate the virtual environment:

$ source ~/software/venv/tf_cpu/bin/activate

5) Install TensorFlow:

$ pip install --upgrade pip
$ pip install tensorflow==2.4

This will automatically install some packages that are required for Tensorflow, like SciPy or NumPy. If you need additional packages (for example, Pandas), you can type

$ pip install pandas

6) Install TensorFlow Jupyter Kernel:

$ python3 -m ipykernel install --user --name tf_cpu --display-name TensorflowCPU

Test TensorFlow Jupyter Kernels

1) Log into OpenOD and click go to Interactive Apps on the nav bar then click Jupyter For TensorFlow.

2) When choosing server resources make sure you have GPU in your selection if you want to use the GPU TensorFlow kernel.

3) Load the modules needed for the version of tensorflow you are using by filling in the textbox. In this case we are loading the following modules

anaconda3/2019.10-gcc/8.3.1 cuda/11.0.3-gcc/7.5.0 cudnn/8.0.0.180-11.0-linux-x64-gcc/7.5.0

4) Set the path of Tensorflow virtual enviorement we created earlier. In our example it was created at

~software/venv/tf_gpu

5) Once your Jupyter Lab session has started, you should see the TensorFlow kernels in your list of kernels in the Launcher.

3) You are now able to launch a notebook using the one of the TensorFlow with GPU kernel:

For Tensorflow with GPU support, the notebook cell containing tf.config.list_physical_devices('GPU') will produce a non-empty list.

Example Deep Learning - Multiple Object Detections

This is a demonstration for the tensorflow gpu kernel. Steps for non-gpu kernel are similar.

1) Request an interactive session on a GPU node. For example:

$ qsub -I -l select=1:ncpus=16:mem=20gb:ngpus=1:gpu_model=p100,walltime=3:00:00

2) Load the Anaconda module:

$ module load cuda/10.2.89-gcc/8.3.1 cudnn/8.0.0.180-10.2-linux-x64-gcc/8.3.1 anaconda3/2019.10-gcc/8.3.1

3) Activate the conda environment:

$ source activate tf_gpu_env

4) Install supporting conda modules:

$ conda install Cython contextlib2 pillow lxml matplotlib utils pandas

5) Setup TensorFlow's Model directory:

$ cd
$ mkdir tensorflow
$ cd tensorflow
$ wget https://github.com/tensorflow/models/archive/master.zip
$ unzip master.zip
$ mv models-master models
$ module load protobuf/3.11.2-gcc/8.3.1
$ cd models/research
$ protoc object_detection/protos/*.proto --python_out=.
$ cp object_detection/packages/tf2/setup.py .
$ python -m pip install --user --use-feature=2020-resolver .
$ cd ~/tensorflow
$ cp /zfs/citi/deeplearning/multi_object.ipynb .

Open Jupyter Server, change into the tensorflow directory, then open and run the multi_object.ipynb notebook. <!--

Tensorboard

TensorFlow 2+ has tensorboard included with the installation package. To run TensorBoard, you can leverage the same notebook server.

1) Click the + sign near the top left corner of the Jupyter Lab interface to open the Launcher. Select a Terminal.

2) Run the following commands (assuming a tf_gpu installation.)

$ module load anaconda3/2019.10-gcc/8.3.1 cuda/11.0.3-gcc/7.5.0 cudnn/8.0.0.180-11.0-linux-x64-gcc/7.5.0
$ source ~/software/venv/tf_gpu/bin/activate
$ tensorboard --logdir logs --host 0.0.0.0

Pay attention to your allocated Palmetto node, as highlight by the red shape in the above image.

3) Follow the instructions shown in Socket Proxy Access to setup proxy access from your local computer. Open the Firefox browser and go to the node from step 2 at port 6006