Skip to content

OLCF-AMD.md

Latest commit

 

History

History
296 lines (230 loc) · 12.1 KB

OLCF-AMD.md

File metadata and controls

296 lines (230 loc) · 12.1 KB

OLCF Spock Tutorial

Last updated 2021-8-19

This document is built off of the excellent how-to guide created for Princeton's TigerGPU

Building the package

Login to Spock

First, login to the Spock headnode via ssh:

ssh -X <yourusername>@spock.olcf.ornl.gov

Note, -X is optional; it is only necessary if you are planning on performing remote visualization, e.g. the output .png files from the below section. Trusted X11 forwarding can be used with -Y instead of -X and may prevent timeouts, but it disables X11 SECURITY extension controls.

Sample installation on Spock

Check out the Code Repository

Next, check out the source code from github:

git clone https://github.com/PPPLDeepLearning/plasma-python
cd plasma-python

Install Miniconda

At the time of writing, Anaconda and Miniconda are not installed on Spock, therefore one of them must be manually downloaded. In their system documentation, AMD recommends downloading Miniconda.

To install Miniconda, download the Linux installer here and follow the installation instructions for Miniconda on this page

Once Miniconda is installed, create a conda environment:

conda create -n your_env_name python=3.8 -y

Then, activate the environment:

conda activate your_env_name

Ensure the following packages are installed in your conda environment:

pyyaml            # pip install pyyaml
pathos            # pip install pathos
hyperopt          # pip install hyperopt
matplotlib        # pip install matplotlib
keras             # pip install keras
tensorflow-rocm   # pip install tensorflow-rocm

Modules

In order to load the correct modules with ease, creating a profile is recommended

vim frnn_spock.profile

Write the following to the profile:

module load rocm
module load cray-python
module load gcc
module load craype-accel-amd-gfx908
module load cray-mpich/8.1.7
module use /sw/aaims/spock/modulefiles
module load tensorflow

# These must be set before running if wanting to use the Cray GPU-Aware MPI
# If running on only 1 GPU, there is no need to uncomment these lines

# export MPIR_CVAR_GPU_EAGER_DEVICE_MEM=0
# export MPICH_GPU_SUPPORT_ENABLED=1
# export HIPCC_COMPILE_FLAGS_APPEND="$HIPCC_COMPILE_FLAGS_APPEND -I${MPICH_DIR}/include -L${MPICH_DIR}/lib -lmpi -L/opt/cray/pe/mpich/8.1.7/gtl/lib -lmpi_gtl_hsa"

export MPICC="$(which mpicc)"

As of the latest update of this document (Summer 2021), the above modules correspond to the following versions on the Spock system, given by module list (Note that this list also includes the default system modules):

Currently Loaded Modules:
  1) craype/2.7.8      3) libfabric/1.11.0.4.75   5) cray-dsmml/0.1.5         7) xpmem/2.2.40-2.1_2.28__g3cf3325.shasta   9) cray-pmi/6.0.12      11) DefApps/default    13) cray-python/3.8.5.1  15) craype-accel-amd-gfx908  17) rocm/4.1.0
  2) craype-x86-rome   4) craype-network-ofi      6) perftools-base/21.05.0   8) cray-libsci/21.06.1.1                   10) cray-pmi-lib/6.0.12  12) PrgEnv-cray/8.1.0  14) gcc/10.3.0           16) cray-mpich/8.1.7         18) tensorflow/2.3.6

Build mpi4py

If wanting to run on multiple GPUs, mpi4py is needed. At the time of writing, a manual installation of mpi4py is needed on the Spock system. To install mpi4py, do the following:

# Ensure your conda environment is activated:
conda activate your_env_name

# Download mpi4py to your home directory
#cd ~
curl -O -L https://bitbucket.org/mpi4py/mpi4py/downloads/mpi4py-3.0.3.tar.gz

# Untar the file
tar -xzvf mpi4py-3.0.3.tar.gz

cd mpi4py-3.0.3

# Edit the mpi.cfg file
vim mpi.cfg

Include the following segment in the mpi.cfg file:

      [craympi]
      mpi_dir              = /opt/cray/pe/mpich/8.1.4/ofi/crayclang/9.1
      mpicc                = cc
      mpicxx               = CC
      include_dirs         = /opt/cray/pe/mpich/8.1.4/ofi/crayclang/9.1/include
      libraries            = mpi
      library_dirs         = /opt/cray/pe/mpich/8.1.4/ofi/crayclang/9.1/

Build and install mpi4py:

python setup.py build --mpi=craympi
python setup.py install

Next, install the plasma-python package:

#conda activate your_env_name
#cd ~/plasma-python
python setup.py install

Understanding and preparing the input data

Location of the data on Spock

Currently, no public data exists on Spock, but we leave this section in here for the user to understand the input data

The JET and D3D datasets contain multi-modal time series of sensory measurements leading up to deleterious events called plasma disruptions. The datasets are located in the /tigress/FRNN project directory of the GPFS filesystem on Princeton University clusters.

For convenience, create following symbolic links:

cd /tigress/<netid>
ln -s /tigress/FRNN/shot_lists shot_lists
ln -s /tigress/FRNN/signal_data signal_data

Configuring the dataset

All the configuration parameters are summarised in examples/conf.yaml. In this section, we highlight the important ones used to control the input data.

Currently, FRNN is capable of working with JET and D3D data as well as thecross-machine regime. The switch is done in the configuration file:

paths:
    ... 
    data: 'jet_0D'

Older yaml files kept for archival purposes will denote this data set as follow:

paths:
    ... 
    data: 'jet_data_0D'

use d3d_data for D3D signals, use jet_to_d3d_data ir d3d_to_jet_data for cross-machine regime.

By default, FRNN will select, preprocess, and normalize all valid signals available in the above dataset. To chose only specific signals use:

paths:
    ... 
    specific_signals: [q95,ip]

if left empty [] will use all valid signals defined on a machine. Only set this variable if you need a custom set of signals.

Other parameters configured in the conf.yaml include batch size, learning rate, neural network topology and special conditions foir hyperparameter sweeps.

Preprocessing the input data

Preprocessing the input data is currently not required on Spock as the data that is available is already preprocessed.

cd examples/
python guarantee_preprocessed.py

This will preprocess the data and save rescaled copies of the signals in /tigress/<netid>/processed_shots, /tigress/<netid>/processed_shotlists and /tigress/<netid>/normalization

Preprocessing must be performed only once per each dataset. For example, consider the following dataset specified in the config file examples/conf.yaml:

paths:
    data: jet_0D

Preprocessing this dataset takes about 20 minutes to preprocess in parallel and can normally be done on the cluster headnode.

Current signals and notations

Signal name Description
q95 q95 safety factor
ip plasma current
li internal inductance
lm Locked mode amplitude
dens Plasma density
energy stored energy
pin Input Power (beam for d3d)
pradtot Radiated Power
pradcore Radiated Power Core
pradedge Radiated Power Edge
pechin ECH input power, not always on
pechin ECH input power, not always on
betan Normalized Beta
energydt stored energy time derivative
torquein Input Beam Torque
tmamp1 Tearing Mode amplitude (rotating 2/1)
tmamp2 Tearing Mode amplitude (rotating 3/2)
tmfreq1 Tearing Mode frequency (rotating 2/1)
tmfreq2 Tearing Mode frequency (rotating 3/2)
ipdirect plasma current direction

Training and inference

Use the Slurm job scheduler to perform batch or interactive analysis on the Spock system.

Batch job

A sample batch job script for 1 GPU is provided in the examples directory and is called spock_1GPU_slurm.cmd. It can be run using: sbatch spock_1GPU_slurm.cmd Note that, the project/account (-A) and partition (`-p) arugments will need to reflect your project and assigned partition.

Some batch job tips:

  • For non-interactive batch analysis, make sure to allocate exactly 1 MPI process per GPU where X is the number of nodes for distibuted training and the total number of GPUs is X * 4. This configuration guarantees 1 MPI process per GPU, regardless of the value of X.
  • Update the num_gpus value in conf.yaml to correspond to the total number of GPUs specified for your Slurm allocation.

And monitor it's completion via:

squeue --me

Optionally, add an email notification option in the Slurm configuration about the job completion:

#SBATCH --mail-user=<userid>@email.com
#SBATCH --mail-type=ALL

Interactive job

Interactive option is preferred for debugging or running in the notebook, for all other case batch is preferred. The workflow is to request an interactive session for a 1 GPU interactive job:

salloc -t 02:00:00 -A <project_id> -N 1 --gres=gpu:1 --exclusive -p <partition> --ntasks-per-socket=1 --ntasks-per-node=1

Ensure the above modules are still loaded and reactivate your conda environmnt. Then, launch the application from the command line:

python mpi_learn.py

Visualizing learning

A regular FRNN run will produce several outputs and callbacks.

Custom visualization

You can visualize the accuracy of the trained FRNN model using the custom Python scripts and notebooks included in the repository.

Learning curves, example shots, and ROC per epoch

You can produce the ROC curves for validation and test data as well as visualizations of shots by using:

cd examples/
python performance_analysis.py

The performance_analysis.py script uses the file produced as a result of training the neural network as an input, and produces several .png files with plots as an output.

In addition, you can check the scalar variable summaries for training loss, validation loss, and validation ROC logged at /outputdir/<userid>/csv_logs (each run will produce a new log file with a timestamp in name).

Sample notebooks for analyzing the files in this directory can be found in examples/notebooks/. For instance, the LearningCurves.ipynb notebook contains a variation on the following code snippet:

import pandas as pd
import numpy as np
from bokeh.plotting import figure, show, output_file, save

data = pd.read_csv("<destination folder name on your laptop>/csv_logs/<name of the log file>.csv")

from bokeh.io import output_notebook
output_notebook()

from bokeh.models import Range1d
#optionally set the plotting range
#left, right, bottom, top = -0.1, 31, 0.005, 1.51

p = figure(title="Learning curve", y_axis_label="Training loss", x_axis_label='Epoch number') #,y_axis_type="log")
#p.set(x_range=Range1d(left, right), y_range=Range1d(bottom, top))

p.line(data['epoch'].values, data['train_loss'].values, legend="Test description",
       line_color="tomato", line_dash="dotdash", line_width=2)
p.legend.location = "top_right"
show(p, notebook_handle=True)

The resulting plot should match the train_loss plot in the Scalars tab of the TensorBoard summary.

Learning curve summaries per mini-batch

To extract per mini-batch summaries, we require a finer granularity of checkpoint data than what it is logged to the per-epoch lines of csv_logs/ files. We must directly use the output produced by FRNN logged to the standard output stream. In the case of the non-interactive Slurm batch jobs, it will all be contained in the Slurm output file, e.g. slurm-3842170.out. Refer to the following notebook to perform the analysis of learning curve on a mini-batch level: FRNN_scaling.ipynb