Raad2: Ansys

From Research Computing @ TAMUQ
Jump to navigation Jump to search


HPC License Policy

TAMUQ has an Ansys license that includes 128 HPC tokens. The license server configuration provides no easy way to limit the consumption of HPC license tokens on a per-user basis. However, in order to ensure fair usage we require users to restrict themselves to a maximum of “32" HPC tokens at any given time. Please read on in order to gain a better understanding of Ansys HPC licensing, and to learn the job parameters which can ensure compliance with our 32 token limit. We intend to monitor usage of these tokens to ensure users are staying within our requested limits. In case of violations we reserve the right to kill offending jobs in order to facilitate fair access for other users.

Below is the brief description and example scenarios of Ansys licensing.

There are two different types of tokens for Ansys Research.

  1. Ansys Research (aa_r)
    1. For each Ansys Fluent Session, one token is checked out. TAMUQ has 25 of such tokens.
  2. Ansys HPC for research (aa_r_hpc)
    1. One license token per core is used for cores in excess of 16. TAMUQ has 128 of such tokens.

Example Scenarios

Case 01

User submits a single node fluent job requesting for 16 cores or less. In this case, no “aa_r_hpc” tokens are used and only 01 “aa_r” token is used. This doesn’t has any adverse effect on license availability.

Case 02

User submits a single node fluent job requesting for 24 cores. In this case, 8 “aa_r_hpc” tokens are used and only 01 “aa_r” token used. Multiple of such jobs will impact availability of HPC tokens to other users.

Case 03

User submit a multi-node fluent job requesting for 48 cores. In this case, the user has occupied 32 tokens of “aa_r_hpc” and 01 token of “aa_r”. Multiple of such jobs will impact availability of HPC tokens to other users.

Case 04 User submits 5 jobs and each job is using 24 cores. In this case this user has occupied 40 “aa_r_hpc” tokens and 5 “aa_r” tokens.

How to comply with licensing limits?

To keep the utilization fair and giving all the potential users a chance to run the computation, each user should not exceed more than 32 aa_r_hpc licenses.

Below are few examples of the job parameters to stay in the limits. A user is allowed to have maximum of one of below;

  1. A 02 node fluent job using. This will use 32 aa_r_hpc licenses.
  2. Maximum of 04 Single node jobs whereas each job is using 24 cores per node. This will use 8 “aa_r_hpc” licenses per node. Hence limiting per user to 32 tokens.
  3. If your core count per job is less than or equal to 16, you may submit multiple jobs. This will only use “aa_r” tokens only and we have ample number of those tokens.

If you have any specific use case where you think these restrictions are not helpful, please let us know and we will look into this.

Sample Small Job File

#!/bin/bash
#SBATCH -J ansys_j
#SBATCH -p s_long
#SBATCH --qos sl
#SBATCH --time=168:00:00
#SBATCH --ntasks=8
#SBATCH --output=ansys_sim.o%j
#SBATCH --error=ansys_sim.o%j
#SBATCH --hint=nomultithread
##SBATCH --mail-type=ALL
##SBATCH --mail-user=<your_email>@qatar.tamu.edu

## Get job Stats
echo "Starting at "`date`
echo "SLURM_JOBID="$SLURM_JOBID
echo "SLURM_JOB_NODELIST"=$SLURM_JOB_NODELIST
echo "SLURM_NNODES"=$SLURM_NNODES
echo "SLURMTMPDIR="$SLURMTMPDIR
echo "working directory = "$SLURM_SUBMIT_DIR


# Load Ansys Module
module use /lustre/sw/xc40ac/modulefiles
module load ansys/162

# Load Intel Compiler if needed for simulation
# module swap PrgEnv-cray/5.2.82 PrgEnv-intel


# Start Simulation
echo "Starting Simulation.."

fluent 3ddp -g -t 8 < inputfile >& outputfile

echo "Ending at "`date`
echo "Simulation Ended"

Sample Large Job File

#!/bin/bash
#SBATCH -J ansys_j
#SBATCH -p l_long
#SBATCH --qos ll
#SBATCH --time=168:00:00
#SBATCH -N 5
#SBATCH --ntasks-per-node=24
#SBATCH --output=ansys_sim.o%j
#SBATCH --error=ansys_sim.o%j
#SBATCH --hint=nomultithread
##SBATCH --mail-type=ALL
##SBATCH --mail-user=<your_email>@qatar.tamu.edu

## Get job Stats
echo "Starting at "`date`
echo "SLURM_JOBID="$SLURM_JOBID
echo "SLURM_JOB_NODELIST"=$SLURM_JOB_NODELIST
echo "SLURM_NNODES"=$SLURM_NNODES
echo "SLURMTMPDIR="$SLURMTMPDIR
echo "working directory = "$SLURM_SUBMIT_DIR


# Load Ansys Module
module use /lustre/sw/xc40ac/modulefiles
module load ansys/162

# Load Intel Compiler if needed for simulation
module swap PrgEnv-cray/5.2.82 PrgEnv-intel


# Set CRAY Specific environmet variables
# Do not remove below two lines or else your job will fail
export FLUENT_ENABLE_SLURM_SUPPORT=1
export FLUENT_CRAY_MPI_ABI=1



# Start Simulation
echo "Starting Simulation.."

NTASKS=`echo $SLURM_TASKS_PER_NODE | cut -c1-2`
NPROCS=`expr $SLURM_NNODES \* $NTASKS`

if [ $SLURM_NNODES -eq 1 ]; then
    # Single node with shared memory
    fluent 3ddp -g -t $NPROCS < inputfile >& outputfile
else
    # Multi-node
    fluent 3ddp -g -slurm -t $NPROCS -mpi=cray < inputfile >& outputfile
fi

echo "Ending at "`date`
echo "Simulation Ended"