Raad2: Ansys
HPC License Policy
TAMUQ has an Ansys license that includes 450 HPC tokens. The license server configuration provides no easy way to limit the consumption of HPC license tokens on a per-user basis. However, in order to ensure fair usage we require users to restrict themselves to a maximum of 96 HPC tokens at any given time. Please read on in order to gain a better understanding of Ansys HPC licensing, and to learn the job parameters which can ensure compliance with our 96 token limit. We intend to monitor usage of these tokens to ensure users are staying within our requested limits. In case of violations we reserve the right to kill offending jobs in order to facilitate fair access for other users.
Below is the brief description and example scenarios of Ansys licensing.
There are two different types of tokens for Ansys Research.
- Ansys Research (ansys)
- For each Ansys Fluent Session, one token is checked out. TAMUQ has 25 such tokens.
- Ansys HPC for research (anshpc)
- One license token per core is used for cores in excess of 16. TAMUQ has 450 such tokens.
Example Scenarios
Case 01
User submits a single node fluent job requesting 16 cores or less. In this case, no “anshpc” tokens are used and only 1 “ansys” token is used. This doesn’t have any adverse effect on license availability.
Case 02
User submits a single node fluent job requesting 24 cores. In this case, 8 “anshpc” tokens are used and only 1 “ansys” token is used. Multiple such jobs will eventually impact availability of HPC tokens for other users.
Case 03
User submits a multi-node fluent job requesting 48 cores. In this case, the user has occupied 32 tokens of “anshpc” and 1 token of “ansys”. Multiple such jobs will impact availability of HPC tokens for other users.
Case 04 User submits 5 jobs and each job is using 24 cores. In this case this user has occupied 40 “anshpc” tokens and 5 “ansys” tokens.
How to comply with licensing limits?
To keep the utilization fair and to give all the potential users a chance to run the computation, each user should not exceed more than 96 anshpc licenses.
Below are few examples of the job parameters to stay in the limits. A user is allowed to have a maximum of one of the below;
- A 2 node fluent job using. This will use 32 licenses.
- Maximum of 4 Single node jobs whereas each job is using 24 cores per node. This will use 8 “anshpc” licenses per node. Hence limiting per user to 96 tokens.
- If your core count per job is less than or equal to 16, you may submit multiple jobs. This will only use ("ansys” tokens only and we have ample number of those tokens.
If you have any specific use case where you think these restrictions are not helpful, please let us know and we will look into this.
Sample Small Job File
#!/bin/bash
#SBATCH -J ansys_j
#SBATCH -p s_long
#SBATCH --qos sl
#SBATCH --time=168:00:00
#SBATCH --ntasks=8
#SBATCH --output=ansys_sim.o%j
#SBATCH --error=ansys_sim.o%j
#SBATCH --hint=nomultithread
##SBATCH --mail-type=ALL
##SBATCH --mail-user=<your_email>@qatar.tamu.edu
## Get job Stats
echo "Starting at "`date`
echo "SLURM_JOBID="$SLURM_JOBID
echo "SLURM_JOB_NODELIST"=$SLURM_JOB_NODELIST
echo "SLURM_NNODES"=$SLURM_NNODES
echo "SLURMTMPDIR="$SLURMTMPDIR
echo "working directory = "$SLURM_SUBMIT_DIR
# Load Ansys Module
module load ansys/231
# Load Intel Compiler if needed for simulation
# module swap PrgEnv-cray/6.0.10 PrgEnv-intel
# Start Simulation
echo "Starting Simulation.."
fluent 3ddp -g -t 8 < inputfile >& outputfile
echo "Ending at "`date`
echo "Simulation Ended"
Sample Large Job File
#!/bin/bash
#SBATCH -J ansys_j
#SBATCH -p l_long
#SBATCH --qos ll
#SBATCH --time=168:00:00
#SBATCH -N 5
#SBATCH --ntasks-per-node=24
#SBATCH --output=ansys_sim.o%j
#SBATCH --error=ansys_sim.o%j
#SBATCH --hint=nomultithread
##SBATCH --mail-type=ALL
##SBATCH --mail-user=<your_email>@qatar.tamu.edu
## Get job Stats
echo "Starting at "`date`
echo "SLURM_JOBID="$SLURM_JOBID
echo "SLURM_JOB_NODELIST"=$SLURM_JOB_NODELIST
echo "SLURM_NNODES"=$SLURM_NNODES
echo "SLURMTMPDIR="$SLURMTMPDIR
echo "working directory = "$SLURM_SUBMIT_DIR
# Load Ansys Module
module load ansys/231
# Load Intel Compiler if needed for simulation
module swap PrgEnv-cray/6.0.10 PrgEnv-intel
# Set CRAY Specific environmet variables
# Do not remove below two lines or else your job will fail
export FLUENT_ENABLE_SLURM_SUPPORT=1
export FLUENT_CRAY_MPI_ABI=1
# Start Simulation
echo "Starting Simulation.."
NTASKS=`echo $SLURM_TASKS_PER_NODE | cut -c1-2`
NPROCS=`expr $SLURM_NNODES \* $NTASKS`
if [ $SLURM_NNODES -eq 1 ]; then
# Single node with shared memory
fluent 3ddp -g -t $NPROCS < inputfile >& outputfile
else
# Multi-node
fluent 3ddp -g -slurm -t $NPROCS -mpi=cray < inputfile >& outputfile
fi
echo "Ending at "`date`
echo "Simulation Ended"
2nd Sample Large Job File
#!/bin/bash
#SBATCH -J ansys_j
#SBATCH --qos=ll
#SBATCH -p l_long
#SBATCH --time=168:00:00
#SBATCH -N 6
#SBATCH --ntasks-per-node=24
#SBATCH --output=ansys_job.o%j
#SBATCH --error=ansys_job.e%j
#SBATCH --hint=nomultithread
## Get job Stats
echo "Starting at " $(date)
echo "SLURM_JOBID = " $SLURM_JOB_ID
echo "SLURM_JOB_NODELIST = " $SLURM_JOB_NODELIST
echo "SLURM_NNODES = " $SLURM_NNODES
echo "SLURMTMPDIR = " $SLURMTMPDIR
echo "working directory = " $SLURM_SUBMIT_DIR
module unload PrgEnv-cray
module load PrgEnv-intel
# Load Ansys Module
module load ansys/231
export ANSYS_NETWORK_START=15
export ANSYS_NETWORK_COMM=5
export ANS_SEE_RUN_COMMAND=1
export ANSYSLMD_LICENSE_FILE=2353@tamuq-ansys.qatar.tamu.edu
export I_MPI_ROOT=/opt/intel/impi/2019.9.304
export I_MPI_DEBUG=5
export I_MPI_FABRICS=shm:ofi
export I_MPI_FALLBACK=no
export OMP_NUM_THREADS=1
export KMP_AFFINITY=disabled
MACHINELIST=$(srun hostname | sort | uniq -c | \
awk '{print $2 ":" $1}' | \
paste -s -d ":" -)
# Start Simulation
echo "Starting Simulation.."
echo "Machinelist = " $MACHINELIST
# Multi-node
time ansys195 -dis -ppf anshpc -usessh -mpi intelmpi -machines $MACHINELIST -j myjob -s read -b nolist -i myinputfile.txt -o stdout.out
echo "Ending at " $(date)
echo "Simulation Ended"