Run any HPC supported ANSYS® solver such as Fluent™, CFX™, HFSS, Mechanical, or other applications using our flexible and easy to use production-level HPC environment. Customers can use their existing product license (Windows or Linux) or lease short-term licenses from ANSYS®. With the addition of one or more ANSYS® HPC Packs, the customer can run on 8, 32, 128, or more processor cores. Thus increasing speed significantly. ANSYS® Solvers and HPC Packs can be leased directly from ANSYS® for 1, 3, 6, 9, or 12 months. Note: Please contact your ANSYS® sales representative for HPC Pack License pricing.
Topics covered in this article:
- Verify your license is setup
- Fluent Launcher
- Workbench Launcher
- Other Solvers
- ANSYS Performance Issues Related to Licensing and Whole Nodes
You can run ANSYS solvers with Sabalcore HPC Cloud either interactively using the Remote Graphics or in batch mode (command line only). Refer the the instructions below.
ANSYS License Setup
To setup your ANSYS license, refer to the License Server Tunnel instruction page for details about setting up the license tunnel for your ANSYS product.
Fluent and Mechanical Launcher
Your license tunnel must be configured before using ANSYS Fluent. To run other Fluent in interactive graphical mode using rg3d™ follow these simple steps.
- Upload your sim files and open a Remote Graphics session.
- Select Fluent or Mechanical from the Sabalcore menu (located at the bottom of the screen) . The Launcher will appear.
- Choose the the node type, number of nodes and the number of cores per node. This is already filled in by default but should be modified to suit your needs.
- Select “Enable Accelerated 3D-Graphics Support” when visualizing large cell 3-D graphics. This will provide maximum performance using hardware acceleration for live 3-D visual rendering. Otherwise, leave unchecked and use only software rendering which is the default for all node types.
- Click “Start”
- After a few moments the ANSYS will appear. Proceed as you normally would. The system will automatically configure the execution nodes so there is no need to specify a hostfile.
WorkBench Launcher
Your license tunnel must be configured before using ANSYS WorkBench. To run WorkBench in interactive graphical mode using rg3d™ follow these simple steps.
- Upload your sim files and open a Remote Graphics session.
- Select WorkBench from the Sabalcore menu (located at the bottom of the screen) . The Launcher will appear.
- Choose the the node type, number of nodes and the number of cores per node. This is already filled in by default but should be modified to suit your needs.
- Select “Enable Accelerated 3D-Graphics Support” for maximum performance using hardware acceleration. Otherwise, leave unchecked and use only software rendering which is the default for all node types.
- Click “Start”
- After a few moments the ANSYS WorkBench Launcher will appear. Proceed as you normally would.
- When setting up a project to run in parallel, you need to manually set the number of solver processes and the machine file. Select “Distributed Memory on a Cluster” and “File Containing Machine Names”. Then enter the path to the machine file. The path is: “/scratch/<JOBID>/machinefile”. Where <JOBID> is the job ID for your job which is displayed on the Sabalcore Workbench Launcher App.
Other ANSYS Solver Specific Instructions
CFX5 in Batch Mode
To run CFX5 in batch mode, you will need to create a submit script. The script shown below is an example:
#PBS -l nodes=2:ppn=16:cobalt # load the current version of the ANSYS Module module load ansys # Change to your working directory cd $PBS_O_WORKDIR # Determine the total number of cores for this job NP=$(wc -l $PBS_NODEFILE | cut -f 1 -d ' ') # Run CFX in batch mode. Note that CFX5_HOSTLIST includes # all the cores for this job cfx5solve -part $NP -batch -def YOUR_FILE_HERE.def \ -start-method 'Platform MPI Distributed Parallel' \ -par-dist $CFX5_HOSTLIST
Note about CFX5 and CPU Affinity
Some users are limited in the number of cores they can use by their product license. For example, they may only have a license for 8 cores for ANSYS. Some of the nodes have more than 8
cores per node. If you attempt to run a CFX5 process on less than the total number of cores on a node, the process my crash unexpectedly or have poor performance. The reason for this is CFX5 tries to set the CPU affinity of the processes in a way that is incompatible with our HPC environment. For example, when you select 8 cores on a 16 core node in the PBS script (i.e. nodes=1:ppn=8:cobalt), our environment will only allow you to use the 8 cores you requested. However, CFX5 ignores this and tries to set the CPU affinity of the 8 processes across all 16 cores which causes a problem. Because the system only allows 8 cores for your job, the CPU affinity fails and cfx5solve crashes or performs very poorly. The solution is to request all 16 cores of the node and use set “-part 8” option to cfx5solve. For example:
#PBS -l nodes=1:ppn=16:cobalt # load the current version of the ANSYS Module module load ansys # Change to your working directory cd $PBS_O_WORKDIR # License is restricted to 8 cores NP=8 # Run CFX in batch mode. Note that CFX5_HOSTLIST includes # all the cores for this job cfx5solve -part $NP -batch -def YOUR_FILE_HERE.def \ -start-method 'Platform MPI Distributed Parallel' \ -par-dist $CFX5_HOSTLIST
Fluent in Batch Mode
To run Fluent in batch mode, you will need to create a submit script. The script shown below is an example:
# Request 2 cobalt node #PBS -l nodes=2:ppn=16:cobalt # Load the ansys module module load ansys # change to working directory cd $PBS_O_WORKDIR # start fluent on all nodes using a journal file fluent 3d -g -cnf=$PBS_NODEFILE -pib -i counterflow.inp
You should refer to the Fluent documentation from Ansys for more information about other possible command-line options to the fluent command. The -g option tells fluent to not start the graphics or GUI. The -pib option is used to specify the high-speed Infiniband network. Finally, the -i counterflow.inp option specifies the journal file to run.
ANSYS Performance Issues Related to Licensing and Whole Nodes
There is a known issue with ANSYS and the way it binds processes to cores. ANSYS tires to bind the processes on it’s own and ignores binding recommended by MPI and resource managers (PBS).
Sometimes this issue pops-up with users that have 4 and 8 core licenses. In situations where the user is using less than a whole node, for example in the case of using an 8 core license on a 16 core node, more than one process may end up on the same core and thus cause poor performance. This is because ANSYS tries to use cores that were not allocated to the job by the resource manager.
The solution to this issue is to always request a whole node so that ANSYS can bind the processes however it sees fit. Below is an example PBS script for using an 8 core license and a whole node. Cobalt nodes have a total of 16 cores.
#PBS -l nodes=1:ppn=16:cobalt # load the current version of the ANSYS Module module load ansys # Change to your working directory cd $PBS_O_WORKDIR # Since we have an 8 core license, we need to specify 8 cores NP=8 # Run CFX in batch mode. Note that CFX5_HOSTLIST includes # all the cores for this job cfx5solve -part $NP -batch -def YOUR_FILE_HERE.def \ -start-method 'Platform MPI Distributed Parallel' \ -par-dist $CFX5_HOSTLIST