LIGGGHTS® is an Open Source Discrete Element Method Particle Simulation Software provided by CFDEM® project. This article demonstrates the basic steps to run and visualize a simple LIGGGHTS® simulation using Sabalcore Computing’s HPC On-Demand service. Before proceeding with this demo, you should understand the basics submitting batch and interactive jobs and using modules at Sabalcore.
First, run the module help liggghts command to see information about the latest installed version of LIGGGHTS®:
$ module help liggghts ----------- Module Specific Help for 'liggghts/3.3.0' ------------- Name: liggghts Version: 3.3.0 Installed: Wed Nov 25 16:35:48 EST 2015 An example is available for liggghts. To use the example, run the following commands: cp -a /uls/6/liggghts-3.3.0/example/chute_wear . cd chute_wear qsub liggghts.pbs
In this case we have version 3.3.0 available. Also, you can see some information about running the example.
The next step is to copy the example to your home directory with the command:
cp -a /uls/6/liggghts-3.3.0/example/chute_wear .
Now cd to the chute_wear directory and take a look at the liggghts.pbs job script:
#PBS -l nodes=1:ppn=16:cobalt #PBS -l nodes+=1:ppn=24:copper #PBS -l nodes++=1:ppn=12:red module load liggghts cd $PBS_O_WORKDIR mpirun liggghts < in.chute_wear
As you can see, this job is setup to run on 16 cores on the cobalt or 24 cores on copper or 12 cores on the bronze cluster. This is a basic job script which can be modified to meet your specific needs for your LIGGGHTS® jobs.
Now submit the job to the clusters using qsub:
qsub liggghts.pbs
Once the job has been submitted, you can monitor it with the qstat command. After the job has completed, it’s time to visualize the results using Paraview via a Remote Graphics session. To do this, you need to submit an interactive job using either of the following commands:
qsub -I -l nodes=1:graphics:ppn=8:gpn=1
-- OR --
qsub -I -l nodes=1:ppn=1
The first option will use one of the graphics nodes and will utilize a GPU to preform hardware accelerated 3D rendering. You should use this option if you have a very large/complex 3D object to visualize. The second option will use software rendering and is preferred for simple or small 3D images. For this demo you can use the second option.
Once the interactive job has started, you will be logged into one of the compute nodes. At this point you need to load the liggghts and paraview modules:
[09:47 user@n716021 ~]$ module load liggghts [09:47 user@n716021 ~]$ module load paraview [rg3d]: This node does not support hw *accelerated* 3D graphics. [rg3d]: Using software level rendering.
Because we are not on a graphics node with a GPU, we get a warning regarding the missing GPU. You can ignore this warning since we are going to visualize a rather simple model.
Next, cd to the ‘post’ directory under the ‘chute_wear’ directory and run the lpp command (LIGGGHTS® post-processing) to generate VTK data for use with Paraview:
[09:55 user@n716021 post]$ cd ~/chute_wear/post [09:55 user@n716021 post]$ lpp dump*.chute starting LIGGGHTS memory optimized parallel post processing chunksize: 8 --> 8 files are processed per chunk. If you run out of memory reduce chunksize. Working with 16 processes... calculating chunks 1 - 16 of 63 calculating chunks 17 - 32 of 63 calculating chunks 33 - 48 of 63 calculating chunks 49 - 63 of 63 wrote 500 granular snapshots in VTK format time needed: 12.5661559105 sec
Now start up Paraview with the paraview command and open the ‘*.vtk’ files and setup your visualization. You should see something similar to this:
