In this tutorial you will find:

• How to list the available versions
• How to setup WRF 4.4 and higher, or how to set up v4.3 and lower
• Run an example
• WRF advanced topics

Introduction

At Sabalcore Computing, we provide easy to use access to the latest WRF software including many of the most popular post-processing and analysis tools. We have carefully built and optimized the software using the best compilers and libraries for the fastest performance possible. There is no need for you to download and build them yourself. You only have to run a few simple commands to setup the WRF modeling suite of your choice.

Using WRF at Sabalcore Computing may be different than the way you have used it in the past. We have done most of the hard work for you. This document will explain some of the differences and will provide the information you need to get started with WRF on Sabalcore’s HPC systems. If you need to use an alternate version, please contact Support  for assistance.

Finding the version of WRF/WPS you want to use

There are several flavors of WRF and WPS available for you to choose from. Use the following commands to see a list of what is available:

$ module avail WRF
$ module avail WPS

If you don’t see the exact module you would like to use, let us know and we will add it for you.

For  WRF versions 4.4 and higher

Next, if this is the first time you are running WRF/WPS at Sabalcore, you should run the example case that we have provided to learn how to submit WRF jobs to the compute nodes.

This example applies to WRF/WPS versions 4.4 and later –

To quickly get WRF and WPS setup and run a basic example, follow these steps.

Load the WRF and WPS modules:

[user@sci02 ~]$ module load -s WPS/v4.4 WRF/v4.4.1

Setup the WPS and WRF run directories:

[user@sci02 ~]$ mkdir Tutorial
[user@sci02 ~]$ cd Tutorial/
[user@sci02 Tutorial]$ install-wps ./WPS
mkdir: created directory ‘./WPS’
[user@sci02 Tutorial]$ install-wrf ./WRF
mkdir: created directory ‘./WRF’

In the newly created WPS and WRF directories, you will find all the files needed to run a basic example case. In particular, take a look at the batch script files:

./WPS/wps-example.pbs
./WRF/run/wrf-example.pbs

Now you can submit the WPS job which will run geogrid.exe, ungrib.exe, and metgrid.exe.

[user@sci02 Tutorial]$ cd WPS
[user@sci02 WPS]$ qsub wps-example.pbs
1356892.jman

Use the ‘qstat’ command to monitor the job:

[user@sci02 WPS]$ qstat 1356892.jman
Job ID Name User Time Use S Queue
------------------------- ---------------- --------------- -------- - -----
1356892.jman wps-example.pbs user 00:10:14 C dque

After the WPS job has completed, you can submit the WRF job:

[user@sci02 WPS]$ cd ../WRF/run
[user@sci02 run]$ qsub wrf-example.pbs

For  WRF versions 4.3 and lower

Installation in your home directory

The WRF modules are organized by Version/Compiler/Model/Communication/Nesting. For example, the “WRF/4.4.1/intel/ARW/dmpar/basic” module is for version 4.4.1, compiled with Intel, ARW model with Distributed Memory (MPI), and Basic Nesting. You can use “module help <module name>” to get more information about a specific module. For example:

$ module help WRF/4.2.1/intel/ARW/dmpar/basic
------------ Module Specific Help for 'WRF/4.2.1/intel/ARW/dmpar/basic' --------------------------- 
Name: WRF Version: 4.2.1 
Model: ARW Communication: dmpar 
Nesting: basic 
Compiler: intel 
Installed: Wed Nov 02 16:00:34 UTC 2022

To use WRF, you will first need to install WRF and WPS in your home directory, but don’t worry because we have done all the hard work of building them for you. To see which versions are available use the command “module available WRF”. To install WRF and WPS, just use the following commands depending on which version you would like to use:

$ tar xzf /uls/7/WRF/<VERSION>/<MODLE>/WRF.tgz
$ tar xzf /uls/7/WPS/<VERSION>/WPS.tgz

Example:

$ tar xzf /uls/7/WRF/4.2.1/ARW/WRF.tgz
$ tar xzf /uls/7/WPS/4.2/WPS.tgz

Where <MODEL> is either ARW or NMM and <VERSION> is the numerical version. This will extract the WRF and WPS installation directories to your home directory so you can later run them on the compute nodes. You can re-run these commands in other subdirectories if you want more than one installation of WRF/WPS.

This example applies to WRF/WPS versions 4.3 and lower –

Example:

$ cd WRF
$ tar xvf /uls/7/WRF/4.2.1/ARW/example.tgz
$ cd ../WPS
$ tar xvf /uls/7/WPS/4.2/example.tgz

Now take a look at the wps-example.pbs script:

#PBS -l nodes=2:onyx:ppn=12 

## This example needs be done before the wrf-example.pbs is run.

## load the WPS module
module load WPS/4.2.1/intel/dmpar

## Move to your Working directory. This is the directory from which
## you ran "qsub", which should be your WPS directory.
cd $PBS_O_WORKDIR

## Run geogrid.exe in parallel (output is 'geo_em.dxx.nc' - one file
## for each domain)
mpiexec ./geogrid.exe

## Link (with the UNIX command ln) the correct Vtable
ln -sf ungrib/Variable_Tables/Vtable.AWIP Vtable

## Link (with supplied script link_grib.csh) the input GRIB data. The
## data for this example is in the DATA directory.
./link_grib.csh DATA/JAN00/2000012

## Run ungrib.exe serially. (output will be intermediate files - one
## file for each time)
./ungrib.exe

## Run METGRID in parallel on all cores
mpiexec ./metgrid.exe

The “module load WPS/4.2.1/intel/dmpar” line may be new to you. Basically, this line selects which version of the WPS executables you would like to run via this script. In this example, the Distributed (MPI) version compiled with the Intel compilers will be used. Selection of the module is discussed in more detail below. The rest of the commands in this script simply run the usual WPS commands on the input GRIB data. You can customize this script later for your production runs if you like.

Now, submit the script to run on the cluster:

$ qsub wps-example.pbs

You can use the qstat command to monitor the progress of the job. When you see a “C” in the second to last column, that means the job has completed. Once the job has finished, there should be some met files and log files in the WPS directory. Take a look at the last line of the metgrid.log.0000 file to make sure that metgrid ran successfully.

$ tail -1 metgrid.log.0000
2013-09-13 12:06:26.807 --- *** Successful completion of program metgrid.exe ***

Now that you’re done with WPS, you should be ready to run WRF. Move to the WRF run directory and take a look at the wrf-example.pbs script:

$ cd ../WRF/run
$ less wrf-example.pbs

#PBS -l nodes=2:onyx:ppn=12

## Be sure to run the WPS example before you run this one.

## This example assumes you are using a parallel version of WRF.

## load one of the WRF modules
module load WRF/4.2.1/intel/ARW/dmpar/basic

## Change to your working directory, i.e. where you issued the qsub
## command. This should be the WRFV3/run directory for this example.
cd $PBS_O_WORKDIR

## Link or copy (with the UNIX command ln / cp ) the met_em files to
## this directory. This assumes the WPS is at the same directory level
## as the WRFV3 directory.
ln -sf ../../WPS/met_em.d0* .

## Run real.exe (verify that the program runs correctly).You should
## have the following output files (default setup): wrfinput_d01 &
## wrfbdy_d01.
mpiexec ./real.exe

## Run wrf.exe (verify that the program runs correctly) You should
## have the follwoing output files (default setup):
## wrfout_dxx_[initial_date] (one for each domain). Each file (by
## default) will contain all the forecast output times.
mpiexec ./wrf.exe

The “module load WRF/4.0.1/intel/ARW/dmpar/basic” selects which version of the WRF executables to use. In this case version 4.0.1, compiled with Intel, ARW model, with Distributed parallelism and basic nesting is used. Selecting the WRF module is discussed below. The rest of the script simply runs real.exe and wrf.exe.

Now submit the job to the cluster:

$ qsub wrf-example.pbs

Once the job has completed, take a look at the rsl.out.0000 file to verify that it ran successfully:

$ tail -1 rsl.out.0000
d01 2000-01-25_00:00:00 wrf: SUCCESS COMPLETE WRF

WRF Advanced Topics

This section includes instructions for building specific WRF packages Note: this is not required for general WRF use. WRF is already built and available. See here for standard WRF installation and usage.

Supporting Tools

We have most supporting tools already installed on the system such as grads, netcdf, etc. For example: “module avail” or “module avail grads” will show which versions are available.

Post Processing Tools

There are several post-processing tools already installed on the cluster as well including NCARG, RIP4, ARW-POST, UPP, GMPAK, wgrib2, and panoply. For example “module avail” or “module RIP” will display the version that are available. If your package or version is not available, please contact us to have it installed.

Building UPP

 $ qsub -I -l nodes=1:copper:ppn=24

$ cd $SCRATCH

$ wget http://www.dtcenter.org/wrf-nmm/users/downloads/UPPV2.1.tar.gz
$ wget http://www.mmm.ucar.edu/wrf/src/WRFV3.5.TAR.gz

untar them:

$ tar -xzvf UPPV2.1.tar.gz

$ tar -xzvf WRFV3.5.TAR.gz

$ module load jasper openmpi netcdf/4.3.0/gcc

$ cd WRFV3

$ ./compile em_real 2>&1 | tee build.log | grep Error

$ cd ../UPPV2.1

$ mkdir bin lib include

$ cd UPPV2.1

$ echo 8 | ./configure

$ ./compile 2>&1 | tee build.log
 

$ cd ..

$ rsync -av WRFV3 UPPV2.1 ~/some/dir/

exit

Additional Help

If you need additional assistance, require a special setup, or want other WRF software installed, please contact Customer Support.