WRFV221 on dual quad Mac Pro

Here are my recent trials of WRF v.2.2.1 on a dual-quad Mac Pro with Intel Fortran (10.1.012) and gcc. The 64 bit version of the compilers are used. The machine has 8 GB of memory. Trials include OMP and MPI versions, the latter based on mpich2.

--------------------------------------------------------------
netcdf compilation for 64 bit (based on netcdf-3.6.0-p1)
--------------------------------------------------------------

export CC=/usr/bin/gcc
export CPPFLAGS="-O -DNDEBUG -DpgiFortran"
export CFLAGS="-O -m64"
export CXX=/usr/bin/c++
export CXXFLAGS="-O -m64"

export FC=ifort
export F77=ifort
export F90=ifort
export FFLAGS="-O3 -mp"
export F90FLAGS="-O3 -mp"


./configure --prefix=/usr/local/netcdf
make
make test
sudo mkdir /usr/local/netcdf
sudo make install


--------------------------------------------------------------
jasper compilation for 64 bit (based on jasper-1.701.0)
--------------------------------------------------------------

setenv CC /usr/bin/gcc
setenv CFLAGS "-O -m64"
setenv CXX /usr/bin/c++
setenv CXXFLAGS "-O -m64"

./configure --prefix=/usr/local/jasper
make
sudo make install


--------------------------------------------------------------
mpich2 compilation for 64 bit (based on mpich2-1.0.5)
--------------------------------------------------------------

setenv FC ifort
setenv F90 ifort
setenv CC "gcc -m64"
setenv RSHCOMMAND "/usr/bin/ssh"
setenv CXX "/usr/bin/c++ -m64"
setenv FFLAGS "-xP -vec- -fp-model precise"
setenv F90FLAGS "-xP -vec- -fp-model precise"
./configure --with-comm=shared


--------------------------------------------------------------
test run
--------------------------------------------------------------

The test run is a short simulation with three telescoping, two-way domains (142x100, 100x100 and 100x100, with 35 vertical levels). Flags were chosen to reproduce output file cksums of a completely unoptimized simulation requesting strict arithmetic. The OMP version occasionally produces different results, apparently randomly.

Timing plot:


The plot below adds results from a single 2.4GHz quad-core machine, running Mandriva Linux, for OMP runs built using the same configuration file as linked below (save Mac-specific references removed). For the four thread run, the 2.4 GHz run is 33% slower, though the clock speed difference is only 17%. Checksums were the same for all the runs.



Configuration files are here: OMP version, MPI version.

[edited 1 April 2008 to include the compilation for netcdf, jasper and mpich2 and to clarify this is for 64 bit compilers.]

WRFV221 Mac PowerPC w/ IBM xlf and MPI

My recent tests using the presently latest WRF version, v.2.2.1, on a PowerPC-based Mac using XLF and MPI appears to be all good, even with nesting. Two files are needed for this:

Additions for arch/configure.defaults.

Replacement for external/RSL_LITE/rsl_malloc.c. Rename file after downloading.

WRFV22 on a dual quad-core Intel Mac

Here are some notes on a recent attempt to run WRF on a dual quad-core Intel Mac, which I received as a loaner from Apple. In the two weeks I had the machine, I made progress, but it's clear from my notes below I didn't pass the finish line. For one thing, I was unable to get even medium-sized jobs to run in the 64 bit environment -- and the tricks for accessing memory that worked with 32 bits fail. In 32 bit land, there is still a limit on the job size, which is apparently due to Apple's internal memory allocation restrictions.

Executive summary: I was able to get all 8 cpus working for me, tho the scaling wasn't the best, and the key turned out to be: (a) moving to mpich-2; AND (b) using --with-comm=shared. An important goal for me is getting results that do not vary with the number of processors used. With OMP, that required the ifort flag '-fp-model precise'. I also used this flag in the WRF code and when compiling mpich-2.

** These notes assume the 32 bit ifort compiler. I used ifort 10.0.017. These notes presume WRF model changes documented on previous posts. Compilation and execution took place on an HFSX-formatted disk.

(1) netcdf-3.6.2

export CC=/usr/bin/gcc
export CPPFLAGS="-O -DNDEBUG -DpgiFortran"
export CFLAGS="-O"
export CXX=/usr/bin/c++
export CXXFLAGS="-O"

export FC=ifort
export F77=ifort
export F90=ifort
export FFLAGS="-O3"
export F90FLAGS=


./configure --prefix=/usr/local/netcdf
make
make test
make check
sudo mkdir /usr/local/netcdf
sudo make install

(2) MPICH-2 (mpich2-1.0.5p4)

setenv FC ifort
setenv F90 ifort
setenv CC "gcc"
setenv RSHCOMMAND "/usr/bin/ssh"
setenv CXX "/usr/bin/c++"
setenv FFLAGS "-xP -vec- -fp-model precise"
setenv F90FLAGS "-xP -vec- -fp-model precise"
./configure --with-comm=shared
make

(3) Modify external/RSL_LITE/buf_for_proc.c to add "extern" before "char mess" (resolves a problem that crops up specifically with mpich-2)

(4) configure.wrf files I used for these tests: APPLE_LOANER_WRF.zip.

WRFV22 on Mac Intel update... ifort and g95

The ZIP file linked below has been updated to reflect further progress made in getting WRFV22 to work on a Core Duo MacBook using Intel ifort and g95. As before, I am using mpich-1.2.5 for the MPI builds. My latest tests have employed ifort 10.0 and the most recent g95 at this writing. The g95 sections also are usable on PPC Macs.

For the most part, these configurations pass my "consistency tests", with some reservations. Consistency means that the MPI executables generate bitwise identical output, irrespective of the number of processors selected. However, if the domain is too finely subdivided, the results will start varying. I also encountered a test case in which identical results were generated for between 2-24 CPUs, inclusive, but results using only 1 CPU were different, which I cannot yet explain.

Results are not bitwise identical among compilers, or for the g95 versions when built on Intel and PPC architectures. However, I have not noted anything particularly amiss in the output. g95 executables from the PPC architecture can be executed on Intel (albeit slowly), and I've found this yields the same results obtained on PPC.

WRFV22_MacIntel_mods.zip (as of July 5, 2007).

WRFV22 on Mac Intel w/ ifort

The zip archive below contains WRFV22 code modifications for an Intel-based Mac with the Intel Fortran compiler (ifort). I have been able to run the single-threaded and OMP versions without problem, but my tests with the MPI version (RSL and RSL_LITE) with mpich-1.2.5 have not produced results that are the same when one and two processors are requested. This has only been tested on a MacBook with a Core Duo processor. So, use at your own risk. Feedback appreciated.

WRFV22_MacIntel_mods.zip

WRFV22 g95 on Mac supporting programs

Previously, I reported that I had to use tcsh to run WRFV22. Turns out I can run in bash if I specifically set the stacksize and datasize as follows

ulimit -s 65536
ulimit -d unlimited

To go along with the configure.defaults additions for g95 on Mac PPC, here are the flags I used for NetCDF and MPICH to get everything working. Specific versions of these programs I used were 3.6.0-p1 for NetCDF and 1.2.5 for MPICH.

netcdf-3.6.0-p1

* I used tcsh for the compilation

setenv CC /usr/bin/gcc
setenv CPPFLAGS "-O -DNDEBUG -DpgiFortran"
setenv CFLAGS "-O"
setenv CXX /usr/bin/c++
setenv CXXFLAGS "-O"

setenv FC g95
setenv F77 g95
setenv F90 g95
setenv FFLAGS "-O3 -fno-second-underscore"
setenv FCFLAGS "-O3 -fno-second-underscore"
setenv F90FLAGS "-O3 -fno-second-underscore"

./configure
* I then edited "macros.make", adding:
FLIBS = -lSystemStubs
F90LIBS = -lSystemStubs

./make
./make test
[all tests were found to work]
make install


mpich-1.2.5

* I used tcsh for the compilation

setenv FC g95
setenv F90 g95
setenv RSHCOMMAND "/usr/bin/ssh"
setenv CC gcc
setenv CXX gcc
setenv LIBS "-lSystemStubs"
setenv FFLAGS "-fno-second-underscore"
setenv F90FLAGS "-fno-second-underscore"
setenv CFLAGS "-fno-common -DFORTRANUNDERSCORE"

./configure --with-device=ch_p4 --without-romio
./make

* On one Mac PPC machine, I had to add -lstdc++ to LIBS, but another failed to work if that was done

* Although mpich compiles fine, the example programs would not compile without manually adding the "-fno-second-underscore" flag. Example:


mpif77 -fno-second-underscore -c fpi.f
mpif77 -o fpi fpi.o

WRFV22 g95 update 2/21/07

Here is an update on g95 with WRF version 2.2 on Mac OS X (PPC G5) and a Linux (Athlon) box. As noted below, WRFV22 produces usable (but not yet verified correct) runs with nesting on PPC with g95, whereas xlf still does not. This is likely an xlf compiler bug specific to OS X. The test case for this particular experiment consists of a single 150 x 150 grid point domain. Model configurations, independent of compiler, are: RSL_LITE, mpich-1.2.5, allows nesting. The executables represent

• g95 on OS X/PPC (donar_g95)
  
• xlf on OSX/PPC (donar_xlf)
  
• g95 on Mandriva Linux/Athlon (cascade_g95)
  
• ifort on Mandriva Linux/Athlon (cascade_ifort)
  

The first thing I check is to see if the model produces the same results, as verified by checksums and visual inspection of plotted fields, independent of the number of cpus -- at least until the domain becomes too finely subdivided. So far in this experiment, only g95 on OS X/PPC is passing that test. (It is likely that ifort will also pass, but that portion of the experiment is not yet finished.) The checksums are the same for runs with 1 to 24 cpus. For xlf on OS X/PPC, runs with 1, 2, 12 and 18 cpus produce one particular checksum while 4, 6, 8, 10 and 14 cpus result in another. 16 and 20 cpus result in a third checksum, and 24 cpus produces a fourth sum. The 12 hour forecast fields differ in some respects, with patterns that suggest roundoff errors.

Unsurprisingly, g95 is slower than a commercial compiler on the same hardware. The plot below presents timing results obtained thusfar. There are gaps in the data, and some degree of inconsistency regarding how the time function works on OS X and Linux. I will redo these statistics using Brian Jewett's scripts for extracting timings from the rsl.out.0000 files. Also, for g95 on OS X/PPC, I/O is a bottleneck. Running with nio_tasks_per_group > 0 appears to help a lot.

Click the image to open a larger version in a new window.

Here is the latest configure.wrf segment for my g95 OS X PPC runs.