WRFblog

Intel 11.1, OS X 10.6.4, Xcode 3.2.2, and WRF seg faults

2010-06-27T11:33:00.000-07:00

This post was kindly provided by Dr. Lou Wicker of the National Severe Storms Laboratory. Though I have updated to WRFv3.2, I'm still on ifort 11.0, Xcode 3.0 and Mac OS X 10.5.8.

===================================

When compiling WRF 3.2.1 [3.2 ?] using Intel 11.1 (v088) on Mac OS X 10.6.4, we recently kept having segmentation faults during runtime. I managed to track down the problem to memory allocated in Thompson microphysics, where some arrays were allocated (successfully, I checked the status flag), and then a bad address to memory to the first element in the array in a nested do-loop caused the segmentation fault. I "fixed" the Thompson code with static allocations of these arrays for a test (they were small), and then the segmentation fault occured in the solver_em routine.

I then compiled WRF with a beta version of Intel v12 compiler (which I am testing), and this behavior does NOT occur.

I posted to Intel Premier, which I have to say is pretty responsive to the 2-3 issues I have ever posted. The got back to me within a few hours. They pointed me to this 9 May 2010 note:

http://software.intel.com/en-us/articles/intel-fortran-for-mac-os-x-incompatible-with-xcode-322/

Which basically says that Apple, in version 3.2.2 of Xcode changed the loader in some way which is incompatible with Intel v11.1 which causes these segmentation faults. Their simple answer is to either remain at Xcode v 3.2.1, although they say the compiler flag "-use-asm" might fix the problem. In any event, this might save people a lot of hours of time and frustration. BTW, the WRF code ran 10-20 time steps using WSM5 with no problem using the "bad" Xcode version.

Hope that is helpful to all the WRF'ers.

Lou

WRFV311 on Snow Leopard

2009-12-21T08:24:00.000-08:00

I tried compiling WRF on a Snow Leopard (SL) Mac, and found I had to make a few adjustments to my configure.wrf files. First of all, ifort version 11 appears to be needed. I also needed to revert to the (now) older version of the gcc compiler. Change "gcc" or "cc" on the SCC and CC_TOOLS lines to "gcc-4.0". I'm not sure if this is ifort's problem or mine, but at least it appears to be resolved.

The issue this resolved was a segfault in executing tools/registry for the creation of the module_state_description.F file.

WRFV300 update: two items

2008-06-16T19:01:00.000-07:00

(1) In WRFV300 on the Mac, using the Intel Fortran 10.1 compiler, the compiler flag –switch fe_use_rtl_copy_arg_inout causes problems that weren't apparent with earlier WRF versions. Specifically, lateral boundary tendencies for water vapor are zeroed, resulting in problems in this field that grow inward from the boundaries. In the wrfbdy_d01 file, fields like QVAPOR_BXE and related entries possess only zeroes. This file is, of course, created by real.exe.

I added this switch owing to this page. Removing it does no harm. I tried several minor versions of ifort 10.1, explored 32 and 64 bit, etc.; nothing helped fix this issue except removing this switch.

(2) In WRFV300, the Lin microphysics scheme (mp_physics = 2) has issues -- noise issues. After only a few tens of minutes, the pressure field starts displaying small scale noise. Decreasing the time step, implementing 6th order smoothing, and upping the small time step smoothing (epssm) do not help. For the simulations I am examining, the noise does not appear with any other microphysics scheme. With Lin, the noise appears in higher but not in lower resolution simulations.

The Lin code was altered between WRFV221 and WRV300. I have not isolated the code changes that cause this problem.

Playing with WRF3 (WRFV300, WPSV300) - part II

2008-05-01T17:20:00.000-07:00

Some updates: In the post Playing with WRF3 (WRFV300, WPSV300) - part I below, I mentioned that WRFV300's OpenMP was broken on Mac and Linux, using ifort and pgf. This problem is solved. OpenMP isn't working in the stock configuration because the $(OMP) declaration was not defined in FCFLAGS. It was declared in LDFLAGS, so the model seemed to go through the motions of spawning the requested number of threads. But, its absence from FCFLAGS means that OpenMP support was not actually complied in the code.

The remedy for this is to edit arch/postamble_new and insert $(OMP) on the FCFLAGS line.

I also have WPSV300 working on my Intel Mac, because I did manage to get a 64 bit version of NCAR Graphics compiled and running. I will update this post to describe how this was done in the future.

[EDIT 10 June 2008: Some additional information..
* My 64 bit NCAR Graphics binaries built on my MacPro are
here. This is an 18MB gzipped tar file. It does not include all of NCL, just NCAR Graphics..

* For this build of NCAR Graphics, you need to include the -ncarbd flag with ncargf77, ncargf90 or ncargex. Or, when compiling with ifort, you need to link in /usr/local/ncarg64/lib/ncarg/robj/libncarbd.o

* With this NCAR Graphics, I had no trouble building WPSV300 and RIP4 64bit on the MacPro.

* Here are my build notes for NCAR Graphics 64 bit on the MacPro, using ifort. Of course, you need to change locations, etc..

setenv NCARG /case/ncl_ncarg-5.0.0
setenv NCARG_ROOT /case/ncl_ncarg-5.0.0
setenv NCARG_CONFIG /case/ncl_ncarg-5.0.0/config

cd config
./ymake
./Configure
..then, as root
mkdir /usr/local/ncarg64

edited config/Darwin_Intel and ran Configure again

ulimit -s 65530
compile as root, so source ifortvars.sh

had been getting segfaults with fontcap. Resolved by editing yMakefile in /case/ncl_ncarg-5.0.0/common/src/libncarg_c/yMakefile for Darwin section to make it like Linux x86_64 section. That is,

#elif defined(Darwin) && defined(i386)
EXCSRCS = bcopyswap.c logic32.c
EXFSRCS = gbytes.f sbytes.f
EXOBJS = sbytes.o gbytes.o bcopyswap.o logic32.o

WRFV300 on Mac G5 (XLF compiler)

2008-04-16T19:39:00.000-07:00

Here are alterations to the stock WRF version 3.0.0 release I had to get it working on Mac/G5 equipment. This was tested on Macs using Tiger. Everything is 32 bit. The diffwrf program does not compile, but can probably be made to work fairly easily.

(1) Here are replacement files for the arch directory: configure_new.defaults, noopt_exceptions_f, and postamble_new

(2) Replacement file for the external/RSL_LITE directory: makefile

(3) Replacement file for the frame directory: frame.Makefile (rename as Makefile)

(4) Replacement file for the dyn_em directory: module_first_rk_step_part1.F

The file module_first_rk_step_part1.F is altered to remove unnecessary whitespace in the subroutine call to surface_driver. As it is, the number of characters in that call exceeds the allowable maximum for xlf on the Mac.

This image shows performance for the two-domain June 2001 test case on a G5 cluster (mpich1, no fancy interconnects whatsoever), a 12 hour run on the number of cpus indicated. The real times are the relevant numbers.

MM5 on Mac Intel (OpenMP)

2008-04-16T19:24:00.000-07:00

Although this is my WRF on Mac blog, I'm also still using MM5 for some projects, and need to run it on my Macs. Recently, I tried to use it on my 8 core Mac Pro, using Intel Fortran 10 and OpenMP. All this is done on a HFSX (case-sensitive) volume. So far, it appears to be running well, with some caveats. One is that the build is 32 bit. My attempts at 64 bit builds, even after invoking flags like -DDEC_ALPHA and -DSGI_IA64, found referenced on the web, were successful but segfault immediately on execution.

Another is that the model actually takes or requires one more thread than is requested using the OMP_NUM_THREADS specification. Thus, if I request 6 threads, it actually spawns and apparently uses 7. Additionally, despite having 8 cores, I cannot request more than 6 and have the model run successfully. It should not need more than 7 in that instance, and thus should run. I'm not sure why it does not.

Getting MM5 compiled on the Mac Pro was easy once a change was made to the way suffixes are handled in configure.user. Intel Fortran complained about #define type statements in the code that also had comments attached beyond column 72. The workaround was to force each Fortran program file through the CPP preprocessor. That was done by removing the rule at the bottom of configure.user that looks like this:
.F.o: $(RM) $@ $(FC) -c $(FCFLAGS) $*.F
This rule compiles files ending with .F without involving the CPP preprocessor. Removing it causes other rules already in place to first push .F files through the preprocessor, and then compile the .f files that result from that operation. Remember, this has to be done on a case-sensitive volume.

I also had to manually add #include to Util/parseconfig.c to avoid a compilation error. The portion of configure.user that concerns compilation looks like this:

RUNTIME_SYSTEM = "macintel" FC = ifort FCFLAGS = -I$(LIBINCLUDE) -pc32 -O3 -convert big_endian -fp-model precise -openmp -fpp -allow fpp-comments -auto -traceback #-DDEC_ALPHA CPP = /usr/bin/cpp CFLAGS = -O #-DSGI_IA64 CPPFLAGS = -I$(LIBINCLUDE) -I. -C -P -traditional -xassembler-with-cpp LDOPTIONS = $(FCFLAGS) -Wl,-stack_size -Wl,0x20000000 -Wl,-stack_addr -Wl,0xd0000000

Here is a sample timing plot for a triply nested 24 hour simulation, revealing results I am not unhappy with:

Playing with WRF3 (WRFV300, WPSV300) - part I

2008-04-05T22:55:00.001-07:00

WRF v.3.0.0 was released yesterday, and I decided to see if it would be any easier to get it to run well on my Intel-based Mac Pro, with the latest ifort 10 compiler. I also tested it on two Linux machines, one with the same chip type and ifort version, the other with older 32 bit hardware and software (ifort 9). I also built WRF with PGI Fortran on the older machine, for a total of four different combinations of hardware and software (one Mac, three Linux; three ifort, one PGI).

The good news is the configuration system is a lot more slick. The bad news for the Mac Pro is the executables would not build with the stock configure.wrf obtained for ifort and OMP. A few alterations were necessary on my Mac:

(1) I'm using 64 bit compilers, so I needed to add "-m64" to CFLAGS_LOCAL.

(2) To get rid of missing symbols (that didn't draw complaints on either Intel Linux build), I had to add "$(WRF_SRC_ROOT_DIR)/frame/module_domain_type.o $(WRF_SRC_ROOT_DIR)/external/io_grib2/grib2tbls_types.o" to LDFLAGS_LOCAL.

Then, the weirdness started. On the two Core Quad machines (Mac and Linux), the OMP ifort build doesn't spawn more than one thread or capture more than 100% according to top and Activity Monitor, despite the setting for OMP_NUM_THREADS. It seems OMP is broken, and it's not platform-specific.

WRF3 is not backward compatible with respect to input files, and this makes upgrading WPS a must. But the new version of WPS' geogrid program isn't working anywhere. It spits out a segfault (at the same place) even when trying to build the example domain. Same experience with ifort and PGI, on the two Linux machines. I verified that the WPS_GEOG data have not changed, so it's not that.

No Mac WPS yet since I don't have, and have been unable as of yet to successfully build, NCL for 64 bit on the Mac. I tried to create parallel 64 and 32 bit versions of WRF and WPS (and all other necessary software) to circumvent this, but decided not to continue since the new WPS is not working yet on the Linux boxes.

WRFV221 on dual quad Mac Pro

2008-03-26T11:10:00.000-07:00

Here are my recent trials of WRF v.2.2.1 on a dual-quad Mac Pro with Intel Fortran (10.1.012) and gcc. The 64 bit version of the compilers are used. The machine has 8 GB of memory. Trials include OMP and MPI versions, the latter based on mpich2.

--------------------------------------------------------------
netcdf compilation for 64 bit (based on netcdf-3.6.0-p1)
--------------------------------------------------------------
export CC=/usr/bin/gcc export CPPFLAGS="-O -DNDEBUG -DpgiFortran" export CFLAGS="-O -m64" export CXX=/usr/bin/c++ export CXXFLAGS="-O -m64" export FC=ifort export F77=ifort export F90=ifort export FFLAGS="-O3 -mp" export F90FLAGS="-O3 -mp" ./configure --prefix=/usr/local/netcdf make make test sudo mkdir /usr/local/netcdf sudo make install

--------------------------------------------------------------
jasper compilation for 64 bit (based on jasper-1.701.0)
--------------------------------------------------------------
setenv CC /usr/bin/gcc setenv CFLAGS "-O -m64" setenv CXX /usr/bin/c++ setenv CXXFLAGS "-O -m64" ./configure --prefix=/usr/local/jasper make sudo make install

--------------------------------------------------------------
mpich2 compilation for 64 bit (based on mpich2-1.0.5)
--------------------------------------------------------------
setenv FC ifort setenv F90 ifort setenv CC "gcc -m64" setenv RSHCOMMAND "/usr/bin/ssh" setenv CXX "/usr/bin/c++ -m64" setenv FFLAGS "-xP -vec- -fp-model precise" setenv F90FLAGS "-xP -vec- -fp-model precise" ./configure --with-comm=shared

--------------------------------------------------------------
test run
--------------------------------------------------------------

The test run is a short simulation with three telescoping, two-way domains (142x100, 100x100 and 100x100, with 35 vertical levels). Flags were chosen to reproduce output file cksums of a completely unoptimized simulation requesting strict arithmetic. The OMP version occasionally produces different results, apparently randomly.

Timing plot:

The plot below adds results from a single 2.4GHz quad-core machine, running Mandriva Linux, for OMP runs built using the same configuration file as linked below (save Mac-specific references removed). For the four thread run, the 2.4 GHz run is 33% slower, though the clock speed difference is only 17%. Checksums were the same for all the runs.

Configuration files are here: OMP version, MPI version.

[edited 1 April 2008 to include the compilation for netcdf, jasper and mpich2 and to clarify this is for 64 bit compilers.]

WRFV221 Mac PowerPC w/ IBM xlf and MPI

2007-12-20T23:51:00.000-08:00

My recent tests using the presently latest WRF version, v.2.2.1, on a PowerPC-based Mac using XLF and MPI appears to be all good, even with nesting. Two files are needed for this:

Additions for arch/configure.defaults.

Replacement for external/RSL_LITE/rsl_malloc.c. Rename file after downloading.

WRFV22 on a dual quad-core Intel Mac

2007-09-16T15:57:00.000-07:00

Here are some notes on a recent attempt to run WRF on a dual quad-core Intel Mac, which I received as a loaner from Apple. In the two weeks I had the machine, I made progress, but it's clear from my notes below I didn't pass the finish line. For one thing, I was unable to get even medium-sized jobs to run in the 64 bit environment -- and the tricks for accessing memory that worked with 32 bits fail. In 32 bit land, there is still a limit on the job size, which is apparently due to Apple's internal memory allocation restrictions.

Executive summary: I was able to get all 8 cpus working for me, tho the scaling wasn't the best, and the key turned out to be: (a) moving to mpich-2; AND (b) using --with-comm=shared. An important goal for me is getting results that do not vary with the number of processors used. With OMP, that required the ifort flag '-fp-model precise'. I also used this flag in the WRF code and when compiling mpich-2.

** These notes assume the 32 bit ifort compiler. I used ifort 10.0.017. These notes presume WRF model changes documented on previous posts. Compilation and execution took place on an HFSX-formatted disk.

(1) netcdf-3.6.2

export CC=/usr/bin/gcc
export CPPFLAGS="-O -DNDEBUG -DpgiFortran"
export CFLAGS="-O"
export CXX=/usr/bin/c++
export CXXFLAGS="-O"

export FC=ifort
export F77=ifort
export F90=ifort
export FFLAGS="-O3"
export F90FLAGS=

./configure --prefix=/usr/local/netcdf
make
make test
make check
sudo mkdir /usr/local/netcdf
sudo make install

(2) MPICH-2 (mpich2-1.0.5p4)

setenv FC ifort
setenv F90 ifort
setenv CC "gcc"
setenv RSHCOMMAND "/usr/bin/ssh"
setenv CXX "/usr/bin/c++"
setenv FFLAGS "-xP -vec- -fp-model precise"
setenv F90FLAGS "-xP -vec- -fp-model precise"
./configure --with-comm=shared
make

(3) Modify external/RSL_LITE/buf_for_proc.c to add "extern" before "char mess" (resolves a problem that crops up specifically with mpich-2)

(4) configure.wrf files I used for these tests: APPLE_LOANER_WRF.zip.

WRFV22 on Mac Intel update... ifort and g95

2007-07-05T16:09:00.001-07:00

The ZIP file linked below has been updated to reflect further progress made in getting WRFV22 to work on a Core Duo MacBook using Intel ifort and g95. As before, I am using mpich-1.2.5 for the MPI builds. My latest tests have employed ifort 10.0 and the most recent g95 at this writing. The g95 sections also are usable on PPC Macs.

For the most part, these configurations pass my "consistency tests", with some reservations. Consistency means that the MPI executables generate bitwise identical output, irrespective of the number of processors selected. However, if the domain is too finely subdivided, the results will start varying. I also encountered a test case in which identical results were generated for between 2-24 CPUs, inclusive, but results using only 1 CPU were different, which I cannot yet explain.

Results are not bitwise identical among compilers, or for the g95 versions when built on Intel and PPC architectures. However, I have not noted anything particularly amiss in the output. g95 executables from the PPC architecture can be executed on Intel (albeit slowly), and I've found this yields the same results obtained on PPC.

WRFV22_MacIntel_mods.zip (as of July 5, 2007).

WRFV22 on Mac Intel w/ ifort

2007-04-19T19:34:00.000-07:00

The zip archive below contains WRFV22 code modifications for an Intel-based Mac with the Intel Fortran compiler (ifort). I have been able to run the single-threaded and OMP versions without problem, but my tests with the MPI version (RSL and RSL_LITE) with mpich-1.2.5 have not produced results that are the same when one and two processors are requested. This has only been tested on a MacBook with a Core Duo processor. So, use at your own risk. Feedback appreciated.

WRFV22_MacIntel_mods.zip

WRFV22 g95 on Mac supporting programs

2007-04-15T22:50:00.000-07:00

Previously, I reported that I had to use tcsh to run WRFV22. Turns out I can run in bash if I specifically set the stacksize and datasize as follows

ulimit -s 65536
ulimit -d unlimited

To go along with the configure.defaults additions for g95 on Mac PPC, here are the flags I used for NetCDF and MPICH to get everything working. Specific versions of these programs I used were 3.6.0-p1 for NetCDF and 1.2.5 for MPICH.

netcdf-3.6.0-p1

* I used tcsh for the compilation

setenv CC /usr/bin/gcc
setenv CPPFLAGS "-O -DNDEBUG -DpgiFortran"
setenv CFLAGS "-O"
setenv CXX /usr/bin/c++
setenv CXXFLAGS "-O"

setenv FC g95
setenv F77 g95
setenv F90 g95
setenv FFLAGS "-O3 -fno-second-underscore"
setenv FCFLAGS "-O3 -fno-second-underscore"
setenv F90FLAGS "-O3 -fno-second-underscore"

./configure
* I then edited "macros.make", adding:
FLIBS = -lSystemStubs
F90LIBS = -lSystemStubs

./make
./make test
[all tests were found to work]
make install

mpich-1.2.5

* I used tcsh for the compilation

setenv FC g95
setenv F90 g95
setenv RSHCOMMAND "/usr/bin/ssh"
setenv CC gcc
setenv CXX gcc
setenv LIBS "-lSystemStubs"
setenv FFLAGS "-fno-second-underscore"
setenv F90FLAGS "-fno-second-underscore"
setenv CFLAGS "-fno-common -DFORTRANUNDERSCORE"

./configure --with-device=ch_p4 --without-romio
./make

* On one Mac PPC machine, I had to add -lstdc++ to LIBS, but another failed to work if that was done

* Although mpich compiles fine, the example programs would not compile without manually adding the "-fno-second-underscore" flag. Example:

mpif77 -fno-second-underscore -c fpi.f
mpif77 -o fpi fpi.o

WRFV22 g95 update 2/21/07

2007-02-21T07:31:00.000-08:00

Here is an update on g95 with WRF version 2.2 on Mac OS X (PPC G5) and a Linux (Athlon) box. As noted below, WRFV22 produces usable (but not yet verified correct) runs with nesting on PPC with g95, whereas xlf still does not. This is likely an xlf compiler bug specific to OS X. The test case for this particular experiment consists of a single 150 x 150 grid point domain. Model configurations, independent of compiler, are: RSL_LITE, mpich-1.2.5, allows nesting. The executables represent

g95 on OS X/PPC (donar_g95)
xlf on OSX/PPC (donar_xlf)
g95 on Mandriva Linux/Athlon (cascade_g95)
ifort on Mandriva Linux/Athlon (cascade_ifort)

The first thing I check is to see if the model produces the same results, as verified by checksums and visual inspection of plotted fields, independent of the number of cpus -- at least until the domain becomes too finely subdivided. So far in this experiment, only g95 on OS X/PPC is passing that test. (It is likely that ifort will also pass, but that portion of the experiment is not yet finished.) The checksums are the same for runs with 1 to 24 cpus. For xlf on OS X/PPC, runs with 1, 2, 12 and 18 cpus produce one particular checksum while 4, 6, 8, 10 and 14 cpus result in another. 16 and 20 cpus result in a third checksum, and 24 cpus produces a fourth sum. The 12 hour forecast fields differ in some respects, with patterns that suggest roundoff errors.

Unsurprisingly, g95 is slower than a commercial compiler on the same hardware. The plot below presents timing results obtained thusfar. There are gaps in the data, and some degree of inconsistency regarding how the time function works on OS X and Linux. I will redo these statistics using Brian Jewett's scripts for extracting timings from the rsl.out.0000 files. Also, for g95 on OS X/PPC, I/O is a bottleneck. Running with nio_tasks_per_group > 0 appears to help a lot.

Click the image to open a larger version in a new window.

Here is the latest configure.wrf segment for my g95 OS X PPC runs.

Experiments with g95 on OSX/PPC and Linux/x86

2007-02-11T10:33:00.000-08:00

I have been trying to get WRF nesting working properly on a Mac PPC cluster. Eventually, this will expand to a Mac Intel machine/cluster. This effort will be successful when the nesting/MPI code produces: (1) the same results on a given machine, independent of the number of nodes requested; and (2) results that do not diverge significantly from those obtained on other platforms. The first efforts involved the IBM xlf compiler. I have been able to get WRFV212 and WRFV22 to compile and run, but the results vary with the number of modes requested -- at least for runs involving more than one domain. Single domain runs have been fine, independent of the number of processors employed.

Frustration with this led me to consider the g95 compiler. For comparison and context, I'm running WRFV22 with g95 on the Mac PPC cluster ("donar") as well as on an Athlon cluster ("cascade") running Mandriva Linux. Both g95 compilers are version 0.91, the most recent at this writing. The latter also has Intel Fortran (version 9.1.036). Builds of WRF with g95 use netcdf (version 3.6.0-p1) and mpich (version 1.2.5) compiled using g95; ifort builds use ifort-made versions of the same software.

Test case is a two domain run (D1 is 40x40 at 60 km resolution; D2 is 28x28 at 20 km; 31 vertical grid levels), run for 12 hours over the midwest US.

Status as of 11 February 2007: Results vary with the number of cpus requested for all combinations tested thusfar, but the point of divergence depends on the compiler and optimization level. At some extreme, it is possible that dividing up a small domain too finely might provoke rounding errors; this is speculation. Divergence is determined by two criteria, both of which must be present: difference in cksum on the wrfout_d02 files, and differences in fields visualized using GrADS, created using wrf_to_grads. In some cases, wrfout_d02 files have differed only in a few apparently unimportant lines (ascertained by examining dumps with ncdump) for a variable called snow density.

Summary:

cascade/ifort -- results diverge after 6 cpus (results for 1-6 cpus identical)
cascade/g95 standard - results diverge after 4 cpus (results for 1-4 cpus identical)
cascade/g95 debug -- results diverge after 3 cpus (results for 1-3 cpus identical)
donar/g95 debug -- results diverge after 2 cpus (results for 1 and 2 cpus identical)
donar/g95 standard -- results diverge after 6 cpus (results for 1-6 cpus identical)

Results differ for g95 runs between the two platforms even at the same optimization level.

One next step may be to work with a larger domain, to see if it delays divergence as the number of processors requested increases.