This post documents the method I used for building and running the CESM 1.2 on a Mac running OSX 10.9.5 (Mavericks). I also included some debugging notes for my own reference. This took several weeks of tinkering to streamline the process, so I hope this helps other researchers looking for a convenient way to run single column model (SCM) experiments.
Get the system ready to build the model
- install homebrew
- This makes installing any miscellaneous missing libraries very simple
- switching libraries around is also very straightforward (ex. openmpi vs. mpich2)
- install the GNU compiler (gcc) with homebrew (includes gfortran)
- This is what I used to build the model, since pgi and intel are not free
- I used version 4.9.2 from homebrew, but older versions might work with some minor adjustments
- install netcdf-c and netcdf-fortran libraries
- I recommend building from source code (which can be found here)
- Some install notes can be found here and here. It’s very simple if you just make a folder in your home directory called “NETCDF”, and build the libraries there.
- Make sure to modify the PATH environmental variable in your .bashrc or .bash_profile file so that the system knows to use the new libraries you built
- Note: you can install NetCDF with homebrew, but only the C library, not fortran. This is why I found it better to just build from source.
- Edit the “userdefined” machine entry in the “config_compilers” XML file, which can be found at:
The entry that needs to be changed will look like this:
<compiler MACH="userdefined"> <NETCDF_PATH> USERDEFINED_MUST_EDIT_THIS</NETCDF_PATH> <PNETCDF_PATH></PNETCDF_PATH> <ADD_SLIBS># USERDEFINED $(shell $(NETCDF_PATH)/bin/nc-config --flibs)</ADD_SLIBS> <ADD_CPPDEFS></ADD_CPPDEFS> <CONFIG_ARGS></CONFIG_ARGS> <ESMF_LIBDIR></ESMF_LIBDIR> <MPI_LIB_NAME></MPI_LIB_NAME> <MPI_PATH></MPI_PATH> </compiler>
I edited mine so that the environment variable NETCDF_PATH pointed to the netcdf libraries that I built from source. You might need to edit the MPI section of this for doing parallel simulations, but that’s not my concern here.
<compiler MACH="userdefined"> <NETCDF_PATH> ~/NetCDF/bld-netcdf-c-4.3.2 </NETCDF_PATH> <PNETCDF_PATH></PNETCDF_PATH> <ADD_SLIBS># USERDEFINED $(shell $(NETCDF_PATH)/bin/nc-config --flibs)</ADD_SLIBS> <ADD_CPPDEFS></ADD_CPPDEFS> <CONFIG_ARGS></CONFIG_ARGS> <ESMF_LIBDIR></ESMF_LIBDIR> <MPI_LIB_NAME></MPI_LIB_NAME> <MPI_PATH></MPI_PATH> </compiler>
- Create a custom IOP file
- IOP stands for “Intensive Observing Period”. Typically this data comes the sounding array of a field campaign, such as TOGA-COARE, BOMEX, or DYNAMO.
- This is also sometimes referred to as the “forcing” file. It includes everything that the single column model needs to know about what’s happening outside the model domain. For example, moisture and temperature might be converged into the model domain by the large-scale dynamics at different rates depending on the respective gradients and wind field.
- I wanted to try and run the model to radiative-convective equilibrium (RCE), so I made a NCL script to create a synthetic IOP file that set the surface temperature, but otherwise had zero large-scale forcing. I’m happy to share this script with anyone who is interested
Build the model
I automated this part of the process with a hefty PYTHON script. The reason for this is that I’m interested in running large ensembles, which many people advocate. I’m happy to share this script with anyone who is interested.
- Create a new case
This part simply creates the directories and files you need to build the model. Each case is a separate build of the model. You have to invoke the “create_newcase” script in the source code directory. Then you have to specify where the case files will go with the “-case” flag. Note that the case directory here is not where the model output goes. You also must specify the resolution, the machine, and the compset. For this example, I’m calling the new case “SCAM_TEST_T42”.
CESM_SRC/cesm1_2_0/scripts/create_newcase -case Cases/SCAM_TEST_T42/ -res T42_T42 -mach userdefined -compset F_2000
Above I’ve used a simple “F” component set, which is just the atmospheric component (no active ocean or sea ice). However, we actually need a special component set to run CESM in SCM mode. To do this we can replace “-compset F_2000” with a custom on-the-fly component set like this:
This is nearly identical to “F_2000”, except that “CAM5%SCAM” appears.
Then move into the case directory and use the xmlchange script to modify some of the xml files used for configuring and building.
./xmlchange -file env_build.xml -id OS -val darwin ./xmlchange -file env_build.xml -id MPILIB -val mpi-serial ./xmlchange -file env_build.xml -id COMPILER -val gnu ./xmlchange -file env_build.xml -id EXEROOT -val scratch/SCAM_TEST_T42/bld ./xmlchange -file env_run.xml -id RUNDIR -val scratch/SCAM_TEST_T42/run ./xmlchange -file env_run.xml -id DIN_LOC_ROOT -val ~/Model/CESM/inputdata ./xmlchange -file env_mach_pes.xml -id MAX_TASKS_PER_NODE -val 1
Here I’ve specified a “scratch” directory, where the built model executable and output data will go. For larger systems, like NCAR’s Yellowstone, this would be a whole different data storage system, independent from the system that the case directory resides on.
I’ve also specified an “inputdata” directory. Before the build stage, the model will download lots of input data it needs like aerosol concentrations and topography files. The files it downloads depend on compset. Note that these files can be larg
- Configure the Case
The next step is to configure the model. First I specify some aspects of how I want the model to be run.
./xmlchange -file env_run.xml -id RUN_STARTDATE -val 2000-01-01 ./xmlchange -file env_run.xml -id STOP_N -val 5 ./xmlchange -file env_run.xml -id SSTICE_YEAR_START -val 2000 ./xmlchange -file env_run.xml -id SSTICE_YEAR_END -val 2001 ./xmlchange -file env_run.xml -id PTS_MODE -val TRUE ./xmlchange -file env_run.xml -id PTS_LAT -val 0 ./xmlchange -file env_run.xml -id PTS_LON -val 155 ./xmlchange -file env_run.xml -id SSTICE_DATA_FILENAME -val rce_iop.nc
For this example, the model will start on Jan. 1, 2000, and run for 5 days. The model is set to as if it were a point on the equator in the Pacific Ocean at 155oE. I’ve also specified the custom input file that I’ve created called “rce_iop.nc”. You will also need to specify this in the namelist file.
Now the only thing to do is run the configuration script that can be found in the case directory. The “clean” step is only for when you are re-configuring a model after you changed something, but it doesn’t hurt to have it in there if you are automating this whole process with a script.
./cesm_setup -clean ./cesm_setup
- Build the model
There are several scripts in the case directory that start with the case name. The bain build script is one of these. Simple type the name of the script in the terminal, but make sure to use “./” in the front, otherwise the system won’t know that you want to execute the script.
- Run the model
There are two important scripts for running the model:
The submit script is used on larger systems for submitting a parallel job that uses many processors. Since we are only really running in serial on a mac, we don’t need to worry about this script. We just need to edit the run script a little. There will be two lines that are commented out, like this:
#mpiexec -n 1 $EXEROOT/cesm.exe >&! cesm.log.$LID #mpirun -np 1 $EXEROOT/cesm.exe >&! cesm.log.$LID
All we need to do is uncomment one of these. I uncomment the “mpirun” line, but I’m not sure what the difference between mpirun and mpiexec are. After that we can finally run the model!
Hopefully this is enough info to help someone who is just starting out trying to build CESM. You’re welcome to email me with any questions, but I probably won’t be able to answer system-specific issues.
EDIT Jun 4 2015:
I forgot to mention an important bug. When I first did this I kept getting an unexplainable segmentation fault. I was tearing my hair out for weeks! But after much googling I realized that the “stack size” is too small by default. You can check (and set) the stack size with the unix command “ulimit -s“. On an mac the maximum you can set it to is ulimit -s 65532.