CESM: Building CESM 1.2 on Mac OSX

This post documents the method I used for building and running the CESM 1.2 on a Mac running OSX 10.9.5 (Mavericks). I also included some debugging notes for my own reference. This took several weeks of tinkering to streamline the process, so I hope this helps other researchers looking for a convenient way to run single column model (SCM) experiments.

Get the system ready to build the model

  1. install homebrew
    • This makes installing any miscellaneous missing libraries very simple
    • switching libraries around is also very straightforward (ex. openmpi vs. mpich2)
  2. install the GNU compiler (gcc) with homebrew (includes gfortran)
    • This is what I used to build the model, since pgi and intel are not free
    • I used version 4.9.2 from homebrew, but older versions might work with some minor adjustments
  3. install netcdf-c and netcdf-fortran libraries
    • I recommend building from source code (which can be found here)
    • Some install notes can be found here and here. It’s very simple if you just make a folder in your home directory called “NETCDF”, and build the libraries there.
    • Make sure to modify the PATH environmental variable in your .bashrc or .bash_profile file so that the system knows to use the new libraries you built
    • Note: you can install NetCDF with homebrew, but only the C library, not fortran. This is why I found it better to just build from source.
  4. Edit the “userdefined” machine entry in the “config_compilers” XML file, which can be found at:
    cesm1_2_0/scripts/ccsm_utils/Machines/config_compilers.xml

    The entry that needs to be changed will look like this:

    <compiler MACH="userdefined">
     <NETCDF_PATH> USERDEFINED_MUST_EDIT_THIS</NETCDF_PATH>
     <PNETCDF_PATH></PNETCDF_PATH>
     <ADD_SLIBS># USERDEFINED $(shell $(NETCDF_PATH)/bin/nc-config --flibs)</ADD_SLIBS>  <ADD_CPPDEFS></ADD_CPPDEFS>
     <CONFIG_ARGS></CONFIG_ARGS>
     <ESMF_LIBDIR></ESMF_LIBDIR>
     <MPI_LIB_NAME></MPI_LIB_NAME>
     <MPI_PATH></MPI_PATH>
    </compiler>

    I edited mine so that the environment variable NETCDF_PATH pointed to the netcdf libraries that I built from source. You might need to edit the MPI section of this for doing parallel simulations, but that’s not my concern here.

    <compiler MACH="userdefined">
     <NETCDF_PATH> ~/NetCDF/bld-netcdf-c-4.3.2 </NETCDF_PATH>
     <PNETCDF_PATH></PNETCDF_PATH>
     <ADD_SLIBS># USERDEFINED $(shell $(NETCDF_PATH)/bin/nc-config --flibs)</ADD_SLIBS>  <ADD_CPPDEFS></ADD_CPPDEFS>
     <CONFIG_ARGS></CONFIG_ARGS>
     <ESMF_LIBDIR></ESMF_LIBDIR>
     <MPI_LIB_NAME></MPI_LIB_NAME>
     <MPI_PATH></MPI_PATH>
    </compiler>
  5. Create a custom IOP file
    • IOP stands for “Intensive Observing Period”. Typically this data comes the sounding array of a field campaign, such as TOGA-COARE, BOMEX, or DYNAMO.
    • This is also sometimes referred to as the “forcing” file. It includes everything that the single column model needs to know about what’s happening outside the model domain. For example, moisture and temperature might be converged into the model domain by the large-scale dynamics at different rates depending on the respective gradients and wind field.
    • I wanted to try and run the model to radiative-convective equilibrium (RCE), so I made a NCL script to create a synthetic IOP file that set the surface temperature, but otherwise had zero large-scale forcing. I’m happy to share this script with anyone who is interested

Build the model

I automated this part of the process with a hefty PYTHON script. The reason for this is that I’m interested in running large ensembles, which many people advocate. I’m happy to share this script with anyone who is interested.

  1. Create a new case
    This part simply creates the directories and files you need to build the model. Each case is a separate build of the model. You have to invoke the “create_newcase” script in the source code directory. Then you have to specify where the case files will go with the “-case” flag. Note that the case directory here is not where the model output goes. You also must specify the resolution, the machine, and the compset. For this example, I’m calling the new case “SCAM_TEST_T42”.

    CESM_SRC/cesm1_2_0/scripts/create_newcase -case Cases/SCAM_TEST_T42/ -res  T42_T42 -mach  userdefined -compset F_2000

    Above I’ve used a simple “F” component set, which is just the atmospheric component (no active ocean or sea ice). However, we actually need a special component set to run CESM in SCM mode. To do this we can replace “-compset F_2000” with a custom on-the-fly component set like this:

    -user_compset 2000_CAM5%SCAM_CLM40%SP_CICE%PRES_DOCN%DOM_RTM_SGLC_SWAV

    This is nearly identical to “F_2000”, except that “CAM5%SCAM” appears.
    Then move into the case directory and use the xmlchange script to modify some of the xml files used for configuring and building.

    ./xmlchange -file env_build.xml    -id OS           -val darwin 
    ./xmlchange -file env_build.xml    -id MPILIB       -val mpi-serial 
    ./xmlchange -file env_build.xml    -id COMPILER     -val gnu
    ./xmlchange -file env_build.xml    -id EXEROOT      -val scratch/SCAM_TEST_T42/bld
    ./xmlchange -file env_run.xml      -id RUNDIR       -val scratch/SCAM_TEST_T42/run
    ./xmlchange -file env_run.xml      -id DIN_LOC_ROOT -val ~/Model/CESM/inputdata
    ./xmlchange -file env_mach_pes.xml -id MAX_TASKS_PER_NODE -val 1

    Here I’ve specified a “scratch” directory, where the built model executable and output data will go. For larger systems, like NCAR’s Yellowstone, this would be a whole different  data storage system, independent from the system that the case directory resides on.

    I’ve also specified an “inputdata” directory. Before the build stage, the model will download lots of input data it needs like aerosol concentrations and topography files. The files it downloads depend on compset. Note that these files can be larg

  2. Configure the Case
    The next step is to configure the model. First I specify some aspects of how I want the model to be run.

    ./xmlchange -file env_run.xml   -id RUN_STARTDATE        -val 2000-01-01
    ./xmlchange -file env_run.xml   -id STOP_N               -val 5
    ./xmlchange -file env_run.xml   -id SSTICE_YEAR_START    -val 2000
    ./xmlchange -file env_run.xml   -id SSTICE_YEAR_END      -val 2001
    ./xmlchange -file env_run.xml   -id PTS_MODE             -val TRUE 
    ./xmlchange -file env_run.xml   -id PTS_LAT              -val 0 
    ./xmlchange -file env_run.xml   -id PTS_LON              -val 155 
    ./xmlchange -file env_run.xml   -id SSTICE_DATA_FILENAME -val rce_iop.nc

    For this example, the model will start on Jan. 1, 2000, and run for 5 days. The model is set to as if it were a point on the equator in the Pacific Ocean at 155oE. I’ve also specified the custom input file that I’ve created called “rce_iop.nc”. You will also need to specify this in the namelist file.

    Now the only thing to do is run the configuration script that can be found in the case directory. The “clean” step is only for when you are re-configuring a model after you changed something, but it doesn’t hurt to have it in there if you are automating this whole process with a script.

    ./cesm_setup -clean
    ./cesm_setup
  3. Build the model
    There are several scripts in the case directory that start with the case name. The bain build script is one of these. Simple type the name of the script in the terminal, but make sure to use “./” in the front, otherwise the system won’t know that you want to execute the script.

    ./SCAM_TEST_T42.build
  4. Run the model
    There are two important scripts for running the model:

    SCAM_TEST_T42.run
    SCAM_TEST_T42.submit

    The submit script is used on larger systems for submitting a parallel job that uses many processors. Since we are only really running in serial on a mac, we don’t need to worry about this script. We just need to edit the run script a little. There will be two lines that are commented out, like this:

    #mpiexec -n 1 $EXEROOT/cesm.exe >&! cesm.log.$LID
    #mpirun -np 1 $EXEROOT/cesm.exe >&! cesm.log.$LID

    All we need to do is uncomment one of these. I uncomment the “mpirun” line, but I’m not sure what the difference between mpirun and mpiexec are. After that we can finally run the model!

  5. ./SCAM_TEST_T42.run

 


Hopefully this is enough info to help someone who is just starting out trying to build CESM. You’re welcome to email me with any questions, but I probably won’t be able to answer system-specific issues.

EDIT Jun 4 2015:
I forgot to mention an important bug. When I first did this I kept getting an unexplainable segmentation fault. I was tearing my hair out for weeks! But after much googling I realized that the “stack size” is too small by default. You can check (and set) the stack size with the unix command “ulimit -s“. On an mac the maximum you can set it to is ulimit -s 65532.

32 thoughts on “CESM: Building CESM 1.2 on Mac OSX

  1. Inger Helene

    Hi,

    I’m also going to use SCAM, but this is my very first time. I got a lot of help reading your blog, even though I’m using Linux.

    I’ve managed to run the single column model over land, but I get some problems with the sea ice (!) when I try to run it for an area over ocean (Missing grid file for sea ice). Did you have any problems like this? After adding some extra inputfiles in user_nl_cice, I don’t get that error any longer, but this one:

    CalcWorkPerBlock: Total blocks: 1 Ice blocks: 1 IceFree blocks: 0 Land blocks: 0
    MCT::m_AttrVectComms::GSM_gather_: Input GlobalSegMap haloed–not allowed error, stat =1
    000.MCT(MPEU)::die.: from MCT::m_AttrVectComms::GSM_gather_()
    MPI_Abort: error code = 2
    Application 1294440 exit codes: 2
    Application 1294440 resources: utime ~28s, stime ~4s, Rss ~338380, inblocks ~5255558, outblocks ~551409

    Have you got any errors like this, or have any suggestions of what to do?

    Thanks,
    Inger Helene H. Karset (PhD-student at the University of Oslo)

    Reply
    1. Walter Post author

      Inger, I’m glad my post helped!
      Your issue sounds a bit strange. Are you wanting to run the model with the active CICE component?
      In my case I used prescribed sea ice, and if I’m remembering this right, this ultimately comes from whatever file is set to the “ncdata” cam namelist variable. In my case I used a file named:
      cami_0000-01-01_64x128_L30_c090102.nc
      So, I would look into that first. There should be a variable like “SSTICE”.
      Feel free to email me with any more questions.

      Reply
      1. Inger Helene

        Thank you so much for your reply. I understand if you got other things to do than solving my problems, but I give it a try!

        I don’t want to run with active sea ice, but prescribed.

        The file that you are refers to doesn’t contain any SSTICE-variable (the only variables in this file concerning ice are TSICE, TSICERAD, SICTHK, ICEFRAC). I’m using the same file, but I get error messages about missing CICE-files. Do you think the file that’s set in env_run for SSTICE_DATA_FILENAME can be the source to my problem instead? From your blog, I can see that you are using your iop-file here. Does your iop-file contains any information about the sea ice? I also tried to use my iop-file here, but I got error messages about the date. Do you know if there any dates that have to be similar i ncdata-file, iop-file and this SSTICE_DATA_FILENAME-file?

        Reply
        1. Walter Post author

          Inger,
          That’s a curious problem… My synthetic IOP file for the ncdata path has variables for surface air and ground temperature (Tg and Tsair), but nothing specifically about sea/ground ice.

          What version do you have? this can make a huge difference. The CESM people are always changing variables and what they do.

          You could also try use the “stub” or “dead” CICE component. I’d be curious to hear if you could get that to work.

          Reply
          1. Inger Helene

            I’m using CESM1.2.2

            I don’t think I can run the model with DICE since I’m using active atmosphere (so I need CICE%PRES to give the right data to calculate the sst needed by the active atmosphere. See comment in this link: https://bb.cgd.ucar.edu/standalone-atmosphere-component).

            I believe the source to my problem is the SST-file I’m using (the one set in env_run.xml as SSTICE_DATA_FILENAME), so I think I’ll try to change that one. I just don’t know how yet:) All the SST-files I’ve tried out (sst_HadOIbl…-files from NCAR-input data) contains both ice_cov and ice_cov_prediddle.

            You will probably hear from me again soon!

            cheers,

            Inger Helene

  2. shoon

    I find “gmake” at following locations:
    ./Tools/config_definition.xml: value=”gmake”
    gmake complib -j $GMAKE_J MODEL=cam COMPLIB=$LIBROOT/libatm.a USER_CPPDEFS=”$camdefs” -f $CASETOOLS/Makefile || exit 2

    I changed gmake to make in these files. and atm builds longer than before.

    also, I think I should change gmake to make at those places too:
    ./Buildconf/cesm.buildexe.csh:gmake exec_se -j $GMAKE_J EXEC_SE=$EXEROOT/cesm.exe MODEL=driver \
    ./Buildconf/cice.buildexe.csh:gmake complib -j $GMAKE_J MODEL=cice COMPLIB=$LIBROOT/libice.a MACFILE=$CASEROO

    There’s a new error pop up when it builds atm but I guess it is something else. The atm.bldlog shows:
    …/cesm1_1_1/models/atm/cam/src/physics/cam/micro_mg_cam.F90:912:25:
    tnd_nsnow, re_ice, gamma, vqsatd_water, polysvp, &
    1
    Error: Intrinsic ‘gamma’ at (1) is not allowed as an actual argument

    Reply
    1. Walter Post author

      ah ok, maybe this all is because you are using version 1.1.1? That would make sense… sort of.

      Does that mean you are trying to run SP-CAM on a Mac?!? That’s a bad idea, unless you want to run the single-column setup, and even then it’s probably better to do it on a Linux machine.

      Feel free to email me to talk more. walter@hannahlab.org

      Reply
      1. shoon

        Thank you for your suggestion.
        I just try to test the model (SD-WACCM) on my own mac, since it crashes in the second step (I have some coupled physics package with it). I guess I should run without SD/SP settting, because it is very consuming?

        I guess I will update my model to cesm1.2.0. When you talk about SP-CAM when I say I use cesm1.1.1. Does it mean cesm1.2.0 cannot run SP or SD for some reasons?

        Reply
        1. Walter Post author

          The SP stands for super-parameterization. CESM 1.1.1 is a special development version that was used for implementing SP. If you are using a SD-WACCM configuration that you definitely aren’t using SP (which is good).

          It sounds like you would benefit from moving to the latest version (1.2). There were several improvements to how the model is run in version 1.2. It definitely will make it easier to build on a mac.

          Reply
  3. Ray

    Hi, thanks for sharing. I find your blog very useful. Just a dumb question: I am wondering whether a personal computer, like macbook pro or imac, is capable for running or just playing with CESM. I am assuming you built it on your personal mac machine right? Thanks!

    Reply
    1. Walter Post author

      I originally did this on a 2010 macbook pro, so you can definitely use a personal computer. It would be nice to be able to run a variety of single column simulations “right out of the box” so people could play with the model on their own, but there aren’t many options for this. Awhile back I was trying to make a component set which would run the model to radiative-convective equilibrium, which does not require any input or forcing data, but I never got that working. Maybe the people at NCAR will make the next version a little more friendly to tinkering.

      Reply
  4. Frank

    Hi Walter,

    Thanks for your posts, they are very helpful. When building NetCDF, do you know if it is necessary to have NetCDF built to support Parallel IO?

    Best,
    Frank

    Reply
    1. Walter Post author

      Glad I could help! I’ve never had to do anything special in regards to parallel I/O when building any libraries for CESM. I don’t think it’s used in the model. There’s probably a reason for this, but I don’t know enough about the parallel NetCDF libraries.

      Reply
  5. Mortis

    Hi, this is a very useful intro!

    Just one question: in my simulations (BOMEX, DYCOMS) I don’t see any changes in my U and V profiles near the surface (although their forcings are non-zero as indicated by UAP and VAP). Any idea what can be a cause? Is there any flag that controls it (something about updating or relaxing horizontal velocity)?
    Do you maybe have an example xml file for these idealized test cases?

    Mortis

    Reply
    1. Walter Post author

      This is interesting, but I’m not sure what could be causing it. Perhaps it is caused by the convective momentum transport? You could check this by adding ZMMTU and ZMMTV to the output variables. I’m not familiar with the the variables “UAP” or “VAP”, are these specific to the BOMEX forcing file? Maybe those variables need to be changed?
      At one point I made a list of the minimum required variables for an IOP file:
      bdate Base date
      time Time after 0Z on base date
      lev pressure levels
      lat Latitude
      lon Longitude
      phis Surface geopotential
      t Temperature
      divt Horizontal T advective tendency
      q Specific humidity
      divq Horizontal Q advective tendency
      ps Surface pressure
      omega Vertical pressure velocity
      u U wind
      v V windspeed
      usrf U wind at surface
      vsrf V wind at surface
      pt Surface pressure tendency
      These could be different in different versions of CESM, and this is based on my own detective work so it could be completely wrong.
      Lemme know when you figure it out!

      Reply
      1. Mortis

        UAP and VAP are ‘U after physics’ and ‘V after physics’ variables (similar to U_aft_pbl and V_aft_pbl). For some reason they do not become a new state (e.g. state%u, contrary to moisture q) at the end of a time step. In effect, they do not evolve with time…
        My first thought was that there are perhaps some additional flags, which control it. My convection scheme finds non-zero forcings for q, u and v (fu and fv), my state variables get modified after calling for tendencies updates (I’ve checked it), but then, somewhere around the end of a time step or at the beginning of a new time step these changes in u and v are reversed.

        My input data files include all the variables you’ve listed above.

        Reply
        1. Walter Post author

          That’s odd…. Maybe there is some kind of relaxation mode that is undoing those tendencies. I’m not sure where to look for that though. Sorry I can’t be more help.

          Reply
  6. Mortis

    In my setup, scm_relaxation = .false. (default). I don’t even have to explicitly state that in my xml file, but I did it to avoid any surprises. But it didn’t help. That’s weird…

    Have you run any idealized cases? Or do you normally see any evolution of your U and V profiles (different from e.g. input U and V that change in time)?

    Reply
    1. Walter Post author

      To be honest, I haven’t run any SCM experiments in awhile since I’ve been focusing on super-parameterized simulations. I ran into some issues trying to get a working RCE compset, but they were just setup issues, not run-time or data issues. The more I think about this I’m starting to wonder if this is actually the correct behavior, since a SCM can’t really prognose the large-scale wind field. How is the wind in your IOP and initial condition files set? zero?

      Reply
      1. Mortis

        In my IOP file, initial u and v are constant with height (e.g. 7 and -5.5 m/s for DYCOMS). Because my IOP file for idealized runs only has 2 time levels (start and end), the profiles are exactly the same at these two times. These velocities should slow down near the surface, which doesn’t happen.

        I know that for ARM97 case (which is another “idealized” case) the IOP file has a large number of time levels and the U and V profiles strongly evolve with time. And probably because of that, the output U and V follow these changes (so they look more realistic). But I’m not sure if this is just because of some relaxation or also due to boundary layer physics.

        Reply
        1. Walter Post author

          The more I think about this, the more I think that the wind profile should not be able to evolve on its own. I would bet that if you changed the wind profile of the last time in your IOP file, the output winds would change, and basically be an interpolation of the first and last time. The model doesn’t have any of the information it would need to prognose the wind. It doesn’t have the horizontal gradients of wind or geopotential, so it can’t estimate the terms of the momentum budget. I think this might be why the winds are constantly corrected to be the same as the wind profile in your IOP, because SCMs have to be designed this way. Does that make sense?

          Reply
          1. Mortis

            Well, this is a very good question. In my opinion, initial U and V define geostrophic wind and the boundary layer dynamics should be able to modify the profiles accordingly (e.g. due to turbulent transport of momentum). We can start our simulation from an idealized state (similarly to LES), but then the physics should attenuate all the profiles towards a quasi-steady realistic state. Otherwise our SCM models would be useless (?)

            I know that people simulated DYCOMS and BOMEX cases using SCAM, but I don’t think they got correct evolution of moisture without having physical values for U and V. They are critical for the turbulent transport of heat and moisture as TKE can only be produced by buoyancy or shear.

  7. Mortis

    Hm, I just noticed these two variables:
    usrf U wind at surface
    vsrf V wind at surface

    I don’t have them in my IOP files for DYCOMS and BOMEX. But they do exist in the ARM97 case.
    However, I would expect my PBL scheme to be able to find some realistic velocities near the surface (form Monin-Obukhov theory), am I right?

    Reply
      1. Mortis

        Unfortunately, adding usrf and vsrf to the IOP file and setting them to 0 didn’t solve the problem. U and V still don’t evolve with time. Very weird…

        Reply
    1. Walter Post author

      I’m actually not sure what I was using when I wrote this post. Most likely 1.2.2, but I’ve cycled through a several versions now, and they can be quite different underneath the hood, especially when it comes to building them in OSX. I’ve found that all these versions have their own special setup issues and quarks. Sometimes legacy options get disabled, but the model doesn’t warn of anything amiss. I can kind of see why this happens, and I see where they’re headed, but in the interim it’s causing some big headaches.

      But to answer your question more directly, I did get 1.2.2 to work on a mac.

      Reply
  8. Pingback: Port CESM on Mac OS X – Yujia You

  9. Yujia You

    Hi Walter,

    I follow your steps. But when I do ./case_name.run, I encountered problems. Could you please do me a favor. I solved some problems before this step, but I really have no idea about this one. Thank you so much!
    ————————————————————————-
    CESM BUILDNML SCRIPT STARTING
    – To prestage restarts, untar a restart.tar file into /Users/YYJ/EXEROOT/run
    infile is /Users/YYJ/Documents/text1/Buildconf/cplconf/cesm_namelist
    CESM BUILDNML SCRIPT HAS FINISHED SUCCESSFULLY
    ————————————————————————-
    ————————————————————————-
    CESM PRESTAGE SCRIPT STARTING
    – Case input data directory, DIN_LOC_ROOT, is /Users/YYJ/inputdata
    – Checking the existence of input datasets in DIN_LOC_ROOT
    CESM PRESTAGE SCRIPT HAS FINISHED SUCCESSFULLY
    ————————————————————————-
    Wed Sep 14 15:53:05 CDT 2016 — CSM EXECUTION BEGINS HERE
    Wed Sep 14 15:53:05 CDT 2016 — CSM EXECUTION HAS FINISHED
    grep: cpl.log.160914-155239: No such file or directory

    Reply
    1. Walter Post author

      I actually remember getting a similar error, but I don’t remember what the problem was. I bet the solution is simple…. from the timestamp on the “BEGINS” and “FINISHED” lines it appears that the model didn’t actually to do anything.
      Are you using STOP_N or something like STOP_DATE in your namelist?
      You’re not using the submit script right? only the “run” script?

      Reply
    2. Fuchang Wang

      In YOUR_CASE.run, you will find lines below:

      # ————————————————————————-
      # # Run the model
      # # ————————————————————————-
      sleep 5
      cd $RUNDIR
      echo “`date` — CSM EXECUTION BEGINS HERE”
      mpirun -np 480 $EXEROOT/cesm.exe >&! cesm.log.$LID
      echo “`date` — CSM EXECUTION HAS FINISHED”

      Noticing two echo lines happens at the same time: Wed Sep 14 15:53:05 CDT 2016
      indicating command: mpirun didn’t works properly.

      Searching ansers around, this document gives hints:
      http://www1.udel.edu/it/research/files/cluster/workshop/hpc_hanson_Apr2015.pdf
      cesm.exe: error while loading shared libraries: libnetcdf.so.7: cannot open shared object file: No such file or directory

      mpi cannot locate needed certain libnetcdf* libraries。 Usually, eigther your sever didn’t install netcdf properly or NETCDF PATH was misled to other netcdf library,like perl or gcc.

      Reply

Leave a Reply to Walter Cancel reply

Your email address will not be published. Required fields are marked *