ARSC HPC Users' Newsletter 367, August 10, 2007

ARSC Transitioning HPCMP User Support to CCAC

ARSC is transitioning basic customer support for our High Performance Computing Modernization Program (HPCMP) users to the Consolidated Customer Support Center (CCAC). The transition began on 12 July, 2007 and is scheduled to be completed by 24 August, 2007.

After August 24th, the CCAC will become the first point of contact for support for users that get their accounts through the HPCMP. The CCAC will be able to answer many basic issues such as kerberos kit installation and usage and problems with compilers, libraries and some software packages. ARSC, in turn, will continue to have on-site support staff available to answer questions that are specific to our systems and environment. We will also have local CCAC liaisons with access to both ARSC's and CCAC's ticketing systems to ensure that your problems and questions continue to be answered as quickly as possible.

We anticipate that the transition will go smoothly and encourage you to visit the CCAC web site to review the services and documentation that they will provide. You are welcome to continue to contact us directly during the transition with any questions or concerns that you might have.

Contact information for the CCAC is:

Toll-free Phone:

877-CCAC-039 (877-222-2039)

Phone:

937-255-0679

Fax:

937-656-9538

Help Desk email:

help@ccac.hpc.mil

Accounts email:

accounts@ccac.hpc.mil

Web site:

http://www.ccac.hpc.mil

Please note that the CCAC will be handling support for HPCMP users only. If you are an Academic ARSC user, we ask that that you continue to contact the ARSC Help Desk directly for support.

The Future of NetCDF


  [by: Kate Hedstrom] 
    

I was recently in Boulder, Colorado and was lucky enough to attend a NetCDF workshop there. For those interested, the talks are all at:

    http://www.unidata.ucar.edu/software/netcdf/workshops/2007/

I was already rather familiar with the NetCDF3 material. One thing that was new to me is the effort to standardize the structure of the files, an effort that still continues:

    http://www.cfconventions.org/

The guys said they used to not endorse any one convention, but they now stand behind the CF conventions. I look forward to some of the changes on the horizon for support of more general model grids.

NetCDF is a standard file format we have been using for many years now. It still serves us well, as is, but there is a list of features we'd like to see added. It turns out there's an effort underway to obtain all those fancy features from the HDF5 file format:

    http://hdfgroup.com/HDF5/

HDF5 is the complete update of the HDF standard, and incompatible with HDF4.

NetCDF4 is the version which will be based on HDF5 and it is out in beta test now. If you don't change anything in your code, you will still get a NetCDF3 file, readable from all your old NetCDF3 tools. If you recompile and relink your code against the NetCDF4 library, the only change is that you will now have to add -lhdf5_hl -lhdf5 after -lnetcdf.

The way that NetCDF can maintain some backward compatibility is to decide at file creation time whether it is a "classic" NetCDF file or a newer HDF file. The NetCDF team has minimized the changes to the API (applications programmer interface) for files that don't contain any of the new features. They have done this by building an interface layer that calls the NetCDF3 library or the HDF5 library for you.

Why not keep using the NetCDF3 files forever, you ask? One of the bottlenecks to scalability for the code I use is in the I/O. Parallel I/O through MPI-I/O seems like something we have to start doing and NetCDF4 is the path we have chosen to get there. It requires the creation of HDF5 files, not the classic files. The new files also allow the creation of user-defined types in the file, though such things may take longer for full support to be available in all our standard pre- and postprocessing NetCDF tools. It's probably safer to stick with the classic data structures until they no longer match our needs. The new files and old data structures do allow for the use of compression and chunking as done in HDF5.

Resolving Linker Errors - Part III


  [by: Don Bahls]

Many scientific codes use more than one programming language. Typically most of the code is written in Fortran with a small part of the code handled in C or C++. While this design methodology allows developers to take advantage of the strengths of each language, it occasionally can lead to linking problems when one the C routines don't match the name mangling convention of the Fortran compiler.

Name Mangling

When Fortran subroutines and functions are compiled into object format, the routine name is typically altered in some way. Perhaps the most common way of doing name mangling for Fortran routines is to make the routine all lower case and to add a single underscore at the end of the routine name. This makes it less likely that the linker will accidentally try to use a C routine when attempting to link to a Fortran routine.

Below is a simple code which we will use to look at how a few different compilers do name mangling:


   midnight % cat name_mangling.f

         subroutine MySub

         end subroutine

         subroutine My_Sub

         end subroutine

         subroutine My_Sub_

         end subroutine

Here's the resulting symbol names for these three routines from six different Fortran compilers:


  Description              MySub     My_Sub     My_Sub_
  =======================  ========  =========  ==========
  GNU F77 (g77)            mysub_    my_sub__   my_sub___ 
  PathScale (pathf90)      mysub_    my_sub__   my_sub___

  GNU F90 (gfortran)       mysub_    my_sub_    my_sub__
  Portland Group (pgf90)   mysub_    my_sub_    my_sub__ 
  Sun Studio (sunf90)      mysub_    my_sub_    my_sub__

  XL Fortran (xlf90)       .mysub    .my_sub    .my_sub_

You'll notice that there are three different name mangling styles used by the six compilers listed above. Because of this discrepancy, there is always a chance that you could run into a new linking error when you move to a new compiler even if everything compiles fine. These linking errors will probably be limited to either Fortran code which calls C subroutines or vice versa.

PathScale Name Mangling

The default compiler on the midnight, the new Linux cluster at ARSC, is the PathScale compiler. Because the PathScale compiler uses a different name mangling style than some other common Fortran compilers, some codes have linking problems that aren't present with other compilers.

Perhaps the most obvious way of dealing with the difference is to simply switch the name mangling style. The "-fno-second-underscore" compiler option tells the compiler to only add one underscore to the symbol name.


    midnight% pathf90 -fno-second-underscore name.o -c     
    midnight% nm name.o
    0000000000000010 T my_sub_
    0000000000000020 T my_sub__
    0000000000000000 T mysub_

Using this option, we find that the name mangling now matches what we get from the PGI and other compilers. If your code doesn't have many library dependencies and it is a serial or OpenMP code, this may be a reasonable solution. However for MPI codes, this solution might not work without some help. The MPI libraries and other Fortran libraries installed on midnight use the standard PathScale Fortran name mangling style, so changing the name mangling convention might make the linking process fail to find a different set of symbols.

If you use a code which makes use of Fortran 90 modules (e.g. USE MPI), the compiler will know the proper name mangling style. However if your code uses Fortran 77 style includes, you might find that the "-fno-second-underscore" option breaks things:

e.g.

    # Using "include mpif.h"
    midnight% mpif90 -fno-second-underscore hello_fortran.f -o hello_fortran
    hello_fortran.o(.text+0x3f): In function `MAIN__':
    : undefined reference to `mpi_init_'
    hello_fortran.o(.text+0x55): In function `MAIN__':
    : undefined reference to `mpi_comm_rank_'
    hello_fortran.o(.text+0x6b): In function `MAIN__':
    : undefined reference to `mpi_comm_size_'
    hello_fortran.o(.text+0x165): In function `MAIN__':
    : undefined reference to `mpi_finalize_'
    collect2: ld returned 1 exit status
    # produces a linking error
e.g.

    # Using "use mpi" in hello_fortran.f90
    midnight% mpif90 -fno-second-underscore hello_fortran.f90 -o hello_fortran
    # compiles and links successfully.

Fortunately the PathScale compiler has an option which lets you avoid having to change your code to resolve these linking errors, but it does involve a little work on your part. The "-fdecorate" option lets you specify how a symbol should be decorated (i.e. what name mangling scheme should be used).

Here's a decorate file for the linking error above:


    midnight% cat mpi_decorate 
    mpi_init        2
    mpi_comm_rank   2
    mpi_comm_size   2
    mpi_finalize    2
 

This file tells the compiler that each of the symbols in the file should get two underscores at the end of the name. Each line contains a routine name followed the corresponding symbol name. The "2" is a short hand way of saying the symbol has two underscores at the end. The mangling for a routine can also be explicitly specified:

e.g.

    
    mpi_init    mpi_init__

Using the decorate file, we see that the first example above now compiles.

e.g.

    
    midnight% mpif90 -fdecorate mpi_decorate -fno-second-underscore \ 
           hello_fortran.f -o hello_fortran

Normally I avoid using "-fno-second-underscore" and simply use "-fdecorate" alone. The advantage of this is that you just have to deal with the C routines which get called from Fortran rather than trying to figure out all of the external library routines used by a code.

Here's one technique that can be used to build a decorate file for C routines called from Fortran:

  1. Attempt to compile the code with no changes to the compile system. Hopefully this will build everything including the C routines.
    
       midnight% make 
    
  2. If there are name mangling issues, the compile will fail during the linking step. At this point, we can try to find all of the object file created from C source files. Here's one way of doing this with bash:
    
       midnight% find . -name \*.c 
     while read f; 
                do 
                   o=${f%.c}.o; 
                   if [ -f $o ]; then echo $o; fi;
                done
    
    The PathScale compiler suite comes with a command called "fsymlist" which dumps symbols that look like Fortran symbols from an object file. Using the previous command as a basis, we'll find all of the symbols in the C files which look like they are meant to be linked from Fortran. Here "echo" is simply replaces "fsymlist" and the resulting output is sent to a file called "fortran_symbols".
    
        midnight % find . -name \*.c 
     while read f; 
                do 
                   o=${f%.c}.o; 
                   if [ -f $o ]; then fsymlist $o; fi;
                done  > fortran_symbols
    
  3. Next we use the "fortran_symbols" file to build a decorate file.
    
        midnight% cat fortran_symbols 
     while read sym; do 
                echo "$sym 1"; 
              done > decorate
    
    Here we build a file with the symbol name followed by "1", where the "1" designates that the symbol has a single underscore. This tells the compiler that the C routines called from Fortran will have a single underscore at the end of the symbol name. NOTE: you can combine steps 2 and 3 if you want to, but it gets to be a fairly long command line statement.
  4. Next update the Fortran flags in the build system. e.g.
    
       FFLAGS += -fdecorate ./decorate
       
    

    Depending on how your code is compiled, you may need to include the full path to the decorate file.

    If you don't want to change your build system at all, you can use the environment variable PSC_FFLAGS to add a flag for the Fortran compiler on the command line.

    e.g.
    
       midnight% export PSC_FFLAGS="-fdecorate $PWD/decorate"
       
    
  5. Last do a clean build of the code and then rebuild.
    
       midnight% make clean
       midnight% make
       
    

This method works fairly well when you're looking for Fortran Code that calls C, however if your code has C routines which call Fortran, you also need to account for those routines. Because the C code may be looking for a routine with a single underscore while the Fortran compiler could be adding two underscores to the symbol name.

One other alternative that can be used in this situation is to add C macros to alter the name mangling for particular symbols. The WRF model uses the "F2CSTYLE" macro in portions of the C code to do this.

e.g.

   #ifdef F2CSTYLE
   #define my_subroutine_ my_subroutine__
   
   #endif

   void my_subroutine_
   ...
   ...

When the C source code is compiled with -DF2CSTYLE, the C preprocessor will rename the specified routines, to ensure the correct name mangling is used.

Quick-Tip Q & A



A:[[ Regarding commenting in C/C++ code: I've seen code that uses /*
  [[ to start comments in some places and /** to start them in other
  [[ places. Why use /** ?
  [[


  #
  # Sean Ziegeler
  #

  It has no effect on the compiler.  That code is using a documentation
  generator.  Doxygen, as one example, scans the code and tags
  any special comments that start with /** or /// (instead of //)
  as information about the following variable, function, class, etc.
  It scans all specified code files, then generates a document (usually
  HTML, but can be PDF, PS, man page, ...).  The most common use is
  to generate interlinked web pages where you can click on names of
  classes, functions, variables, etc. and get information about it.
  It can also do things like graphical class hierarchies.

  Many open source projects use it for online developers'
  guides.  It's quite handy for large projects with multiple
  developers.  See 
www.doxygen.org
 or for other alternatives see
  
http://en.wikipedia.org/wiki/Documentation_generator


  #
  # Chris Young
  #

  /** is used to start document comments for certain documentation
  generation tools.  OnlineDoc, KDOC and likely many others use the
  syntax for C++ documentation generation.  They often attribute the
  syntax to javadoc, but I don't know the actual origins.  Depending
  upon the capabilities of the documenter tool you'll often see other
  special flags within the document comment.

  #
  # Anton Kulchitsky
  #
  
  C/C++ code comments start with /** (the ending can be "*/" ) when a
  developer prepares the code for a documentation generator. Wikipedia
  defines a documentation generator as a programming tool that
  generates documentation intended for programmers (API documentation)
  or end users (End-user Guide), or both, from a set of specially
  commented source code files, and in some cases, binary files. In such
  comments, programmer puts some tags that describe the code followed
  by this comment block. For example, function purpose, inputs and
  outputs, developer name etc. Documentation generator then generates
  documentation from the source files in different formats like
  html, pdf, LaTeX, RTF, man pages and in many others. Sophisticated
  usage of doc generators can really produce an extensive and nice
  documentation without writing documentation separately from the code
  which allows you to save time and follow commenting standards in
  team work. There are some very illustrative examples below showing
  what kind of documentations can be produced:

  
http://kdevelop.org/HEAD/doc/api/html/

  (KDevelop documentation by Doxygen)
 
http://www.naturaldocs.org/documentation/html/files/NaturalDocs-.html

  (NaturalDocs)

  The examples of free generators are Doxygen
  
http://www.stack.nl/~dimitri/doxygen/
 (supports at
  least C++, C, Java, Objective-C, Python), Sun JavaDoc
  
http://java.sun.com/j2se/javadoc/
 (Java), Epydoc
  
http://epydoc.sourceforge.net/
 (Python), NaturalDoc
  
http://www.naturaldocs.org/
 (supports 19 languages!) and many others.

  If you are interested in documentation generators,
  you might also be interested in Literate Programming (see
  
http://en.wikipedia.org/wiki/Literate_programming
), "a philosophy of
  computer programming based on the premise that a computer program
  should be written similar to literature, with human readability
  as a primary goal" (Wikipedia). The first practical literate
  programming tool named WEB was created by Donald Knuth in 1981. The
  idea of literate programming is going much further than just making
  documenting easier for a developer and clearer to read. The literate
  programming code often looks like a scientific article illustrated
  with some code rather than code with comments. This type of document
  can be directly processed as a documentation or as a program. Some
  program compilers directly support this approach. For example GHC,
  the Glasgow Haskell Compiler by University of Glasgow . This famous
  compiler is itself written in literate programming style and can
  process the literate programming code when asked without additional
  preprocessing.

  Donald Knuth says about literate programming: "Let us change
  our traditional attitude to the construction of programs:
  Instead of imagining that our main task is to instruct a
  computer what to do, let us concentrate rather on explaining
  to human beings what we want a computer to do."  Please,
  see below a couple of links to the Literate Programming
  web resources: 
http://www.literateprogramming.com/
http://en.literateprograms.org/LiteratePrograms:Welcome

Q: I can't seem to remember what optimization flags I used to compile
   my program.  Is there any way for me to tell by just looking at the
   executable?
   
   

  [[ Answers, Questions, and Tips Graciously Accepted ]]

Current Editors:
Ed Kornkven ARSC HPC Specialist ph: 907-450-8669
Kate Hedstrom ARSC Oceanographic Specialist ph: 907-450-8678
Arctic Region Supercomputing Center
University of Alaska Fairbanks
PO Box 756020
Fairbanks AK 99775-6020
E-mail Subscriptions: Archives:
    Back issues of the ASCII e-mail edition of the ARSC T3D/T3E/HPC Users' Newsletter are available by request. Please contact the editors.
Back to Top