ARSC HPC Users' Newsletter 339, April 28, 2006

IBM Users: ScicomP/SP-XXL Conference Announcement

Important dates:

Abstract submission deadline: April 28. This is today, so get busy or you'll be late. :-)

Early Registration Deadline: June 30.

This is a co-located meeting: ScicomP and SP-XXL run parallel, but separate, programs. For details:

http://www.cisl.ucar.edu/info/scicomp/index.html

About ScicomP ScicomP meetings target computational scientists and engineers interested in achieving maximum performance and scalability on all of IBM's HPC platforms. of IBM technologies

About SP-XXL The SP-XXL group is a membership-based organization with systems-oriented interests in large-scale scientific technical computing on IBM hardware. Members of SP-XXL work together to increase the capabilities of these machines and to provide guidance to IBM on essential development and support issues.

NCAR Command Language (NCL)

[ Thanks to Kate Hedstrom of ARSC for this article. ]

I first heard of NCL a few years ago when Dennis Shea and Sylvia Murphy came up to teach a class on it. The story goes that the climate department at NCAR was looking for a useful tool, something with the functionality of IDL, but without the stiff licensing fees. They were rather surprised to find that the visualization department downstairs had the tool that they finally chose - NCL.

The NCL home page is at http://www.ncl.ucar.edu/ , which has documentation, examples, and a link to the download site at Earth Systems Grid. The binaries are available for free for a variety of architectures, in both 32- and 64-bit versions. They do not provide source code because one person only knows how to get the libraries it uses to work together.

So what is NCL? It is a scripting language with a matrix syntax, it can make graphics, and it has a very nice interface to NetCDF and HDF files, and can even read Grib files. It is also extensible, being able to call C and Fortran routines.

To get started, I highly recommend the "Getting Started" section under http://www.ncl.ucar.edu/Document/Manuals/ . You'll also the reference manual there. NCL is available on the ARSC IBM and Sun Fedora Core systems and you will need to set the NCARG_ROOT environment variable to point to its home in /usr/local/pkg/ncl/current. (Note that this does not point to the latest version on the ARSC IBMs yet.)

So what's new in NCL? Well, last fall they brought out a new version which includes support for some command line parsing. Also relatively new is the ability for it to make contour plots of data on a triangular mesh. In this article I'll give an example of using the command line to pass the name of a file to process. I needed to extract information from hundreds of NetCDF files, so passed in the file name as an argument to ncl:


touch foo2
ncl 'dfile="nep4_001.nc"' ice_2.ncl >> foo2
ncl 'dfile="nep4_002.nc"' ice_2.ncl >> foo2
   :
ncl 'dfile="nep4_038.nc"' ice_2.ncl >> foo2
   :
and so on. Here, ice_2.ncl is the name of the ncl script I want to run, and its output is being sent to the file foo2. Internal to ice_2.ncl, it uses the variable dfile for the name of the datafile to open:

f    = addfile (dfile, "r")
It then extracts the variables aice, hice, and ocean_time from that file with:

x    = f->aice
h    = f->hice
time = f->ocean_time
Note that I couldn't get this to work with any other combination of quotes. An exercise for the reader would be write a shell loop to loop through all the files successfully. I got lazy and decided to just put all the ncl commands into a shell script.

Once this script was run, I used a perl script to extract the numbers from that foo2 output file, then as second NCL script to make a plot of the resulting timeseries of the ice extent and volume. NCL also has a version with Python built in for such string manipulation tasks, but I haven't yet learned how to use it. It is called PyNGL (pronounced pingle), described at http://www.pyngl.ucar.edu/ .

Book Review: "Wicked Cool Java"

[ Thanks to Lee Higbie of ARSC for another book review. ]

Review of:

Wicked Cool Java, Brian Eubanks, No Starch Press, $30 list in lie-flat paper binding.

(And two chances to win a copy--see below.)

"No Starch Press publishes the finest in geek entertainment" claims its web page. They also have one of the best titles in "Wicked Cool Java" and I recommend the book for its content.

Ten years ago, during the Java 1.1 era, Java programmers expected to know the entire language and all its packages or libraries. By Java 1.2, also called Java 2, this became difficult and I don't think it is possible since Java 1.3. We are now at Java 5.

In Wicked Cool, Eubanks mostly covers unusual new features of Java 1.4 and 5 and some of the obscure Java packages not documented at java.sun.com. Unfortunately, Eubanks does not always provide pointers to download sources when he mentions a package. There are pointers to all the libraries at www.wickedcooljava.com or you can google the package names.

Background first: in an HPC context, what are the applications for Java? I think its primary use is for GUIs, programs to allow users to access HPC programs in an intuitive fashion. In Java, such user interface codes can run on Windows, Linux, Unix or Mac with no problems. Some HPC platforms support Java, which may facilitate inter-platform operations. I know of no direct interface between Fortran and Java, but JNI provides the functionality. C-Java interoperability is well defined so Fortran <==> C <==> Java would be one way to directly interface Fortran and Java, if that is essential. Where I see Java being used it will read and create files that Fortran programs will create and read.

Some of the wicked cool features that Eubanks describes are new looping constructs and default operations in the basic language. New in Java 5, for example, are enum; for loops with listed index values; variable and generic procedure arguments; and a high resolution timer. All of these are likely to be useful for control systems.

Some of the packages he describes are the important new ones for regular expressions and new operations on strings. If I am correct about our applications of Java, then the regular expression operations are needed. My choice of Java has often been instead of Perl. I expect to use it more now it has the basic string functions that we need. Eubanks describes these string operations nicely.

Eubanks offers a useful chapter on scientific and mathematical applications. It describes packages for functors and one for inter-unit operation, that should interest all of us who have to deal with both metric and English measurements.

However, Eubanks reveals that he may be just a tad removed from high-performance computing, as he launches the chapter with this statement: "Amazingly, in some scientific communities the Fortran programming language still rules supreme (I know, it makes me cringe too)." In Fortran computing, where we all know speed is king and space is duke, or queen and duchess if you prefer, I tested Java on a single simple calculation, one of the 3D structures described in the CUG 2005 paper "Simple Loop Performance on the Cray X1." My results:

  1. The Java code would only run with grid size one fifth the size of the grid I used in Fortran and C.
  2. The Java code ran about one fourth as fast as the Fortran code for my test case.

For the record, this test case created only a single object so there were few of the typical object oriented overhead operations or data that would occur in most programs. And, as I said at the start, I do recommend this book.

--

WIN A COPY!!!

I have two copies of Wicked Cool Java to give away.

RAFFLE:

One copy will simply be raffled off. Send me an email stating you'd like to participate in the raffle and you'll be entered in the random drawing.

CONTEST:

For the second copy, do either of the following exercises:

  1. Write an example of a Java code that can or could be used to control a Fortran program. The sort of thing I'd like is a GUI to select files or test cases, build the appropriate script to run the HPC program, etc..
  2. Test the performance of Java and C/C++ or Fortran on whatever type of operations you care to examine. In this context the "performance" includes the relative execution speed, size of problem that can be executed, and the development time, complexity, and maintainability of the code.
I will judge the entries for interest, clarity, maintainability, flexibility, and other factors in keeping with the way contests are judged in Ester, Alaska, where I live. Contest entries may be shared with the HPC community via future articles in this newsletter. So please, teach us something!

The deadline for all entries is 31 May, 2006. Send your contest or raffle entry to: higbie at arsc dot edu.

X1 Compilers on Skagway

We encourage all X1 users to check out the new X1 cross-compiler system, skagway, to compile codes for execution on the X1. Skagway provides:

  • Approximately 10x faster compilation and linking of codes compared with the compilers on klondike.
  • Identical programming environment to the X1:
    • "modules" command
    • Cray compilers and libraries
    • Cray man pages
  • Linux OS and faster Unix/Linux utilities such as tar and zip.
  • emacs, vim, vi. (Did I mention emacs?)

"News" items on skagway and the following document will help you get started:

http://www.arsc.edu/support/howtos/usingx1xc.html

--

Note: We'll be upgrading PrgEnv.new next Weds. The X1 compilers on klondike and skagway will be kept in sync.

Iceberg: Compiler Upgrades

On May 10th, the compilers on iceberg will be upgraded. The new versions are currently available via the "debug" class for testing.

Following the upgrade the compiler versions will be:


  Compiler                    Version
  ==========================  =========== 
  C/C++ (xlc,mpcc_r,etc.)     7.0.0.4
  Fortran (xlf,mpxlf_r,etc.)  9.1.0.4
The current compiler versions are:

  Compiler                    Version
  ==========================  =========== 
  C/C++ (xlc,mpcc_r,etc.)     7.0.0.2
  Fortran (xlf,mpxlf_r,etc.)  9.1.0.3

For more information see "news compilers" or contact the ARSC help desk at consult@arsc.edu.

Quick-Tip Q & A



A:[[ I'm new to C programming and just discovered that you can't nest
  [[ comments.  So the following isn't legal:
  [[
  [[ /* comment A /* comment B */  */
  [[ 
  [[ Is there a way to comment out part of my code without getting rid of
  [[ the existing comments or having to add an absurd number of C style
  [[ comments?
  [[
  [[ Say I want to comment out the second two printf statements in the
  [[ code below without losing the existing comments.  
  [[
  [[ #include <stdio.h>
  [[ #include <stdlib.h>
  [[  
  [[ int main()
  [[ {
  [[     int a;
  [[     a=100;
  [[  
  [[     printf("%-8s = %d\n","a",a);          /* print out a */
  [[     printf("%-8s = %d\n","a*a",a*a);      /* print out a*a */
  [[     printf("%-8s = %d\n","a*a*a",a*a*a);  /* print out a*a*a */
  [[ }

   
#
# Thanks to Sean Ziegeler, Kurt Carlson, Brad Chamberlain, Derek Bastille and
# Ed Kornkven for sharing a technique using #if and #ifdef preprocessor
# directives
#

There're a couple ways to accomplish this.  My preferred way, assuming
I'm going to build in some debug code segments anyway, is to #if the
code with a -D:

    $ cat x.c
    #include <stdlib.h>
    #include <stdio.h>
    main()
    {
    #if defined(DEBUG)
        printf("this is only a test\n"); /* debug code */
    #endif
        printf("hello world\n");
        exit(0);
    }

    $ cc -o x x.c; ./x 
    hello world

    $ cc -DDEBUG -o x x.c; ./x
    this is only a test
    hello world

For this specific question, wrapping the unwanted code with:

    #if !defined(ignore_stuff)
    #endif

might be more appropriate, use 'cc -Dignore_stuff'.


#
# Ed Kornkven and Brad Havel pointed out that C++ style comments are a
# part of the C99 standard.
#

Use a version of C that allows C++ style comments.

  // printf("%-8s = %d\n","a",a);          /* print out a */
  // printf("%-8s = %d\n","a*a",a*a);      /* print out a*a */

For example, gcc supports this and it is part of the C99 standard.



Q: [
Thanks to Derek Bastille for this one... 
] 

   How can I run a remote command on a system and use the remote values
   for $HOME, $WRKDIR, etc. instead of the local values?

   For example, I'd like this command to work as I obviously intend it
   to work (even if WRKDIR is defined differently on the local and
   remote systems):

     scp -r remotesys:$WRKDIR/mymegamodel/answers .

[[ Answers, Questions, and Tips Graciously Accepted ]]


Current Editors:
Ed Kornkven ARSC HPC Specialist ph: 907-450-8669
Kate Hedstrom ARSC Oceanographic Specialist ph: 907-450-8678
Arctic Region Supercomputing Center
University of Alaska Fairbanks
PO Box 756020
Fairbanks AK 99775-6020
E-mail Subscriptions: Archives:
    Back issues of the ASCII e-mail edition of the ARSC T3D/T3E/HPC Users' Newsletter are available by request. Please contact the editors.
Back to Top