ARSC HPC Users' Newsletter 328, November 04, 2005

ARSC at SC05

ARSC will be at SC05 in Seattle from November 12-18. We will be in booth 112, so stop by to see what we are doing. If you have story ideas or Quick-Tip questions for the HPC Users' Newsletter, we would enjoy hearing from you.

Introduction to PMPI Part I

The MPI Standard includes provisions to allow developers to create profiled libraries on top of the system implementation of MPI. This is done by providing a name shifted entry point for each MPI functions. All functions beginning with the prefix "MPI_" are required by the MPI standard to be accessible through an alternate name beginning with the prefix "PMPI_".

E.g. MPI_SEND is also available as PMPI_SEND.

This allows developers to create a profiled implementation of an MPI subroutine which uses the underlying implementation of MPI without having access to the source code for the MPI library.

Profiled libraries could be useful for a number of reasons including:

  • performance timing and statistics.
  • to provide additional debugging information without changing the source code for your model.
  • to trap invalid conditions that are not caught by the default implementation of MPI.

Here's a simple example which counts the number of MPI_SEND and MPI_RECV calls and displays the totals when MPI_FINALIZE is called.


klondike 1% cat prof_mpi.f90
module perf_data
! module to store performance data on a task by task basis
!
    IMPLICIT NONE
    INTEGER :: i_my_rank
    INTEGER :: i_recv_calls
    INTEGER :: i_send_calls
end module

subroutine MPI_INIT(ierror)
! alternate implementation of MPI_INIT which initializes data
! in the perf_data module.  
!
    use perf_data
    IMPLICIT NONE
    INTEGER, INTENT(OUT) :: ierror
    INTEGER err   

    i_recv_calls=0
    i_send_calls=0
    i_my_rank=-1
    CALL PMPI_INIT(ierror)
    CALL SET_RANK
    PRINT *,"Starting task", i_my_rank 

end subroutine

subroutine SET_RANK
! sets i_my_rank in the perf_data module
!
    use perf_data
    use mpi
    IMPLICIT NONE
    INTEGER err
    CALL PMPI_COMM_RANK(MPI_COMM_WORLD, i_my_rank, err)

end subroutine

subroutine MPI_SEND(buf, count, typ, dest, tag, comm, err)
! alternate implementation of MPI_SEND which tracks the number 
! of MPI_SEND calls.
!
    use perf_data 
    IMPLICIT NONE
    INTEGER, INTENT(IN) :: buf, count, typ, dest, tag, comm
    INTEGER, INTENT(OUT) :: err
    INTEGER :: total_tasks, ierr
   
    i_send_calls=i_send_calls+1
    CALL PMPI_SEND(buf, count, typ, dest, tag, comm, err)

end subroutine

subroutine MPI_RECV(buf, count, typ, source, tag, comm, status, err)
    use perf_data
    IMPLICIT NONE
    INTEGER, INTENT(OUT) :: buf, count, typ, source, tag, comm 
    INTEGER, INTENT(OUT) :: status(*), err

    INTEGER :: total_tasks
    INTEGER :: ierr

    i_recv_calls=i_recv_calls+1
    CALL PMPI_RECV(buf, count, typ, source, tag, comm, status, err)

end subroutine

subroutine MPI_FINALIZE(ierror)
! alternate implementation of MPI_FINALIZE which displays the 
! number of MPI_SEND calls that were made by each task
!
    use perf_data
    IMPLICIT NONE
    INTEGER, INTENT(OUT) :: ierror
    
    PRINT *, "Task:", i_my_rank, " MPI_SEND Calls: ", i_send_calls  
    PRINT *, "Task:", i_my_rank, " MPI_RECV Calls: ", i_recv_calls 
    call PMPI_FINALIZE(ierror)

end subroutine

Notice that each subroutine calls the underlying system subroutine somewhere in the body of the routine. This ensures that the output of the profiled routines are the same as the default implementations.

The profiled subroutines can be compiled with an MPI compiler:

E.g. Cray X1 (with the mpt module loaded):     klondike 2% ftn prof_mpi.f90 -c IBM AIX     iceberg2 1% mpxlf90_r -qsuffix=f=f90 prof_mpi.f90 -c

This results in an object file which has profiled implementations of MPI_INIT, MPI_SEND, MPI_RECV and MPI_FINALIZE. At this point we could build a profiled MPI library by replacing the MPI routines in the default MPI library. This task can be done trivially on the Cray X1 using the 'ar' command, but requires a number of steps on AIX due to the "unusual" design of shared libraries on AIX. See reference B below for details on how to do this on AIX. For this example we will simply add prof_mpi.o to the compile statements and use the default MPI library to resolve the other symbols.

Here's a simple code which sends a message from even numbered tasks to odd numbered tasks.


iceberg2 3% cat mpi-send-recv.f90 
PROGRAM MPI_SEND_RECV
    use mpi
    INTEGER ierr
    INTEGER task,total_tasks
    INTEGER stat(MPI_STATUS_SIZE)
    REAL IN(10)
    REAL BUF(10)
    INTEGER II
    
    CALL MPI_INIT(ierr)
    CALL MPI_COMM_RANK(MPI_COMM_WORLD,task,ierr)
    CALL MPI_COMM_SIZE(MPI_COMM_WORLD,total_tasks,ierr)
    
    DO II=1, 10
        IN(II)=task
    END DO
    CALL MPI_BARRIER(MPI_COMM_WORLD)
    
    IF ( mod(task,2) == 0 ) then
        CALL MPI_SEND(IN, 10, MPI_REAL, task+1, 99, MPI_COMM_WORLD, ierr)
    ELSE
        CALL MPI_RECV(BUF, 10, MPI_REAL, task-1, 99, MPI_COMM_WORLD, stat, ierr)
    END IF
    
    CALL MPI_FINALIZE(ierr)
    
END PROGRAM

This code does not produce any output by default. If we link against the profiled subroutines in prof_mpi.o, the executable will display a message from each task as it starts and the total number of MPI_SEND and MPI_RECV calls when MPI_FINALIZE is called.

Here are example compile statements for klondike and iceberg.

Cray X1:     klondike 3% ftn mpi-send-recv.f90 prof_mpi.o IBM AIX:     iceberg2 4% mpxlf90_r -qsuffix=f=f90 mpi-send-recv.f90 prof_mpi.o

Running the resultant executable reports the numbers of sends and receives from each task.


 iceberg2 72% poe ./a.out -hostfile hosts -procs 4 
 Starting task 0
 Starting task 1
 Task: 1  MPI_SEND Calls:  0
 Task: 1  MPI_RECV Calls:  1
 Starting task 2
 Task: 2  MPI_SEND Calls:  1
 Task: 2  MPI_RECV Calls:  0
 Starting task 3
 Task: 3  MPI_SEND Calls:  0
 Task: 3  MPI_RECV Calls:  1
 Task: 0  MPI_SEND Calls:  1
 Task: 0  MPI_RECV Calls:  0

In the next part of this series will take a look how PMPI can be used to trap invalid conditions in MPI calls.

References:

  1. MPI 1.1 Standard:     http://www.mpi-forum.org/docs/mpi-11-html/mpi-report.html
  2. IBM Parallel Environment for AIX 5L: MPI Programming Guide:    Version 4 Release 2; pg 11-13; SA22-7945-02

Utilization Information Available

Utilization information is now available on both iceberg and klondike. The 'show_usage' command will display allocation, foreground utilization, background utilization and remaining allocation by project.


iceberg2 1% show_usage
Allocation for 'projecta' as of Nov 4 2005 at 04:00:01
  Allocation:                  50000.00
  Usage:                         704.69
  Remaining Allocation:        49295.31
  ------------------ 
  Background Usage:                0.00

Allocation for 'projectb' as of Nov 4 2005 at 04:00:01
  Allocation:                  10000.00
  Usage:                        2890.01
  Remaining Allocation:         7109.99
  ------------------ 
  Background Usage:                0.00

For a complete breakdown by project, please contact the
ARSC help desk:
    email: consult@arsc.edu
    phone: 907-450-8602

Quick-Tip Q & A


A:[[ I love vim's syntax highlighting, but, to save my eyes, I always work
  [[ with light text on a dark background.  Unfortunately, vim uses dark
  [[ blue for C comments and they practically disappear. (So, of course, I
  [[ end up burning my eyes out squinting and getting too close!)
  [[
  [[ Is there some way to change vim's colors for syntax highlighting?

#
# Thanks to Jed Brown:
#

The Short answer--
  :h hi

Longer--
  The "highlight" command is what you are looking for. For this problem,
  I have

    :highlight Comment ctermfg=cyan cterm=NONE

  in my vimrc which makes comments appear as nonbold cyan when running
  vim in the console. (The GUI has a special blue for comments which I
  find very readable.)

#
# From Bob Clark
#

In vim, ':help highlight' will get you all the highlighting
help you ever wanted.

The ':hi' command will display your active settings.

I use white-background xterms and have the following
highlight settings in my .vimrc file:

  " XTerm colors are:
  " 0:black 1:red 2:green 3:yellow 4:blue 5:magenta 6:cyan 7:white
  " (yellow and green do not work well on light backgrounds)
  highlight MoreMsg ctermfg=5
  highlight LineNr ctermfg=5
  highlight Question ctermfg=5
  highlight Statement ctermfg=5
  highlight Type ctermfg=5
  highlight Identifier ctermfg=4
  highlight DiffAdd ctermfg=7

(ctermfg stands for color terminal foreground)

So, assuming you are using an xterm-like terminal, you
could turn your comments green with ':hi Comment ctermfg=2'

If you want more color options, you can try gvim.


#
# For advanced users, here's one from Jesse Niles
#

To use the extended color scheme set in vim:

1) Create the directory ~/.vim if it doesn't exist.
2) Download the color sampler pack located at:

http://www.vim.org/scripts/download_script.php?src_id=4010
 and unpack
it in that directory.  It will create the two necessary subdirectories
plugin and colors, if they don't already exist.
3) Open vim in graphical mode ('gvim' or 'vim -g'), if available, and
browse the color schemes.

4) When you find the one you like, add the line:  
  :colorscheme newcolors 
   
to your ~/.vimrc file ("newcolors" is the name of the color scheme you
like).  Create one if it doesn't already exist.

The next time you fire up vim or gvim, you'll get the new color
scheme.  Note that the colors of the same scheme may differ between vim
and gvim, so you'll want to check both.  If you just want the comments
to be a different color in the same scheme:

1) Copy the color scheme file from the system vim colors file to your
~/.vim/colors directory and open it in a text editor (you might as well
use vim because it has syntax highlighting for vim files).
2) Find the line that looks like:

  highlight Comment    ctermfg=8                        guifg=#ff00ff

This line specifies the color that vim will use first, followed by what
gvim will use.  If you want to change the gvim color to white for
example, just change the hex value.  If you are using terminal-style
vim, you can just set the ctermfg value equal to the name of one of the
common colors available on terminals (e.g. white, black, blue, etc.).
To make it White, you can simply change the line to:

  highlight Comment    ctermfg=white                    guifg=#ffffff

--

As a bonus, if you'd like certain symbols to stand out, e.g. OpenGL
functions or MPI functions, you can add the functions or keywords to
the language syntax file.  I use OpenGL a lot, and I sometimes make
errors in the capitalization of the symbol names.  In vim you can
create a specialized syntax file that can color arbitrary symbols.
Thus, if you make a typo you won't have to wait until the compiler
whines about it.  If you are as nerdy as I am, you can:

1) Copy the cpp.vim (or whatever language syntax file you are using)
from the system $VIMRUNTIME/syntax directory to your ~/.vim/syntax
directory and open it in a text editor.  
2) Add the following lines to the file where the similar declarations
are, adding the functions/types/constants you want to add, separated by
spaces:

  syn keyword specialFunctions glBegin glAlphaFunc glAreTexturesResident ...
  syn keyword specialTypes GLbitfield GLboolean GLbyte ...
  syn keyword specialConsts GL_2_BYTES GL_2D GL_2_BYTES ...

3) Add the following lines to the section with all of the "HiLink"
statements:

  HiLink specialFunctions  Special
  HiLink specialTypes      Type
  HiLink specialConsts     Constant

It may then look something like:

  HiLink cppAccess              cppStatement
  HiLink cppCast                cppStatement
  HiLink cppExceptions          Exception
  HiLink cppOperator            Operator
  HiLink cppStatement           Statement
  HiLink cppType                Type
  HiLink cppStorageClass        StorageClass
  HiLink cppStructure           Structure
  HiLink cppNumber              Number
  HiLink cppBoolean             Boolean
  HiLink specialFunctions  Special
  HiLink specialTypes      Type
  HiLink specialConsts     Constant

The above changes will get vim to highlight GLbitfield the same way it
would int, float or char, and will highlight GL_2_BYTES as it would
INT_MAX.  Functions in the list above should show up in an entirely
different color so you can easily see errors.  There are much more
advanced and generic ways to color text in vim, but this is as fancy as
I ever needed to get.  Lastly, if you forget what the parameters in one
of the functions is, you can put the cursor over the function and hit
'K' (shift-k) to view the man page.


#
# Editor's solution:
#

Taken from vim's internal help ("*mysyntaxfile-add*"),
create this directory tree and file:  ~/.vim/after/syntax/c.vim .
To retain all default C syntax colors, but turn comments
green, the file would contain this line of text:

  highlight cComment ctermfg=Green guifg=Green




Q:  [[Thanks to Ed Kornkven of ARSC for this week's Quick Tip question]]

    I use a calculator for evaluating arithmetic expressions.  It accepts
    an arithmetic expression as stdin and outputs the answer on stdout.
    E.g.,
        cl 1024*1024*1024*15
         = 16106127360
    [Actually, all "cl" does is 'echo "print $*" 
 python' -- but I digress].
    My only complaint with this "calculator" is that I have a hard time
    reading long-digit numbers.
    
    What I would like is a filter that accepts such a number on stdin, and
    writes to stdout the number formatted with commas separating the
    thousands places.  E.g.,
        cl 1024*1024*1024*15 
 comma_adder
         = 16,106,127,360


[[ Answers, Questions, and Tips Graciously Accepted ]]


Current Editors:
Ed Kornkven ARSC HPC Specialist ph: 907-450-8669
Kate Hedstrom ARSC Oceanographic Specialist ph: 907-450-8678
Arctic Region Supercomputing Center
University of Alaska Fairbanks
PO Box 756020
Fairbanks AK 99775-6020
E-mail Subscriptions: Archives:
    Back issues of the ASCII e-mail edition of the ARSC T3D/T3E/HPC Users' Newsletter are available by request. Please contact the editors.
Back to Top