ARSC T3D Users' Newsletter 90, June 7, 1996

Porting Heterogeneous Distributed Codes to the T3D

[Don Morton, visiting ARSC from Cameron University, contributes this article. See his contact information, provided below.]

Many of us come to the T3D with previously-written heterogeneous codes - a classic example is the master/slave paradigm in which a single, master process "controls" one or more slave processes. Since the T3D requires that we place the same executable on each Processing Element (PE) in a partition, our codes may require modification in order to execute on this platform.

A first approach is to make use of the Y-MP/T3D computing environment, running the "master" on the Y-MP and spawning slaves on the T3D, all easily done by making use of PVM. This is not necessarily the best way to do things, but it serves as a useful starting point. The programs master.f and slave.f, below, illustrate how this is done.

master.f

Since the master process is executed on the Y-MP, we need to insure that it is compiled with "Network PVM," much like the standard PVM many of us have become used to. We spawn the slave processes on the T3D, and note that only the first entry of the array "list" contains a valid TID. By default, the Y-MP will communicate only with PE0 on the T3D, and only PEs that can communicate with the Y-MP have valid TID's listed; the other entries of "list" contain a 1. There are ways to establish connections to other T3D PEs, but these methods rely on additional socket connections between the two machines. CRI recommends that programmers pass Y-MP/T3D data through PE0.

Thus, the remainder of the master program involves initializing some data, sending it to PE0 on the T3D, then waiting for an acknowledgment, which will be sent by PE0 after the other slaves have received the message.

slave.f

In this example, slave.f is intended to run on the T3D in SPMD mode, and therefore makes use of the Cray-specific pvmfgetpe() function. For illustrative purposes only, all PEs call pvmfparent(), but because the default behavior has only PE0 communicating with the Y-MP, only PE0 will have a parent. PE0 receives the data from the Y-MP, then sends it on to the other slave processes. The slaves simply receive the data, then all T3D processes participate in the pvmfbarrier() call. On the T3D, the use of the group 'PVMALL' implies that all PEs in the partition participate - therefore, a "count" argument is not necessary. Finally, PE0 sends an acknowledgment to the Y-MP process and all slaves terminate.

Closing Note

The Y-MP/T3D distributed programming model seems to be shunned by most, although in some cases it is an effective way of partitioning various workloads. For example, with careful programming, vector intensive computations can be performed on the Y-MP while MPP computations are performed on the T3D. However, communication between Y-MP and T3D processes is quite slow for many reasons. Additionally, those who are thinking of migrating to the T3E should consider the fact that there will no longer be a Y-MP front-end.

Related discussion of Y-MP/T3D cooperation via PVM can be found in Newsletters #15 and #16.

program master:


        include 'fpvm3.h'
  
        parameter (NUMPROCS=5, ISIZE=1000)
        dimension vec(ISIZE)
        dimension list(0:NUMPROCS-1)   ! For storing TID's
  
  ccc   Enroll in PVM
        call pvmfmytid(mytid)
  
        print *, 'Hello World, I am task ', mytid
  
  ccc   Spawn slaves
        call pvmfspawn("slave", 0, '*', NUMPROCS-1, list, info)
        print *, 'Task ', mytid, ' spawned ', info, ' slaves...'
        print *, 'TID List of Slaves:'
        do i=0,info-1
           print *, list(i)
        enddo
  
  ccc   Initialize a vector for transmission to slaves      
        do i=1,ISIZE
           vec(ISIZE) = real(i)
        enddo            
  
  ccc   Send it to slave process on PE0
        msgtag = 1
        call pvmfinitsend(PVMDATADEFAULT, info)
        call pvmfpack(REAL8, vec, ISIZE, 1, info)
        call pvmfsend(list(0), msgtag, info)
        print *, 'Task ', mytid, ' sent message...'
  
  ccc   Wait for ACK from PE0
        msgtag = 2
        call pvmfrecv(list(0), msgtag, info)
        print *, 'Task ', mytid, ' received an ACK ...'
      

  ccc   exit PVM
        call pvmfexit(info)
        print *, 'Task ', mytid, ' exiting...'
  
        end
program slave:

    
        include 'fpvm3.h'
  
        parameter (ISIZE=10)
        dimension vec(ISIZE)
  
  ccc   Enroll in PVM
        call pvmfmytid(mytid)
        call pvmfgetpe(mytid, mype)
        call pvmfparent(myparent)
        print *, 'Hello World, I am task ', mytid
  
  ccc   By default, only PE0 is guaranteed to have the Y-MP process as
  ccc   a PVM parent.  Therefore, all data transfers between the Y-MP
  ccc   and T3D will go through PE0.  This is recommended in CRI's PVM
  ccc   Manual.
       
        print *, 'Task ' , mytid, '  Parent ', myparent
  
        msgtag = 1
        if (mype .eq. 0) then
  ccc      Receive data from master process
           call pvmfrecv(myparent, msgtag, ibuf) 
           call pvmfunpack(REAL8, vec, ISIZE, 1, info)
           print *, 'Task ', mytid, ' received data ...'
         
  ccc      Now, relay this data to the other T3D slave processes
           call pvmfinitsend(PVMDATADEFAULT, info)
           call pvmfpack(REAL8, vec, ISIZE, 1, info)
           call pvmfbcast(PVMALL, msgtag, info)
        else
  ccc      Receive data from PE0                
           call pvmfrecv(0, msgtag, ibuf) 
           call pvmfunpack(REAL8, vec, ISIZE, 1, info)
           print *, 'Task ', mytid, ' received data ...'
        endif
  
  ccc   Wait for all slaves to reach this point 
        call pvmfbarrier(PVMALL, -1, info)
  
  ccc   PE0 sends ACK to Y-MP process
        if (mype .eq. 0) then
           msgtag = 2
           call pvmfinitsend(PVMDATADEFAULT, info)
           call pvmfpack(INTEGER8, 1, 1, 1, info)
           call pvmfsend(myparent, msgtag, info)
        endif
  
  ccc   Everybody exits PVM
        call pvmfexit(info)
        print *, 'Task ', mytid, ' exiting...'
  
        end
Makefile:

  all: master slave
  
  slave: slave.f  
          (export TARGET=cray-t3d;                            \
          /mpp/bin/cf77 -I/usr/include/mpp -o slave slave.f)  
          cp slave ${HOME}/pvm3/bin/CRAY/
  
  master: master.f  
          (export TARGET=target;                              \
          cf77 -I/usr/include/pvm3 -o master master.f -lpvm3)
From:

  =======================================================================
  
    Don Morton                              Email:  morton@arsc.edu  

  
    Visiting Scientist  (summer)            Voice: (907) 474-5507    

  
    Arctic Region Supercomputing Center     Fax  : (907) 450-8601    

  
    University of Alaska                                             

  
    Fairbanks, AK 99775                                              

  
    http://grizzly.cameron.edu/morton.html                           

  =======================================================================

The "TARGET" Environment Variable

If you do development work on both the Y-MP and the T3D, and switch back and forth between the two, you might have some handy aliases defined. For instance, users of the korn shell might have the following in their .kshrc file:

  alias t3d1='TARGET=cray-t3d;MPP_NPES=1;PVM_PE_LIST=0'
  alias t3d2='TARGET=cray-t3d;MPP_NPES=2;PVM_PE_LIST=0'
  alias t3d4='TARGET=cray-t3d;MPP_NPES=4;PVM_PE_LIST=0'
  alias t3d8='TARGET=cray-t3d;MPP_NPES=8;PVM_PE_LIST=0'
The following alias for switching back to the Y-MP environment, however, would be a mistake (were it not commented out):

  #alias ymp='TARGET=cray-ymp'
The correct Y-MP alias is:

  alias ymp='TARGET=target'
The setting, "target," is the configured default, which should be set correctly for each site. The setting, "cray-ymp," sounds good, but according to the man page for the UNICOS command, "target," is for the "CRAY Y-MP model 832."

I discovered this when cf77 suddenly balked on a test program, and gave the following message:


  ldr-159 cf77: CAUTION 
  The program in file 'a.out' requires 40150829 words of memory to begin
  execution.  The target computer system contains only 33554432 words of
  memory.
The linker refused to produce an executable because it thought that denali had only 33 MWs of memory -- less than required for the given program. As we know, though, denali has, not 33 MWs, but 1000 MWs of memory.

The problem was that I had set TARGET to "cray-ymp" in order to switch it back from "cray-t3d." I should have set it to "target."

The following (abridged) listing shows the "cray-ymp", "target", and "cray-t3d" configurations, as known on denali.


  denali$ 
  denali$ TARGET=cray-ymp
  denali$ target
    Primary machine type is:  CRAY-Y-MP
      banks  = 256
      numcpus  = 8
      ibufsize = 32
      memsize  = 33554432
      memspeed = 17
      clocktim = 6000
      numclstr = 9
      bankbusy = 5
  denali$ TARGET=target
  denali$ target
    Primary machine type is:  CRAY-Y-MP
      banks  = 256
      numcpus  = 8
      ibufsize = 32
      memsize  = 1073741824
      memspeed = 27
      clocktim = 6000
      numclstr = 9
      bankbusy = 20
  denali$ TARGET=cray-t3d
  denali$ target
    Primary machine type is:  CRAY-T3D
      memsize  = 8388608
  denali$ 

Setting Environment Variables for a Single Sub-Process

While we're discussing environment variables... Here's something for everyone who's made the switch to the Korn shell.

The Korn shell lets you change the setting of environment variables for the scope of single sub-processes, without altering their global values. You simply preface any command with one or more variable definitions. The environment for that command will be set accordingly.

Examples:


  denali$ TARGET=cray-t3d target 
  denali$ TARGET=cray-ymp target

  denali$ TARGET=target cf77 -I/usr/include/pvm3 -o master master.f -lpvm3
  denali$ TARGET=cray-t3d /mpp/bin/cf77 -I/usr/include/mpp -o slave slave.f

  denali$ TMPDIR=/tmp/baring/960607  MPP_NPES=4  a.out
  denali$ TMPDIR=/tmp/baring/960607  MPP_NPES=8  a.out
Using this approach, we have an alternative structure for makefiles with multiple targets. The following example is based on Don Morton's makefile:

  # Makefile

  all: master slave

  slave: slave.f  
          TARGET=cray-t3d /mpp/bin/cf77 -I/usr/include/mpp -o slave slave.f  
          cp slave ${HOME}/pvm3/bin/CRAY/

  master: master.f  
          TARGET=cray-ymp cf77 -I/usr/include/pvm3 -o master master.f -lpvm3

European CRAY MPP Workshop

This announcement came through CUG's MPP Mailing List:

> 
>          *********************************************
>          *    PROGRAMME AND CALL FOR PARTICIPATION   *
>          *   FOR THE 2ND EUROPEAN CRAY MPP WORKSHOP  *
>          *                                           *
>          *   http://www.epcc.ed.ac.uk/t3d/workshop   *
>          *********************************************
>  
> 
> The programme for The 2nd European Cray MPP Workshop has now been
> finalised.  Most of the major Cray MPP sites in Europe are represented
> with speakers from Finland, France, Germany, Italy, Spain, Switzerland,
> and the UK.  There will also be talks from Cray US and Pittsburgh
> Supercomputer Center about early experiences with the Cray T3E.  The
> programme is available at
> 
>   http://www.epcc.ed.ac.uk/t3d/workshop/programme.html
> 
> We currently have over 100 registrations, so as space is limited, please
> register as soon as possible if you are interested in attending; the
> deadline is June 21.  A registration form is available at our WWW site. 
> 
> Although we can no longer accept offers of talks, there is still some
> space left for the poster session.  We are interested in posters
> describing any aspects of Applications in Science and Engineering,
> Systems Management, Programming Models or Performance Optimisation on
> Cray MPP platforms.  See the WWW pages for more details. 
> 
> 
> Please distribute this information to anyone who might be interested. 
> We look forward to meeting you in Edinburgh in July,
> 
> Dr David Henty
> T3D In-Depth Support Team
> Edinburgh Parallel Computing Centre
> workshop@epcc.ed.ac.uk
> 

A Call for Material

If you have discovered a good technique or information concerning the T3D and you think it might benefit others, please send it to the e-mail address above and it will be passed on through this newsletter.
Current Editors:
Ed Kornkven ARSC HPC Specialist ph: 907-450-8669
Kate Hedstrom ARSC Oceanographic Specialist ph: 907-450-8678
Arctic Region Supercomputing Center
University of Alaska Fairbanks
PO Box 756020
Fairbanks AK 99775-6020
E-mail Subscriptions: Archives:
    Back issues of the ASCII e-mail edition of the ARSC T3D/T3E/HPC Users' Newsletter are available by request. Please contact the editors.
Back to Top