ARSC HPC Users' Newsletter 335, March 3, 2006
X1: Capturing missing stdout
[ This article is inspired by the PET Consolidated X1 FAQ ]
The Cray X1 buffers stdout, which can lead to missing information in your output files when the program crashes before the buffer is flushed. You could add explicit flush statements to your code to ensure stdout is written, however the assign command provides a much more elegant way accomplishing this task. Since stderr is not buffered, we can simply direct stdout (unit 6) to stderr to ensure that output is written. NOTE: there could be considerable performance impact doing this, so this technique should only be used for debugging purposes.
First, set the file to store the assign command to:
setenv FILENV ./myassign
Next, use the assign command to direct stdout to stderr:
assign -D stderr u:6
Finally, run the executable as you normally would.e.g.
aprun -n 4 ./a.out
If you redirect stdout to a file, be sure to redirect stderr as well:
aprun -n 4 ./a.out 2> output
aprun -n 4 ./a.out >& output
To remove an assign statement from the file, use the following:
Here's a PBS script which ties everything together.
#PBS -q default #PBS -l mppe=4 #PBS -l walltime=30:00 #PBS -j oe #PBS -S /bin/ksh cd $PBS_O_WORKDIR export FILENV=./assign.debug assign -D stderr u:6 aprun -n 4 ./a.out
Since the environment variable FILENV is set within the script, there is no need to remove the assign statement stored in the file unless you are using the same file to store other assign statements.
NOTE: this trick doesn't work with C based codes.
The PET X1 Consolidated FAQ is available at:https://okc.erdc.hpc.mil
To get to the Consolidated X1 FAQ:
- Select the "CE" link under the "Cross-Cutting Areas" heading (in the "Menu" frame on the left side of the window), then
- Then the "Papers/Pubs" link (in the "CE" frame), and finally
- the "Cray X1 Frequently Asked Questions" link (in the list of papers).
ARSC also maintains a direct link to the FAQ from the klondike how to page:
Three-part GIS sessions March 27th ,29th, and 31st
Title: Workshop of Online 3D Terrain Visualization and GIS tools Location: WRRB 010 Conference Room Dates: Monday, March 27 Wednesday, March 29 Friday, March 31 Time: noon - 3:00 pm (lunch provided)
This workshop is intended to create earth science, earth science education/outreach, and earth science data access applications using online 3D terrain visualization and GIS tools such as EarthSLOT, Google Earth, World Wind and GeoFusion. Your own Windows laptop computer is required (the needs of participants needing Macintosh access will be covered, but development will occur on PCs for consistency). The format of the workshop will be tailored to the nature of the participants' projects.
To attend, please send an email to matt.nolan (at) uaf.edu that outlines the nature of your project, the types of data you hope to use, the target audience, what you would like to accomplish in the workshop, and how you envision using these tools in the future. We will then work with you before the workshop to refine your ideas, ensure your data are in easily ingestible formats, ensure that your computer has all of the necessary software, and suggest some advance reading if appropriate.
The workshop is limited to 10 participants. If the number of applicants exceed 10, preference will be given to graduate students with a clear vision of what they want to accomplish and who have all the necessary data in hand to finish a pilot project by the end of the workshop. Additional workshops will be held as demand warrants, and it is hope that the participants of this first workshop will serve as the nucleus of a UAF community built around use and advancement of these tools.
In the last few months, ARSC has added new hardware to both iceflyer and nelchina. These additions have increased the available computation resources of both systems by 2 times or more.
4- p575 nodes have been added to iceflyer. Each of these nodes has 8- 1.9 Ghz Power5 processors and 16 GB of RAM. Currently the Power5 nodes are running a variety of operating systems including: AIX 5.2, AIX 5.3, and SuSE Linux Enterprise Server (SLES) 9.2.
Both AIX 5.3 and SLES 9.2 can exploit the SMT (simultaneous multithreading) support available within the Power5 architecture. This functionality allows the operating system to schedule two threads per processor. Preliminary tests done by ARSC staff show that SMT reduces run time for some OpenMP applications over 10% when doubling the number of threads (i.e. setting OMP_NUM_THREADS=16 rather than 8).
These nodes are now open to all iceflyer users via the "p5" class.
For more information on the power5 nodes iceflyer users should see:
During the past two weeks ARSC and Cray staff have been busy adding three more chassis to the Cray XD1, nelchina. Each of the new chassis has 6 nodes with 2 dual core AMD Opteron processors per node.
Once the upgrade is complete, nelchina will have 108 PEs in the following configuration.
6- nodes with 2 AMD Opteron 250 Processors (Single Core) with FPGAs 12- nodes with 2 AMD Opteron 250 Processors (Single Core) 18- nodes with 2 AMD Opteron 280 Processors (Dual Core)
The upgrade also includes a change to the fat tree network topology which improves performance for all-to-one, one-to-all, or all-to-all communications by providing a direct link between each node.
Users interested in using nelchina should contact the ARSC help desk at email@example.com.
Quick-Tip Q & A
A:[[ I want stdout and stderr from a command to go to a file AND to my [[ screen. These commands, [[ [[ CSH: [[ ./prog >& prog.out [[ [[ KSH: [[ ./prog > prog.out 2>&1 [[ [[ would merge stdout and stderr to one file. But, as I said, I'll [[ like to see it on the screen at the same time. We got a ton of responses to this, but, as professors lament at finals time, several people didn't read the question too carefully. The real problem here was dealing with stderr (not just stdout and "tee"). Thanks to all: Chris Young, Brad Chamberlain, Harper Simmons, Steve Finn, Rich Griswold, Greg Newby, Kurt Carlson, Hank Happ, Lorin Hochstein, and Martin Luthi. # # From Brad Chamberlain, here's a csh solution: # Try the following (CSH format): ./prog & tee prog.out "tee" is a program that feeds its standard input to a filename and to stdout. Thus, what we're doing here is piping the standard ( ) and error (&) output of prog into tee, which is piping it to prog.out and to stdout. # # From Kurt Carlson, ksh: # ./prog 2>&1 tee prog.out # # Lorin Hochstein reminds us of another possibility, tail -f. A problem # with this, tail -f doesn't terminate with the program, which could be # misleading. # Terminal 1: touch prog.out tail -f prog.out Terminal 2: ./prog >& prog.out The "tail" programs allows a user to view the file on Terminal 1 as data is being appended. (When the file is done, Ctl-C is required to terminate the tail program.) Q: On some systems, when I use vi/vim to open a file and scroll down to a particular line and then exit, vi/vim remembers the position within the file. The next time I open the file it returns me to that position. I can't get this feature working everywhere. Do you know how one enables this?
[[ Answers, Questions, and Tips Graciously Accepted ]]
Ed Kornkven ARSC HPC Specialist ph: 907-450-8669 Kate Hedstrom ARSC Oceanographic Specialist ph: 907-450-8678 Arctic Region Supercomputing Center University of Alaska Fairbanks PO Box 756020 Fairbanks AK 99775-6020
Subscribe to (or unsubscribe from) the e-mail edition of the
ARSC HPC Users' Newsletter.
Back issues of the ASCII e-mail edition of the ARSC T3D/T3E/HPC Users' Newsletter are available by request. Please contact the editors.