Job Chaining in Three Steps

Job Chaining is a convenient tool used for automating the preparation of input files for compute jobs (pre-processing), launching the compute job, then archiving the new output files (post-processing).  

The following 3 steps describe how to set up a basic job chain to run on the pacman HPC system at ARSC.

 

Step 1 - Create a Pre-processing Job Submission Script

To begin the job chain, a pre-processing job submission script needs to be created to  batch_stage all archived input files, then copy them to your $CENTER.  Following the copy, this pre-processing script then automatically submits your regular job submission script to the queue.

#!/bin/bash 
#PBS -q transfer 
#PBS -l walltime=24:00:00  
#PBS -l nodes=1:ppn=1 
#PBS -j oe 

cd $PBS_O_WORKDIR 

batch_stage $ARCHIVE/results.tar 
/usr/bin/rcp "bigdip-s:$ARCHIVE/results.tar" $CENTER || exit 1 

# Submit your regular job to the compute nodes 
qsub my_job.pbs 

Step 2 - Modify Your Existing Job Submission Script

Your regular job submission script should remain the same, plus a few new lines at the end of the file. These new lines test whether your compute process completed successfully.  If the process ended successfully, a post-processing job will be automatically submitted to the transfer queue to copy the output files from $CENTER to $ARCHIVE.

#PBS -q standard_4 
#PBS -l nodes=1:ppn=4 
#PBS -l walltime=8:00:00 
#PBS -j oe 

cd $PBS_O_WORKDIR 

# Run the executable:   
mpirun ./a.out 

# Check for error status: 
#  If the application exited successfully, submit the post processing  
#  script. 
# 
if [[ $? -eq 0 ]]; then 
   qsub postProcess.pbs 
else 
   echo "Error: The application exited with an error!" 
fi  

Step 3 - Create a Post-processing Job Submission Script

If your compute job completes successfully, the script from step 2 will submit the postProcess.pbs script to the queue automatically.  This postProcess.pbs script will copy the output files back to $ARCHIVE.

#!/bin/bash 
#PBS -q transfer 
#PBS -l walltime=24:00:00  
#PBS -l nodes=1:ppn=1 
#PBS -j oe 

cd $PBS_O_WORKDIR 

tar -cvf mydata.tar $CENTER/myOutputFiles 
/usr/bin/rcp $CENTER/mydata.tar "bigdip-s:$ARCHIVE/" || exit 1 

More Information

A more complex example using job chaining is described in the ARSC HPC Users' Newsletter Issue 322.

Back to Top