Module: rje_hpc Description: High Performance Computing job farming Version: 1.1.1 Last Edit: 22/02/18

Copyright © 2014 Richard J. Edwards - See source code for GNU License Notice

Imported modules: rje rje_obj rje_seq

See SLiMSuite Blog for further documentation. See rje for general commands.

Function

This module contains the generic code for farming jobs out over multiple processors and/or nodes. It is based on rje_iridis Version 1.10, updated to the new RJE_Object structure and made more generic. The recommended tool for actual job farming is SLiMFarmer, which inherits the rje_hpc.JobFarmer object and adds extra bells and whistles.

Commandline

Generic HPC Job Farming pptions

subsleep=X : Sleep time (seconds) between cycles of subbing out jobs to hosts [1]
subjobs=LIST : List of subjobs to farm out to HPC cluster []
hpcmode=X : Mode to be used for farming jobs between nodes (rsh/fork) [fork]
iolimit=X : Limit of number of IOErrors before termination [50]
memfree=X : Min. proportion of node memory to be free before spawning job [0.0]
keepfree=X : Number of processors to keep free on head node [1]

SLiMSuite and SeqSuite SeqBySeq program options

seqbyseq=T/F : Activate seqbyseq mode - assumes basefile=X option used for output [False]
farm=X : Program to farm out using seqbyseq mode. []
seqin=FILE : Input sequence file to farm out [None]
basefile=X : Base for output files - compiled from individual run results [None]
outlist=LIST : List of extensions of outputs to add to basefile for output (basefile.*) []
pickhead=X : Header to extract from OutList file and used to populate AccNum to skip []
startfrom=X : Sequence ID at which to begin the SeqBySeq farming [None]
rjepy=T/F : Whether program is an RJE *.py script (adds log processing) [True]
jobini=FILE : Ini file to pass to the called program [None]
pypath=PATH : Path to python modules ['/home/re1u06/Serpentry/']

See also rje.py generic commandline options.

History Module Version History

    # 1.0 - Initial Compilation based on rje_iridis V1.10.
    # 1.1 - Disabled memory checking in Windows and OSX.
    # 1.1.1 - Added output of subjob command to log as run.