Helpdesk

Many of the programs available on our systems for bioinformatics can take a considerable amount of time to run. Some may take days or even weeks to complete their analysis. For this reason, it may be desirable to place such jobs "in the background." This is a way of running a program that allows you to continue working on other tasks (or even log out) while still keeping the program running. Furthermore, backgrounded jobs are not dependent on your session remaining open, so even if your computer crashes, your job will continue uninterrupted.

NOTE! Due to problematic interactions between OpenMosix kernels and threading used by perl/python/java, you may experience strange and random segfaults when running programs written in these languages. Please use the -g option to bgrun for such programs to prevent them from running on OpenMosix nodes.

There is one consideration you should keep in mind when you're planning to run a job in the background - is it interactive or non-interactive? Interactive jobs require you to provide input after you enter the command. An example of an interactive job is the passwd command - after you enter the command, you have to answer some prompts (ie your password). A non-interactive job is one that doesn't require any further input after entering the command. An example is the ls command - after you enter the command, it does its thing, and then ends.

Non-interactive Jobs

Non-interactive jobs are easy to run in the background and there are several ways to go about it. The best method for most cases is to use the bgrun command to preface your usual command. For example:

bgrun mendel

bgrun is a special program that will take all of it's arguments as a command and submit them to the cluster batch queue. This ensures that the cluster is utilized as efficiently as possible.

After invoking the command, you will be given some information about the job you just submitted, including the identifier and where the output of the program (if any) will go.

your job 8202 ("mendel.31187") has been submitted
Job submitted. Identifier: 31187 ; output redirected to mendel.31187.output

You can check on the status of your submitted jobs by entering the following command:

qstat -u myusername

If your command generates output normally, you can also preview that output by looking at the output file using the tail command. In the example above, you would use tail mendel.31187.output to look at the last 10 lines of that file.

When the job has completed running, you will no longer see the job listed via the qstat command. You can also cancel a job by entering qdel myjobid. The job id is the first number in the bgrun output above, ie 8202.

Once the job is running in the cluster queue, you do not have to remained logged in. You can log out and your job will continue to run.

The bgrun command has a few options itself. They can be seen if you enter bgrun as a command with no arguments. They include -m which will cause bgrun to generate an email message once the job is complete (this requires an email address in your ~/.forward file); -x which will exclude the head node from running the job (it will instead select one of the backing, usually less busy, cluster nodes); and -o OUTFILE which will let you specify a name for the output file instead of the default.

Interactive Jobs

Interactive jobs are harder to run in the background, since if they are running in the background, there is no way for it to receive input from you. However, some interactive programs can be made to run in a non-interactive way. Some, for example, allow you to answer all prompts as a series of command-line options. If this is possible, then you should investigate the options required to do this and then run the command non-interactively.

Other commands can accept input from a file instead of from your direct input. In these cases, you would put all of the answers you would type during the interactive run into file and then "pipe" these answers into the program via input redirection, eg command < input.file. Again, in this case, you can then run the command as if it were non-interactive.

If there is no way to make a program run in a non-interactive fashion, you can essentially background your entire session using the screen command. To do so, you have to first initialize a screen session by entering screen. It will look like nothing happened. Now run your interactive job. Once you have provided all required inputs and the job has begun to run, you can background the entire session by pressing Control-A followed by D. If you do this successfully, you should see a [detached] message to indicate your session was put in the background.

At this point, you can log out.

To re-attach your session, enter screen -r. Your session should pick up right where it left off. If your job is complete, you can close out your session by entering exit. Remember to close out all screen'd sessions! You can see a list of all your screen sessions via screen -list.