Administering Platform LSF - SAS

Handling Job Exceptions
eadmin script
Chapter 5
Working with Queues
You can configure queues so that LSF detects exceptional conditions while jobs
are running, and take appropriate action automatically. You can customize
what exceptions are detected, and the corresponding actions. By default, LSF
does not detect any exceptions.
When an exception is detected, LSF takes appropriate action by running the
script LSF_SERVERDIR/eadmin on the master host. You can customize eadmin
to suit the requirements of your site. For example, in some environments, a job
running 1 hour would be an overrun job, while this may be a normal job in
other environments. If your configuration considers jobs running longer than
1 hour to be overrun jobs, you may want to close the queue when LSF detects
a job that has run longer than 1 hour and invokes eadmin. Alternatively,
eadmin could find out the owner of the problem jobs and use bstop -u to
stop all jobs that belong to the user.
Job exceptions LSF can detect
Default eadmin actions
If you configure exception handling, LSF detects the following job exceptions:
◆ Job underrun—jobs end too soon (run time is less than expected).
Underrun jobs are detected when a job exits abnormally
◆ Job overrun—job runs too long (run time is longer than expected)
By default, LSF checks for overrun jobs every 5 minutes. Use
EADMIN_TRIGGER_DURATION in lsb.params to change how frequently
LSF checks for job overrun.
◆ Idle job—running job consumes less CPU time than expected (in terms of
CPU time/runtime)
By default, LSF checks for idle jobs every 5 minutes. Use
EADMIN_TRIGGER_DURATION in lsb.params to change how frequently
LSF checks for idle jobs.
LSF sends email to the LSF administrator. The email contains the job ID,
exception type (overrrun, underrun, idle job), and other job information.
An email is sent for all detected job exceptions according to the frequency
configured by EADMIN_TRIGGER_DURATION in lsb.params. For example,
if EADMIN_TRIGGER_DURATION is set to 10 minutes, and 1 overrun job and
2 idle jobs are detected, after 10 minutes, eadmin is invoked and only one
email is sent. If another overrun job is detected in the next 10 minutes, another
email is sent.
Administering Platform LSF 109