Administering Platform LSF - SAS

Automatic Job Suspension
Automatic Job Suspension
Jobs running under LSF can be suspended based on the load conditions on the
execution hosts. Each host and each queue can be configured with a set of
suspending conditions. If the load conditions on an execution host exceed
either the corresponding host or queue suspending conditions, one or more
jobs running on that host will be suspended to reduce the load.
When LSF suspends a job, it invokes the SUSPEND action. The default
SUSPEND action is to send the signal SIGSTOP.
By default, jobs are resumed when load levels fall below the suspending
conditions. Each host and queue can be configured so that suspended
checkpointable or rerunnable jobs are automatically migrated to another host
instead.
If no suspending threshold is configured for a load index, LSF does not check
the value of that load index when deciding whether to suspend jobs.
Suspending thresholds can also be used to enforce inter-queue priorities. For
example, if you configure a low-priority queue with an r1m (1 minute CPU run
queue length) scheduling threshold of 0.25 and an r1m suspending threshold
of 1.75, this queue starts one job when the machine is idle. If the job is CPU
intensive, it increases the run queue length from 0.25 to roughly 1.25. A highpriority
queue configured with a scheduling threshold of 1.5 and an unlimited
suspending threshold will send a second job to the same host, increasing the
run queue to 2.25. This exceeds the suspending threshold for the low priority
job, so it is stopped. The run queue length stays above 0.25 until the high
priority job exits. After the high priority job exits the run queue index drops
back to the idle level, so the low priority job is resumed.
When jobs are running on a host, LSF periodically checks the load levels on
that host. If any load index exceeds the corresponding per-host or per-queue
suspending threshold for a job, LSF suspends the job. The job remains
suspended until the load levels satisfy the scheduling thresholds.
At regular intervals, LSF gets the load levels for that host. The period is defined
by the SBD_SLEEP_TIME parameter in the lsb.params file. Then, for each job
running on the host, LSF compares the load levels against the host suspending
conditions and the queue suspending conditions. If any suspending condition
at either the corresponding host or queue level is satisfied as a result of
increased load, the job is suspended. A job is only suspended if the load levels
are too high for that particular job’s suspending thresholds.
There is a time delay between when LSF suspends a job and when the changes
to host load are seen by the LIM. To allow time for load changes to take effect,
LSF suspends no more than one job at a time on each host.
Jobs from the lowest priority queue are checked first. If two jobs are running
on a host and the host is too busy, the lower priority job is suspended and the
higher priority job is allowed to continue. If the load levels are still too high
on the next turn, the higher priority job is also suspended.
360
Administering Platform LSF