DARPA ULTRALOG Final Report - Industrial and Manufacturing ...

Load balancing issues have not been paid much attention
in MAS studies [7][8]. There are few papers in multiagent
load balancing. Schaerf et. al. [9] studies how an agent can
adapt to the environment. They separated the resources
from the agents. In their model, agents assign their jobs to
these resources. Using reinforcement learning, they
showed that agents could adapt to each other under fixed
or even for dynamical loads. Chow and kwok [7] devise an
agent reallocation algorithm, called ‘Comet’ algorithm,
which select agents to be moved to other machines.
Agents are distributed on multiple machines. The comet
algorithm chooses agents based on credits, which are
continuously evaluated for each agent. The agent with low
credit will be moved. The credit will decrease as the
agent’s workload increases or the agent has more
communication with other agents on other machines. We
consider a similar agent system environment with chow
and kwok. However, we added a feature of agent’s selfregulation
on the workload.
3 QUEUEING MODEL FOR WORKLOAD DYNAMICS
We conjecture that workload dynamics could be modeled
as a queueing system. A service request from outside or
other agents could be seen as a customer in a queueing
system. While a request is being served, the later incoming
service requests will wait in the queue. We consider a
situation in which agents have multiple alternative
algorithms to provide their service. Those algorithms trade
off between computation time and quality of solution.
Thus, depending on the workload in the queue, an agent
can choose an optimal algorithm to improve the overall
performance measure. This is similar to anytime algorithm
composition [10]. In anytime algorithm, we have to
determine time duration in which the algorithm solves the
problem. Here, we assume that problem solving time is not
predetermined. Instead, it is a statistical characteristic of
the algorithm. From the queueing model perspective, this
could be seen as a service rate control problem [11].
Multiagent system infrastructure can have a facility, where
each machine works as a server by assigning
computational resources (run right) to the agents for CPU
time-sharing. We call the server as a node. This could be
seen as a polling model, which has been used to model for
time-sharing in a computer operating system or link
sharing in a communication network. The node could give
priority to a certain agent by visiting the agent more
frequently. The node could monitor the amount of workload
or arrival rate of service requests through agents. Based
on the detected changes, it can change the priority of the
agent.
Imbalance among machines can be controlled by
reallocation of agents from a high loaded machine to a low
loaded machine for better performance. However, in this
paper, we considered only agent and machine level
control.
4 DECENTRALIZED CONTROL
In view of the above mentioned workload dynamics, load
control in multiagent systems could be seen as a
decentralized stochastic control problem [12]. The
decentralized control system consists of multiple control
posts. They locally sense and control some part of the
system they take charge of. However, their controls
influence the system dynamics collectively. Thus, in order
to operate the system optimally, decisions taken by
controllers should be compatible and coherent. The
information sharing between agents is treated as the main
issue. In order for the controllers to obtain global optimal
control decisions, exchange of all the local information is
inevitable. Each controller makes a decision by solving a
larger problem in which other agents’ movement is
considered. However, it is unrealistic in the case of
multiagent systems because of the long communication
time. Finding optimal controls might be very difficult due to
the size of the problem. It is difficult or almost impossible to
find a purely decentralized optimal control policy for a
multiagent system in this way. There are few systems in
which locally made decisions could be globally compatible
[13]. However this is very limited to some specific problem
only. Thus, we need a control structure in which each
control component (for example, an agent) takes control
decisions by communicating only with closely connected
components, while well coordinated decisions could be
generated. In this paper, we suggest a hierarchical control
structure which aims at achieving the above mentioned
expectations.
5 HIERARCHICAL CONTROL
5.1 General Description
In order to manage the complexity of large-scale problems,
hierarchical control approach have been studied in various
areas [14][15]. For multiagent systems, hierarchical control
has been adopted as an intermediate form between
centralized and decentralized control as a tradeoff between
the advantages of the two approaches [14]. Hierarchical
control can reduce the computation of gathering
information and finding an optimal control than centralized
control. On the other hand, it has better coordination
capabilities than decentralized control.
We consider three levels of hierarchy – the entire system,
nodes and agents. There are usually multiple
subcomponents under a higher-level controller i.e. there
are multiple agents under a node and multiple nodes under
a top controller. Agents and a node controller have direct
communication connection by which they share
information. The information sharing is restricted between
the components, which are connected in the hierarchy. In
this case, an agent reports its workload and performance
(see 5.2) and the node controller announces control (the
visit order and frequency) or state information such as
estimates about environment parameters, which could be
more effectively observed by the node rather than agents.
We could also consider similar information exchanges
between nodes and the top controller. Here, nodes report
their node level workload trend to the top controller. On the
other hand, the top controller could inform the system level
environment parameters and order to move agents from
one node to another.
We assume the control frequency to be different at
different levels. It is higher in lower levels compared to the
higher levels. Control decisions on service rate are more
frequent than the changes on the CPU time assignment
policy in the node level. For a given arrival rate and
configuration, CPU time assignment policy in node level
will not be changed until the arrival rate and configuration
change. At higher level, the frequency of events is less and
the time intervals between events become longer. This
could be said to have multi-time scale depending on the
level [16]. The difference of our problem from other multitime
scale problems is that there are multiple components
in the lower levels. In addition, it is reasonable to assume
that the environment does not change very frequently, in
such a manner that the system may not be able to
estimate environment parameters and control it.
A Higher-level controller has coarser information than the
lower level subsystems. A higher-level controller could
have better global information over its territory because it
collects information from its subcomponents. However, it
will not use the information gathered as a system state