Bayesian Programming and Learning for Multi-Player Video Games ...

9.2.4 Inter-game Adaptation (Meta-game)
In a given match, and/or against a given player, players tend to learn from their immediate
mistakes, and they adapt their strategies to each other’s. As this can be seen as a continuous
learning problem (reinforcement learning, exploration-exploitation trade-off), there is more to
it. Human players call this the meta-game*, as they enter the “I think that he thinks that I
think...” game until arriving at fixed points. This “meta-game” is closely related to the balance
of the game, and the fact that there are several equilibriums makes the interest of StarCraft.
Also, clearly, for human players there is psychology involved.
Continuous learning
For all strategic models, the possible improvements (subsections 7.5.3, 7.6.3, 7.7.2) would include
to learn specific sets of parameters against the opponent’s strategies. The problem here is that
(contrary to battles/tactics) there are not much observations (games against a given opponent) to
learn from. For instance, a naive approach would be to learn a Laplace’s law of succession directly
on P (ET echT rees = ett|P layer = p) = 1+nbgames(ett,p)
#ET T +nbgames(p) , and do the same for EClusters,
but this could require several games. Even if we see more than one tech tree per game for the
opponent, a few games will still only show a sparse subset of ET T . Another part of this problem
could arise if we want to learn really in-game as we would only have partial observations.
Manish Meta [2010] approached “meta-level behavior adaptation in RTS games” as a mean
for their case-based planning AI to learn from its own failures in Wargus. The opponent is not
considered at all, but this could be an interesting entry point to discover bugs or flaws in the
bot’s parameters.
Bot’s psychological warfare
Our main idea for real meta-game playing by our AI would be to use our models recursively. As
for some of our models (tactics & strategy), that can be used both for prediction and decisionmaking,
we could have a full model of the enemy by maintaining the state of a model from them,
with their inputs (and continually learn some of the parameters). For instance, if we have our
army adaptation model for ourselves and for the opponent, we need to incorporate the output
of their model as an input of our model, in the part which predicts the opponent’s future army
composition. If we cycle (iterate) the reasoning (“I will produce this army because they will have
this one”...), we should reach these meta-game equilibriums.
Final words
Finally, bots are as far from having a psychological model of their opponent as from beating the
best human players. I believe that this is our adaptability, our continuous learning, which allows
human players (even simply “good” ones like me) to beat RTS games bots consistently. When
robotic AI will start winning against human players, we may want to hinder them to have only
partial vision of the world (as humans do through a screen) and a limited number of concurrent
actions (humans use a keyboard and a mouse so they have limited APM*). At this point, they
will need an attention model and some form of hierarchical action selection. Before that, all the
problems arose in this thesis should be solved, at least at the scale of the RTS domain.
176