Lecture 11: Learning in Games - Fictitious Play 1 Introduction 2 ...

6.254: Game Theory with Engineering Applications March 13-18, 2008 

Lecture 11: Learning in Games - Fictitious Play 

Lecturer: Asu Ozdaglar 

1 Introduction 

Most economic theory relies on equilibrium analysis based on Nash equilibrium or its refinements. 

The traditional explanation for when and why equilibrium arises is that it results 

from analysis and introspection by the players in a situation where the rules of the game, 

the rationality of the players, and the payoff functions of players are all common knowledge. 

In this lecture, we develop an alternative explanation why equilibrium arises as the longrun 

outcome of a process in which less than fully rational players grope for optimality over 

time. We investigate theoretical models of learning in games. Our focus will be on the process 

of fictitious play (and its variants), which is a widely used model of learning. In this process, 

agents behave as if they think they are facing a stationary, but unknown, distribution of 

opponents’ strategies. We examine whether fictitious play is a sensible model of learning 

and the asymptotic behavior of play when all players use fictitious play learning rules. 

2 Fictitious Play 

One of the earliest learning rules studied is fictitious play. Fictitious play was introduced by 

Brown [1]. It is a “belief-based” learning rule, i.e., players form beliefs about opponent play 

and behave rationally with respect to these beliefs. 

The model for fictitious play is given below. Here we focus on the case where we have 

two players only, see [2] for the analysis of fictitious play for multiple players. 

• Two players, i = 1, 2, play the strategic form game G at times t = 1, 2, . . .. 

• Denote the stage payoff of player i by g i (s i , s −i ). 

11-1

• For t = 1, 2, . . . and i = 1, 2, define the function η t i : S −i → N, where η t i(s −i ) is 

the number of times player i has observed the action s −i before time t. Let η 0 i (s −i ) 

represent a starting point (or fictitious past). 

Example 1: Assume that S 2 = {U, D}. If η 0 1(U) = 3 and η 0 1(D) = 5, and player 2 plays 

U, U, D in the first three periods, then η 3 1(U) = 5 and η 3 1(D) = 6. 

Each player assumes that his opponent is using a stationary mixed strategy. 

Players 

choose actions in each period (or stage) to maximize that period’s expected payoff given 

their prediction of the distribution of opponent’s actions, which they form according to: 

µ t i(s −i ) = 

η t i(s −i ) 

∑¯s −i ∈S −i 

η t i (¯s −i) , 

i.e., player i forecasts player −i’s strategy at time t to be the empirical frequency distribution 

of past play. 

Given player i’s belief/forecast about his opponents play, he chooses his action at time t 

to maximize his payoff, i.e., 

s t i ∈ arg max 

s i ∈S i 

g i (s i , µ t i). 

This choice is myopic, i.e., players are trying to maximize current payoff without considering 

their future payoffs. Note that myopia is consistent with the assumption that players are 

using stationary mixed strategies. 

Example 2: Consider the fictitious play of the following game: 

L 

R 

U 3, 3 0, 0 

D 4, 0 1, 1 

Note that this game is dominant solvable (D is a strictly dominant strategy for the row 

player), and the unique Nash equilibrium is (D, R). Assume that η 0 1 = (3, 0) and η 0 2 = (1, 2.5). 

Period 1: Then, µ 0 1 = (1, 0) and µ 0 2 = (1/3.5, 2.5/3.5), so play follows s 0 1 = D and s 0 2 = L. 

Period 2: We have η 1 1 = (4, 0) and η 1 2 = (1, 3.5), so play follows s 1 1 = D and s 1 2 = R. 

Period 3: We have η 1 1 = (4, 1) and η 1 2 = (1, 4.5), so play follows s 2 1 = D and s 2 2 = R. 


Since D is a dominant strategy for the row player, he always plays D, and µ t 2 converges 

to (0, 1) with probability 1. Therefore, player 2 will end up playing R. 

Remark: The striking feature of the fictitious play is that players don’t have to know 

anything about their opponent’s payoff. They only form beliefs about how their opponents 

will play. 

3 Convergence of Fictitious Play 

Let {s t } be a sequence of strategy profiles generated by fictitious play. In this section, we 

study the asymptotic behavior of the sequence {s t }, i.e., the convergence properties of the 

sequence {s t } as t goes to infinity. 

We first define the notion of convergence to pure strategies. 

Definition 1 The sequence {s t } converges to s if there exists T such that s t = s for all 

t ≥ T . 

The next proposition formalizes the property that if the fictitious play sequence converges, 

then it must converge to a Nash equilibrium of the game. 

Proposition 1 Let {s t } be a sequence of strategy profiles generated by fictitious play. 

(a) If {s t } converges to ¯s, then ¯s is a pure strategy Nash equilibrium. 

(b) Suppose that for some t, s t = s ∗ , where s ∗ is a strict Nash equilibrium of G. Then 

s τ = s ∗ for all τ > t. 

Part (a) is straightforward. We provide a proof for part (b): 

Proof of part (b): Let s t = s ∗ . We will show that s t+1 = s ∗ . Note that 

µ t+1 

i 

= (1 − α)µ t i + αs t −i = (1 − α)µ t i + αs ∗ −i, 

where, abusing the notation, we used s t −i to denote the degenerate probability distribution 

and 

α = 

1 

∑s −i 

η t i (s −i) + 1 . 


Therefore, by the linearity of the expectation, we have for all s i ∈ S i , 

Since s ∗ i 

g i (s i , µ t+1 

i ) = (1 − α)g i (s i , µ t i) + αg i (s i , s ∗ −i). 

maximizes both terms (in view of the fact that s ∗ is a strict Nash equilibrium), it 

follows that s ∗ i will be played at t + 1. Q.E.D. 

Note that the preceding notion of convergence only applies to pure strategies. We next 

provide an alternative notion of convergence, i.e., convergence of empirical distributions or 

beliefs. 

Definition 2 The sequence {s t } converges to σ ∈ Σ in the time-average sense if for all i 

and for all s i ∈ S i , we have 

[number of times s t i = s i for t ≤ T ] 

lim 

T →∞ 

T + 1 

i.e., µ T −i(s i ) converges to σ i (s i ) as T → ∞. 

= σ(s i ), 

The next example illustrates convergence of the fictitious play sequence in the timeaverage 

sense. 

Example 1: (Matching Pennies) Consider the fictitious play of the following matching 

pennies game: 

Consider the following sequence of play: 

H 

T 

H 1, −1 −1, 1 

T −1, 1 1, −1 

η1 t η2 t Play 

0 (0,0) (0,2) (H,H) 

1 (1,0) (1,2) (H,H) 

2 (2,0) (2,2) (H,T) 

3 (2,1) (3,2) (H,T) 

4 (2,2) (4,2) (T,T) 

5 (2,3) (4,3) (T,T) 

6 · · · · · · (T,H) 


Play continues as (T,H), (H,H), (H,H) - a deterministic cycle. The time average converges 

( 

) 

to (1/2, 1/2), (1/2, 1/2) , which is the unique Nash equilibrium. 

The next proposition extends the basic convergence property of fictitious play, stated in 

Proposition 1, to mixed strategies. 

Proposition 2 Suppose a fictitious play sequence {s t } converges to σ in the time-average 

sense. Then σ is a Nash equilibrium of G. 

Proof: Suppose s t converges to σ in the time-average sense. Assume that σ is not a Nash 

equilibrium. Then there exist some i, s i , s ′ i ∈ S i with σ i (s i ) > 0 such that 

g i (s ′ i, σ −i ) > g i (s i , σ −i ). 

Choose ɛ > 0 such that 

ɛ < 1 2 

[ 

] 

g i (s ′ i, σ −i ) − g i (s i , σ −i ) , 

and T sufficiently large that for all t ≥ T , we have 

∣ 

∣µ T ɛ 

i (s −i ) − σ −i (s −i ) ∣ < 

max s∈S g i (s) , 

which is possible since µ t i → σ −i by assumption. Then, for any t ≥ T , we have 

g i (s i , µ t i) = ∑ s −i 

g i (s i , s −i )µ t i(s −i ) 

≤ ∑ s −i 

g i (s i , s −i )σ −i (s −i ) + ɛ 

< ∑ s −i 

g i (s ′ i, s −i )σ −i (s −i ) − ɛ 

≤ ∑ s −i 

g i (s ′ i, s −i )µ t i(s −i ) = g i (s ′ i, µ t i). 

This shows that after T , s i is never played, implying that as T → ∞, µ t −i(s i ) → 0. But this 

contradicts the fact that σ i (s i ) > 0, completing the proof. □ 

It is important to realize that convergence in the time-average sense is not necessarily a 

natural convergence notion, as illustrated in the following example. 

Example 2: (Mis-coordination) Consider the fictitious play of the following game: 


A 

B 

A 1, 1 0, 0 

B 0, 0 1, 1 

Note that this game had a unique mixed Nash equilibrium 

( 

) 

(1/2, 1/2), (1/2, 1/2) . Consider 

the following sequence of play: 

η1 t η2 t Play 

0 (1/2,0) (0,1/2) (A,B) 

1 (1/2,1) (1,1/2) (B,A) 

2 (3/2,1) (1,3/2) (A,B) 

3 · · · · · · (B,A) 

Play continues as (A,B), (B,A), . . . - again a deterministic cycle. The time average 

( 

) 

converges to (1/2, 1/2), (1/2, 1/2) , which is a mixed strategy equilibrium of the game. 

But players never successfully coordinate! 

The following proposition provides some classes of games for which fictitious play converges 

in the time-average sense (see Fudenberg and Levine [2]; see also [9], [6], [7]). 

Proposition 3 Fictitious play converges in the time-average sense for the game G under 

any of the following conditions: 

(a) G is a two player zero-sum game. 

(b) G is a two player nonzero-sum game where each player has at most two strategies. 

(c) G is solvable by iterated strict dominance. 

(d) G is an identical interest game, i.e., all players have the same payoff function. 

(e) G is an (exact) potential game. 

In Section 5, we consider a continuous time variant of fictitious play and provide the 

proof of convergence for two player zero-sum games. 


4 Nonconvergence of Fictitious Play 

The next proposition shows that there exists a game for which fictitious play does not 

converge. 

Proposition 4 (Shapley [12]) In a modified version of Rock-Scissors-Paper game, fictitious 

play does not converge. 

Example 3: Shapley’s Rock-Scissors-Paper game has payoffs: 

R S P 

R 0, 0 1, 0 0, 1 

S 0, 1 0, 0 1, 0 

P 1, 0 0, 1 0, 0 

Suppose that η 0 1 = (1, 0, 0) and that η 0 2 = (0, 1, 0). Then in period 0, play is (P,R). In period 

1, player 1 expects R, and 2 expects S, so play is (P,R). Play then continues to follow (P,R) 

until player 2 switches to S. Suppose this takes k periods. Then play follows (P,S), until 

player 1 switches to to R. This will take βk periods (verify this), with β > 1. Play then 

follows (R,S), until player 2 switches to P. This takes β 2 k periods. And so on, with the key 

being that each switch takes longer than last. 

Shapley’s proof of the nonconvergence of fictitious play in the above game explicitly 

computes time spent in each stage of the sequence. Here, we provide an alternative insightful 

proof due to Monderer et al. [8]. For this we will make use of the following lemma. Define 

the time-average payoffs through time t as: 

U t i = 1 

t + 1 

Define the expected payoffs at time t as: 

t∑ 

g i (s τ i , s τ −i). 

τ=0 

Ũ t i = g i (s)i t , µ t i) = max 

s i ∈S i 

g i (s i , µ t i). 


Lemma 1 Given any ɛ > 0 and any i ∈ I, there exists some T such that for all t ≥ T , we 

have 

Ũ t i ≥ U t i − ɛ. 

Proof: Note that 

Ũi t = g i (s t i, µ t i) ≥ g i (s t−1 

i , µ t i) 

1 

= 

t + 1 g i(s t−1 

i 

1 

= 

t + 1 g i(s t−1 

i 

, s t−1 

−i ) + 

, s t−1 

−i ) + 

t 

t + 1 g i(s t−1 

i , µ t−1 

t 

t + 1 Ũ t−1 

i . 

i ) 

Expanding Ũ t−1 

i 

and iterating, we obtain 

Ũ t i ≥ 1 

t + 1 

t∑ 

τ=0 

g i (s τ i , s τ −i) − 1 

t + 1 g i(s t i, s t −i). 

Choosing T such that 

completes the proof. 

□ 

ɛ > 1 

T + 1 max 

s∈S g i(s), 

We can now use the preceding lemma to prove Shapley’s result. 

Proof of Shapley Nonconvergence: In the Rock-Scissors-Paper game, there is a unique 

Nash equilibrium with expected payoffs of 1/3 for both players. Therefore, if the fictitious 

play converged, then Ũ t i → 1/3 for i = 1, 2, implying that Ũ t 1 + Ũ t 2 → 2/3. But under 

fictitious play, realized payoffs always sum to 1, i.e., 

U t 1 + U t 2 = 1, for all t, 

thus contradicting the lemma above, and showing that fictitious play does not converge. 

Q.E.D 

For extensions of fictitious play (e.g., stochastic fictitious play, derivative action fictitious 

play) and their convergence behavior, see Fudenberg and Levine [2] and also Shamma and 

Arslan [11]. 


5 Convergence Proofs 

We now consider a continuous time version of fictitious play and provide a proof of convergence 

for zero-sum games. 

5.1 Continuous Time Fictitious Play 

Let us introduce some notation to facilitate the subsequent analysis. We denote the empirical 

distribution of player i’s play up to (but not including) time t by 

p t i(s i ) = 

∑ t−1 

τ=0 I{sτ i = s i } 

. 

t 

We use p t ∈ Σ to denote the product distribution formed by the p t i. 

In continuous time fictitious play (CTFP), the empirical distributions of the players are 

updated in the direction of a best response to their opponents’ past action: 

where 

where 

dp t i 

dt ∈ BR i(p t −i) − p t i, 

BR i (p t −i) = arg max 

σ i ∈Σ i 

g i (σ i , p t −i). 

Another variant of the CTFP is the perturbed CTFP defined by 

dp t i 

dt = C i(p t −i) − p t i, 

[ 

] 

C i (p t −i) = arg max g i (σ i , p t −i) − V i (σ i ) , (1) 

σ i ∈Σ i 

and V i : Σ i → R is a strictly convex function and satisfies a boundary condition, which 

we informally state as follows: “as σ i approaches the boundary of the probability simplex, 

‖∇V i (σ i )‖ goes to infinity.” Note that because the perturbed best-response C i is uniquely 

defined, the perturbed CTFP is described by a differential equation rather than a differential 

inclusion. 


5.2 Convergence of (perturbed) CTFP for Zero-sum Games 

In this section, we provide a proof of convergence of perturbed CTFP for two player zero-sum 

games. The proof presented here is due to Shamma and Arslan (see also Johari [5]). 

We consider a two player zero-sum game with payoff matrix M, where the payoffs are 

perturbed by the functions V i (see the previous section), i.e., the payoffs are given by 

Π 1 (σ 1 , σ 2 ) = σ ′ 1Mσ 2 − V 1 (σ 1 ), (2) 

Let {p t } be generated by the perturbed CTFP, 

Π 2 (σ 1 , σ 2 ) = −σ ′ 1Mσ 2 − V 2 (σ 2 ). (3) 

dp t i 

dt = C i(p t −i) − p t i, 

[cf. Eq. (1)]. We use a Lyapunov function approach to prove convergence. In particular, we 

consider the function 

W (t) = U 1 (p t ) + U 2 (p t ), 

where the functions U i : Σ → R are defined as 

U i (σ i , σ −i ) = max Π i (σ i, ′ σ −i ) − Π i (σ i , σ −i ), 

σ i ′ ∈Σ i 

(i.e., the function U i gives the maximum possible payoff improvement player i can achieve 

by a unilateral deviation in his own mixed strategy). Note that U i (σ) ≥ 0 for all σ ∈ Σ, and 

U i (σ) = 0 for all i implies that σ is a mixed Nash equilibrium. 

For the zero sum game with payoffs (2)-(3), the function W (t) takes the form 

We will show that 

W (t) = max Π 1 (σ 1, ′ p t 2) + max Π i (p t 1, σ 2) ′ + V 1 (p t 1) + V 2 (p t 2). 

σ 1 ′ ∈Σ 1 

σ 2 ′ ∈Σ 2 

dW (t) 

dt 

all initial conditions p 0 , we have 

We need the following lemma. 

≤ 0 with equality if and only if p t i = C i (p t −i), showing that for 

( ) 

lim p t i − C i (p t −i = 0 i = 1, 2. 

t→∞ 


Lemma 2 (Envelope Theorem) Let F : R n × R m → R be a continuously differentiable 

function. Let U ⊂ R m be an open convex subset, and u ∗ (x) be a continuously differentiable 

function such that 

Let H(x) = min u∈U F (x, u). Then, 

F (x, u ∗ (x)) = min F (x, u). 

u∈U 

∇ x H(x) = ∇ x F (x, u ∗ (x)). 

Proof: The gradient of H(x) is given by 

∇ x H(x) = ∇ x F (x, u ∗ (x)) + ∇ u F (x, u ∗ (x))∇ x u ∗ (x) 

= ∇ x F (x, u ∗ (x)), 

where we use the fact that ∇ u F (x, u ∗ (x)) = 0 since u ∗ (x) minimizes F (x, u). 

Using the preceding lemma, we have 

[ 

] 

d 

max Π 1 (σ 

dt 

1, ′ p t 2) 

σ 1 ′ ∈Σ 1 

= ∇ σ2 Π 1 (C 1 (p t 2), p t 2) ′ dpt 2 

dt 

□ 

= C 1 (p t 2) ′ M dpt 2 

dt 

= C 1 (p t 2) ′ M (C 2 (p t 1) − p t 2). 

Similarly, 

[ 

] 

d 

max Π 2 (p t 

dt 

1, σ 2) 

′ = −(C 1 (p t 2) − p t 1) ′ MC 2 (p t 1). 

σ 2 ′ ∈Σ 2 

Combining the preceding two relations, we obtain 

dW (t) 

dt 

= −C 1 (p t 2) ′ Mp t 2 + (p t 1) ′ MC 2 (p t 1) + ∇V 1 (p t 1) ′ dpt 1 

dt + ∇V 2(p t 2) ′ dpt 2 

dt . (4) 

Since C i (p t −i) is a perturbed best response, we have 

C 1 (p t 2) ′ Mp t 2 − V 1 (C 1 (p t 2)) ≥ (p t 1) ′ Mp t 2 − V 1 (p t 1), 

−(p t 1) ′ MC 2 (p t 1) − V 2 (C 2 (p t 1)) ≥ −(p t 1) ′ Mp t 2 − V 2 (p t 2), 

11-11

with equality if and only if C i (p t −i) = p t i, i = 1, 2 (the latter claim follows by the uniqueness 

of the perturbed best response). Combining these relations, we have 

−C 1 (p t 2) ′ Mp t 2 + (p t 1) ′ MC 2 (p t 1) ≤ ∑ i 

[V i (p t i) − V i (C i (p t −i))] 

≤ 

∑ i 

∇V i (p t i) ′ (C i (p t i) − p t i) 

= − ∑ i 

∇V i (p t i) ′ dpt i 

dt , 

where the second inequality follows by the convexity of V i . The preceding relation and Eq. 

(4) imply that dW dt 

≤ 0 for all t, with equality if and only if C i (p t −i) = p t i for both players, 

completing the proof. 

References 

[1] G. Brown, “Iteartive solution of games by fictitious play,” in Activity Analysis of Production 

and Allocation, T. Koopmans, editor, John Wiley and Sons, New York, New 

York, 1951. 

[2] Fudenberg D. and Levine D. K., The Theory of Learning in Games, MIT Press, 1999. 

[3] Hart S. and Mas-Colell A., “A simple adaptive procedure leading to correlated equilibrium,” 

Econometrica, vol. 68, no. 5, pp. 1127-1150, 2000. 

[4] Hart S., “Adaptive Heuristics,” Econometrica, vol. 73, no. 5, pp. 1401-1430, 2005. 

[5] Johari R., Lecture Notes, 2007. 

[6] Miyasawa K., “On the convergence of learning processes in a 2x2 nonzero sum game,” 

Research Memo 33, Princeton, 1961. 

[7] Monderer D. and Shapley L., “Fictitious play property for games with identical interests,” 

Journal of Economic Theory, vol. 68, pp. 258-265, 1996. 

[8] Monderer D., Samet D., and Sela A., “Belief affirming in learning processes,” Mimeo, 

Technion, 1994. 


[9] Robinson J., “An iterative method of solving a game,” Annals of Mathematical Statistics, 

vo. 54, pp. 296-301, 1951. 

[10] Shamma J. S. and Arslan G., “Unified convergence proofs of continuous-time fictitious 

play”, IEEE Transactions on Automatic Control, pp. 1137-1142, July 2004. 

[11] Shamma J. S. and Arslan G., “Dynamic fictitious play, dynamic gradient play, and 

distributed convergence to Nash equilibria”, IEEE Transactions on Automatic Control, 

pp. 312-327, March 2005. 

[12] Shapley L., “Some Topics in Two-Person Games” Advance in Game Theory M. 

Drescher, L.S. Shapley, and A.W. Tucker (Eds.). Princeton: Princeton University Press, 

1964.

Lecture 11: Learning in Games - Fictitious Play 1 Introduction 2 ...

Create successful ePaper yourself

Delete template?

Save as template?