Get - Institute of Computer Engineering - Technische Universität Wien

Research Report 38/2004, TU Wien, Institut für Technische Informatik February 4, 2005
Distributed Construction of Fault-Tolerant Overlay
Networks: Propose Modules
Bernd Thallner
Technische Universität Wien
Embedded Computing Systems Group E182/2
Treitlstrasse 3, A-1040 Vienna, Austria
Email: thallner@ecs.tuwien.ac.at
Tel: +43-1-58801-18203, FAX: +43-1-58801-18297
February 4, 2005

Abstract
In [TS04] we present a distributed algorithm for constructing a sparse overlay network that
has fixed node-degree and facilitates efficient fault-tolerant multi-hop communication in
large-scale distributed systems. This paper deals with different Propose Modules implementations
and presents simulation results.
3

Chapter 1
Propose Modules
This chapter discusses different propose modules, their implementations and their properties.
A propose module is an independent and highly network-specific part of our topology
construction. It can be adapted to various characteristics of the underlying wireless network,
e.g. memory consumption, number of messages versus message size, size of neighborhood
or the reaction time to network changes. Furthermore the propose modules define the quality
(efficiency) of the generated overlay network and the complexity of construction.
Typically, all propose module implementations use simple search strategies to explore
the search space. The implementations are sometimes complicated since the search is done
entirely message driven, in a distributed way and the required information (edge weights) is
distributed among different nodes. In the following, we present some propose modules with
different properties and analyze their complexity.
Recall that from the description and analysis of our algorithm in [TS04], it is apparent
that the propose module is crucial for liveness and performance albeit not for convergence,
correctness and fault-tolerance. In this chapter we discuss some propose modules and their
worst case performance. Propose modules can be classified according to (1) Search space
exploration:
• Perfect: We call a propose module perfect if it eventually infinitly often generates
a proposal corresponding to a group in the final minimum admissible overlay graph
arbitrary often. Hence in some scenarios it potentially searches the whole search space.
If there is a perfect propose module on each node and leader we can guarante that
each group of the minimal admissible overlay graph is proposed until all groups are
constructed.
• Non-perfect: Propose modules that have a restricted search space (e.g. to achieve low
message complexity) are called non-perfect. Note that the construction of the minimal
admissible overlay graph cannot be guaranteed if non-perfect propose modules are
employed.
(2) Distribution
• Global: A propose module is called global if it uses any kind of central coordinator
so that group proposals are proposed in a fully serialized order (i.e., with increasing
5

6 CHAPTER 1. PROPOSE MODULES
group weight) or such that former built groups cannot be destroyed through later group
constructions.
• Local: If the group construction is fully distributed, such that suboptimal groups may
be built that have to be destroyed later again, the propose module is called local.
• Locally agreed: A propose module is called locally agreed if the construction is as
concurrent as possible in order to ensure that no group is ever destroyed by later group
constructions.
Note that local, global, and locally agreed propose modules can be either perfect or nonperfect.
1.1 Introduction
We assume that each node knows a subset of the edgeset Λ ID ⊂ Λ where its ID is an
edge endpoint: (x, y, w) ∈ Λ ID ⇔ x =ID ∧ y ∈ Π. The edges are undirected, therefore
if (x, y, w) ∈ Λ x → (y, x, w) ∈ Λ x . Note that the edgeset is time-variant. Furthermore,
we assume that each node knows the minimal edge weight: w min = min w (x, y, w) ∈ Λ. A
proposal is released to the topology construction algorithm (received by the topology construction)
if the proposal is stored to the variable P [gid]. Recall, that a proposal P consists
of the member set P.Members, the group connection set P.Connections, the terminal node
set P.T erminals and a group weight P.W eight. The propose module implementations require
a unique leader of a group, which is the terminal node with the smallest ID, such that
the function leader of(gid) can be implemented as min(gid).
In order to keep our code clear we intentionally left out all code for the handling of node
crashes and recovery in our propose module implementations. In particular since the propose
modules are not critical for correctness and convergence of the topology, a simple solution
for handling node crashes could be to restart the proposal generation upon failure detection.
1.2 Sub-functions
In this section we introduce some basic sub-functions for all remaining propose module
implementation. Our pseudo code notation uses sets (keyword: set) and arrays (keyword:
array). An array a consists of elements a 1 , a 2 ,. . . , a i−1 , a i , a i+1 ,. . . , a n , where a[i] refers to
the i-th element a i in a if 0 < i ≤ n. For adding elements at the end of the array a we use
the notation a ← a ∪ a n+1 , and checking whether an element a i is in a is donated a i ∈ a. If
the array index i is not an integer, then array is used as an associative array where i as the
corresponding key.
Our propose module algorithms require a communication subsystem without message loss
that provides the primitives send(), if received() and wait for receive.
The function unique id() returns an id, which is unique on the calling node (e.g. a global
counter).
The Figures 1.1 and 1.2 depict the pseudo code implementation of function calculate
proposal(). This function calculates for a given set of members fix and an edgeset edges

1.2. SUB-FUNCTIONS 7
the complete group information consisting of the group connections P.Connections, the terminal
node set P.T erminals, and the group weight P.W eight. If fetch missing edges is
enabled, required edges still missing in edges are fetched from the corresponding nodes via
GET EDGES messages, whereas if disabled, missing edges are assumed to have minimum
edge weight k (line 7-14). The function checks all possible combinations for connecting
the group members via terminal nodes and returns the cheapest alternative. The functions
source() and destination() assign the terminal nodes to the connection (line 46-55 and 58-
67). If all connections are created (line 26), the terminal set is calculated (line 31-38). If
the calculated weight is smaller than min weight it is adjusted according to Definition 1
in [TS04] (line 18). The complexity of this function depends only on k.
0 global P out ←⊥
1
2 function calculate proposal ( array fix, min weight,set edges, fetch missing edges)
3
4 P out .Connections←∅, P out .T erminals←∅, P out .W eight←∞
5 set t = {∃x : x ′ ∈ fix : x ∈ x ′ }
6
7 for x ∈ t /∗ complete edge set ∗/
8 if fetch missing edges = true ∧ t \ {y : (x, y, w) ∈ edges} ≠ ∅
9 tag←unique id()
10 send (GET EDGES, t \ {y : (x, y, w) ∈ edges}, tag) to x
11 wait for reply (EDGES, edges in , tag in ) from x with tag in = tag
̸
12 edges←edges ∪ edges in
13 else
14 edges←edges ∪ {(x, y, w min ) : y ∈ t∧ ∃w : (x, y, w) ∈ edges}
15
16 source(1, 2, ∅, 0, fix, edges)
17 if P out .W eight < min weight
18 P out .W eight←min weight + ɛ
19 return P out
20
21 if received request (GET EDGES, nodes, tag) from j
22 reply (EDGES, {(x, y, w) ∈ Λ ID : x =ID, y ∈ nodes}, tag) to j
Figure 1.1. First part of the pseudo code for function calculate proposal().
Figure 1.3 depicts the function potential member set(). This function calculates a set pms
consisting of nodes and groups of ps, which allow to build groups with the members in fix,
and with less weight than bound. The value edges is the set of edges known to the function.
First of all, if filter is true, nodes are removed with id lower than the id of the smallest
node in fix. This is done to avoid multiple searches for the same proposal by different
propose modules (line 72-75). Then, all nodes and groups are added to pms which allows to
build groups with lower weights below bound (line 80-84). After that, nodes and groups are
removed from pms set for which no group of k members from pms exists (line 86-89). The
worst case complexity for this function is O(|ps| k ).
Figures 1.4 shows the implementation of the function get group ids(). The function takes

8 CHAPTER 1. PROPOSE MODULES
23 function source(a, b,set connections, weight,array fix,set edges) /∗ find connection source ∗/
24 c with c.from, c.to
25
26 if a = |fix| − 1 ∧ b = |fix| /∗ all internal connections are built ∗/
27 if P out .W eight > weight
28 P out .W eight←weight
29 P out .Connections←connections
30 P out .T erminals←⊥
31 for i ∈ fix /∗ calculate terminal nodes ∗/
32 if i ∈ Π
33 P out .T erminals←P out .T erminals ∪ i
34 else
35 set terminal←i
36 for j ∈ connections
37 terminal←terminal \ (j.from ∪ j.to)
38 P out .T erminals←P out .T erminals ∪ terminal
39 return
40
41 if b = |fix| /∗ next source/destination pair ∗/
42 a←a + 1
43 b←a
44 b←b + 1
45
46 if fix[a] ∈ Π
47 c.from←fix[a]
48 destination (a, b, connections, weight, fix, edges, c)
49 else
50 set available terminals←fix[a] /∗ probe all source terminals ∗/
51 for i ∈ connections
52 available terminals←available terminals \ (i.from ∪ i.to)
53 for i ∈ available terminals
54 c.from←i
55 destination (a, b, connections, weight, fix, edges, c)
56
57 function destination (a, b,set connections, weight,array fix,set edges, c) /∗ find destination ∗/
58 if fix[b] ∈ Π
59 c.to←fix[b]
60 source(a, b, connections ∪ c, weight + w : (c.from, c.to, w) ∈ edges, fix, edges)
61 else /∗ probe all destination terminals ∗/
62 set available terminals←fix[b]
63 for i ∈ connections
64 available terminals←available terminals \ (i.from ∪ i.to)
65 for i ∈ available terminals
66 c.to←i
67 source(a, b, connections ∪ c, weight + w : (c.from, c.to, w) ∈ edges, fix, edges)
Figure 1.2. Second part of function calculate proposal().
a set of nodes ps and returns a set which contains all group ids of higher level groups of the
nodes in ps. The maximum number of nodes in the set ps is n and the maximum number of
groups in G ′ is ⌊ n−1
k−1⌋
= O(n) by Theorem 4 in [TS04] , therefore the function has a worst

1.3. LOCAL PERFECT PROPOSE MODULE 9
68 function potential member set ( array fix,set ps,set edges, bound, filter)
69 set pms←⊥
70
71 if filter
72 if fix ∩ Π ≠ ∅ /∗ avoid duplicate solutions, from multiple nodes in the same proposal ∗/
73 for i ∈ ps
74 if (i ∈ Π) ∧ (i < min(fix ∩ Π))
75 ps←ps \ i
76
77 ps←ps \ {x ∈ ps : x ′ ∈ x : y ∈ fix : y ′ ∈ y : x ′ = y ′ }
78 /∗ remove members in ps already member or submember in fix ∗/
79
80 for i ∈ ps /∗ add potential candidates to pms for a group with | fix |+1 members
81 and smaller group weight than bound ∗/
82 P ′ ←calculate proposal(fix ∪ {i}, 0, edges, false)
83 if P ′ .W eight ≤ bound
84 pms←pms ∪ {i}
85
86 if k − |fix| > 1 /∗ remove candidates from pms for which no group with k members exist ∗/
87 for i ∈ pms
88 if |potential member set(fix ∪ {i}, pms \ {i}, edges, bound, false)| < k − |fix| − 1
89 pms = pms \ {i}
90
91 return pms
Figure 1.3. Pseudo code for function potential member set().
case message complexity of O(n 2 ) and the worst case time complexity of 1 + ⌊ n−1
k−1⌋
+ 1,
hence O(n).
1.3 Local Perfect Propose Module
In this section we present a local perfect propose module. We start with the following
definitions:
Definition 1. Let (G ′ ) e be an incomplete built overlay graph at time e. A local propose
module on a node or on the leader of a group gid at time e generates a proposal consisting
of gid and members in (G ′ ) e .
This property reduces the search space since only proposals which include the initial
node/group gid are built.
Definition 2. A propose module is perfect, if it eventually generates infinitly often a proposal
which corresponds to a group in the minimal admissible overlay graph.
Note that, in some scenarios each perfect propose module eventually has to search the
whole search space.

10 CHAPTER 1. PROPOSE MODULES
92 function get group ids (ps)
93 array tags
94
95 for x ∈ ps /∗ fetch all group ids of nodes in ps ∗/
96 tags[x]←unique id()
97 send (GET GROUP IDS, {x},ID, x, x, tags[x]) to x
98
99 while ps ≠ ∅
100 wait for receive (GROUP IDS, group ids, x, tag) with x ∈ ps ∧ tag = tags[x]
101 pms←pms ∪ group ids
102 ps←ps \ {x}
103
104 return pms
105
106 if received (GET GROUP IDS, group ids, source, x, gid, tag)
107
108 if Group[gid].P arentID ≠⊥
109 send (GET GROUP IDS, group ids ∪ {Group[gid].P arentID}, source, x, Group[gid].P arentID, tag) \\
110 to leader of (Group[gid].P arentID)
111 else
112 send (GROUP IDS, group ids, x, tag) to leader of(source)
Figure 1.4. Pseudo code for get group ids() function.
Our implementation of a local perfect propose module is based upon the following idea:
To make a proposal, the propose module of a node/group leader gid adds itself to a list
of proposed members and calculates its potential member set pms gid . This set contains all
nodes/groups that are candidates for forming a group with less weight than its current parent
group. Then, gid sends a SEARCH message to all nodes/leaders in the potential member
set. Every node/leader j receiving such a SEARCH message does the same as gid (except
that it bases the calculation of its pms j upon pms gid instead of (G ′ ) e gid ). This process of
forwarding is repeated k −1 times or the potential member set becomes empty. In the former
case, a RESULT message containing the proposed members set is sent back to the originator
gid. Otherwise, an extinction notification for this particular forwarding “thread” is sent. The
proposal comprising the members of the RESULT message with the lowest group weight is
eventually proposed by gid.
The pseudo code of the perfect local propose module is depicted in Figure 1.5. The global
value reply set stores for each search step (fix) the nodes/groups to which a SEARCH message
has been sent, and in best proposal the currently best received proposal for this search
step is stored (line 113).
The initiator initiates pms with its potential member set pms and their corresponding
parent group ids and starts the search with a SEARCH messages to itself. If a node or
leader receives a SEARCH messages, it sets the minimal weight according to Definition 1
in [TS04] (line 128) and reduces the bound if its current group weight is lower than bound
(line 130). If the number of group members (|fix|) is k, this search branch is completed
and a RESULT message is sent back to the predecessor (line 134-138). If the proposal is not

1.3. LOCAL PERFECT PROPOSE MODULE 11
113 set reply set,set best proposal
114
115 procedure local perfect proposal (gid)
116 bound←∞
117
118 if Group[gid].P arentID ≠⊥
119 bound←Group[Group[gid].P arentID].W eight
120 set pms←potential member set({gid}, Π, Λ ID , bound, true)
121 pms←get group ids(pms);
122
123 send (SEARCH, {gid}, pms, bound, 0, ∅, gid) to leader of(gid)
124
125 if received (SEARCH,array fix,set ps, bound, min weight,set relevant edges, gid)
126 reply set[fix]←∅, best proposal[fix]←⊥
127
128 if min weight < Group[gid].W eight /∗ store minimal possible group weight ∗/
129 min weight←Group[gid].W eight
130 if Group[gid].P arentID ≠⊥ ∧ Group[Group[gid].P arentID].W eight < bound
131 bound←Group[Group[gid].P arentID].W eight /∗ store maximum possible group weight ∗/
132
133 if min weight < bound
134 if |fix| = k /∗ proposal is complete ∗/
135 P ′ ←calculate proposal(fix, min weight, relevant edges ∪ Λ ID , true)
136 if P ′ .W eight > bound
137 P ′ ←⊥
138 send (RESULT, P ′ , gid, ⋃ |fix|−1
x=1
fix[x]) to leader of(fix[|fix| − 1])
139 return
140 else /∗ calculate next member ∗/
141 for x ∈ ps
142 set pms←potential member set(fix ∪ {x}, ps \ {x}, relevant edges ∪ Λ ID , bound, true)
143 if |pms| + |fix ∪ {x}| ≥ k
144 send (SEARCH, fix ∪ {x}, pms, bound, relevant edges ∪ \\
145 {(x, y, w) ∈ Λ ID : x ∈ID ∧ y ∈ {z : z ′ ∈ fix : z ∈ z ′ }}, x) to leader (x)
146 reply set[fix]←reply set[fix] ∪ {x}
147 if reply set[fix] = ∅ ∧ |fix| > 1
148 send (RESULT, ⊥, gid, ⋃ |fix|−1
x=1
fix[x]) to leader of(fix[|fix| − 1])
149
150 if received (RESULT, P ′ , j,array fix)
151 if P ′ ≠⊥ ∧ (best proposal(fix) =⊥ ∨ best proposal(fix).W eight > P ′ .W eight)
/∗ store current best proposal ∗/
152 best proposal(fix)←P ′
153
154 reply set[fix]←reply set[fix] \ {j}
155 if reply set[fix] = ∅ /∗ if all RESULT messages received, send best proposal ∗/
156 if |fix| = 1
157 P [gid]←best proposal(fix) /∗ release proposal ∗/
158 else /∗ send proposal to predecessor ∗/
159 send (RESULT, best proposal(fix), fix[|fix|], ⋃ |fix|−1
x=1
fix[x]) to leader of(fix[|fix| − 1])
Figure 1.5. Pseudo code for perfect local propose module.

12 CHAPTER 1. PROPOSE MODULES
completed, node/leader gid calculates its potential member set pms based on the received ps
set and sends a SEARCH message to all nodes/leaders in the new pms set. Thereby gid adds
itself to fix and includes some edges to the relevant edge set, to improve future search (line
140 to 145). Each node/group to which a SEARCH messages is sent is added to the reply set
(line 146) in order to determine when all RESULT messages are received. If no SEARCH
message is sent an empty RESULT message is immediately sent back to the predecessor (line
147-148).
If a node or leader receives a RESULT message, it checks whether this result is better than
the currently best proposal and if so, stores it in best result (line 152). When all results are
received either the best result is sent to the predecessor (line 159) or it is released as new
proposal (line 157).
Lemma 1. The worst case message complexity of the local perfect propose module
generating a single proposal is O(n k−1 ).
for
Proof. Let (G ′ ) e gid be the incompletely built overlay graph at time e as perceived by node
gid. The worst case message complexity of the initialization (potential member set() and
get group ids()) is O(n 2 ). Since the maximum number of nodes is n and the maximum
number of groups is ⌊ n−1
k−1⌋
, the pms set in line 123 and therefore the maximum number of
recipients of a SEARCH message is n + ⌊ n−1
k−1⌋
. In the worst case the SEARCH and RESULT
messages are sent to every node/leader so we obtain a message complexity of at most O(n 2 )+
2 · ∑k−1
i=0 (n + ⌊ n−1
k−1⌋
) i = O(n k−1 ).
Lemma 2. The worst case time complexity of the local perfect propose module for generating
a single proposal is O(n).
Proof. Analogous to the proof of Lemma 1 the worst case time complexity for the initialization
is O(n). In the search phase, in worst case, each SEARCH and RESULT message is
forwarded k − 1 times, resulting in a time complexity of 2 · (k − 1) = O(1). Hence the
claimed time complexity of O(n).
Theorem 1. There are at most ∏⌊ n−k
k−1
i=0
for a given graph G.
⌋
( n−i·(k−1)
)
k different admissible overlay graphs G
′
Proof. For the first group there are ( n
k)
different combinations. A group removes k members
and adds a new possibility, therefore for the second group there are ( )
n−(k−1)
k combinations,
the third has ( )
n−2·(k−1)
k combinations and so on. Multiplying all possible combinations, we
have a maximum of ∏⌊ ⌋
n−k (
k−1 n−i·(k−1)
)
i=0 k different admissible overlay graphs G ′ for a given
graph G.
Theorem 2. The worst case message and time complexity for generating a minimal admissible
overlay network G ′ with the regular topology construction algorithm using local perfect
propose modules is O(n k·n ).

1.4. LOCAL NON-PERFECT PROPOSE MODULE 13
Proof. Assume that the proposal from a local perfect propose module which correspond to a
global minimal group is delayed and many other proposals are released. Furthermore assume
that the edge weights are assigned so that there exists an order for building admissible graphs
so that a bulk of the possible admissible graphs from Theorem 1 can be constructed if the
proposals are generated according to this order. So until the first globally perfect and therefore
final group is built not more than ∏⌊ ⌋
n−k (
k−1 n−i·(k−1)
) ∏
i=0 k = O( n−k
( n
) (
i=0 k ) = O( n
) n
) =
k
O((n k ) n ) = O(n k·n ) admissible graphs can be constructed. The complexity for building
a non minimal admissible graph ⌊ n−1
k−1⌋
= O(n), the complexity for generating a proposal
(Lemma 1 or 2) and the complexity for building the missing groups of the minimal admissible
graph ⌊ n−1
k−1⌋
= O(n) can be neglected.
1.4 Local Non-Perfect Propose Module
This section presents a deterministic implementation of a local non-perfect propose module.
We use a depth first search for finding proposals. Rather than to all nodes/leaders in
the potential member set, as in the local perfect propose module, a node/leader sends the
SEARCH message only to the node/group corresponding to its lowest-weight edge.
The pseudo code for the local non-perfect propose module is given in Figure 1.6. The
node/leader that initiates the search, sends a SEARCH message to itself. The array fix contains
the for each search step i an array of the current memebers fix[i], hence fix is a
array of arrays. Each node/leader, which receives a SEARCH message, first of all calculates
the weight of the group with the current members (fix, line 167) and checks if the current
bound (bound2) is not exceeded (line 173).
If the number of members (fix) is k, a RESULT message is sent back to the initial
node/leader. If the number of members is lower, the potential member set pms—based on
the last received potential set ps—is calculated. If pms is large enough to build a complete
group, a SEARCH message is sent to the node corresponding to the lowest-weight edge (line
177-185).
If the bound is exceeded or the pms was too small, the SEARCH message is forwarded to
either the parent group (line 189) if it exists, or to the previous entry in fix (line 195). Each
time a search message is sent the excluded set is extended by the current node/group (gid),
so that each node/group is tried only once.
Lemma 3. The worst case message complexity of our local non-perfect propose module for
generating a single proposal is O(n).
Proof. Let (G ′ ) e gid be the incompletely built overlay graph at time e as viewed by node gid.
Due to DFS search, each of the at most n + ⌊ n−1
k−1⌋
nodes/leaders is asked only once for the
participation in the member set of the new proposal and only generates a single SEARCH
or RESULT message. Therefore the worst case message complexity for generating a single
proposal is hence at most 2 · (n + ⌊ n−1
k−1⌋
) = O(n).
Lemma 4. The worst case time complexity of the local non-perfect propose module for
generating a single proposal is O(n).

14 CHAPTER 1. PROPOSE MODULES
160 procedure local non-perfect proposal(gid)
161 send (SEARCH, ∅, {Π}, {gid}, {∞}, {0}, ∅, gid) to leader of(gid)
162
163 if received (SEARCH,array fix,array ps,array excluded, \\
164 array bound,array min weight,set relevant edges, gid)
165
166 P ′ ←calculate proposal(fix[|fix|] ∪ {gid}, max(min weight ∪ {Group[gid].W eight}), \\
167 relevant edges ∪ Λ ID , true)
168
169 bound2←bound[|fix|]
170 if Group[gid].P arendID ≠⊥
171 bound2←min(bound2, Group[Group[gid].P arentID].W eight)
172
173 if P ′ .W eight < bound2
174 if |P ′ .Members| = k /∗ send back final result ∗/
175 send (RESULT, P ′ ) to leader of(fix[|fix|][1])
176 exit
177 else if |P ′ .Members| < k /∗ calculate next search step ∗/
178 set pms←potential member set(fix ∪ {gid}, ps[|ps|] \ (excluded ∪ {gid}), \\
179 relevant edges ∪ Λ ID , bound2)
180 if |fix ∪ {gid} ∪ pms| ≥ k
181 set j←y : min w (x, y, w) ∈ Λ ID : x ∈ID∧y ∈ {z : ∃z ′ ∈ pms : z ∈ z ′ }
182 send (SEARCH, fix ∪ {fix[|fix|] ∪ {gid}}, ps ∪ {pms}, excluded ∪ {gid}, bound∪ \\
183 {bound2}, min weight ∪ {Group[gid].W eight}, relevant edges∪ \\
184 {(x, y, w) ∈ Λ ID : x =ID∧y ∈ {z : ∃z ′ ∈ fix : z ∈ z ′ }}, j) to leader of(j)
185 exit
186
187 if Group[gid].P arentID ≠⊥ /∗ forward search to parent group ∗/
188 send (SEARCH, fix, ps, excluded ∪ {gid}, bound, min weight, relevant edges, \\
189 Group[gid].P arentID) to leader of(Group[gid].P arentID)
190 else /∗ send back search ∗/
191 if |fix| > 1
192 fix2←fix[|fix|]
193 send (SEARCH, ⋃ |fix|−1
x=1
fix[x], ⋃ |ps|−1
x=1
ps[x], excluded ∪ {gid}, \\
⋃ |bound|−1
194
x=1
bound[x], ⋃ |min weight|−1
x=1
min weight[x], relevant edges, fix2[|fix2|]) \\
195 to leader of (fix2[|fix2|])
196
197 if received (RESULT, P ′ )
198 P [gid]←P ′ Figure 1.6. Pseudo code for non-perfect local propose module.
Proof. Since by Lemma 3 in the worst case only O(n) messages are sent, the worst case
time complexity cannot be larger than O(n).
Theorem 3. The worst case message and time complexity for generating an admissible overlay
network G ′ with the regular topology construction algorithm using local non-perfect
propose modules is O(n k·n ).
Proof. The proof is analogous to the proof of Theorem 2.

1.5. GLOBAL PROPOSE MODULES 15
A major disadvantage of the purely local propose modules is the unsatisfying worst case
complexity for constructing the topology, caused by the fully distributed way of construction.
In the next sections we introduce propose modules that avoid this problem.
1.5 Global Propose Modules
Definition 3. Let (G ′ ) e be an incompletely built overlay graph at time e, then a global propose
module at time e returns the best proposal of all local propose modules and consisting
of members in (G ′ ) e .
A single node (in our case the one with the lowest id) is used to collect all proposals from
the local propose modules. Only the minimal proposal is returned by the global propose
module.
The implementation of the global propose module in Figure 1.7 is straight forward and
simply returns the minimum proposal of the received local perfect or non-perfect propose
modules.
199 define local proposal (gid) ← local perfect proposal (gid) or local non−perfect proposal(gid)
200
201 function global proposal (gid)
202 set ps←Π, array tags, P ′ ←⊥
203
204 P ′ .W eight←∞
205 if gid = min(Π)
206 ps←get group ids(ps);
207
208 for x ∈ ps /∗ request all local proposals from nodes/leaders in ps ∗/
209 tags[x]←unique id()
210 send (REQUEST, gid, x, tags[x]) to leader of(x)
211
212 while ps ≠ ∅
213 wait for receive (REPLY, P ∗ , x, tag) with x ∈ ps ∧ tag = tags[x]
214 if P ∗ .W eight < P ′ .W eight
215 P ′ ←P ∗
216 ps←ps \ {x}
217
218 return P ′
219
220
221 if received (REQUEST, source, gid, tag) /∗ return local proposal ∗/
222
223 P [gid]←⊥
224 local proposal (gid)
225 wait for P [gid] ≠⊥
226 send (REPLY, P [gid], gid, tag) to leader of(source)
Figure 1.7. Pseudo code for the global propose module.
The node with the lowest id initiates ps with the nodeset and all group ids and then

16 CHAPTER 1. PROPOSE MODULES
sends a REQUEST message to all nodes and leaders (line 210). If a node/leader receives
a REQUEST message it returns a REPLY message with its local proposal (either local
perfect proposal(gid) or local non perfect proposal(gid)) to the originator (line 221-
226). When all local group proposals are received (REPLY messages), the minimal one is
returned.
Lemma 5. The worst case message complexity of our global propose module for generating
a single non-perfect resp. perfect global proposal is O(n 2 ) resp. O(n k ).
Proof. In the worst case the initiation required O(n 2 ) messages from the get group ids()
function. Each of the at most n+ ⌊ n−1
k−1⌋
= O(n) nodes/leaders generates a single non-perfect
resp. perfect proposal, using at most O(n) resp. O(n k−1 ) messages each. Additionally the
central coordinator node causes O(2 · (n + ⌊ n−1
k−1⌋
)) = O(n) REQUEST/REPLY messages.
Hence the claimed overall message complexity follows: O(n 2 + n + n · n) = O(n 2 ) resp.
O(n 2 + n + n · (n k−1 ) = O(n k ).
Lemma 6. The worst case time complexity of the global propose module for generating a
single non-perfect resp. perfect global proposal is O(n).
Proof. The GET GROUP IDS() function as well as the REQUEST/REPLY phase cause a worst
case time complexity of O(n). Together with the time complexity of the local propose modules
O(n) by Lemma 4 and 2 the worst case time complexity of O(3·n) = O(n) follows.
In sharp contrast to local propose modules, we obtain only polynomial message complexity
for constructing the topology here:
Theorem 4. The worst case global message complexity for constructing the non-minimal
resp. minimal admissible overlay network G ′ with the regular topology construction algorithm
and the global propose module using non-perfect respectively perfect local propose
module is O(n 3 ) respectively O(n k+1 ).
Proof. Obviously, the global propose module causes the topology tree to be constructed in
the order of the minimum-weight groups. Hence, no existing group is ever destroyed later
on, recall Definition 1 in [TS04] and Theorem 6 in [TS04] . By Theorem 4 in [TS04] , there
are ⌊ n−1
k−1⌋
= O(n) groups to be built, and the worst case complexity for a global proposal is
O(n 2 ) resp. O(n k ) by Lemma 5.
Theorem 5. The worst case construction time for the non-minimal resp. minimal admissible
overlay network G ′ with the regular topology construction algorithm and the global propose
module using non-perfect resp. perfect local propose module is O(n 2 ).
Proof. By Theorem 4 in [TS04] , there are ⌊ n−1
k−1⌋
= O(n) groups to be built, and the worst
time complexity for a global proposal is O(n) by Lemma 6. Hence the claimed time complexity
follows.
The obvious drawback of the global propose module is that the topology is constructed in
a fully serialized way. Independent parts of the overlay network cannot be constructed in parallel.
In the next section we introduce a locally agreed propose module, which circumvents
this problem.

1.6. LOCALLY AGREED PROPOSE MODULE 17
1.6 Locally Agreed Propose Module
The locally agreed propose module permits concurrent group construction of parts where
this is feasible without reconstructing groups and serializes the topology construction in parts
where independent construction is not possible.
Definition 4. Let (G ′ ) e be an incompletely built overlay graph at time e, then a group corresponding
to a proposal generated by a locally agreed propose module is not destroyed
through later group constructions if the incompletely built overlay graph (G ′ ) e and the communication
graph G remain stable.
The locally agreed propose module is based upon two sets: L and L ′ . Initially each
node/leader gid generates its own local proposal P ∗ [gid] and the corresponding L[gid] and
L ′ [gid] sets. In L[gid] all nodes/groups are stored, which allow building a group with a
weight below P ∗ [gid].W eight, therefore all nodes/leaders in this set have to be asked if they
have a proposal with less weight than P ∗ [gid].W eight. Set L ′ [gid] contains all nodes/leader
from L that have confirmed that P ∗ has a lower weight than their current local proposal.
Each node/leader gid forwards the PROPOSAL message, consisting of P ∗ [gid], L[gid] and
L ′ [gid], to any node/leader in L[gid] \ L ′ [gid]. If a PROPOSAL message with a lower weight
proposal is received, the variables P ∗ , L and L ′ are taken over and L is extended with all
nodes/groups that have to be asked additionally. If L[gid] = L ′ [gid], all potential candidates
for a group with lower weight than P ∗ confirmed that this P ∗ [gid] is the minimal one, thus
the proposal is released.
Figure 1.8 depicts the pseudo code for the locally agreed propose module. Each
node/leader periodically calls the function locally agreed proposal() and initiates the search
for a new proposal with an empty PROPOSAL message to itself (line 231).
If a node or leader receives a PROPOSAL message and P ∗ [gid] is empty, it first generates
its local perfect or non-perfect proposal P ∗ [gid] and the corresponding set L and L ′ (line
235-238). If the received proposal Pin ∗ is equal to the local proposal, the sets of L and L′
are merged (line 241-244). Whereas, if Pin ∗ has lower weight than the local proposal, it is
taken over and set L is extended with nodes/leaders that have to be asked additionally (line
244-249) and set L ′ with nodes/groups that have been asked already.
If, after that, L[gid] is equal to L ′ [gid] all potential nodes/groups for a group with weight
below P ∗ [gid] confirmed that P ∗ [gid] is minimal. The proposal P ∗ [gid] can be released and
all nodes/groups in L ′ [gid] have to set P ∗ to ⊥ and start the search for the next proposal
(initiated by an empty PROPOSAL message, line 251-261 and 267-269). If L[gid] ≠ L ′ [gid],
the PROPOSAL message is sent to any node/leader in L[gid] which is not in L ′ [gid] (line
263-265). If the received proposal has a higher weight nothing is done.
Lemma 7. The worst case message complexity of the locally agreed propose module for
generating a single non-perfect resp. perfect proposal is O(n 2 ) resp. O(n k ).
Proof. Let (G ′ ) e be the incompletely built overlay graph at time e. Each of the at most n +
⌊ n−1
k−1⌋
= O(n) nodes/leaders generate a single local non-perfect resp. perfect proposal, using
at most O(n) resp. O(n k−1 ) messages each. In the worst case L for the lowest local proposal

18 CHAPTER 1. PROPOSE MODULES
227 define local proposal (gid) ← local perfect proposal (gid) or local non−perfect proposal(gid)
228 P ∗ [gid]←⊥, set L[gid]←⊥, set L ′ [gid]←⊥
229
230 procedure locally agreed proposal (gid) /∗ periodically initiate search for new proposal ∗/
231 send (PROPOSAL, ⊥, ∅, ∅, gid) to leader of(gid)
232
233 if receive (PROPOSAL, Pin ∗ in, set L ′ in , gid)
234
235 if P ∗ [gid] =⊥ /∗ generate local proposal for gid ∗/
236 P ∗ [gid]←local proposal(gid)
237 L[gid]←potential member set({gid}, Π, Λ ID , P ∗ [gid].W eight, false)∪{gid}
238 L ′ [gid]←{gid}
239
240 if Pin ∗ =⊥ ∨ P in ∗ .W eight ≤ P ∗ [gid].W eight
241 if Pin ∗ = P ∗ [gid] /∗ merge proposals ∗/
242 L[gid]←L[gid] ∪ L in
243 L ′ [gid]←L ′ [gid] ∪ L ′ in
244 else if Pin ∗ ≠⊥ /∗ adopt proposal ∗/
245 P ∗ [gid]←Pin
∗
246 L[gid]←L in ∪ potential member set({gid}, Π, Λ ID , P ∗ [gid].W eight, false)
247 if Group[gid].P arentID ≠⊥
248 L[gid]←L[gid] ∪ {Group[gid].P arentID}
249 L ′ [gid]←L ′ in ∪ {gid}
250
251 if L[gid] = L ′ [gid] /∗ release proposal, all potential candidates confirmed minimality ∗/
252 array tags
253 for x ∈ L[gid]
254 tags[x]←unique id()
255 send (RESET, gid, x, tags[x]) to leader of(x)
256 while L[gid] ≠ ∅
257 wait for receive (RESET OK, x, tag) with x ∈ L[gid] ∧ tag = tags[x]
258 L[gid]←L[gid] \ {x}
259 P [gid]←P ∗ [gid] /∗ release proposal ∗/
260 for x ∈ L ′ [gid] /∗ restart search for next proposal ∗/
261 send (PROPOSAL, ⊥, ∅, ∅, x) to leader of(x)
262
263 else /∗ send to next node/leader in L and not in L ′ ∗/
264 x← arbitrary element of (L[gid] \ L ′ [gid])
265 send (PROPOSAL, P ∗ , L[gid], L ′ [gid], x) to leader of(x)
266
267 if received (RESET, source, gid, tag) /∗ reset local P ∗ [gid], L[gid] and L ′ [gid] set ∗/
268 P ∗ [gid]←⊥, L[gid]←⊥, L ′ [gid]←⊥
269 send (RESET OK, gid, tag) to leader of(source)
Figure 1.8. Pseudo code for locally agreed proposal module.
can include all nodes and groups so that the PROPOSAL message is forwarded n + ⌊ n−1
k−1⌋
− 1
times. If the second lowest proposal is generated and forwarded to all nodes and groups
except the one with the minimal local proposal before the lowest, it can cause n + ⌊ n−1
k−1⌋
− 2
PROPOSAL messages, and so on. Summing up to O(n 2 ) PROPOSAL messages in the worst
case. Obviously, the worst case number of messages for the reset and restart phase is 3·O(n).

1.6. LOCALLY AGREED PROPOSE MODULE 19
Hence the worst case message complexity for a single locally agreed non-perfect proposal is
O(n) · O(n) + O(n 2 ) + 3 · O(n) = O(n 2 ) and O(n) · O(n k−1 ) + O(n 2 ) + 3 · O(n) = O(n k )
for a single locally agreed perfect proposal.
Lemma 8. The worst case time complexity of the locally agreed propose module for generating
a single proposal is O(n 2 ).
Proof. The set L can include at most n + ⌊ n−1
k−1⌋
= O(n) nodes and groups. Each
node/leader in L generates a local non-perfect or perfect proposal with time complexity
O(n) by Lemma 2 and 4 and forwards a PROPOSAL message to O(n) nodes/leaders in L.
Hence the worst case time complexity of O(n 2 ) follows. The worst case time complexity for
the reset and restart phase is O(3 · n) and can be neglected.
Theorem 6. The worst case global message complexity for constructing the non-minimal
resp. minimal admissible overlay network G ′ with the regular topology construction algorithm
and the locally agreed propose module using non-perfect resp. perfect local propose
module is O(n 3 ) resp. O(n k+1 ).
Proof. Proof analogous to Theorem 4.
Theorem 7. The worst case construction time of the non-minimal resp. minimal admissible
overlay network G ′ with the regular topology construction algorithm resp. the locally agreed
propose module using non-perfect resp. perfect local propose module is O(n 3 ).
Proof. By Theorem 4 in [TS04] , there are ⌊ n−1
k−1⌋
= O(n) groups to be built, and the worst
time complexity for a single locally agreed proposal is O(n 2 ), the claimed time complexity
follows.
Note that the locally agreed proposal module significantly improves the average case for
constructing the overlay network G ′ , since local proposals are only generated by potential
members instead of all nodes/leaders in (G ′ ) e as in the global proposal module.
Next, we show that the implementation in Figure 1.8 is a locally agreed propose module
according to Definition 4.
Lemma 9. If L[gid] = L ′ [gid], the set L[gid] includes all nodes/groups that allow constructing
a group with a weight lower than P ∗ [gid].W eight and a member of L[gid].
Proof. Each node/leader initially calculates its local proposal P ∗ [gid], adds itself to
L ′ [gid] and sets L[gid] to the potential member set() with fixed member gid and bound
P ∗ [gid].W eight. The function potential member set() returns all nodes/groups which allows
to build a group consisting of the fixed members without exceeding the bound. If a
PROPOSAL message is received by a node/leader it either adopts P ∗ , L and L ′ and extends
L[gid] with nodes/groups from its potential member set() including itself as fixed member
and adds itself to L ′ [gid]. Or it leaves L ′ [gid] unchanged and thereby prevents L ′ [gid] from
getting equal to L[gid] for this proposal. Therefore, each node/group which allows to constructed
a group with weight below P ∗ [gid].W eight for any node/group in L[gid] must be
in L[gid] if L[gid] = L ′ [gid].

20 CHAPTER 1. PROPOSE MODULES
Lemma 10. If L[gid] = L ′ [gid], all nodes/leaders in L[gid] agreed upon the fact that P ∗ [gid]
is the cheapest local proposal of all nodes/leaders in L[gid].
Proof. If a node/leader receives a PROPOSAL message and the weight of the received proposal
Pin ∗ is lower than the local proposal P ∗ [gid] it adopts P ∗ , L and L ′ and adds itself to
L ′ [gid] thereby agrees that the weight of Pin ∗ is lower than the weight of its current proposal
P ∗ . If a node/leader receives a PROPOSAL message and the received Pin ∗ is equal to its local
P ∗ [gid] the sets L and L ′ are merged. Whereas if a received Pin ∗ has higher weight than
P ∗ [gid], the received set L ′ in is discarded and therefore L in can not become equal to L ′ in for
this received proposal.
Theorem 8. A group corresponding to a proposal generated by a locally agreed propose
module is not destroyed through later group constructions if the groups in the incompletely
built overlay graph (G ′ ) e and the communication graph G remain stable.
Proof. By contradiction. Let us assume that there exists an alternative proposal P alt that
destroys a group corresponding to previous released proposal P ∗ [gid] of a locally agreed
propose module. Then the weight of the P alt has to be lower than the weight of P ∗ [gid],
and P alt and P ∗ [gid] must have a member in common, i.e. ∃j ∈ P alt .Members : j ∈
P ∗ [gid].Members by the topology construction algorithm (Section 3 in [TS04] ). If P ∗ [gid]
was released by a locally agreed propose module the sets L[gid] and L ′ [gid] were equal.
Therefore L[gid] had to include all nodes/groups which allows to built a group that has
lower than P ∗ [gid].W eight by Lemma 9. But if L[gid] = L ′ [gid], all nodes/leaders in L[gid]
agreed upon the fact that P ∗ [gid] is the cheapest local proposal of all nodes/leaders in L[gid]
by Lemma 10, which is a contradiction to the existence of a lower weight proposal P alt .
1.7 Local Probabilistic Non-Perfect Propose Module
In addition to the previous deterministic propose modules we now present a simple probabilistic
local non-perfect propose module. The propose module operates in two phases:
The first phase tries to build proposals consisting of nearby nodes and groups. If this phase
fails, the second phase generates a proposal consisting of k nodes and groups currently in
no groups. For the first phase a list L is used where all nodes and known groups are sorted
according to their distance. Initially the first k entries in L are used for a new proposal.
Nodes or groups which reject the proposal are randomly exchanged by their next entry in L.
The new proposal is checked again until all members accept (all bounds are ok) or if they
reject, phase two is started. In phase two the propose module uses a list L2 consisting of
nodes/groups without parent group. If this list is smaller than k, nodes and groups in L are
checked to find new groups without parent group. If L2 is large enough, the first k entries
are used for a proposal.
First of all, if the current node/group gid is member in a group, any search for possibly
better groups is immediately suspended (line 275 and 294). If the search for a proposal
is initiated by calling procedure probabilistic proposal and there is currently no search in
process (L[gid] =⊥) all global variables are initialized. For the first phase L[gid] is set to

1.7. LOCAL PROBABILISTIC NON-PERFECT PROPOSE MODULE 21
270 array L[gid], array L2[gid], array L2 old [gid], array min weight[gid], array bound[gid]
271 set reply set[gid], next[gid], phase[gid], P ∗ [gid]
272
273 procedure probabilistic proposal (gid)
274
275 if Group[gid].P arentID =⊥ ∧ L[gid] =⊥
276 phase[gid]←1 /∗ initiate local data for search ∗/
277 L[gid]←sort nodes in Π according to increasing w : (x, y, w) ∈ Λ ID : x = ID ∧ y ∈ Π
278 L2[gid]←gid
279 next[gid]←gid
280 reply set[gid]← ⋃ k
x=1 L[gid][x]
281 min weight[gid]←⊥
282 bound[gid]←⊥
283 P ∗ [gid]←calculate proposal(reply set[gid], Group[gid].W eight, Λ ID , true)
284 for i=reply set[gid]
285 send (CHECK, gid, i) to leader of(i) /∗ query group weight, bound and ParentID ∗/
286
287 if received (CHECK, i, gid) /∗ return group weight, bound and ParentID ∗/
288 if Group[gid].P arentID =⊥
289 send (RETURN, Group[gid].W eight, ∞, ⊥, gid, i) to leader of(i)
290 else
291 send (RETURN, Group[gid].W eight, Group[Group[gid].P arentID].W eight, \\
292 Group[gid].P arentID, gid, i) to leader of(i)
Figure 1.9. First part of the local probabilistic propose module.
the sorted list of nodes. The first proposal is generated consisting of the first k entries in L
(line 276-285). After that, we send a CHECK message (line 285) to all members to query
their current group weight and bound (weight of parent group).
If such a (CHECK) is received a (RETURN) message including current group weight, parent
group weight and parent group id is returned (line 287-292).
If a (RETURN) message is received the min weight and bound is stored and the id and its
parent id is inserted into L and L2 (line 295-302). If all (RETURN) messages are received,
the weight of the proposal is adjusted according to Definition 1 in [TS04] . If the propose
module is in phase one (line 307-320), we calculate the reject set consisting of all members
which reject that proposal. If the reject set is empty the proposal is released. Whereas if
the reject set is not empty, a random member is exchanged by its next element in L and the
new member is checked again. If there is no next member, the propose module switches to
phase two. In phase two (line 320-334) it is checked if the L2 set is larger than k. If this
is not the case new nodes/groups without parent group are searched via CHECK messages to
nodes and groups in L. If the set L2 is large enough, the first k entries are used for a new
proposal. The proposal members are checked and if the L2 set does not change the proposal
is released (line 334).
It is not necessary to check the whole L set in phase one until we switch to phase two.
In simulation we found that the total weight of the topology does not change significantly if
we switch to phase two after phase one has searched 10% of L. This limitation of the L set
reduces the average number of messages.

22 CHAPTER 1. PROPOSE MODULES
293 if received (RETURN, weight, bound, P arentID, i, gid)
294 if Group[gid].P arentID =⊥ /∗ only search if in no group ∗/
295 if P arentID ≠⊥ /∗ enqueue P arentID and i ∗/
296 L[gid]←add P arentID to L[gid] after first node x with x ∈ P arentID
297 L2[gid]←L2[gid] \ {i}
298 else
299 L2[gid]←L2[gid] ∪ {i}
300 reply set[gid]←reply set[gid] \ {i}
301 min weight[gid][i]←weight
302 bound[gid][i]←bound
303
304 if reply set = ∅ /∗ all returns received ∗/
305 if P ∗ [gid].W eight < max(min weight[gid]) /∗ set minimum group weight ∗/
306 P ∗ [gid].W eight←max(min weight[gid]) + ɛ
307 if phase[gid] = 1
308 set reject←x ∈ P ∗ [gid].Members : bound[gid][x] ≤ P ∗ [gid].W eight
309 if reject = ∅
310 P [gid]←P ∗ [gid] /∗ release proposal ∗/
311 else
312 i←randomly choose a node/group in reject
313 P ∗ [gid].Members←exchange node/group i in P ∗ [gid].Members with next in L[gid]
314 if there is no next element in L[gid]
315 phase[gid]←2 /∗ change to phase 2 ∗/
316 else
317 P ∗ [gid]←calculate proposal(P ∗ [gid].Members, Group[gid].W eight, Λ ID , true)
318 send (CHECK, gid, i) to leader of(i)
319 reply set←{i}
320 if phase[gid] = 2
321 if |L2[gid]| < k /∗ search for nodes/groups in no group ∗/
322 next←next node/group next in L[gid]
323 send (CHECK, gid, next) to leader of(next)
324 reply set←{next}
325 else if L2[gid] = L2 old [gid] /∗ check if L2 has changed ∗/
326 L2 old [gid]←L2[gid]
327 for i= ⋃ k
x=1 L2[gid][x]
328 send (CHECK, gid, i) to leader of(i)
329 reply set←reply set ∪ {i}
330 min weight[gid]←⊥
331 bound[gid]←⊥
332 P ∗ [gid]←calculate proposal(reply set, Group[gid].W eight, Λ ID , true)
333 else
334 P [gid]←P ∗ [gid] /∗ release proposal ∗/
Figure 1.10. Second part of the local probabilistic propose module.
Lemma 11. The worst case message complexity of the local probabilistic propose module
for generating a single proposal is O(n).
Proof. In the worst case the L set includes all nodes and groups (n + ⌊ n−1
k−1⌋
) and each initial
proposal member is exchanged by all its successors in L. So no more than k · (n + ⌊ n−1
k−1⌋
) =
O(n) (CHECK) and (RETURN) messages are caused by phase one. For phase two no more

1.8. GLOBAL PROBABILISTIC NON-PERFECT PROPOSE MODULE 23
than |L| (CHECK) and (RETURN) messages are possible until the L2 set is large enough to
build a proposal. Note that a proposal from phase two is always accepted by the construction
algorithm (since their members have no parent group and hence no bounds). The claimed
complexity follows if we add the message complexity of phase one and two.
Lemma 12. The worst case time complexity of the local probabilistic propose module for
generating a single proposal is O(n).
Proof. Since only O(n) messages are transmitted for a single proposal by Lemma 11, the
worst case time complexity cannot be worse than O(n).
Theorem 9. The worst case message and time complexity for generating an admissible overlay
network G ′ with the regular topology construction algorithm using the local non-perfect
probabilistic propose modules is O(n 2 ).
Proof. After O(n) messages each of the at most n + ⌊ n−1
k−1⌋
nodes/groups switches from
phase one to phase two. In phase two the proposal consist only of members currently in no
group, hence the proposal will always be constructed by the topology construction algorithm.
Since there are not more than n + ⌊ n−1
k−1⌋
= O(n) nodes and groups the worst case messages
complexity of O(n 2 ) follows. Obviously the claimed worst case time complexity follows by
the same argument as for Lemma 12.
1.8 Global Probabilistic Non-Perfect Propose Module
In the next section we present a global probabilistic propose module with low worst case
complexity. The propose module uses a list L containing all available nodes and groups
(without parent group) and a second list L2 which is repeatedly forwarded, sorted and reduced
until it has size k. If L2 consists of k nodes and groups a corresponding proposal is
calculated.
Figure 1.11 depicts the pseudo code for the global probabilistic propose module. The
search is started with an INIT message to a random node containing a list L containing all
nodes. Upon reception of the INIT message, a L2 list with the nearest INIT SIZE nodes/-
groups is generated and forwarded randomly (PROB message) to any in L2. If a PROB is
received the node/group corresponding to the most expensive edge is removed from L2. If
L2 has size k a proposal is calculated, the proposal members are removed from L and the
new group is added to L. Next, the search for a new proposal is initiated by a new INIT
message. If L2 is larger than k L2 is randomly forwarded to any node/leader in L2.
Lemma 13. The worst case message and time complexity for a single proposal from the
global probabilistic propose module is O(1).
Proof. The set L2 is initiated with INIT SIZE elements from L and each PROB message
reduces the size of L2 by one. After (init size − k) PROB messages a proposal is generated.
List L2 contains only nodes/groups from L without parent group, hence the generated
proposal is always accepted by all its members.

24 CHAPTER 1. PROPOSE MODULES
335 function global probabilistic proposal () /∗ initially initiate search ∗/
336 i← select id in Π
337 send (INIT, Π) to leader of(i)
338
339 if received (INIT, L) /∗ generate L2 from L ∗/
340 L←sort ids in L according to increasing w : (x, y, w) ∈ Λ ID : x = ID ∧ ∃z ∈ L : y ∈ z
341 L2←trim(L, min(|L|−INIT SIZE, |L|)
342 i← select random id in L2
343 send (PROB, L, L2) to leader of(i)
344
345 if received (PROB, L, L2)
346 if |L2| > k /∗ reduce L2 ∗/
347 L2←sort ids in L2 according to increasing w : (x, y, w) ∈ Λ ID : x = ID ∧ ∃z ∈ L2 : y ∈ z
348 L2←trim(L2, 1)
349 if |L2| = k /∗ generate proposal from L2 ∗/
350 P [gid]←calculate proposal(L2, Λ ID , true)
351 L←L \ L2
352 L←L ∪ {P [gid].T erminals}
353 i← select random id in L
354 send (INIT, L) to leader of(i)
355 else if |L2| > k /∗ forward L2 to random node/group in L2 ∗/
356 i← select random id in trim(L2,|L2|/2)
357 send (PROB, L, L2) to leader of(i)
Figure 1.11. Pseudo code for global probabilistic propose module.
Theorem 10. The worst case message and time complexity for generating an admissible
overlay network G ′ with the regular topology construction algorithm using the global probabilistic
propose modules is O(n).
Proof. Obvious, since list L initially contains n nodes and each generated proposal reduces
L by k − 1 nodes/groups.
Note that this propose module maintains a list of nodes/groups without parent groups
instead of exploring this information from the network. If there are concurrent group constructions
or group destructions e.g. caused through crashing nodes or concurrently executed
propose modules, it can not be guaranteed that the overlay graph is constructed completely
by this propose module.
1.9 Propose Module for the Extended Topology
For constructing an admissible overlay graph G ′ with a topology construction algorithm
that uses the extension of Section 6 in [TS04] (extended groups), new group proposals need
not necessarily have lower group weight than the current group. Let (G ′ ) e be an incompletely
built overlay graph at time e. Let i be a member of a regular or extended group
g i : i ∈ members(g i ) in (G ′ ) e . Then, if a new group proposal g j with higher weight
than g i is proposed to leader of group g i , an extended group gi ∗ is built instead of rejecting
the proposal as with the regular topology construction algorithm. The new ′
group

1.9. PROPOSE MODULE FOR THE EXTENDED TOPOLOGY 25
gi ∗ consists of the members of g ′ i and of g j , so that the terminal nodes of T gi keep unchanged
(T gi = T g ∗ ) and the new members of g i ′ j become internal members in group gi ∗ ′:
I g ∗ = I
i ′ gi ∪ {x : x ∈ members(g j ) ∧ x /∈ members(g i )}. Thus, even if groups with higher
weight than the current group are proposed, the new members are added to the topology
without destroying a group. Recall that extended groups with more than 2 · k − 2 members
are split into two regular groups.
A simple propose module that does not guarantee that a proposed group has lower weight
for all group members is presented in Figure 1.12. The propose module simply proposes a
group consisting of members corresponding to the k − 1 lowest-weight edges.
358 procedure local non-perfect proposal for extended groups(gid)
359
360 if Group[gid].P arentID ≠⊥
361 return /∗ if already in a group no new group is proposed ∗/
362
363 set fix←{gid}
364 while |fix| < k
365 set ps←Π \ {∃x ′ ∈ fix : x ∈ x ′ }
366 fix←fix ∪ {y : min w (x, y, w) : x ∈ gid ∧ y ∈ ps}
367 P [gid]←calculate proposal(fix, Λ ID , true)
Figure 1.12. Pseudo code for the local non-perfect proposal module for a topology with
extended groups.
It is easy to see that the message complexity for such a proposal is O(1) and time complexity
is O(k).
Theorem 11. The worst case message and time complexity for constructing the non-minimal
admissible overlay network G ′ with extended groups using the presented proposal module of
Figure 1.12 is O(n).
Proof. Each node and each group generates such a local non-perfect proposal at most once,
since each proposed group causes their node/group to get member in a group. A new group
is only proposed for nodes/groups without a parent group. Recall that in this constructing
scheme no group is ever destroyed. Therefore at most n + ⌊ n−1
k−1⌋
= O(n) nodes/leaders
generate a single proposal with O(k), hence the claimed complexity follows. Note that the
reconstruction of groups with more than 2 · k − 2 members (cf. Section 6 in [TS04]) is
independent of n, therefore does not cause additional complexity.
Note that even if proposed groups with higher weight than the current group weight are
accepted, this does not mean that no optimization takes place. The proposed groups are
constructed to achieve low weight although they are not minimal, and there is a second
optimization taking place if an extended group with more than 2 · k − 2 members is split into
two regular groups. This is a kind of delayed optimization and it takes place in reverse order
(top down), in contrast to the regular topology construction.

26 CHAPTER 1. PROPOSE MODULES
1.10 Improvements
In this section we present some optimizations for all propose modules that might significantly
improve the average case albeit not the worst case.
• A simple trick to avoid the global arrays with index gid is to include them into the
Group[gid] data structure. So the allocated memory space is automatically freed if a
group is destroyed.
• If a node/group has no parent group, the bound for calculating a proposal can be set
to a low value and incremental increases until the first proposal is found, instead of
setting the bound to ∞.
• The size of the potential member set is critical for the performance of most propose
module. It can be strongly shortened if the weight of unknown edges is estimated
instead of using the lowest overall edge weight k for calculation.
• The caching of edge weights of other nodes reduces the number of GET EDGES messages,
shortens the potential member set if fetch missing edges = false and might
reduce message size.
• For larger k the complexity of the calculate proposal() function gets significant, even
if independent from n. To improve performance a bound can be implemented (e.g.
externally passed from potential member set() or internally from currently best calculated
proposal) to prematurely prune useless search branches.
• If function calculate proposal() is called from potential member set it is not necessary
to calculate the terminal set.
• In the local perfect proposal module it is not necessary to initially generate a complete
pms set with function get group ids(), the necessary group ids can also be collected
during the search.
• The performance of the locally agreed propose module can be increased if each
node/leader locally keeps a set of nodes/groups that were in P ∗ [gid].Member and
Pin.Member ∗ when a node/leader took over the received proposal. If upon reset a
node/leader rests all nodes/leaders in the new set, the number of proposed groups
which will be rejected by the topology construction algorithm can be significantly
reduced.
• If one wants to allow some rare group reconstruction with the locally agreed propose
module, the performance can be increased by exchanging line 249 with L ′ [gid]¡-L ′ in ∪
(L[gid] ∩ L ′ [gid]).
1.11 Summary
Table 1.1 summarizes the worst case complexities for all presented propose modules.

1.11. SUMMARY 27
Complexity Topology Topology Time Message Time Message
regular/ perfect/ Proposal Proposal Topology Topology
extended non-perf.
Local Perfect regular perfect O(n k−1 ) O(n) O(n n·k ) O(n n·k )
Local Non-Perfect regular non-perf. O(n) O(n) O(n n·k ) O(n n·k )
Global Perfect regular perfect O(n k ) O(n) O(n k+1 ) O(n 2 )
Global Non-Perfect regular non-perf. O(n 2 ) O(n) O(n 3 ) O(n 2 )
Locally Agreed Perfect regular perfect O(n k ) O(n 2 ) O(n k+1 ) O(n 3 )
Locally Agreed Non-Perf. regular non-perf. O(n 2 ) O(n 2 ) O(n 3 ) O(n 3 )
Local Probabilistic regular non-perf. O(n) O(n) O(n 2 ) O(n 2 )
Global Probabilistic regular non-perf. O(1) O(1) O(n) O(n)
Prop. mod. for ext. groups extended non-perf. O(1) O(1) O(n) O(n)
Table 1.1. Overview of the worst case complexities.

28 CHAPTER 1. PROPOSE MODULES

Chapter 2
Simulation
Analytical results, like our worst case results, are not sufficient to evaluate the practical
use of our algorithm. Of similar interest are questions about the quality of the generated
overlay graph, and the average case complexity of its construction. This section explains our
MATLAB [MAT] simulation environment 1 and presents simulation results.
2.1 Simulation Environment
We developed a modular simulation environment based on MATLAB which allows to
simulate our construction algorithm and different propose modules on various networks.
The simulation environment provides:
• Node: The main function allows to create, suspend and destroy nodes at any time. A
node has a unique ID and a random position in the simulation area. It consists of a
separate memory space and has access to different subsystem functions.
• Message subsystem: The simulation provides a message subsystem with different message
queuing strategies (unordered, FIFO) and consists of a send and receive interface
for each node. To analyze the complexity of different propose modules, the message
subsystem is able to protocol each message.
• Node scheduler: Our simulation continuously executes the active nodes in a random
and fair order.
• Weight matrix: The communication weight matrix calculates the weight of each communication
edge. Any function for assigning weight to edges can be used (e.g. for our
simulation results we used the Euclidean distance). Each node knows the subset of the
weight matrix where the node is endpoint of an edge.
• Mobility: A node and the main function can change the position of a node. The weight
matrix is automatically updated according to the new position of the node.
1 The source code of the simulation environment can be downloaded from the web: http://www.ecs.
tuwien.ac.at/W2F/.
29

30 CHAPTER 2. SIMULATION
• Connections: The simulation environment maintains a global connection matrix where
each node can make and cancel connections to other nodes. Upon each change the
maximum node degree of each node is checked and violations are reported. After the
stabilization time, the simulation checks if each connection is bidirectional.
• Stabilization: Since our algorithms continuously try to adapt to network changes and
improve the topology, we can not define an exact termination point for our algorithms
(e.g. no messages in transit). Therefore the simulation provides some support for determining
the stabilization of the overlay graph like the monitoring of the timing behavior
of the group constructions.
On top of this simulation environment we implemented the construction algorithm from
[TS04] with the atomic commit function and all propose modules of Section 1.
With this simulation environment we are capable of analyzing the following system properties:
• The number of joins and leaves, as well as the number of released proposals.
• The number of sent messages, the number of messages in transit, the number of messages
per node and the number of certain types of messages.
• The total weight of the connection matrix and the spanning factor of specific paths.
• With our visualization module we can monitor connection and group construction and
termination, the moving of nodes and the final topology.
• To analyze the effectivity of the locally agreed propose module we can visualize the
size of the L list and highlight the nodes and groups in L.
2.2 Results
For the following simulations we randomly placed the indicated number of nodes on a
100 × 100 square. The edge weight between two nodes is set to the square of their distrance.
2.2.1 Example Overlay Graphs
First of all, we depict some snapshots of our overlay graph with propose modules.
Figure 2.1 shows four overlay graphs for the same example network with n = 50 and
k = 3, where cycles are regular nodes and squares represent gateway nodes. For the overlay
graph in Figure 2.1(a) we use perfect propose modules, overlay graph 2.1(b) uses non-perfect
propose modules, overlay graph 2.1(c) uses local probabilistic propose modules and the overlay
graph in Figure 2.1(d) was constructed using a global probabilistic propose module. A
deeper look at the topology example yields the underlying group structure.
2.2.2 Connectivity
Figure 2.2(a) depicts the perfect overlay graph with k = 3 and in Figure 2.2(b) with k = 4
for the same example network with n = 30.

2.2. RESULTS 31
(a)
(b)
(c)
(d)
Figure 2.1. Different topologies for an example network.
2.2.3 Total Weight
Next we want to compare the minimality of our overlay graph with respect to the unique
minimal k-connected and k-regular overlay graph. Since the problem of finding the minimum
cost overlay graph proved to be NP-complete 2 , we used a specific and well studied
subproblem. For k = 2 our overlay graph is comparable to the Traveling Salesman Problem
(TSP) except that the TSP finds a closed path and our algorithm generates an open path.
We hence add the weight for the connection between the two gateway nodes, which is not
generated by our algorithm, to the total graph weight to compare the weight with the exact
solution of the TSP problem. For small n (around 50 to 100) there exit tools [TSP] that
are available to calculate the exact solution of the TSP in reasonable time. Table 2.1 shows
the average total weight factors (total weight of the overlay graph plus the weight for the
2 Since equivalent to TSP for k = 2.

32 CHAPTER 2. SIMULATION
(a)
(b)
Figure 2.2. Overlay graph with k = 3 and k = 4.
gateway interconnection divided by the weight of the exact TSP solution) from 20 random
graphs with n = 50 and k = 2 using different propose modules.
Propose Module perfect/non-perfect factor
Perfect propose module perfect 1.11
Non-Perfect propose module non-perfect 1.17
Local Probabilistic propose module non-perfect 1.31
Global Probabilistic propose module non-perfect 1.30
Table 2.1. Overview of the average total weight factors.
2.2.4 Message Complexity
Figure 2.3 and 2.4 illustrate the relationship between number of nodes in the overlay graph
and the average number of messages per node for construction. Figure 2.3 plots the average
number of messages of all propose modules in a double-logarithmic scale and Figure 2.4
depicts the number of messages for the probabilistic propose modules in linear scale. Note
that the absolute number of messages is only a rough indication for the complexity, since
the number of messages strongly depends on the propose module implementation (e.g. the
use of caches or larger messages instead of multiple messages can decrease the absolute
number of messages). For this results we used the same implementation as presented in
Chapter 1. These results confirm that the average case is much better than the analytical
worst case results. An interesting aspect is that in the average case the non-synchronized
local propose modules perform better than their corresponding synchronized propose modules
(locally agreed or global) even if their worst case complexity is much larger. Hence the
additional synchronization complexity of the global and locally agreed propose module for

2.2. RESULTS 33
generating the proposals in the perfect order is not necessary in the general case. For the
local probabilistic propose module we discover a nearly linear average message complexity.
10 3 Number of nodes
Number of messages per node
10 2
10 1
10 0
local perfect
local non−perfect
global perfect
global non−perfect
locally agreed perfect
locally agreed non−perfect
local probabilistic non−perfect
global probabilistic non−perfect
10 −1
10 0 10 1 10 2
Figure 2.3. Average number of messages per node (double-logarithmic scale).
2.2.5 Weight Distribution
Besides the total weight of the overlay graph, the distribution of the edge weights among
the nodes is an important issue. For example, an overlay graph that has minimal total weight
but also contains connections with very high weight might not be feasible as network topology,
since the corresponding nodes will prematurely run out of battery power. Figure 2.5
shows the average distributions of the connection weights per node for 20 random networks
with n = 50 and k = 3.
2.2.6 Spanner Property
In the following simulations we study the spanner property of our overlay graphs. We
therefore calculate the stretch factors of random source and destination pairs. The stretch
factor is the ratio between the weight of the shortest path in G ′ and the shortest path in G
(hence the weight of the direct edge between source and destination). In Table 2.2 we show
the average and maximum spanning factors for each node-disjoint path from 50 random
source and destination pairs in 20 random networks with n = 50 and k = 3. Although our
overlay graph is not a spanner (has a bounded stretch factor) the results depict a good spanner
property.

34 CHAPTER 2. SIMULATION
20
18
16
Number of messages per node
14
12
10
8
6
4
2
local probabilistic non−perfect
global probabilistic non−perfect
0
0 20 40 60 80 100
Number of nodes
Figure 2.4. Average number of messages per node (linear scale).
Average number of nodes
0 10 20 30 40 50 60 70 80 90 100
Weight
Figure 2.5. Weight distribution per node.
2.2.7 Group Destructions
If we use local propose modules it happens that a constructed group is destroyed through
later proposed groups. In Figure 2.6 we plot the number of groups of the completely con-

2.2. RESULTS 35
Stretch factor shortest path second shortest path longest path
ave. max. ave. max. ave. max.
Perfect propose module 2.0675 7.5739 3.2863 10.5328 6.1817 18.6312
Non-Perfect propose module 2.0781 7.7823 3.3203 12.8245 6.3020 18.7564
Local Probabilistic propose module 2.0026 6.3044 3.1066 14.0739 4.9470 19.3306
Global Probabilistic propose module 2.446 9.1306 3.9401 18.7618 6.5578 23.1523
Table 2.2. Average and maximum stretch factors.
structed overlay graph and the average number of groups destroyed during the construction
of G ′ for 20 random networks with k = 3. So the number of unnecessarily constructed
groups grows linear with n, and does not necessitate the additional synchronization of the
global and locally agreed propose module.
25
Number of group destructions
20
15
10
local perfect
local non−perfect
local probabilistic non−perfect
number of groups
5
0
5 10 15 20 25 30 35 40 45 50
Number of nodes
Figure 2.6. Average number of group destructions.
2.2.8 Locally Agreed Propose Module
The last simulation depicts the effectivity of the locally agreed propose module. Figure 2.7
plots the group weight ω(g) and the size of the L list for each proposed group of the locally
agreed propose module. We observed that the size of list L increases with increasing group
weight and hence larger groups.

36 CHAPTER 2. SIMULATION
Group weight
Size of L
0 2 4 6 8 10 12 14 16 18 20
Group
Figure 2.7. Group weight and size of L.

Bibliography
[MAT] MathWorks MATLAB. http://www.mathworks.com/.
[TS04] Bernd Thallner and Ulrich Schmid. Distributed construction of fault-tolerant
overlay networks. Research Report 35/2004, Technische Universität Wien, Institut
für Technische Informatik, Treitlstr. 1-3/182-2, 1040 Vienna, Austria, 2004.
http://www.ecs.tuwien.ac.at/W2F/RR35-2004.pdf.
[TSP] Collection of TSP software to compute the optimal solution.
http://www.or.deis.unibo.it/research pages/tspsoft.html.
37