Tofu: A 6D Mash/Torus Interconnect - Fujitsu

1

Highly scalable and usable direct network (6D mesh/torus)
10 redundant high BW links, 4 RDMA engines (4x2 simultaneous transfer)
Good collective communication performance with Tofu original algorithms
Tofu barrier for barrier & reduction in H/W
Direct attached interconnect controller
SPARC64 TM IXfx
CPU
C
Y
Inter- Y
Connect X
B ×
Controller
X
(ICC)
5GB/s x 2
x10 links XYZ
20GB/s x 2 A Z Z B
Tofu realizes scalable systems beyond 100,000 nodes
With low power consumption, low latency, and high BW
2
ABC
Copyright 2011 FUJITSU LIMITED

Y+
Z+
X-
Y-
Z-
Y+
Z+
X-
Y-
Z-
Y+
Z+
X-
Y-
Z-
C
C
C
X+
A
X+
A
X+
A
C
C
C
A
A
A
3
B
B
B
X-
Y-
Z+
Z-
B
Y+
C
A
X+

ABC 3D Mesh
X -
Y -
Z+
Z -
4
B
Y+
C
X+
A
XYZ 3D Mesh

xyz
abc
5

abc
6

abc
b→c →a → x → y → z ⇒ a → c → b
abc
7

From To
3 example paths out of 12 possible paths
8

source fault
destination
9

Example)
One application dimension rendered on
two dimensional slice of a machine
0
2
1
4 dimensional slice view
11
3
4
10
10
6
5
2 dimensional slice view
0
2
11
9
7
3
8
10
6
9
1 4 5 8
7

0 11 10 9
0 11 10 9
0 11 1 10 4 9 5 8
0 11 1 10 4 9 5 8
1 4 2 5 3 8 6 7
1 4 2 5 3 8 6
2 3 6 7
7
2
3
6
7
Application X-Axis
The 6D submesh can be reused.
One of the app-dimensions is
degraded by one hop.
When a node failure occurs,
the running job is force quitted
and restarted from the user’s
checkpoint. 0 10 9
0 10 9
0 10 1 9 4 8 5
0 10 1 9 4 8 5
1 4 2 5 3 6
1 4 2 5 3 6
11
2
2
3
3
6
6
7
7
8
7
Mapping Hole
8
7

Application Y-Axis
2
1
0
2
1
3
4
5
10
3
4
9
6
5
Application X-Axis
the board to be replaced
8
7
mapping hole
mapping hole

13
Copyright 2010 FUJITSU LIMITED