A Scalable Computer Architecture for On-line Pulsar Search on the ...

A <strong>Scalable</strong> <strong>Computer</strong> <strong>Architecture</strong> <strong>for</strong>
<strong>On</strong>-<strong>line</strong> <strong>Pulsar</strong> <strong>Search</strong> on the SKA
- Draft Version -
G. Knittel, A. Horneffer MPI <strong>for</strong> Radio Astronomy Bonn
with help from:
M. Kramer, B. Klein, R. Eatough

GPU-Based <strong>Pulsar</strong> Timing
FFT –DeDisp– IFFT –
Full Stokes – Folding
GPU
DISK
GPU
User‐Mem
DMA
Fast Bit‐
Reversal
mory
Syst tem Me
CPU
From
ADC
NIC

GPU-Processing
Processing
FFT
Coherent Dedispersion
DM
IFFT
Full Stokes Parameters
Folding
Pulse Period

Per<strong>for</strong>mance:
610M Samples/s
(2 GPUs)
GPU-Based <strong>Pulsar</strong> Timing

<strong>Pulsar</strong> <strong>Search</strong>
FFT
Coherent Dedispersion
DM
IFFT
Full Stokes Parameters
Folding
Pulse Period

<strong>Pulsar</strong> <strong>Search</strong>
FFT
Coherent Dedispersion
Trial DM
Loop
IFFT
Stokes I Parameter
Folding
Trial
Pulse Period

Binary Systems <strong>Search</strong>
FFT
Coherent Dedispersion
Trial DM
Loop
IFFT
Stokes I Parameter
Folding
Trial
Pulse Period,
Orbital
Parameters

<strong>Pulsar</strong> <strong>Search</strong>
• Idea:
<strong>Pulsar</strong> <strong>Search</strong> by Massively-Parallel
Folding (in Time Domain)

<strong>Pulsar</strong> <strong>Search</strong>
• Idea:
<strong>Pulsar</strong> <strong>Search</strong> by Massively-Parallel
Folding
• Binary Systems:
Make Length of Phase Bins variable
(similar to Time Sequence Resampling)

<strong>Pulsar</strong> <strong>Search</strong>
• Idea:
<strong>Pulsar</strong> <strong>Search</strong> by Massively-Parallel
Folding
• Binary Systems:
Make Length of Phase Bins variable
• High-Dimensional <strong>Search</strong> Space:
Complete Coverage not possible.

<strong>Pulsar</strong> <strong>Search</strong>
• Frequency Domain:
<strong>Pulsar</strong> <strong>Search</strong> by Massively-Parallel
Harmonic Summation
• Binary Systems:
Process Range of neighboring Frequency
Bins
• Use same Hardware!

<strong>Pulsar</strong> <strong>Search</strong>
• Frequency Domain:
<strong>Pulsar</strong> <strong>Search</strong> by Massively-Parallel
Harmonic Summation
• Binary Systems:
Process Range of neighboring Frequency
Bins
• Use same Hardware!
• Not completely worked out yet.

<strong>Pulsar</strong> <strong>Search</strong> on the SKA
• Add Coherent Beam<strong>for</strong>ming

<strong>Pulsar</strong> <strong>Search</strong> on the SKA
Polyphase Filterbank
FFT
Coherent Beam<strong>for</strong>ming
Coherent Dedispersion
Polyphase Filterbank
FFT
Coherent Beam<strong>for</strong>ming
Coherent Dedispersion
IFFT
Stokes I Parameter
Folding
Power Spectrum
Harmonic Sum

<strong>Pulsar</strong> <strong>Search</strong> on the SKA
Polyphase Filterbank
FFT
Coherent Beam<strong>for</strong>ming
Coherent Dedispersion
IFFT
Stokes I Parameter
Folding
FPGA
CPU
CPU
GPU
GPU
GPU
ASIC
Polyphase Filterbank
FFT
Coherent Beam<strong>for</strong>ming
Coherent Dedispersion
Power Spectrum
Harmonic Sum

Mode of Operation (Time Domain)
Telescope 0
Telescope 1
Telescope 127

Parallelization: Timeslicing
Telescope 0
Telescope 1
Telescope 127
Timeslice to Timeslice to
Timeslice to
„Rank 0“ „Rank 1“
„Rank 0“

Parallelization
Telescope 0
Telescope 1
Telescope 127
Required Processing Time defines Number of Ranks

Parallelization
Data from all Telescopes
Chain Network
Rank 0 Rank 1 Rank 2 Rank 63
<strong>Architecture</strong> scales endlessly

Parallelization
Telescope n
Polyphase Filterbank
16 Subbands
Compute Node 15
Compute Node 1
Compute Node 0

Parallelization
<strong>On</strong>e Rank
Compute Node 0
Data Capture Phase
Compute Node 1
Compute Node 15
8 Telescopes each

Parallelization
<strong>On</strong>e Rank
Compute Node 0
Filtering and
Subband Distribution
Compute Node 1
Ring Network
Compute Node 15

Coherent Beam<strong>for</strong>ming
Compute Node k
Local
Mem
FPGA
Polyphase
Filterbank,
Data Exchange
System
Mem
Beam 0 Subb k
Beam 1 Subb k
Beam 2 Subb k
Beam n Subb k
Tel 0 Subb k
Tel 1 Subb k
Tel 2 Subb k
Tel 127 Subb k
FFT,
Coherent
Beam<strong>for</strong>ming
CPU
CPU

Coherent Beam<strong>for</strong>ming on CPUs
• Per<strong>for</strong>mance using AVX:
128 Input Spectra, single-precision float,
256k Elements
• 128 Beams of same Size:
2.1s per Core @ 3.5GHz (prel. Results)

GPU-Processing
Processing
Video Video DeDisp, IFFT, SI
Mem Mem
System
Mem
GPU0 GPUn
Local
Mem
FPGA
Beam 0 Subb k
Beam 1 Subb k
Beam 2 Subb k
Beam n Subb k
Tel 0 Subb k
Tel 1 Subb k
Tel 2 Subb k
Compute Node k
FFT,
Coherent
Beam<strong>for</strong>ming
CPU
CPU
Polyphase
Tl127 Tel Subb bbk
Filterbank,
Data Exchange

GPU-Processing
Processing
• Per<strong>for</strong>mance:
2 Spectra, horz/vert, single-precision float,
4M Elements
• Total Power Time Sequence of same Size:
Total Power Time Sequence of same Size:
5ms

GPU-Processing
Processing
• How to output the Results to the ASICs

GPU-Processing
Processing
• How to output the Results to the ASICs
• All PCIe-Slots are already taken (GPUs,
FPGAs)

GPU-Processing
Processing
• How to output the Results to the ASICs
• All PCIe-Slots are already taken (GPUs,
FPGAs)
• Write to Screen Buffer to be output via
Write to Screen Buffer, to be output via
Monitor Cable

GPU-Processing
Processing
Mini‐DisplayPort
17.28 Gbit/s
~ 70 Gbit/s
Equiv. 1 PCIe x16 Slot

• Does it work
GPU-Processing
Processing

GPU-Processing
Processing
• Does it work Yes, but...
GPU Kernel
Screen
Via DVI:
2.7 Gbit/s
(Video)

Massively-Parallel Folding
Local
Mem
Local
Mem
SI Time Sequence
ASIC0
ASICn
Monitor
Cable
Video
Mem
Massively‐Parallel
Folding
GPU0

Massively-Parallel Folding
Compute Node 0
Compute Node 1
ASIC‐PC
Up to 16 Monitor Cables
Compute Node 15
GPU‐PC

Folding – Time Domain
Hypothetical
Pulse Period P
time
Dt t S lit P l
Detects Solitary <strong>Pulsar</strong>s
having P +‐ small P

Folding – Acceleration <strong>Search</strong>
Hypothetical
Acceleration
Variable Bin Length
(# of Samples per Bin)
time
Equiv. to Time Sequence Resampling

Harmonic Summation
f 2f 3f 4f
f 0 2f 0 3f 0 4f 0
freq
Dt Detects t Solitary <strong>Pulsar</strong>s
between f 0 and f 0 +f
f 0
f 0 + f

Harmonic Sum - Acceleration <strong>Search</strong>
f 2f 3f 4f
f 0 2f 0 3f 0 4f 0
freq
Dt Detects t Binary Systems
with max. Acceleration f
f 0
f 0 + f

Folding Processor
Broadcast Bus
SI Time Sequence or Power Spectrum
Accumulator
Programmable
Set of
Counters and
Incrementers
Memory
64 x 32 bits
To / from
local Memory

<strong>Pulsar</strong> Detector ASIC
10 3 ‐ 10 5
Folding
Processors

ASIC Network
Ring Network
Rank 0 Rank 1 Rank 2
Rank 63
ASIC‐PC

RFI Mitigation
• Subband-relative:
Accumulation is per Subband
• Beam-relative

The <strong>Pulsar</strong> <strong>Search</strong> Machine
• 128 Telescopes
• 128 Beams
• 100 DMs
• 50.000 Orbits
• 64x10 6.4 8 hypothetical <strong>Pulsar</strong>s

The <strong>Pulsar</strong> <strong>Search</strong> Machine
• PC Cluster
• Switch-less Design, helps Scalability
• GPU-PC + ASIC-PC = Compute Node
• 64 Ranks of 16 Compute Nodes
• 2048 PCs, 4096 CPUs, 8192 GPUs,
16384 ASICS
• 32.5 M€

Thanks!

The <strong>Pulsar</strong> <strong>Search</strong> Machine

Folding Processor

Costs

Supercomputer Costs 2005
• Sandia National Laboratories Red Storm: $90
million
• Los Alamos National Laboratory ASCI Q: $215
million
• Earth Simulator Center, Japan: $250 million
• IBM Blue Gene/L: $290 million
various (unreliable) Internet Sources