DNK7_F5_8_Cluster Xilinx Kintex 7 Rack Mount HPC Cluster

PC3-1600
PC3-1600
PC3-800
DDR3
PC3-800
DDR3
DDR3
(128M DDR3x 16)
(128M x 16)
PC3-800 PC3-1600
DDR3
DDR3
(128M x 16) (128M x 16)
(128M x 16) (128M x 16)
DDR3 DDR3
DDR3
DDR3
DDR3
DDR3
(128M DDR3 x 16) (128M DDR3 x 16)
(M x 16) (M x 16)
(M x 16) (M x 16)
(128M x 16) (128M x 16)
DDR3
DDR3
(M x 16)
(M x 16)
Kintex-7
Kintex-7
7K410T/7K325T/
46
7K410T/7K325T/
46
7K160T
7K160T
Kintex-7
FPGA 1
FPGA 2
6
(FFG676)
(FFG676)
7K410T/7K325T
FPGA 0
(FFG900)
46
8 CFG Bus
46
JTAG
6
1
0 2
Kintex-7
Kintex-7
78
7K410T/7K325T/
46
7K410T/7K325T/
3
7K160T
7K160T
FPGA 4
FPGA 3
CPLD 4
PCIe
(FFG676)
(FFG676)
DDR3
DDR3
DDR3
DDR3
(M x 16) (M x 16)
(M x 16) (M x 16)
DDR3
DDR3
DDR3
DDR3
(M x 16) (M DDR3 x 16)
(M x 16) (M DDR3 x 16)
PC3-800
(M x 16)
PC3-800
(M x 16)
PCIe
I 2 C 2
PC3-1600
PC3-1600
Bridge
Serial
cfg Prog
5
HPCLK
(FF484)
EEProm
OSC
DDR3 Clocks
(all FPGAs)
cfg Prog
5
Flash
HRCLK
FPGA
OSC
Config
FPGA 1
SYSCLK (all FPGAs)
4-lane PCIe
PCIe
(GEN1 / GEN2)
SFB Clock (CFG Bus Clocking)
FPGA
DNK7F5PCIe
PCIe
Godzilla’s Bad Hair Day
Block Diagram v1.10
Product Brief
Oct. 2012
Ver. 1.0
Features
• 5U Rackmount Chassis containing:
TM
- 1 Intel Xeon E3-1275 processor card
- 8 DNK7_F5PCIe FPGA cards each
with 5 Xilinx Kintex 7
FPGAs (7K325T)
• PCIe 4-lane (GEN1/GEN2)
• 40 FPGAs in total, 100% dedicated to
application
- Other configurations with different CPU to
FPGA ratios are available
- 2 bays for SATA-3 hard drives
• Processor card
TM
- Intel Xeon E3-1200 series (Sandy Bridge)
processors, 3.4 GHz
• Quad-Core, 8MB shared L2 cache
• 4 GB DDR3 memory
(4 GB total)
- Options to 32 GB DDR3 memory
(32 GB max)
• VGA with standard D-Sub connector
• 10/100/1000BASE-T Ethernet (2 ports)
• USB 2.0 (4 ports total)
- 2 ports on front panel
- 2 ports on back bracket
• Supports most all Linux distributions
• DNK7_F5PCIe FPGA HPC Acceleration card
-PCI Express (4-lane) FPGA-based algorithm
acceleration peripheral with 5 Kintex-7 FPGAs
•4 Xilinx Kintex-7 FPGAs: 7K325T-1 (FFG676)
-7K410T-3,-2,-1, 7K325T-3,-2,-1, or 7K160T-3,-2,-1
•1 Xilinx Kintex-7 FPGA: 7K325T-1 (FFG900)
-7K410T-3,-2,-1 or 7K325T-3,-2,-1
DNK7_F5_8_Cluster
Xilinx Kintex 7 Rack Mount HPC Cluster
- Fixed 4-lane PCIe interface and controller
• PCIe GEN1/GEN2
• Full mastering DMA
- 2 transmit (host memory -> card)
- 2 receive (card -> host memory)
- Xilinx FPGA Kintex-7 7K325T-1 - 5 total user FPGAs
•508,400 flip-flops per FPGA
- 254K flips-flops with 6-input LUT
•1,540, 25x18 multipliers + 48-bit accumulator per FPGA
•1,590, 18 Kbit block RAM (2 Mbytes) per FPGA (or 445, 36 Kbit blocks)
- Fully dual-ported
- Each block RAM configurable as:
- 32K x 1, 16K x 2, 8K x 4, 4K x 9 (or 8),
- 2K x 18 (or 16), 1K x 36 (or 32), or 512 x 72 (or 64)
-- 5 separate 256Mb x 16 DDR3 memories for each field FPGA
• 3 memories PC3-1600
• 2 memories PC3-800
• Each memory has separate address, data, and control
- 6 separate 256Mb x 16 DDR3 memories for Dataflow Manager FPGA
• 3 memories PC3-1600
• 3 memories PC3-800
• Each memory has separate address, data, and control
- Two independent low-skew global clock networks
• distributed differentially and balanced
- Fast and Painless FPGA configuration via PCIe
• On-board battery for AES bitstream encryption
- Full support for embedded logic analyzers via JTAG interface
• ChipScope, and other third-party debug solutions
- FPGA-controlled LEDs
• Enough light to use as LED-based flashlight
DNK7_F5PCIe
PCIe (GEN1/GEN2)
4 4 4 4 4
4 4 4
Block Diagram
PCIe Fan Out
Switch
8 8
PCIe
PCIe Fan Out
Switch
Xeon E3-1275
(CPU1)
8 GB DDR3
8 GB DDR3
8 GB DDR3
8 GB DDR3
Platform
Controller
Hub
(3420)
6
LAN 1
10/100/1000 base-T
LAN 2
USB 0
USB 1
Video (SandyBridge)
HDD
Up to 6 HDD (SATA 3)
DNK7_F5_8_Cluster
Block Diagram v1.00
1 the DINI group

DNK7_F5PCIe Block Diagram
PC3-800
DDR3
(256M DDR3 x 16)
(256M x 16)
DDR3
(256M x 16)
PC3-1600
DDR3
(256M DDR3 x 16)
(256M x 16)
DDR3
(256M x 16)
PC3-800
DDR3
(256M x 16)
DDR3
(256M x 16)
DNK7_F5_8_Cluster Xilinx Kintex 7 Rack Mount HPC Cluster
PC3-1600
DDR3
(256M DDR3x 16)
(256M x 16)
DDR3
(256M x 16)
PC3-800
DDR3
(256M x 16)
DDR3
(256M x 16)
PC3-1600
DDR3
(256M DDR3x 16)
(256M x 16)
DDR3
(256M x 16)
LED
Kintex-7
7K410T/7K325T
FPGA 0
(FFG900)
46
Kintex-7
7K410T/7K325T
FPGA 1
(FFG676)
46
6
Kintex-7
7K410T/7K325T
FPGA 2
(FFG676)
46
46
8
CFG Bus
JTAG
1
0 2
CPLD 4
PCIe
3
78
Kintex-7
7K410T/7K325T
FPGA 4
(FFG676)
6
46
Kintex-7
7K410T/7K325T
FPGA 3
(FFG676)
PCIe
Bridge
(FF484)
I 2 C
2
Serial
EEProm
Flash
FPGA
Config
DDR3
DDR3
DDR3
DDR3
(256M x 16) (256M x 16)
(256M x 16) (256M x 16)
DDR3
DDR3
DDR3
DDR3
(256M x 16) (256M DDR3 x 16)
(256M x 16) (256M DDR3 x 16)
PC3-800
(256M x 16)
PC3-800
(256M x 16)
PC3-1600
PC3-1600
cfg
cfg
Prog
OSC
Prog
OSC
FPGA 1
5
5
HPCLK
HRCLK
DDR3 Clocks
(all FPGAs)
SYSCLK (all FPGAs)
Overview
Description
PCIe
4-lane PCIe
(GEN1 / GEN2)
PCIe
FPGA
SFB Clock (CFG Bus Clocking)
DNK7_F5_PCIe
Block Diagram v1.2
The DNK7_F5_8_Cluster is a complete, 5U rack mount FPGA acceleration cluster. The standard configuration
integrates the following:
Trenton TSB7053 Xeon processor card (boots to Linux). Other cards are also available.
8 – DNK7_F5PCIe Xilinx Kintex 7 FPGA cards with 5 7K325T FPGAs per card.
2 TB SATA II Hard Drive
This system contains the maximum number of cost effective FPGA that can be reasonability integrated into a 5U
chassis. Power and cooling are the constraining variables. High performance data paths between FPGA boards
enable data movement under algorithmic control that is wholly separate from the host processor, enabling FPGAbased
acceleration of whole new classes of data intensive algorithms.
In short, the DNK7_F5_8_Cluster is a massive number of large, low cost FPGAs integrated with an excellent dual
Xeon-based processor host.
A partial list of possible applications includes:
- bioinformatics
- Genomic search
- financial analytics
• low latency analysis
• derivative calculations
- image processing
- signal processing
- radar
- scientific computing
- video compression
- encryption/decryption (cryptography)
the DINI group
2

DNK7_F5_8_Cluster Xilinx Kintex 7 Rack Mount HPC Cluster
The Processor Card – single/dual Xeons
Central to the DNK7_F5_8_Cluster is the Trenton TSB7053 host processor card. This single-board computer
has Intel Xeon E3 processor, clocked at 3.4GHz. The processor board has 4 DIMM slots that can be stuffed with
up to 32GB DDR3 RAM, max 8GB of memory per slot. The processor card has two 10/100/1000 Base-T
Ethernet ports, along with 2, USB2.0 ports. The chassis can host up to 6 SATA drives. Power and cooling are
provided for up to 8 DNK7_F5PCIe cards. Power is cabled to the FPGA cards separately and also drawn from
the motherboard, allowing us to exceed the 25W slot PCIe limitation. The power budget is 50W per board. Note that
this requires a lot of airflow and the fans are noisy. Fully populated, the system is perhaps too noisy to be in close
quarters with an engineer.
The DNK7_F5PCIe – 5 Xilinx Kintex 7 FPGAs
The DNK7_F5PCIe is Xilinx Kintex-7 based FPGA board optimized for algorithmic acceleration applications
requiring FPGAs with high performance local memory. Data movement to/from the FPGA grid is accomplished via
a fixed 4-lane, GEN1/GEN2 PCIe bridge. Each field Kintex-7 FPGA (FPGAs 1-4 in the block diagram) has five
separate 256M x 16 DDR3 (4 Gb) memories. The Dataflow Manager FPGA (FPGA 0 in the block diagram) has six
256M x 16 DDR3 memories. The DNK7_F5_8_Cluster can host 8 of these cards.
Dedicated PCIe, 4-lane controller (GEN1 or GEN2)
We ship the DNK7_F5PCIe with a fixed, full function, 4-lane master/target PCIe controller. The PCI controller has
two mastering DMA engines, 2 for transmit (board -> host) and 2 for receive (host -> board). Drivers with 'C' source
for several operating systems are included at no cost.
Kintex-7 FPGAs from Xilinx - Performance and Low Power
The Xilinx Kintex-7, 28 nm FPGAs are utilized. We use the second largest member of this cost effective (read:
CHEAP) family. The Kintex-7 FPGA family has an impressive price/performance ratio for hardware-in-the-loop
accelerators with excellent device power consumption properties. Operation frequency is approximately twice that
of the previous low cost Xilinx FPGAs - Spartan-6.
Features of Kintex-7 include efficient, dual-register 6-input look-up table (LUT) logic, 36 Kb block RAMs, and
second generation DSP slices which contain 25 x 18 multipliers along with a 48-bit accumulator.
We use the second largest device from this family, the 7K325T, in the FFG900 and FFG676 packages. 100% of the
FPGA resources are dedicated to your application. All FPGAs, excluding the PCIe controller, are configured via
PCIe. The PCIe FPGA can be updated in the field.
Memory - DDR3
The availability of large amounts of local high speed memory is pivotal to FPGA-based algorithmic acceleration
applications. The DNK7_F5PCIe is optimized accordingly. Each of the four field FPGAs (FPGAs 1 thru 4) has a
total of five, 4 Gb DDR3 memories. Each memory is 256M x16 with separate data, address and control. Three of
these DDR3 memories are connected to FPGA pins capable of 800 MHz (1600 Mb/s per data pin) and remaining
two are connected to FPGA pins capable of 400 MHz (800 Mb/s per data pin). The Xilinx Memory Interface
Generator (MIG) works fine. The five memories can be used independently or grouped in any manner that best fits
your application. The Dataflow Manager FPGA (FPGA 0) has a total of six, 4 Gb DDR3 memories. Three of these
memories are connected to FPGA pins capable of 800 MHz (1600 Mb/s per data pin) and three are connected to
FPGA pins capable of 400 MHz (800 Mb/s per data pin).
As always, we provide examples and reference designs to help you with all of your memory interface issues. Please
check with us to make sure that what we ship for no charge meets your requirements.
3 the DINI group

DNK7_F5_8_Cluster Xilinx Kintex 7 Rack Mount HPC Cluster
Power Consumption
The PCI Express specification limits slot power to 25 watts. The DNK7_F5PCIe is capable of consuming power
significantly beyond that. In addition to the PCIe fingers, a separate connector adds a second path for power.
This product is shipped with adequate heat sinks to consume TBD watts, but airflow is required in the chassis to
dissipate the heat. Contact the factory if you require high reliability, no-fan heatsinks.
Status LEDs, Debug
Although no specific testing was performed, sophisticated statistical finite element models and back of the envelope
calculations are showing the number of status LEDs to be bright enough to work as a flashlight. Contact the factory
for more information about this sophisticated feature and make sure an adult is present during operation. These LEDs
are user controllable from the FPGAs so can be used as visual feedback in addition to emergency lighting. A JTAG
connector provides an interface to ChipScope and other third party debug tools.
the DINI group
For technical applications and sales support, call 858.454.3419
7469 Draper Ave.
La Jolla, CA 92037-5026
Phone: 858.454.3419
Fax: 858.454.1728
E-Mail: sales@dinigroup.com
Web: http://www.dinigroup.com
The DINI Group reserves the right to make changes to the product(s) or information contained herein without notice. No liability is
assumed as a result of their use or application. No rights under any patent accompany the sale of any such product(s) or information.
4