Power and Cooling in the Data Center: Hyper-density ... - OpenMPE

openmpe.com

Power and Cooling in the Data Center: Hyper-density ... - OpenMPE

Power and Cooling in

the Data Center:

Hyper-density Trends and Solutions

Ken Baker

Infrastructure Technologist

MrBlade@hp.com

© 2004 Hewlett-Packard Development Company, L.P.

The information contained herein is subject to change without notice


Modern Datacenter Design

• Raised floor, forced air cooling through perforated panels

Power and network wiring may be under floor or overheard

• Designed for 5-10 year lifecycles

• Not designed for additional growth in power and cooling

capacity

Cooling efficiencies average between 40-50%

2


Average Datacenter Density

• Lawrence Berkley

National Labs study

found no

datacenters in their

study that exceeded

1950W /sq meter

• Historical trends

have shown loading

in the 250-

500W /sq meter

• Customers have not

seen a need to

invest in support for

density until now.

3


Energy Density of Datacenters

Datacenter

capacity is not

rising at the

same rate as

power density in

servers

• Indicates there is

a resistance to

dense

deployments in

volume

4


Customer Landscape

3000 Sanger Institute

Customer Facility Examples

Watts per square meter

2000

1500

Dreamworks Palo Alto

Nacio

Cutten Road

365 Main

1050

950

Site6.2

PNNL

Los Alamos

Verizon

Raytheon

Oracle

850

750

Watts

Site6.1

650

550

450

350

Site4.2

Site4.1

Site3

Site8.1

Morgan Stanley

Goldman

Deutsche Bank

Wells Fargo

BofA

Site8.2

Site2

Site1

State Farm

Citigroup

250

50

Site9

Site7

Size of Datacenter

300 600 1200 2000 3000 5000 9000 14000

5


Datacenter Trends

•Four years ago

• Average servers per rack = 4 – 6

• Average U size was 5 – 7U

• Average watts per rack = 1500 – 3000

•Today

• Average servers per rack = 8-24

• Average U size is moving down to 2U

• Average watts per rack = 5000 – 6000

• Future

Power densities of 1000+ watts per U are imminent

6


Server Trends

Generation over generation density growth rates

DL360 G2 DL360 G3

246 W 389 W

58%

2p, 4gb, 2hdd, 1pci

1U

DL380G2

362 W

DL380 G3

581 W

60 %

2p, 4gb, 6hdd, 2pci

2U

DL580 G1

456 W

DL580 G2

754 W

65%

4p, 8gb, 4hdd, 3pci

4U

7


Intel Processor Roadmap

8+ Way

’03 ’04 ’05 ’06

Common Platform

CSI based

4 Way

Gallatin

2M/130nm

2.0GHz

Gallatin

4M

4M/130nm

3GHz

T case

71 ºC

TDP 85 W

Potomac

8M/90nm

3.6GHz

T case

65 ºC

TDP 115 W

667 MHz

Tulsa

Dual core

8M/90nm

3.4GHz

T case

69 ºC

TDP 175 W

32bit

Whitefield

16M/90nm

TDP 100-130W

DP

Prestonia

512K/130nm

2.8GHz

Prestonia

2ML3

130nm

3.2 GHz

T case

71 ºC

TDP 94W

Nocona

1M/90nm

3.6GHz

T case

71 ºC

TDP 103W

Dempsey

DP DC

65nm

T case

? ºC

TDP 150W

Woodcrest

4M

(mobile dual)

65nm

TDP 70W

400 MHz

533 MHz

800 MHz

8


Impact of Future Dense Server

Designs

• Dense designs drive up the

Watts/ square meter

requirement

• A typical rack requires 1.3

square meters of space

• At bl30p watt densities of

35000 watts per cabinet

(max theoretical), the watt

density for a facility of

bl30p’s could exceed

• >26,000 watts per

square meter!

9


Understanding the Limits of Data Center

Designs

• HP commissioned study

−Impact of increasing power density

−What can a facility successfully support?

−How can we improve the efficiency of a

datacenter?

−Initial data indicates supporting room level

power densities above 2000 to 2500 watts per

square meter is not practical using classic raised

floor designs

10


CM750-FL – BC

11

Y-Plot at 78” from the raised floor

Cold Aisle

Cold Aisle

Cold Aisle

Cold Aisle

Cold Aisle

R1C13-C14 max = 78.5 o F


CM750 - FL – Solved

One extra tile

Layout

PDU 1 PDU 2

Cold Aisle

PDU 4 PDU 3

Legend:

= deactivated tiles = 56% tiles = 25% tiles = 30KW cabinets

12


R1C12 - C13 max = 68.3 o F

CM750-FL – Solved

Y-Plot at 78” from the raised floor

Note: Maximum allowable rack inlet temperature is 77 o F

13


PDU 3

PDU 4

CM1500 – FL - BC

Layout Top View

14

Extra AHU

All floor tiles 25% open

AHU1

AHU2 AHU9 AHU3 AHU4

Hot Aisle

Cold Aisle

Hot Aisle

Cold Aisle

Hot Aisle

Cold Aisle

Hot Aisle

Cold Aisle

Hot Aisle

Cold Aisle

Hot Aisle

AHU8 R3C1 AHU7 AHU10 AHU6 AHU5

Extra AHU

PDU 2

PDU 1


CM1500 – FL - BC

Y-Plot at 78” from the raised floor

Cold Aisle

Cold Aisle

Cold Aisle

Cold Aisle

Cold Aisle

•Max. inlet temperature occurs at R9C1: 84.1 o F

R9C1

R10C1

15


R1C14

CM1500 – FL – Solved – HE

Layout – Top View

Extra AHU

AHU1 R1T14 AHU2 AHU9 AHU3 R9T14 AHU4

R10C14

R1C1

PDU1 PDU2

R2T1

PDU4 PDU3

R2C1

R3T1

AHU8 AHU7 AHU10 AHU6

R9C1

AHU5

Extra AHU

16


CM1500 – FL – Solved – HE

Y-Plot at 78” from the raised floor

•Max. inlet temperature occurs at R9C1: 71.8 o F

R9C1

R10C1

Note: Maximum allowable rack inlet temperature is 77 o F

17


CM3000-FL – BC

Y-Plot at 78” from the raised floor

•R4 – R5 C13: 95.2 o F

R4C13

R5C13

18


CM3000-FL - Solved

Layout – Floor tiles

AHU 10 AHU 9 AHU 8 AHU 7 AHU 6

PDU 1 PDU 2

AHU 11

PDU 4

AHU 12

PDU 3

AHU 5

AHU 4

AHU 3

AHU 2

AHU 1

De-activated tiles

25% open tiles

All other tiles are

56% open tiles

19


CM3000-FL – Solved

Y-Plot at 78” from the raised floor

AHU 10

AHU 9

AHU 8

AHU 7

AHU 6

AHU 11

AHU 12

AHU 5

AHU 4

AHU 3

AHU 2

AHU 1

Highly localized

region of 77.3 o F

All hot air temperatures observed in the base case

have now come down below 68 o F

Note: Maximum

allowable rack inlet

temperature is 77 o F

20


CM150 – FL - BC

Pressure plot at 2” below the raised floor

• Max. 0.11” of H 2 O observed below the supply of AHU

21


Data Center Thermal Modeling

Non-intuitive flow patterns (HPL Palo Alto Smart Data Center Project)

5% open vent tile results in

cooler inlet temperature

Front View

5% open tile

95% open vent tile results in

hotter inlet air temperature

95% open tile

Front View

22


Data Center Thermal Modeling

Dreamworks Redwood City (DL360 G3)

Solution:

This issue was solved by strategically

repositioning inlet air vents (the only

degree of freedom allowed by the

customer)

23


Data Center Thermal Modeling

Impact of load or cooling changes

Modeling has the capability to show the

impact of:

• Turning on new machines

• AC failure

100 %

load

Region Size

• AC shutdown for scheduled

maintenance

Factoid: For a given data center (at 2250

W /sq meter) with an A /C failure, the

reaction time to reduce load is less than ~

35 seconds to prevent redlining & shutdown

50% load

24


Key Findings

• Use of intuition is counterproductive in managing

thermal situations

• Costs to support datacenter designs above 1000

watts/sq meter can become prohibitive

• Harvesting of low hanging fruit lies in improving

existing efficiencies

• Thermal modeling can provide significant energy

savings and allow for increased densities without

spending on new equipment

25


Co-produced by:

More magazines by this user
Similar magazines