IBM Intelligent Bricks - - UNM Center for Advanced Research ...

Project Scope• IceCube/Kybos or “Intelligent Bricks” project formally originatedin IBM Almaden in 2001– Collaboration between IBM’s Almaden Research, Yorktown CSS, E&TS(Mainz), and ~ 20 external vendors• It is a research project, not an IBM product• It’s all about ‘Brick-based’ architectures– It’s not just about cool hardware !– Integrates several new ideas in system hardware, software,communications technology, scalability, reliability and managementinterface designARC External Website, Intelligent Bricks, 2005

Team Members• Winfried W. Wilcke, Claudio Fleiner, Robert Garner,, RichardGoulding, Jody Glider, Deepak Kenchammanna-Hosekote, TedWong, Ralph Becker-Szenday, Omer Zaki, K.K. Rao, Jim Hafner,Manny Hernandez, Ken Fernandez, Moidin Mohiuddin, (IBMAlmaden Res.)• Harald Huels, Klaus Smolin, Manfred Ries, Heinz Lenke, GabyPersch-Schuy (IBM E&TS Mainz)• Barry Rubin, Tom Picunko, Vincent Arena (IBM Watson Res.)ARC External Website, Intelligent Bricks, 2005

Storage Enterprise Customer’s Top Issues• Easier System Management => reduced Total Cost of Ownership• Better Scalability - but at low cost• Better Environmentals– floor space, power, cooling and noise• Better Reliability and Availability• Goal: Order-of-Magnitude Improvements…ARC External Website, Intelligent Bricks, 2005

How much (unique) Information ? (2002)• Gigabyte => Terabyte => Petabyte => Exabyte => Zetabyte =Yottabyte10 9 10 12 10 15 10 18 10 21 10 24• Worldwide phone calls: 17.3 Exabytes (15% wireless)• Radio & TV new programs 3.5 & 69 Petabytes, resp.• E-mails per year 400 Petabytes (11 PB in ’99)• New Home Videos per year: 2600 Petabytes ( 400 PB in ’99)• Office Documents per year: 195 Petabytes (’99)• World Wide Web (surface) 170 Terabytes• World Wide Web (deep) 80 Petabytes• Medical images (digital , USA) 80 PetabytesIn 2003, 5 Exabytes of unique new information was created ** Source: UC Berkeley School of Information Mgt. Systems Study etalARC External Website, Intelligent Bricks, 2005

ARC External Website, Intelligent Bricks, 2005Keep things as simpleas possible(but not simpler)

Zen of Simplicity• Be transparent – no surprises– Give a coherent mental model of how the thing works• Restraint– Cut non-essential features• Modularity• Encapsulate unavoidable complexity=> Small number of building block typesARC External Website, Intelligent Bricks, 2005

Brick Architectures for Simplicity (1)• Encapsulate hardware into Lego-like bricks– Each brick contributes to processing, storage, & networking• Small number of brick types– That is very different from today, where disks, controllers, switches etcall are different (field replacable and customer visible) partsARC External Website, Intelligent Bricks, 2005

Brick Architectures for Simplicity (2)• Long-lived, public interfaces– Network protocol, Physical, Interfaces,– Careful architecture work required to define ‘families’ of interoperablebricks and relations between families• Bricks should be low cost (use commodity components)– Minimize redundancy within a brick– Deal with failures at a higher, system level• Any brick must be allowed to fail without noticeably impactingsystem operation, as long as ‘most’ of the bricks are alive– System software must deal with and hide the failure– “Fail-in-place’ a natural extensionARC External Website, Intelligent Bricks, 2005

Generic 3-D BrickNo connectors, fans, cables, fibers, lasers...Commodity components @ low costHeat (out)Power (in)Communications(in,out)zyxARC External Website, Intelligent Bricks, 2005

Intelligent Bricks: PrototypeARC External Website, Intelligent Bricks, 2005

Storage Brick - Prototype20 cmARC External Website, Intelligent Bricks, 2005

Intelligent Bricks PrototypeARC External Website, Intelligent Bricks, 2005External Communicationthrough cube surface(s)

Storage Brick Block Diagram (Prototype)Coupler3 x4-wayCTRLPCI-BUS12 80 GB Disks (2.5”)CPUInterfaceCouplerDRAM512MB8-PortSwitchChipCouplerCouplerCouplerCouplerGatewayexternalworldARC External Website, Intelligent Bricks, 2005

Intelligent Bricks Brick Internals - PrototypeARC External Website, Intelligent Bricks, 2005

Multi-Petabyte Design Mockup960 Bricks4-6 PB in 2007ARC External Website, Intelligent Bricks, 2005

Intelligent Bricks: Cooling Structure10x10x7 bricks(example)Cold RailorBricksPower baseCoolant AirARC External Website, Intelligent Bricks, 2005

Fluid CoolingARC External Website, Intelligent Bricks, 2005

Traditional Air-Cooled Server RoomChillerA/CairRacksairA/COffice A/C...Coolant pipes“Air-cooled” data centers include chilled coolant pipes.ARC External Website, Intelligent Bricks, 2005

Reliability Concepts in Brick Architectures“The Biological Muse”ARC External Website, Intelligent Bricks, 2005

A system (‘747’) may be more reliable thancomponents (‘engines’)ARC External Website, Intelligent Bricks, 2005

Reliability through Redundant Components• Not a new story – cells,neural networks, internet, airplanes, RAID• Bricks analogous to Cells– Any Individual brick may break – as long as “most” survive.• One can build reliable systems out of commodity componentsNeural NetworkInternet Routers Intelligent BricksARC External Website, Intelligent Bricks, 2005

“Maintenance-Free” Brick Systems• The following slides walk you through how one can builda brick-based storage system which requires no ongoingmaintenanceARC External Website, Intelligent Bricks, 2005

Red brick about to fail…livedeadabout to failBackup datafor red brick(data copies, parity…)ARC External Website, Intelligent Bricks, 2005

Data rebuild complete…livedeadrecently diedExisting backupdata for red brick2 nd generationBackup dataARC External Website, Intelligent Bricks, 2005

Systems without on-going maintenance• Leave a failed brick in the system– Until convenient to replace, which may be never• Software must deal with failed brick without humanintervention• Architecture must ensure that a failed brick is forced off-line• Pulling the wrong RAID disk is a dominant cause of data loss• We presented this no-maintenance concept to ~70 customers– Overwhelming majority loves the concept– All want to have visibility into the system state• Note: Grid computing must deal with disappearing resources,tooARC External Website, Intelligent Bricks, 2005

Simulation of Imperfect 3-D MeshMeasures of goodnessSYSTEM DYINGSYSTEM OKA typical plot relating a measure of performance on the y-axis to the percentage of live bricksin a 3-dimensional mesh system. In this case, up to 40% of bricks may die before the system isnon-linearly degraded. This result is typical for all relevant performance metrics in 3-Dsystems. 2-D systems are less resilient.ARC External Website, Intelligent Bricks, 2005

Reliability ProjectionsARC External Website, Intelligent Bricks, 2005

Operating Time before Required MaintenanceYears between MaintenanceCompute BrickStorage BrickFailure Rate per YearNote: 4% “storage” brick failure rate is 2% for disks plus 2% for electronicsARC External Website, Intelligent Bricks, 2005

Reliability via “Distributed RAID”• Failed disk/brick’s data are rebuilt across all bricks, utilizing sparecapacity• For an extreme target of 0.5 failures per ExaByte-year (1 data lossevent in 5 years in 25,000 100-disk systems), KK Rao & Jim Hafner@ARC have recommended a choice of 3-failure-tolerant, distance-4redundancy codes:• 1-failure-tolerant (RAID5) in bricks and 2-failure-tolerant (“Dist 3”)code between bricks– 40 - 75 % storage efficiencyOR• 0-failure-tolerant (NoRAID) in bricks and 3-failure-tolerant (“Dist 4”)code between bricks– 50 - 75 % storage efficiencyARC External Website, Intelligent Bricks, 2005

Example Markov Failure Model(Distance 3, No RAID in brick)03µ N1Nκ N(N-1)dκ d (1-h Nd )µ dµ N4Ndκ dµ d2(N-1)κ N (1-h NN )(N-1)κ N (1-h dN )5µ N(N-2)(dκ d +κ N )(N-2)(dκ d +κ N )(N-1)(dκ d h Nd +κ N h NN )(N-1)(dκ d h dd +κ N h dN )(N-2)(dκ d +κ N )7(N-1)dκ d (1-h dd )µ d6(N-2)(dκ d +κ N )ARC External Website, Intelligent Bricks, 2005

Distributed RAID Scheme Reliabilities1.E+02Data Loss Events per PB-Year1.E+011.E+001.E-011.E-021.E-031.E-041.E-051.E-061.E-071.E-08BrickNoRAIDBrickRAID5Target1.E-09Dist 2 Dist 3 Dist 4Hamming Distance (Fault Tolerance + 1)ARC External Website, Intelligent Bricks, 2005

Sensitivity to Electronics Failure Rate1.E+00Data Loss Events per PB-Year1.E-011.E-021.E-031.E-041.E-051.E-061.E-071.E-08Dist 3, 100kHr-Disks,NoRAID in BricksDist 3, 750kHr-Disks,NoRAID in BricksDist 3, 100kHr-Disks,RAID5 in BricksDist 3, 750kHr-Disks,RAID5 in BricksDist 4, 100kHr-Disks,NoRAID in BricksDist 4, 750kHr-Disks,NoRAID in BricksTarget1.E-09100,000 250,000 500,000 750,000 1,000,000 1,250,000Brick Electronics MTBFARC External Website, Intelligent Bricks, 2005

Observation• Not everyone wants very high reliability, because it isn’t free• Distributed RAID gives users choices in a reliability/costtradeoffARC External Website, Intelligent Bricks, 2005

Intelligent Bricks NetworkARC External Website, Intelligent Bricks, 2005

Intelligent Bricks’s 3-D MeshNo (expensive) External Switch -Switch Chip in each Brickgoodbad≥1 GigaByte/seceach direction,each linkARC External Website, Intelligent Bricks, 2005

Internal Routingz+y+z+y+x+SwitchSwitchChipChipx-x+SwitchSwitchChipChipx-Not Involvedy-z-CPUy-z-CPUSource BrickIntermediate Brickz+y+DestinationBrickx+SwitchSwitchChipChipx-y-ARC External Website, Intelligent Bricks, 2005z-CPU

PerformanceARC External Website, Intelligent Bricks, 2005

Performance• The Intelligent Bricks architecture is designed for extremely highperformance– Scalability, bandwidth, power, cooling capability etc.– Larger cubes support many 100’s of GB/s of internal and external BW– It would be quite straightforward to build a PetaFLOP Cube• The prototype uses five year-old chips– Even so, the prototype Cube could –for example--store 5000 DVDquality movies and stream 900 simultaneously at full Mpeg2 (DVD)quality.– It is about 10x slower (per brick) than what could be built todayARC External Website, Intelligent Bricks, 2005

Cube 3D-Network PerformanceCube Peak & Average Internal & Surface BandwidthsBandwidth (GB/s, one-way)80070060050040030020010006-Surface BWOut->In Rand BWStorage Total BW3-Surface BW2-Surface BWHistoricalSupercomputer I/O1000:1=Flops:IO/s3x3x34x4x45x5x56x6x67x7x78x8x69x9x5Cube Dimensions8x8x89x9x910x10x10Assumes 500-MB/s aggregate disk bandwidth per brick.ARC External Website, Intelligent Bricks, 2005

Capacitive CouplerARC External Website, Intelligent Bricks, 2005

Brick-to-Brick Capacitive Coupling• Bit-serial data is transferred via capacitive couplers on brick surfaces– Differential coupler pair for receive and differential pair for transmit– 4 channels for 10 Gb/s Ethernet/Infiniband/FC• Capacitors formed by small, insulated plates gently touching• No active electronics required at couplerSwitchChipSwitchChipARC External Website, Intelligent Bricks, 2005

Capacitive 10-Gbit/s (total) Half-Coupler4 differentialinput pad pairs@3.125 Gbps each4 differentialoutput pad pairs@3.125 Gbps each10 Gb/s XAUI Ethernetoperates on 4 parallellanes @3.125 Gb/seachARC External Website, Intelligent Bricks, 2005

Capacitive Coupler Differential PairARC External Website, Intelligent Bricks, 2005

Why Capacitive Coupling ?• How else would you do it in a 3D brick geometry ??– Conventional connectors don’t work in this geometry– Optics too expensive, alignment sensitive• Properties….– Fast and very low cost– Rugged– FCC/EMI properties need to be studied more closely, solvable issue– High-pass filter– Blocks DC, low frequencies, and stops ground loops– Assumes universally used 8b/10b signal encoding• Invented by IBM, used in Ethernet, Fibrechannel, Infiniband…• Never more than consecutive five 1’s or five 0’sARC External Website, Intelligent Bricks, 2005

EMSim Coupler Frequency Responses0-2-4-6-810 100 1000 10000 1000003.125-Gbit/channelCoupler8mm x 10mmAirGap ~ 7.0 µm(used in 10 Gb/s links)12.5-Gbit/chCoupler Design(for 40 Gb/s links)ARC External Website, Intelligent Bricks, 2005

3.2 Gbit/s/ch (=10 Gb/s Ethernet) Eye DataEYEARC External Website, Intelligent Bricks, 2005

“Deliberately Noisy” TestbedARC External Website, Intelligent Bricks, 2005

Intelligent Brick SoftwareARC External Website, Intelligent Bricks, 2005

Practical Experience in Prototype• Capacitive couplers have been functioning flawlessly in theprototype• FCC compliance studies are to be done– initial field strength measurements are encouraging.ARC External Website, Intelligent Bricks, 2005

Software Goal“A distributed, scale-out file service,running on brick-based systems,that dramatically decreases administrative costsand increases system efficiency.”By Jody Glider, Mgr. Kybos Software ProjectARC External Website, Intelligent Bricks, 2005

Software for Intelligent Bricks (today)– Kybos– GPFS– Operating EnvironmentKybosRouterLinuxGPFSApplicationsServers/Hosts(eg. Blue Gene)Sys-AdminIceCube Firmware & BIOSHardwareARC External Website, Intelligent Bricks, 2005

Operating Environment: Control StructuregoodbadBricks check each otherin-band and dynamicallydetermine topology.Receives in-band“I’m alive” messagesfrom all bricksBaseForces shutdownreboot of badbricks (out-of-band)Redundant N+1 DC power supply& Sensor & Control UnitARC External Website, Intelligent Bricks, 2005“007”Monitor Processor(license to kill bricks)

Operating Environment• Linux (2.4) based• Dual images + software upgrade server for easier upgrades• Monitor processor plus Opteron22 system manages sensors, power,side-band network• Position server maps brick addresses onto brick x,y,z co-ordinates• Cube shortest path routing protocol (not spanning tree)ARC External Website, Intelligent Bricks, 2005

Screen Dump of Cube MonitorARC External Website, Intelligent Bricks, 2005

Higher Level Software today (2005)• GPFS provides the fundamental I/O functionality– Reads/writes data from/to disks, replicates data as required, providesparallelism and interface to external world, etc.– Each brick contains one instance of GPFS’s NSD (Network StorageDevice) service• Kybos provides self-management– Translates the human-intuitive language of ‘resource pools’ into GPFSactions and resource configuration– Provides system monitoring services for its own and human use– Adapts to failures and new resources by controlling GPFS configurationand actionsARC External Website, Intelligent Bricks, 2005

IntelligentBricksSoftwareOverviewAdministrator workstationWeb browserKybosadministrationrequestsand responsesCubeBluegene(or other applicationserver)GPFS remotemountGPFSCube brickGPFS I/O requestsand dataGPS Network StorageDeviceBrick sensor readingand reportingBrick mesh routingcontrolLinux OSNetwork configprotocolsSensor monitoringBrick controlCube management processorWeb server and userinterfaceEvent handling andrecoveryBase sensormonitoringEvent determinationSensor aggregationand storageGPFSCube DHCP serviceCube power controlBrick position serviceCube sensormonitoringLinux OSARC External Website, Intelligent Bricks, 2005

Background: What is GPFS ?IBM’s shared disk, parallel cluster file system.Product available on RS/6000 SP supercomputer and on Linux clusters.Used on many of the largest supercomputers in the world.• Cluster: 2000+ nodes, fastreliable communication,common admin domain.• Shared disk: all data andmetadata on diskaccessible from any nodethrough disk I/O interface.• Parallel: data andmetadata flows from all ofthe nodes to all of thedisks in parallel.GPFS File System NodesSwitching fabric(System or storage area network)Shared disks(SAN-attached or networkblock device)ARC External Website, Intelligent Bricks, 2005

Administrative Cost ReductionExample: Add physical storage & make available to apps:Current Systems…Intelligent Bricks…• Define requirements• Define requirements• Install storage drawer(s) in rack• (possibly) install disks in drawers• Add brick(s) to cube• Modify resource pool• (possibly) install SAN switches• Attach power and network cablesSys Administrators tasks should scalewith # of applications, not # of disks !• Configure RAID arrays• Create virtual disks– size, LUN masking• Initialize virtual disks for use in file system• Define or modify storage pool(s) to include newstorage• Modify SAN switch zoning• Modify virtual disk LAN masking• Initiate rediscovery of SAN devices inapplication servers• Modify application quotasARC External Website, Intelligent Bricks, 2005

Adminstrative Cost ReductionESS37 days12 6 19requirementsdesignmanagement & implementationKybos19days12 2 5-18 days-48%Summary comparison chart*Thanks to NIWS Corporation whichanalyzed this task indetail and allowed us to usetheir chartsARC External Website, Intelligent Bricks, 2005

Kybos Resource Pools• A pool is a virtual collection of storage visible to applications andusers• Defined by a set of requirements– CapacityResour cepool– Performance– ReliabilityAPBrickAPBrickAP• System maps pools onto physical hardwareBrickBrick– Taking requirements into accountARC External Website, Intelligent Bricks, 2005

Kybos Self-management Monitor (Simulator)System of 5 bricks &2 pools: 1 brickfailed, data is rebuildon 4 remaining bricksARC External Website, Intelligent Bricks, 2005

Overall Cube Software Structure (May 2005)Active instance of Kybos ManagementARC multi-Gigabit LANInactive instanceGPFSNSDKybosMgt.GPFSNSDGPFSNSD. .GPFSBlueGeneGPFSNSDGPFSNSDGPFSNSD. .GPFSNSDGPFSNSDGPFSNSD.GPFSMore Host Servers. . .Active and replicated instances ARC External of Website, resource Intelligent database. Bricks, 2005 Any brick cantake over the management function any time, since it knows the system status.

Intelligent Brick Futures• Intelligent Bricks is still an IBM research project– It may or may not become a product• The following are examples of brick systems under discussionARC External Website, Intelligent Bricks, 2005

An Intelligent Bricks Family Vision *10 PBToo small….oneBrick Tower Wall Cube1PB100 TB10 TB11..8 bricks6..32 bricks6..2000bricksAir-cooledLiquidcooled ?Common Host/User Interface1 TBNon-sealed bricksw/ internalRAID‘Sealed, internally non-redundant bricksSame bricks may be used in water or aircooled systems0D 1D 2D 3D* This is not an IBM product plan or roadmapARC External Website, Intelligent Bricks, 2005

Modular Cubes @ 1/4 PB each (2007)Air-cooled versionAirself-aligningcapacitivecouplers32 Bricks= 1/4 Petabyte @ 1TB/diskAir(Option)2x4 Basewith Blower Arrayand N+1 Power UnitARC External Website, Intelligent Bricks, 2005more modularBases…

Intelligent Brick Compute Servers• The Cube architecture is particularly well suited for high-poweredcompute servers– Extreme cooling capacity (project several hundred kW/m 3 )– TeraBytes/second bandwidth in large systems– Dense packaging– It would be rather straightforward to build a Petaflop Cube• Storage and compute bricks should be mixed in the same cubeARC External Website, Intelligent Bricks, 2005

Compute Brick (conceptual)40 Gb/s couplersSkin4-12 -way CPU Brick2-4 kW/brick“Heatbox”(= boxshapedheatpipe)Graphite-foam filledmetal can (“book”)RAMBackplaneDC-DC convertersw/ internal switch Cell CPU’sColdrailARC External Website, Intelligent Bricks, 2005

External References• Two major papers on intelligent bricks will be published at the IBMJournal of R&D, Special Issue on Exploratory Systems, April 2006• Numerous references on the web– Google ibm intelligent bricks or ibm icecube• Computerworld First Horizon Award, 2005ARC External Website, Intelligent Bricks, 2005

Summary• Google keywords: ibm icecube or ibm intelligent bricks• It is NOT about packaging, it’s about software, hardware and management ofbrick based architectures• Has biologically inspired system concepts ( bricks = cells )• Prototype 3x3x3 system : Fully functional since 1/2005• Software: Kybos+GPFS+Operating Environment• Platform for Storage and Data-intensive Computing– Extremely scalable (multi Petabytes)– Easy to manage• Customer question #1 ‘when can one buy it’ ?Thank you ! The Intelligent Bricks teamARC External Website, Intelligent Bricks, 2005