Disk server

it.ba.de

Disk server

Herzlich willkommen!

17.4. München

26.4. Düsseldorf

03.5.

Dr. Bernd Panzer-Steindel

Frankfurt


Storage Strategien am CERN

Bernd Panzer-Steindel

Computing Fabric Manager

April 2007

Dr. Bernd Panzer-Steindel


“Conseil Européen pour la Recherche Nucléaire”

“European Organisation for Particle Physics”

Located on top of the French-Swiss

border in Geneva (Switzerland)

2600 Staff members and Fellows

plus 6800 visitors on-site

CERN

Basic Research Laboratory

World’s largest particle physics centre

Founded in 1954

1300 MCHF (800 million Euro)

Annual Budget

Dr. Bernd Panzer-Steindel

www.cern.ch


Europe:

296 institutes

4716 users

Elsewhere:

224 institutes

2059 users

CERN has some 6,800

visiting scientists from more

than 500 institutes and 80

countries from around the

world

Dr. Bernd Panzer-Steindel


Basic building

blocks of nature

Dr. Bernd Panzer-Steindel


Stars and Planets only

account for a small percentage

of the universe !

Dr. Bernd Panzer-Steindel


Open Questions

‣Why are the parameters of the size as we observe them ?

‣What gives the particles their masses ?

‣How can gravity be integrated into a unified theory ?

‣Why is there only matter and no anti-matter in the universe ?

‣Are there more space-time dimensions than the 4 we know of ?

‣What is dark energy and dark matter which makes up 95% of the universe ?

finding the Higgs and possible new physics with LHC will give answers !

Dr. Bernd Panzer-Steindel


Particle Accelerator

The most powerful microscope in the world

‘snapshot of nature’

Creating conditions similar

to the Big Bang

Dr. Bernd Panzer-Steindel


The Large

Hadron

Collider

‣27 kilometer long circle

‣1700 super-conducting magnets

‣7000 km super-conducting cables

‣5000 tons of liquid Helium

The largest super-conducting

installation in the word

9 km

Dr. Bernd Panzer-Steindel

100m below ground


The ATLAS Experiment

Diameter

25 m

Barrel toroid length 26 m

End-wall chamber span 46 m

Overall weight 7000 Tons

Electronic channels 150 million

Dr. Bernd Panzer-Steindel


LHC

1000 million

‘snapshots of nature’

(=events) per second

4 Experiments

We are looking for 1 ‘good’ snapshot

in 10 000 000 000 000 ‘photos’

1000/s 1600/s

Create sub-samples

Filter and first selection

800 selected ‘snapshots’

per second (= 1 CD, 800/s)

to the CERN computer center

The Dataflow

100000/s

World-Wide Analysis

Physics

Explanation of nature

Store on disk and tape

Export copies

0

σ

_

f f

0 sΓ

σ _ ≈σ

_ ×

2

f f f f ( s-m ) +

12π

Γ Γ

=

m

2

Z

ee ff

2

ΓZ

and

2

Z

Γ

ff

2

Z

2 2

s ΓZ

/m

2

z

with

3

G mZ

2 2

= F

× (vf

+ af

) × N


2

col

Dr. Bernd Panzer-Steindel


Tasks

simplify the ‘snapshots’ and extract the physics

all ‘snapshots’ are independent of each other

simplifies the computing , ‘embarrassingly parallel’

there is lot’s of ‘noise’ to be subtracted

need to understand precisely the environment (detectors, accelerator, etc.)

is the measured effect really the underlying basic physics or just

an artifact from the measurement itself ?!

Dr. Bernd Panzer-Steindel


Tides

Stray currents

Precision !

The 27 km length of the ring

is sensitive to


CERN Computing Fabric

Configuration

large scale !

Wide Area Network

Purchasing

Physical installation

Installation

Linux

Tape Server

Mass

Storage

Network

Linux

Service

Nodes

Batch

Scheduler

Monitoring

Node repair

and

replacement

Logistic !

Linux

Disk Server

Shared File

System

Fault Tolerance

Linux

CPU Server

automation !

Electricity

Space

Cooling

1000 boxes per year to be moved

Dr. Bernd Panzer-Steindel


Hardware Building Blocks

commodity market components

not cheap but cost effective !

simple components, but many of them

CPU server

dual CPU, dual core,

8 GB memory

TCO

Tape server

=

CPU server + fibre channel connection

+ tape drive

market trends more important

than technology trends

Disk server

=

CPU server + RAID controler +24 SATA disks

Dr. Bernd Panzer-Steindel


Software ‘glue’

‣ management of the basic hardware and software :

installation, configuration and monitoring system

Which version of Linux ? How to upgrade the software ?

What is going on in the farm ? Load ? Failures ?

‣ management of the processor computing resources :

Batch system (LSF from Platform Computing)

Where are free processors ? How to set priorities between

different users ? sharing of the resources ? How are the

results coming back ?

‣ management of the storage (disk and tape) : CASTOR

(CERN developed Hierarchical Storage Management system)

Where are the files ? How can one access them ?

How much space is available ? what is on disk, what is on tape ?

Dr. Bernd Panzer-Steindel

TCO buy or develop ?!


NAS Disk Server

Charactaristics

~ 5 TB usable disk space

22 SATA disks (server quality)

3 file systems

RAID5 configuration (6+2 disks)

RedHat Enterprise Linux

Total costs ~1.2 Euro/GB

Operation

Disk space is always filled, > 90 %

Mixture of sequential and random IO

Frequent change of access patterns

Disk MTBF measured : ~200000h

two disk replacement per day

Production

600 NAS server with 4000 TB raw space

4 different generations

Assumed life time 3 years

130 TB data moved per day (peak 400 TB)

one million files read or written per day

Dr. Bernd Panzer-Steindel

TCO space, cost and reliability

large scale disk failure investigations :

http://216.239.37.132/papers/disk_failures.pdf

http://www.usenix.org/events/fast07/tech/schroeder/schroeder_html/index.html

Fibre channel and SCSI disks

two times better performance

sequential and random IO

higher MTBF numbers (on paper)

but

4-6 times more expensive CHF/GByte


Network

Everything is based on Ethernet

30000 devices connected on site

160 Gigabits/s

WAN

CERN campus

Backbone

Routers

1000 Gigabit switches from HP

100 10-Gigabit routers from Force10

Disk Server

Dr. Bernd Panzer-Steindel

10Gigabit uplinks

CPU Server

Tape Server

Gigabit Ethernet Switches

1000 Mbits/s


WINDOWS, ORACLE,

FC disks, SAN

Administrative Data

2 million electronic documents

70000 electronic signatures per month

80000 emails per day

250000 orders per year (650 million CHF)

backup

per hour and per day

Users

Storage

accessibility 24*7*52 = always

tape storage forever

LINUX, SATA disks, Gigabit

continuous storage

Physics Data

10000 TB and

20 million files per year

Dr. Bernd Panzer-Steindel


Compliance

‣ Internal audit several times a year

‣ External audit once per year

‣ National laws (Swiss AND France)

‣ EU laws

‣ Contracts with companies from > 80 countries world-wide

‣ ‘meta-laws’ from the physics community

All data and documents need to be always accessible and stored forever

Dr. Bernd Panzer-Steindel


Security

5 person security team

plus security effort for all services

regular security patches, weekly, monthly plus

emergency at any time

several levels of firewalls

detailed automatic

monitoring

250

200

Timeline for Security Incidents May 2000 - August 2006

Blaster Worm

variants

(Windows)

IRC Based Hacker

Networks

(ALL platforms)

encryption of

sensitive data

Number of incidents

150

100

50

Code Red Worm

(Webservers)

Suckit Rootkits

(LINUX)

Compromised

Machines

0

Jan-00 Jul-00 Jan-01 Jul-01 Jan-02 Jul-02 Jan-03 Jul-03 Jan-04 Jul-04 Jan-05 Jul-05 Jan-06 Jul-06

Change

in trend

Non-centrally managed

Systems exposed in firewall

laptops & downloaded

caused most incidents

code caused most incidents

Focus : protection of sensitive data (administration)

hijacking of large clusters

Dr. Bernd Panzer-Steindel


CERN Computer Center

ORACLE Data Base Server

2.5 MW Electricity

and Cooling

240 CPU and disk server

50 TB

80 Gbits/s

CPU Server

8500 processors

Tape Server

and

Tape Library

Network router

Disk Server

100 tape drives , 20000 tapes

10000 TB capacity

Dr. Bernd Panzer-Steindel

600 NAS server, 4000 TB

14000 disks


RAW data

Bookkeeping

RAW data copies

Sub-samples

Enhanced copies

Enhanced copies

‣10000 million ‘snapshots’ created per year

=10 000 000 000 000 000 Bytes = 10 Petabytes

‣Distributed world-wide to over 500 institutes

‣Each and every ‘snapshot’ is catalogued and needs to be traced

Dr. Bernd Panzer-Steindel


Resources for Computing

~120000 processors

today

~100000 disks

~100000 TeraBytes

Dr. Bernd Panzer-Steindel

CERN can only contribute ~15% of these resource

need a world-wide collaboration


Physical and logical coupling

Complexity

Components

10

Hardware

CPU+disk+memory

+motherbord

Software

Operating system

WINDOWS LINUX

PC

30000

Local Area Network

Resource Management

software

Cluster

1000000

Wide area network

Grid middleware

World Wide

Cluster

Dr. Bernd Panzer-Steindel


Solution: the Grid

The World Wide Web provides

seamless access to information that

is stored in many millions of

different geographical locations

Tim Berners-Lee

invented the

World Wide Web

at CERN in 1989

The Grid is an infrastructure that

provides seamless access to

computing power and data storage

capacity distributed over the globe

Dr. Bernd Panzer-Steindel


Grid history

• Name “Grid” chosen by analogy with

electric power grid (Foster and

Kesselman 1997)

• Vision: plug-in computer for processing

power just like plugging in toaster for

electricity.

• Concept has been around for decades

(distributed computing, metacomputing)

• Key difference with the Grid is to realize

the vision on a global scale.

Dr. Bernd Panzer-Steindel


What is the problem ?

I want to analyze the LHC measurements

?

?

Where are the data ?

Where is a free computer ?

How do I access them ?

How do I get the results back ?

Am I allowed to work in this center ?

?

?

?

There are many different centers !

Each one with different hardware and software !

Dr. Bernd Panzer-Steindel


How does the Grid work?

• It relies on advanced

software, called middleware.

• Middleware automatically

finds the data the scientist

needs, and the computing

power to analyse it.

• Middleware balances the

load on different resources. It

also handles security,

accounting, monitoring and

much more.

Dr. Bernd Panzer-Steindel


Grid Projects Collaborating in

LHC Computing Grid

Today

Active sites : > 230

Countries involved : ~ 50

Available processors : ~ 35000

Available disk space : ~ 10 PB

Dr. Bernd Panzer-Steindel

More magazines by this user
Similar magazines