Large Scale Biomolecular Dynamics Simulations - prace

XXL-BIOMD
Large Scale Biomolecular Dynamics Simulations
David van der Spoel, PI
Aatto Laaksonen
Peter Coveney
Siewert-Jan Marrink
Mikael Peräkylä
Uppsala, Sweden
Stockholm, Sweden
London, UK
Groningen, Netherlands
Kuopio, Finland
onsdag, 2009 maj 13

Molecular Dynamics
1. Start with a system of particles with given
coordinates
2. Compute forces on each particle due to a
(classical) energy function
3. Integrate the particle positions
4. Save coordinates and energies etc.
5. Go to 2.
Particles != Atoms
onsdag, 2009 maj 13

Protein BBA5:
4000 Waters:
Total:
400 atoms
12000 atoms
12400 atoms
onsdag, 2009 maj 13

onsdag, 2009 maj 13

BBA5 folding: CPU Time
• 12400 atoms
• 100 interactions per atom
• 50 flops / interaction
• 2 x 10 7 integration steps
• 1.24 Petaflop (NOW: 1 day on 8 cores)
• 10,000 copies of the simulation (in 2004
using Folding@Home with GROMACS)
onsdag, 2009 maj 13

Fraction folded structures
Fold Fraction over time
0.6
0.4
Folding time
from slope
4-5 μs
0.2
lagtime
0.0 0
ns
10
ns
20
ns
30
ns
Rhee et al. Proc. Natl. Acad. Sci. U.S.A. 101 (2004) p. 6456
onsdag, 2009 maj 13

Simulation vs. Experiments
Experiments
Efficient averaging
Bond vibrations
torsions
single ion passes
through a channel
Where we
need to be
Less detail
-15 -12 -9 -6
-3 0
3
10 s 10 s 10 s 10 s 10 s 10 s 10 s
Simulations
Extreme detail
Sampling issues?
Parameter quality?
Where we are
Fast protein
folding
Normal protein
folding
Where we
want to be
Interesting
protein folding
Biologically
interesting stuff
onsdag, 2009 maj 13

GROMACS
• The world’s fastest MD code - and it’s GPL!
• Estimated 5,000-10,000 academic and
industrial users
• Used in Folding@Home - 250,000 CPUs
(2008)
• Part of SPECfp & PRACE benchmark suites
• PRACE project for improvements on
GROMACS (CSC, Erik Lindahl)
onsdag, 2009 maj 13

DPPC & Cholesterol:
130k atoms
BlueGene: 6ns/day,
using 2000 CPUs
GROMACS only
achieves 2ns/day ...
...on a single dual dual
core Opteron node!
onsdag, 2009 maj 13

Parallel Domain Decomposition
• Partition space, instead of atoms, over
nodes
• Supported in version 4 of Gromacs
• Good for load balancing
• Bad for communication bandwidth
• Each node ‘imports’ coordinate and
exports forces from neighbors within a
sphere with radius=cutoff (expensive)
Data must be imported
from whole sphere, although
it can be optimized to half
onsdag, 2009 maj 13

The Eighth-Sphere Method
• Smarter way to communicate
•
Don’t calculate interactions on a
home node, but in general on
“neutral territory” (David Shaw)
• Drastically reduced communication
bandwidth needs for dom. dec.
• 2D example to the right
•
In 3D, we need to import data
from 1/8 sphere to the central cell
cells send data to
central (purple) cell, where
interactions are calculated
Red/Yellow
• Implemented in GROMACS 4
onsdag, 2009 maj 13

Efficient PME Parallelization
• Almost all accurate simulations today
use Particle-Mesh Ewald lattice
summation
• Small 3D Fourier Transforms scale
bad - all-to-all communication
• Direct space & PME are mostly
independent, though!
• Dedicate a subset of nodes to run a
separate PME-only version of the
program to improve scaling
X
Y
• FFT over 5 instead of 25 nodes!
Original implementation with
help of RZG of MPI
PME nodes
onsdag, 2009 maj 13

Scaling - DHFR 23558 atoms
•1 fs time step
•Constraints on H-
Bonds
•PME every other step
(NAMD, Desmond)
resp. every step
(GROMACS)
Hess et al. J. Comp.Theor. Chem. 4 (2008) 435-447
onsdag, 2009 maj 13

Blue Gene/P scaling
200
0.43M atoms
60
•Coarse grained system
•Cut-off 2.6 sigma
steps / second
100
3.4M atoms
40
20
steps / second
0
0 512 1024 1536 2048
#cores
0
Amdahl’s law:
•0.43 Matoms on 1024 cores - 28% time in global sum
•3.4 Matoms on 2048 cores: 33% time in global sum
onsdag, 2009 maj 13

Deisa Extreme
Computing Initiative
• Larger systems
• Longer simulation times
onsdag, 2009 maj 13

Dynamics of a
Virus Capsid
Daniel Larsson
Lars Liljas
David van der Spoel
Uppsala University
Sweden
onsdag, 2009 maj 13

Why study viruses?
• New viral diseases may be on the
way - Bird flu, SARS, Mexican
• 40 existing drugs against viruses -
20 against AIDS
• Most drugs target viral
reproduction cycle (reverse
transcriptases, proteases)
• Interesting features - self assembly
• Packaging of RNA/DNA
onsdag, 2009 maj 13

Satellite Tobacco Necrosis Virus
• Discovered 1967
• Icosahedral plant
virus
• Satellite virus, TNV is
the helper
• Transmitted by a
fungus
• Small: 18 nm
diameter
• Simple model system
onsdag, 2009 maj 13

5 ns simulation. Blue: protein. Green: Cl - . Red Na +
Computer time estimate: 12 Petaflop
onsdag, 2009 maj 13

What can simulations
contribute with?
• Non-averaged aspects
• Non-symmetrical aspects
• Missing pieces of the structure (Res. 1-11)
• Effect of structural Ca 2+ on stability
• RNA binding / Salt effects
• Dynamics – the fourth dimension
onsdag, 2009 maj 13

onsdag, 2009 maj 13

RNA Secondary Structure Prediction
Bringloe et al. J. Gener. Virol. 79 (1998) p. 1539
onsdag, 2009 maj 13

The goal is not merely reproducing
experimental results.
onsdag, 2009 maj 13

Unfortunately, reproducing
experimental results is difficult.
onsdag, 2009 maj 13

Simulation Details
• 1,000,000+ particles
• OPLS/AA force field + TIP3P water
• Particle mesh Ewald
• 5 fs timestep
• Dodecahedron simulation box
onsdag, 2009 maj 13

Hardware Specs
Louhi (CSC,
Espoo)
HECToR
(Edinburgh)
Neolith
(Linköping)
Vendor Cray Cray HP
CPU type
AMD Opteron
2.3 GHz
AMD Opteron
Intel Xeon E534
(2.33 GHz, 8
MB L2 cache)
#Cores 10800 12000 6440
Interconnect Cray Cray
Infiniband
ConnectX
OS Cray Linux Cray Linux Centos 5 Linux
Top500 (11-08) 32 47 56
Price ? ? ?
Time (2008) 400 kh (DECI) 800 kh (DECI) 3600 kh (VR)
onsdag, 2009 maj 13

Superposition of 60
monomers after 100
ns MD
Superposition of 12
pentamers
after 100 ns MD
•Main part of the protein acts like a rigid body.
•Residue 1-24 forms a flexible arm.
onsdag, 2009 maj 13

onsdag, 2009 maj 13
500 ns: ~1.2 Exaflop, 30 core-years

onsdag, 2009 maj 13
Size of the virus particle

Water flow analysis
• Is the virus capsid leaky?
• Can water molecules pass?
• Can ions pass?
• Is the flow concentrated to specific
regions?
onsdag, 2009 maj 13

onsdag, 2009 maj 13
Physiological Conditions

Conclusions
• N-terminal arm is flexible
• Ionic strength is important for stability but
may not be enough. RNA!
• Capsid is very leaky
• (interfering with capsid stability is a
possibility for antiviral therapy, but not
proven yet)
onsdag, 2009 maj 13

Questions & future
plans...
• What structural changes facilitate the
water flow?
• The RNA structure - 2D to 3D structure
• Affinities (ΔGbind) between the proteins -
quantitative stability analysis
• Pathways for virus assembly (simplified
models?)
onsdag, 2009 maj 13

Extreme Computing
• If you can not solve a small problem, try a
bigger problem
• How about accuracy of the results & the
predictive power?
• HPC will force many codes to reconsider the
physics
• Amdahl’s law needs to be taken care of in MD
onsdag, 2009 maj 13

Molecular Biophysics, Uppsala
Daniel Larsson
onsdag, 2009 maj 13