Multi Region failover scenarios in the Cloud - DrupalCon Munich 2012

Multi Region failover scenarios in the Cloud - DrupalCon Munich 2012

Bonus agenda:Basic DemoMultiregion-Shot recipe:01- 2 cloud servers (AWS us-east)- 2 cloud servers (AWS eu-west)- Ubuntu/Linux- Gluster + Unison- Lamp Stack- Drupal- Tungsten

Why the cloud?

Scaling Drupal"It is not always easy toscale Drupal -- notbecause Drupal sucks,but simply becausescaling the LAMP stack(including Drupal) takesno small amount ofskill."Dries Buytaert

in a Cloudinfrastructure...

..or movingInfrastructure?

A site is a livingthing,always changingand evolving!

A spike canruin the day

ut a bad setupcan ruin themonthHTTP/1.1 503 Service UnavailableDate: Mon, 04 May 2012 22:32:02 GMTServer: NginxCache-Control: max-age=1Expires: Mon, 04 May 2012 22:32:03 GMTContent-Length: 2615Connection: closeContent-Type: text/html

Are you takingchances?

Cloud computing- Infrastructure as a Service (IaaS)- Platform as a Service (PaaS)- Software as a Service (SaaS)

Cloud: RegionsUS West(NorthernCalifornia)US West(Oregon)US East(NorthernVirginia)SouthAmerica(São Paulo)Europe(Ireland)Asia Pacific(Singapore)AsiaPacific(Tokyo)AWS Regions

Availability Zones & Regionsdev1DevCloud

Availability Zones & Regionsfsdb1web1fsdb2web2 bal2bal1Managed Cloud

Availability Zones & Regionsfsdb1bal1web1bal3web3fsdb3fsdb2web2fsdb4bal2web4bal4Multi-RegionManaged Cloud

The Cloud Providers

Cloud providersAmazon Ec2ReliacloudGoDaddyRackspaceGoGridSoftlayer OpSourceNephoscaleHosting.comSource:

Dedicated Drupal Cloud Hosting

Acquia01Dedicated Drupal Hosting

Acquia Managed Cloud:By The Numbers600 customerson 2300 Servers7.7 BILLION+ requestsserved per month1.4 BILLION+ page viewsper month133+ terabytes of contentdelivered per month90Mrequests inone day73Mrequests inone day51Mrequests inone dayLarge Media &EntertainmentorganizationGlobal politicalmagazineLarge newsorganization

CloudApplicationLAMP StackVirtual InfrastructureNetworkPhysical MediaPhysical SecuritySecurity Availability Compliance

PantheonNew dedicated DrupalCloud Hosting.- Recently raised $5M in Series Afinancing, led by the FoundryGroup.- Targeting on SaaS (Software asa Service) based on Drupal.Promising!

Omega8.ccHosting startup was born in 2009.- Platform is based on AEGIR and supports it.- Actively contributing to Barracuda and Octopusinstallers.- Flag on speed and performance.

Follow the discussion on:

The Clouds OutagesThe hard journey of hosting

A very largenumber ofcompaniesdon't have adisasterrecovery plan

21 April 2011Major outage causing a huge lackof confidence on AWS.Providers like Rackspace take the opportunity topromote their services.

December 2011Mass preventive relaunches dueto instance degradation.

June 2012Massive outage damaginghundreds of data volumes andinstances.US Virginia's datacenter hit by astorm.

Plan for failure!

Production scenario

Staging Scenario


The Failover

DNS / CDNInternetCDNRegion AIP: BIP:

DNS / CDNInternetDNS withlow TTLRegion AIP: BIP:

EIP / DNS01Always use Elastic IP’s• EIP's guarantee instance IP and names stick on restart.• Use the EIP internal name for consistent DNS results.

The Replication Software

GlusterFSGlusterFS is a scale-out NAS file system. It is freesoftware, licensed under the GNU GPL v3. Itaggregates various storage servers over Ethernet orInfiniband RDMA interconnect into one large parallelnetwork file system. It has found a variety ofapplications including cloud computing, biomedicalsciences and archival storage.

GlusterFSSERVER 1:$sudo mkdir /mnt/brick1$sudo mkdir /mnt/gfs$sudo apt-get -y install glusterfs-server$sudo service glusterfs-server startSERVER 2$sudo mkdir /mnt/brick2$sudo mkdir /mnt/gfs$sudo apt-get -y install glusterfs-server$sudo service glusterfs-server start$sudo gluster peer probe SERVER-1$sudo gluster peer statusSERVER 1:$sudo gluster peer probe SERVER-1$sudo gluster peer status

GlusterFS$sudo gluster volume create volume1 replica 2$sudo gluster volume info$sudo gluster volume start volume1$sudo mount -t glusterfs /etc/glusterd/vols/volume1/volume1-fuse.vol /mnt/gfs/ -o rw,allow_other,default_permissions,max_read=131072,_netdevSERVER 2:$sudo mount -t glusterfs /etc/glusterd/vols/volume1/volume1-fuse.vol /mnt/gfs/ -o rw,allow_other,default_permissions,max_read=131072,_netdevFSTAB (for boot mount)/etc/glusterd/vols/volume1/volume1-fuse.vol /mnt/gfs/ fuse.glusterfs rw,allow_other,default_permissions,max_read=131072 00

VideoP2- Gluster Install

GlusterFSPrepare EBS volumes for Performance• Network connected storage• 1Gb/sec assuming no noisy neighbors• Per volume limit of ~20MB/sec for I/O => ~32KB• Slows to ~2MB/sec @ I/0 ~2KB• Always build mdadm arrays of 8 devices• ~80MB/sec with a single, 8 volume array.

Unison ReplicationConfiguration?! AH!

Unison ReplicationSetup cron to run every x minutesunison ssh://Server4//var/www /var/www \-rsync -times -prefer=newer \-auto -batch \-ignore "Path tmp/*" \> /tmp/unisonrepliction.log 2>&1

VideoP3- Lamp Unison Drupal

Tungsten ReplicatorDatabase replication

Tungsten ReplicatorTungsten Replicator is a high performance, open source,data replication engine for MySQL. It offers a set offeatures that surpass any open source replicatoravailable today: global transaction IDs to supportfailover, flexible transaction filtering, extensibletransaction metadata, sharding, multiple replicationservices per process...

AlwaysOnlineProcessing services command...NAMEVALUE---- -----appliedLastSeqno: 50844930appliedLatency : 0.527role: masterserviceName : fsdbmeshAserviceType : localstarted: truestate: ONLINENAMEVALUE---- -----appliedLastSeqno: 745199appliedLatency : 0.431role: slaveserviceName : fsdbmeshBserviceType : remotestarted: truestate: ONLINENAMEVALUE---- -----appliedLastSeqno: 16599appliedLatency : 0.781role: slaveserviceName : fsdbmeshCserviceType : remotestarted: truestate: ONLINENAMEVALUE---- -----appliedLastSeqno: 16619appliedLatency : 4.678role: slaveserviceName : fsdbmeshDserviceType : remotestarted: truestate: ONLINEFinished services command...

Tungsten Replicatorsql_mode on settings.php(...)"driver" => "mysql","database" => "jblogsmr","username" => "jblogsmr","password" => "zzzxzzzzxzxzzxxzz","host" => "fsdbmesh-aaaa","port" => "$mysql_port","prefix" => "","init_commands" =>array ("sql_mode" => "SET sql_mode=\"REAL_AS_FLOAT,PIPES_AS_CONCAT,IGNORE_SPACE,STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER\"",),

VideoP4- Tungsten Install

Tungsten ReplicatorAWS Performance tips:Use arrays of Elastic Block Storage volumes• Dramatically improves performance.• Consider pre-warming the array• Raid Arrays of 8 volumes gives better timings.• Use ec2-consistant-snapshot for snapshots. the largest instance size in each class• Limits the potential impact of a noisy neighbor.• m1.xlarge, m2.4xlarge

VideoP5- Installation end

"Do it yourself"Scenario.

Self-hosted + scalable site?Admiral AckbarBattle of Endor in Return of the Jedi

or the smart choice:

With whatever you do...DevOpsMulti Region failover scenariosin the CloudChoose H.A. in...Ricardo AmaroCloud Systems Engineerricardo.amaro@acquia.comthe Cloud

Multi Region failoverscenarios in the CloudQuestions?ricardo.amaro@acquia.comtwitter: @ricardoamaro

Multi Region failoverscenarios in the CloudLocate this session on theDrupalCon Munich website: you!ricardo.amaro@acquia.comtwitter: @ricardoamaro

More magazines by this user
Similar magazines