OSS at Unibet.key - Jfokus

Unibet.com ArchitectureOpen Source at Unibet.com:10x scalability at half the coststefan.norberg@unibet.com@stnor

About the speaker

Unibet Open Sourcestrategy

• Open source software and open standards shouldalways be our first choice• Avoid vendor lock-in - software we use should have aright-to-use license without any cost attached

• Commercial, proprietary software need to showexceptional business value (over Free solutions) in orderto be considered• Contribute to the community by buying support fromcompanies backing FOSS solutions or paying forproduct improvements

CustomersPartners &AffiliatesBannersBrowserFlash/FlexContent Deliver NetworksFan-out Push ServersDDos ProtectionSingle-sign-onPaymentWeb CMSREST API:sPoker Casino BingoAsync Betting APIMessagingCustomerSystemWalletSystemBonusSystemBettingEngine

CustomersPartners &AffiliatesBannersBrowserFlash/FlexContent Deliver NetworksFan-out Push ServersDDos ProtectionSingle-sign-onPaymentWeb CMSREST API:sAsync Betting APIMessagingCustomerSystemWalletSystemBonusSystemBettingEngine

Unibet in numbers• 100% online - www.unibet.com• 30+ web sites in 27 languages• Gross win: 250 million dollars / year• “Technical” growth is >100% / year

Why is this hard?

Because users aregrowing impatient

2006:4 secondsSource: Juniper Research

2009:2 secondsSource: Forrester Research

Two main challenges• Latency - data too far away from where it’sneeded• Bottlenecks - resource contention

Avg 40 ms RTT

Meet Kevin• Kevin is a customer to the candy store• Kevin wants to buy 40 pcs for his allowance• Kevin has extremely small pockets (theyonly fit one piece of candy)• He also wants to buy two chocolate barsand one bubble gum• The candy store is far far away from Kevin’shouse

Idea #1Sell bags of candy(Caveat: no bars or bubblegums in bags!)

The bag is a big JS file

The bag is a big CSS file

The bag is a CSS imagesprite*

4.9.3/images/uskin1/uspritesheadertabs.pngblackTab

The 100% optimized e-commerce web page:4 requestsConn 1250 ms 30 msHTMLCSSConn 2250 msJSTotal time: 530 ms30 msIMG

Some real world data• amazon.com: 57 requests (6)• ebay.com: 41 requests (4)• unibet.com: 41 requests (1)

Ouch.

Idea #2Sell bags on streetcorners(Caveat: no bars or bubblegums in bags!)

There are many streetcorners

Street corners are very nearall customers

The street corner is aContent Delivery Networknode

GOOD SPOT35 msavg latency

BAD SPOT70 msavg latency

BAD SPOTBandwidthLatencyRedundancyPricing

Tune tune tune thefront-end!• 80-90% of the end-user response time isspent downloading and processing all thecomponents in the page• Use best practice to optimize your frontend(YSlow, Page Speed etc)• Next step to improve performance is totweak delivery

How we do it• Be close to your users - use a ContentDelivery Network (CDN)• Prepend CDN proxy name to all staticfiles: /foo/bar -> http://cdn.com/foo/bar• Set a one year cache-time (no 304:s)• Version your static content• Stripe objects over several CNAMES

15 msavg latency10 msavg latency10 msavg latencyNOT SO BAD SPOT

• By using a CDN we serve our users quicklyfrom anywhere in the world• Capacity is not an issue• We have cut our delivery costs to 25%• Use several CDN vendors for load balancing /redundancy

Next steps...

Live Bettingdata distribution• Bet while the game is on“Who gets the next corner”, “who scores the next goal” etc• 10000:s of real-time clients need priceupdates almost every second• 20 concurrent games• 10+ offers within one game• Product growth 60-80% per year

BettingEngine

Fan-outServerFan-outServerBettingEngine

KaazingRabbitMQKaazingRabbitMQRabbitMQBettingEngine• End-to-end messaging• RabbitMQ• Kaazing Enterprise Gateway• Google Protocol Buffers

RabbitMQ• Free Open Source and Open Standardsprotocol (AMQP)• Highly available, highly scalable (Erlang/OTP)• Excellent & friendly support available

Kaazing• Open Source and Open Standards protocol(HTML5 WebSockets)• Supported AMQP clients for Flex, browers• Connection offloading• Great at overcoming last mile hurdles(firewalls, proxies, browsers)• Excellent & friendly support available

Protocol Buffers• Flexible, efficient, automated mechanism forserializing (binary)• Schema support (.proto files)• Language neutral (we use Java to Actionscript)• Invented and used by Google• Open Sourced in July 2008

Java/SpringBettingEngineRabbitMQ RabbitMQ KaazingFan-outnodeAS3Live BettingClientProtobufs/AMQP

Let’s get back to theCandy store...

Datacenter• Ok, the candy delivery problem is solved!• However, we cannot serve the customerschocolate and bubble gum quickly enough• Queues are building up due to ourbottlenecks

The chocolate bar is acacheable read

The bubble gum is anon-cacheable read ora write

The candy store checkoutis an app server

The bubble gum machineis a database system

Example #1More hardware!(and sw licenses)

Kids are invading ourstore...

Food for thoughtpaying per core is paying for peaks......is not acceptable?

You cannot buy aproduct to solve thisproblem

Need to rearchitecture!

Example #2Near-cache

Terracotta• Network Attached Memory• Extends the Java threading model acrossJVM:s• In-memory speed access to data• Fully coherent, with stored to diskguarantees• Writes are sent as deltas across nodes

Keep data stored/cachedwhere it’s needed for processingJVMHeap“L1”BettingEngineBettingEngine...BettingEngineSoREventRepositoryBet OfferRepositoryTerracotta L2Live BetsEventuallyconsistentReporting/Legacywrite-behindOracle (MySQL)

Keep data stored/cachedwhere it’s needed for processingJVMHeap“L1”BettingEngineBettingEngine...BettingEngineupdateSoREventRepositoryBet OfferRepositoryTerracotta L2Live Betswrite-behindReporting/LegacyOracle (MySQL)

Terracotta• L2 server can become bottleneck• Shard (free) or stripe ($$$)

Example #3Offload reads toseparate systems

BettingEngineBettingEngine...BettingEngineHistoryServer...HistoryServerEventRepositoryBet OfferRepositoryTerracotta L2Live BetsR R RBet HistoryOracle (MySQL)SettleBetMySQLBackoffice

Example #4Affinity +multi master replication

cus-client.jarcus-client.jaruser1 user2 user3user4CustomerSystemCustomerSystemCustomerSystemCustomerSystemLDAPLDAPLDAPLDAPdata center 1data center 2

Example #5Partition the data toscale writes

C C C• Need to get settlementsdone quickly• More than 10000 writes / sHistoryServer...HistoryServer•Scale out by using multipleMySQL instancesLive BetsBet HistoryroutingS1 S2 S3MySQL

Combining the pieces

BrowserBetting ClientFlash BettingClientHTMLGeo-distributedFan-out PushServers (Kaazing)History APISettled bets100% readBetting APILive datacommandsprice updatesand repliesBettingHistoryshardedwith localMySQLwrite-behindon settlementBettingEngineBettingEngineBettingEngineShared memory for events, betoffers and customer bets(Terracotta)payoutsAMQPBroker(RabbitMQ)write-behindAccountServersDatabase(MySQL)

Recap:Two main challenges• Latency - data too far away from where it’sneeded• Bottlenecks - resource contention

Recipes used at Unibetinformation delivery• Optimize your web apps!• Get rid of static object traffic• Move web data closer to customers (CDN)• Last-mile fan-out messaging

Recipes used at Unibetin the data center• Move data to business logic• Read-only farms• Multi-master replication for read-mostlydata• Sharding to scale writes

Open Source Software and Services isused by Unibet extensively in missioncritical applications because...

...we are convinced it’s better andmore cost effective!

Thanks!(we’re hiring)@stnorstefan.norberg@unibet.comstnor.wordpress.comwww.linkedin.com/in/stnor