Real-time Engineering at Uber and the Evolution of an Event-Driven Architecture Presentation

Real-timeEngineeringat Uberwolski@uber.com

Lords of the TripReal-time Engineering

Anatomy of a TripLooking for a ride

Anatomy of a TripRequesting a ride

Anatomy of a TripDispatching a driver

Anatomy of a TripPicking you up

Service-OrientationGeospatial Proxy, Car Locator, Pricing, ETAs,Ranking, Dynamic Configuration, Map-Matching,Reverse Geocoding, User Caching, ...

User Cache… for fault tolerance

End-UserReal-timeServicesUser CacheUser ServicePOST /tripsHappy Path

End-UserReal-timeServicesUser CacheUser ServicePOST /tripsGET /users/123Happy Path

End-UserReal-timeServicesUser CacheUser ServicePOST /tripsGET /users/123GET /users/123Happy Path

End-UserReal-timeServicesUser CacheUser ServicePOST /tripsGET /users/123GET /users/123Happy Path

End-UserReal-timeServicesUser CacheUser ServicePOST /tripsGET /users/123GET /users/123HMSET users:123StoreHappy Path

End-UserReal-timeServicesUser CacheUser ServicePOST /tripsGET /users/123GET /users/123HMSET users:123StoreHappy Path

End-UserReal-timeServicesUser CacheUser ServicePOST /tripsSad Path

End-UserReal-timeServicesUser CacheUser ServicePOST /tripsGET /users/123Sad Path

End-UserReal-timeServicesUser CacheUser ServicePOST /tripsGET /users/123GET /users/123Sad Path

End-UserReal-timeServicesUser CacheUser ServicePOST /tripsGET /users/123GET /users/123Sad Path

End-UserReal-timeServicesUser CacheUser ServicePOST /tripsGET /users/123GET /users/123HMGET users:123StoreSad Path

End-UserReal-timeServicesUser CacheUser ServicePOST /tripsGET /users/123GET /users/123HMGET users:123StoreSad Path

Cache-Served Requests

AirlockVarnish Cache inspired HTTP probehttp://github.com/uber/airlock

Instantiate your Airlock1 var airlock = new Airlock({2 title: ‘geocod’,3 threshold: 3,4 window: 5,5 maxWait: 64 * 10006 });

Seal your HTTP request1 var options = {2 host: ‘http://geocod.uber.local’,3 path: ‘/reverse/geocode’,4 params: { lat: 45.543, lng: -122.654 },5 timeout: 15006 };78 airlock.seal(options, function(err, addr) {9 if (err) return callback(err);10 // do something interesting with addr11 });

Coupling, the Quiet KillerA Push Notifications Case Study

“Your Uber is on the way. Jeff (4.9stars) will pick you up in 2 minutes.”smsClient.sendDispatchAccepted(...);

“Your Uber is on the way. Jeff (4.9stars) will pick you up in 2 minutes.”smsClient.sendDispatchAccepted(...);pushClient.sendDispatchAccepted(...);

“Your Uber is on the way. Jeff (4.9stars) will pick you up in 2 minutes.”smsClient.sendDispatchAccepted(...);pushClient.sendDispatchAccepted(...);dispatchAccepted.publish(...)

1 dispatchAccepted.publish({2 driverFirstName: ‘Jeff’,3 driverRating: 4.9,4 eta: 120,5 smsEnabled: false6 });

End-UserReal-timeServicesKafkaNotificationServiceSMS ServiceAccept DispatchThe Handoff

End-UserReal-timeServicesKafkaNotificationServiceSMS ServiceAccept DispatchPublishThe Handoff

End-UserReal-timeServicesKafkaNotificationServiceSMS ServiceAccept DispatchPublishSubscribeThe Handoff

End-UserReal-timeServicesKafkaNotificationServiceSMS ServiceAccept DispatchPublishSubscribeSendThe Handoff

End-UserReal-timeServicesKafkaNotificationServiceSMS ServiceAccept DispatchPublishSubscribeSendPushThe Handoff

Real-timeServicesKafka LeafKafkaAggregateOut-of-BandConsumerpublish(to: topicwith: data)ZYXStoreForwardBA Z Y Xsubscribe(to: topic)DiskKafka Architecture

Event-Driven APIAlerts, Analytics, ETAs, Fraud, Pricing,Marketplace Metrics, Visualizations

DO make events a first-class citizendispatchAccepted.publish({driver: driver,trip: trip});

DO make it easy to adorn yourevents with common propertiesdecorate(event).with(‘common’).with(‘driver’).with(‘vehicle’);

DO consider throughput andlatency

DO give yourself “an out” by versioning your eventsDO give your consumers the rawest of data

DON’T forget that Kafka is a dependency tooDON’T publish events for actions that aren’tinitiated by your serviceDON’T pollute your event stream withunnecessary information

StormRedistribute your data

Kafka Topics Storm Spouts Storm Bolts RedisChannelstrip_requestedS1trip_beganS2B1trip_changestrip_endedS3

Kafka Topics Storm Spouts Storm Bolts RedisChannelstrip_requestedB3S1trip_beganS2B2B1trip_changestrip_endedB4S3B5

Stream MergingFor application-specific streams

Kafka EventStreamStorm Spouts Storm Bolts Kafka DataStreamtrip_requestedS1driver_dispatchedMdispatch_distancesS2

Outpace Complexity

Thanks for listening!Oh, and we’re hiring:wolski@uber.com