Splunk at Macy's.com

Splunk at Macy’s.com
Operational Visibility Delivers Up-time for Top Retail Website
The Business
Facing one of the most difficult holiday seasons in
decades, the Macy’s IT team, who manage two of the
world’s highest-grossing retail websites, needed the
visibility across their infrastructure to ensure peak
website performance and continuous up-time. This
was a challenge considering that their sites
experienced outages on both Black Friday and the last
day of a free shipping promotion the past two years.
In fact, the retailer's websites had experienced holiday
downtime for six straight seasons. As its online sales
exceed $1 Billion per year, seconds of downtime meant
thousands of dollars in lost revenue and an impact to
the customer experience.
Challenges
To better handle the load of up to 130 orders per
minute generated by over a million unique daily
visitors, the operations team tripled the number of
servers and fortified its environment with failover
capabilities. Before Splunk, the operations team spent
hours manually locating issues or finding the source of
problems. For example, benign conditions like a hung
thread could be the source of the problem. This
sometimes harmless, sometimes devastating
condition, which is a normal part of recycling the
system, is a lot like a traffic jam. It can clear up by itself
or bring an entire ten-mile stretch of highway to a
complete halt.
Overview
Industry
• Retail - e-commerce
Splunk Use Cases
• Application Troubleshooting
• IT Operations Management
Business Impact
• Delivered the IT team end-to-end visibility
across their entire technology stack
• Enabled 100% up-time for two straight
seasons with a 50% increase in
transactions
• Achieved a 90% efficiency improvement
by automating previously manual
troubleshooting processes
• Established proactive infrastructure
monitoring across to detect issues before
they impact users
• Supplied role-specific, dashboard views to
100 users across IT
Data Sources
• System error, system out, native standard
error, native standard out, application logs
• .txt files
• Multi-processing modules (MPM) stats
from IBM HTTP servers
“We’ve recouped the Splunk license cost over and over again. Today we spend two
or three minutes looking for an error, versus five or six hours in the past."
Camille Bali, Architecture Team
Customer Success Story: Macy’s.com Copyright © 2010, Splunk Inc.

Enter Splunk
Macy’s deployed Splunk as part of their Holiday
Readiness Project to monitor and alert on system
performance. The organization built a 24/7 NOC, with
Splunk at the heart of the effort. During this peak
period, the team relied on Splunk dashboards for endto-end
visibility across their systems. When an anomaly
appeared, an analyst would drill down into the
problem using Splunk to determine its source and
remedy it before it brought the system down and
effected users.
In addition, the team set up proactive alerts for hung
threads and other abnormalities. An immediate
notification to the support operations center
prompted them to investigate, and if necessary, take a
server cluster out of rotation without impacting the
customer experience.
The Macy’s.com and Bloomingdales.com sites send
Splunk all their production logs and critical system
metrics and stats, this includes data from custom
application logs. By monitoring metrics and stats from
their IBM webserver environment, the team has
visibility into any resource constraints. The Macy’s team
focuses on maintaining the maximum number
connections coming through each web server.
Over 100 employees have access to Splunk dashboards
across the testing, performance and production teams
– even the Group Vice President of Technology
monitors several dashboards. While there are 100 users,
there are only two Splunk administrators.
Breakthroughs
For the first time in six years, Macy’s, a Fortune 100
retailer experienced no downtime during its peak
holiday shopping season. And that’s despite a 50%
increase in traffic over the prior year and several record
breaking days including one over 90,000 orders. As a
result, web sales increased almost 40% for December
“Splunk has helped us to achieve
100% stability during our peak
holiday shopping season for two
straight years."
Camille Bali, Architecture Team
and 29% for the full year. The company was recently
ranked the #28 retailer in terms of online spending.
Splunk dashboards and alerts played a critical role in
keeping the online environment up and running
throughout the entire holiday season and beyond.
Splunk continues making a difference at Macy’s.
Proactive alerts set up across their environment, can
now be tweaked on the fly. Instead of a ten-person
team monitoring systems, one individual follows forty
to fifty systems from a single user interface. And it now
takes 90% less time to troubleshoot issues.
As average daily sales grow north of $2.7 million, the
investment in Splunk has already come back to the
Macy’s many times over.
The business impact of Splunk is built on Macy’s new
level of operational visibility. With the help of Splunk,
Macy’s has gained real-time visibility across their
infrastructure and established a proactive way to
monitor it.
Get Started Today!
• Free download: www.splunk.com/download
• Toll Free: +1.866.GET.SPLUNK (+1 866.438.7758)
• Direct: +1.415.848.8450
• Email: info@splunk.com
Customer Success Story: Macy’s.com Copyright © 2010, Splunk Inc.

Customer Success Story: Macy’s.com Copyright © 2010, Splunk Inc.