data warehouse - Supporting Advancement

Skill Building: Building a Data Warehouse – Part 1Brian Dowling

Learning Objectives• What is a data warehouse, why do we need one and how should we goabout building it?• Challenges, Benefits, Investments & Costs• Key Performance Indicators and Management Dashboards• Some Live Demos• Caveats, Excuses, Limitations:• Broad Topic, High Level View• Limited Time to Discuss• Many Variations on the Theme• Everyone Uses Different ToolsBrian Dowling, Blackbaud International Conference | Page #2

Agenda• Building a Data Warehouse – Part 1 (This Session)Introductions/Agenda (5 Min.)Definitions & Concepts (20 Min.)Data/Info. Challenges & DW Benefits (25 Min.)Foundation/Getting Started (15 Min.)Evil Gods of Demos Permitting (10 Min.)A Few Resources (For Your Reference)• Building a Data Warehouse – Part 2Introductions/Agenda (5 Min.)Case Study (5 Min.)FoundationSecond FoundationFoundation and EmpireA Few Resources (For Your Reference)• Ask Questions Anytime (More Fun For All)• If We Can’t Get to a Question, We’ll Log and Email AnswersBrian Dowling, Blackbaud International Conference | Page #3

Who are you? Who, who?Senior Vice President for Finance & Information SystemsVGH & UBC Hospital Foundationwww.worldclasshealthcare.caPrincipal and FounderSupportingAdvancement.Comwww.supportingadvancement.comFounding Board MemberAssociation of Advancement Services Professionalswww.advserv.orgbrian.dowling@supportingadvancement.comhttp://twitter.com/SupportAdvancehttp://youtube.com/SupportAdvancementhttp://www.facebook.com/brian.dowlingBrian Dowling, Blackbaud International Conference | Page #4

Personal Experience, BiasMost of career in Information Systems and FinanceReasonably GoodHands on Technical Skills(DBA, Developer,Report Writer,Data Analyst)Small, Medium&LargeOrganizationsConsultingPart-TimePrivate SectorHigher EducationNot for ProfitExperience in Building About 8 “Significant” Reporting/DataWarehouse/Information Deployment Environments(Amongst Other Activities)brian.dowling@supportingadvancement.comhttp://twitter.com/SupportAdvancehttp://youtube.com/SupportAdvancementhttp://www.facebook.com/brian.dowlingBrian Dowling, Blackbaud International Conference | Page #5

The Benefits of Experience• Learn from Past Mistakes• Realize that Every Organization has a Variety of Issues that Need to beSolved (Every Project has its Own Variations)• Most Organizations have Similar Challenges• Approach Needs to be Based on What Needs to Be Solved• Typically Lot of Data Cleanup• Will Typically be Changes in Jobs, Roles and Responsibilities• Change Management and Communication Plans are Key• It Takes a Lot of Time and Effort• Ongoing Engagement and Support from Sponsors is Criticalbrian.dowling@supportingadvancement.comhttp://twitter.com/SupportAdvancehttp://youtube.com/SupportAdvancementhttp://www.facebook.com/brian.dowlingBrian Dowling, Blackbaud International Conference | Page #6

Definitions & ConceptsBrian Dowling, Blackbaud International Conference | Page #7

Data Information• DataThe term data means groups of information that represent thequalitative or quantitative attributes of a variable or set of variables.• InformationRecords are a specialized form of information. Essentially, records areinformation produced consciously or as by-products of businessactivities or transactions and retained because of their value. Primarilytheir value is as evidence of the activities of the organization but theymay also be retained for their informational value.Sound records management ensures that the integrity of records ispreserved for as long as they are required.(Wikipedia)Brian Dowling, Blackbaud International Conference | Page #8

Information Knowledge• Information - Basis of Competitive Advantage• Information - Foundation for Innovation• Information - Changes OrganizationalDirection• Information - Focuses us on Goals• Information Strategies Need to Enhance andImprove Knowledge Across the OrganizationBrian Dowling, Blackbaud International Conference | Page #9

Data Information KnowledgeBrian Dowling, Blackbaud International Conference | Page #10

The Concept of Real Time• “Any time you get information in less than real time, you need tosignificantly re-architect your information environment.” Ralph KimballBrian Dowling, Blackbaud International Conference | Page #11

Brian Dowling, Blackbaud International Conference | Page #12

Brian Dowling, Blackbaud International Conference | Page #13

Data Warehouse• A data warehouse is a repository of an organization's electronicallystored data. Data warehouses are designed to facilitate reporting andanalysis.A data warehouse houses a standardized, consistent, clean andintegrated form of data sourced from various operational systems in usein the organization, structured in a way to specifically address thereporting and analytic requirements.(Wikipedia)Brian Dowling, Blackbaud International Conference | Page #14

Data Warehouse – Is Also …• A place where you can combine your data in ways that are not possiblein transactional and other systems.DataWarehouseBrian Dowling, Blackbaud International Conference | Page #15

Data Warehouse – Is Also …• A Set of Reporting Tables and Views Designed for More EfficientReporting• An Operational Data Store• Fundraising or Similarly Described Data Mart• Any others?Brian Dowling, Blackbaud International Conference | Page #16

Business Intelligence• Business Intelligence (BI) refers to computer-based techniques usedin spotting, digging-out, and analyzing business data, such as salesrevenue by products and/or departments or associated costs andincomes.BI technologies provide historical, current, and predictive views ofbusiness operations. Common functions of Business Intelligencetechnologies are reporting, online analytical processing, analytics, datamining, business performance management, benchmarking, textmining, and predictive analytics.(Wikipedia)Brian Dowling, Blackbaud International Conference | Page #17

Key Performance Indicators• A Key Performance Indicator (KPI) is a measure of performance.Such measures are commonly used to help an organization define andevaluate how successful it is, typically in terms of making progresstowards its long-term organizational goals.KPIs can be specified by answering the question, "What is reallyimportant to different stakeholders?" KPIs may be monitored usingBusiness Intelligence techniques to assess the present state of thebusiness and to assist in prescribing a course of action.The act of monitoring KPIs in real-time is known as business activitymonitoring (BAM). KPIs are frequently used to "value" difficult tomeasure activities such as the benefits of leadership development,engagement, service, and satisfaction.KPIs are typically tied to an organization's strategy using concepts ortechniques such as the Balanced Scorecard.Brian Dowling, Blackbaud International Conference | Page #18

Key Performance Indicatorshttp://www.supportingadvancement.com/reporting/general/brian_dowling_metrics_strategy.htmBrian Dowling, Blackbaud International Conference | Page #19

DashboardsBrian Dowling, Blackbaud International Conference | Page #20

Dashboards• Web Deployed• Graphical Presentation Wherever Possible• Automated Refresh (Daily)• Drop Down Prompts to Select Sub-Sets of Information• Ability to Drill Down to See Details Wherever Possible• Structure from High Level Views Down to Low LevelBrian Dowling, Blackbaud International Conference | Page #21

Visual Analytics• Visual Analytics is the science of analytical reasoning supported byhighly interactive visual interfaces.People use visual analytics tools and techniques to synthesizeinformation into knowledge; derive insight from massive, dynamic, andoften conflicting data; detect the expected and discover the unexpected;provide timely, defensible, and understandable assessments; andcommunicate assessments effectively for action.(From the VisWeek Web Site.)Brian Dowling, Blackbaud International Conference | Page #22

Visual Analyticshttp://www.supportingadvancement.com/reporting/general/brian_dowling_metrics_strategy.htmBrian Dowling, Blackbaud International Conference | Page #23

Visual Analyticshttp://oakland.crimespotting.org/Brian Dowling, Blackbaud International Conference | Page #24

Data WarehouseBrian Dowling, Blackbaud International Conference | Page #25

Data WarehouseBrian Dowling, Blackbaud International Conference | Page #26

Addendum: Facts, Dimensions, Surrogate Keys• Fact – Gift Amount• Dimension – Gift Tender Type Description• Surrogate Key – Gift Date + Gift Tender Type Code• Terms are not Critical in Moving Forward• What are the questions that I need to be able to find answers quickly?http://www.supportingadvancement.com/reporting/data_warehouse_environment/brian_dowling_star_schema.htmBrian Dowling, Blackbaud International Conference | Page #27

Data/Information ChallengesData Warehouse BenefitsBrian Dowling, Blackbaud International Conference | Page #28

Many Customers, Everyone Wants Different DataBrian Dowling, Blackbaud International Conference | Page #29

Data Warehouse Benefits• A data warehouse can be designed to work with organizational structureand “end customers” simultaneously.Example: Appeals & Packages & Integration of Complex Rules andBusiness ProcessesHow did we ask?Which department in our organization made the ask?Who is managing the relationship?Who needs to manage the relationship?Who gets credit for the gift?What are the next steps in management of the relationship?Annual Giving asks with a direct mail appeal, a gift comes in for $50 andthe donor’s cumulative giving reaches $10,000 as a result of the gift.How much of our complex business processes are automatically informed?Brian Dowling, Blackbaud International Conference | Page #30

Many StakeholdersBrian Dowling, Blackbaud International Conference | Page #31

Data Warehouse Benefits• A data warehouse can be planned around the concept of multiplestakeholders.Example: Donor stewardship reports.Donor stewardship reports can include information from the fundraisingsystem, the financial system (account balances, investment returns,new gifts, expenditures.)Most detailed customer focus with the least amount of work.Multiple stakeholders simultaneously involved and engaged with data.Helps with approvals, sign-off and accuracy.More people end up understanding the mechanics and importance ofthe business process.Brian Dowling, Blackbaud International Conference | Page #32

Data IntegrationBrian Dowling, Blackbaud International Conference | Page #33

Data Warehouse Benefits• A data warehouse can help to automate data integration using an ETL(extract, transform, load) tool.Automation can significantly reduce the amount of work to load andintegrate data into both the data warehouse and fundraising database.Data can be transformed according to business rules which cansignificantly reduce the amount of programming on the data warehouseside.Example: Gift information from fundraising system can be reformatted inan ETL tool and be automatically uploaded into the financial system.)Brian Dowling, Blackbaud International Conference | Page #34

Shadow or Supplemental SystemsBrian Dowling, Blackbaud International Conference | Page #35

Data Warehouse Benefits• A data warehouse can provide compelling benefits for those that insiston using and maintaining shadow or supplemental systems.http://www.supportingadvancement.com/web_sightings/esolicitations/rk_e_analysis.htmBrian Dowling, Blackbaud International Conference | Page #36

Data Synchronization of Supplemental SystemsBrian Dowling, Blackbaud International Conference | Page #37

Data Warehouse Benefits• By providing incremental benefits back to users of supplementalsystems and by tracking synchronization and data quality coming fromsupplemental systems – users will eventually move away from them.Tracking as to last updated information in the supplemental systemcompared to the main system.Showing where data may be missing in the supplemental system that isin the main system.Automated duplicate checking with the main system.Automated reporting on key information kept in supplemental systems.Brian Dowling, Blackbaud International Conference | Page #38

Data Warehouse Benefits - Information Leveragehttp://www.supportingadvancement.com/reporting/information_leverage/information_leverage.htmBrian Dowling, Blackbaud International Conference | Page #39

Data Warehouse Benefits• A data warehouse can substantially improve data integrity in thetransactional system when users begin to rely more on key informationin the data warehouse – and – that information is provided to them ondemand in a final pixel perfect format.Example: Using the User Table in the Transactional SystemRule: Every System User Must have an Employment RecordData Warehouse: Checks for Rules, Formats Data, User Department,Name, Employment Status and Other Information.Deployment: Other Tables in Data Warehouse such as Data EntryStats, Contact Us List on Web Site, Other Staff ListsBrian Dowling, Blackbaud International Conference | Page #40

Complexity of Data is IncreasingBrian Dowling, Blackbaud International Conference | Page #41

Data Warehouse Benefits• A data warehouse can be used to combine data in creative ways thatnormally require a lot of manual effort.Stock Market IndicesReal Estate Market IndicesGifts and PledgesFinancial Information from Public Revenue AgenciesAll of the above combined in a data warehouse to answer thequestion(s):How are we doing with fundraising compared to broad economicindicators, what is the correlation and how are we doing against likecompetitors within our geographic market segment?Brian Dowling, Blackbaud International Conference | Page #42

Complex Data Models in Transactional SystemsBrian Dowling, Blackbaud International Conference | Page #43

Data Warehouse Benefits• A data warehouse (highly denormalized) can help to simplify the use ofthe data model in the transactional system (highly normalized).Columns can be renamed for more clarity and consistency betweendifferent tables.Decoded descriptions can be included as additional columns.Complex calculations can be added as additional columns.The amounts of joins in reports can be significantly reduced.Meta data can be used to describe columns or other data warehousedata elements. (SQL Server Extended Properties.)Users can be “shielded” from more complex business rules in reporting.Brian Dowling, Blackbaud International Conference | Page #44

Complex Logic Imbedded in Reporting/Query ToolsBrian Dowling, Blackbaud International Conference | Page #45

Complex Logic Imbedded in Reporting/Query Tools• Queries in fundraising system don’t produce output in final format.Users spend (waste) a lot of time reformatting.• Query tool in fundraising system doesn’t have all of the calculated fieldsthat we’d like to be able to use.• Users request training on queries more than anything else. They willnever understand all of the complexities of the database.• Traditional report development cycle takes a long time, and it’s neverquite what users want. As soon as they see the information they wantsomething slightly different.• When logic is imbedded in reports, and we change a business rule wehave to change many reports. Very time consuming.• Performance on complex reports can be very slow.• Traditional reports tend to be static and don’t provide the flexibility togain additional insights into the data.Brian Dowling, Blackbaud International Conference | Page #46

Complex Logic Imbedded in Reporting/Query ToolsBelow the Line = Lots of ReworkBrian Dowling, Blackbaud International Conference | Page #47

Data Warehouse Benefits• A data warehouse can contain highly complex business logic whichdoesn’t need to be replicated anywhere else.Reporting and query tools should ideally not contain any business logicand only be used for headings, footings, fonts, colors, subtotals, totals,% of totals and similar simple calculations.If a business rule is changed, it only needs to be changed in 1 placeand all of the reports (output) from the data warehouse will changeautomatically. (Much lower maintenance.)Brian Dowling, Blackbaud International Conference | Page #48

Data/Information ChallengesBrian Dowling, Blackbaud International Conference | Page #49

Data Warehouse Benefits• A data warehouse can help us monitor all key performance indicatorsand “sell” needed investments to top management.Management dashboards and other key performance indicators can bedesigned to change the direction of the organization when somebusiness condition changes or needs attention. (Actionable Intelligence)Brian Dowling, Blackbaud International Conference | Page #50

Data/Information Challenges• For the most part we’re still using systems whose design is based onthe old technical limitations and backwards compatibility for thecustomer base.• Lack of Integrated Data Dictionary• Lack of Easily Configurable Data Rules• Lack of Audit Tables• Lack of Truly Extensible Attributes• Lack of Aggregate/De-normalized Data Tables• Lack of History Tables (Who was assigned, for how long, and when.)• Old Limitations Such as Costs of Disk Storage, Network Bandwidth,Processing Speeds• Others …Brian Dowling, Blackbaud International Conference | Page #51

Data Warehouse Benefits• Processor Speeds• Expensive Disk Space• Limited Bandwidth• Ability to Deploy Easily• In Memory Processing• Complex Tools• End User SkillsBrian Dowling, Blackbaud International Conference | Page #52

We Shall OvercomeBrian Dowling, Blackbaud International Conference | Page #53

FoundationGetting StartedBrian Dowling, Blackbaud International Conference | Page #54

Start With a PlanOrganize Planning ProcessConduct ResearchConstruct PlanFormalize the Planning ProcessImplement PlanEvaluate Metrics and OutcomesRevise PlanBrian Dowling, Blackbaud International Conference | Page #55

Relate Back to Strategic PlanImproveDataQualityWhat are the data bubbles?Where do they fit?How do they match strategy?Cost ofBadRecordContactReportTemplateLunch&Learnshttp://www.supportingadvancement.com/reporting/general/brian_dowling_metrics_strategy.htmBrian Dowling, Blackbaud International Conference | Page #56

Evaluate Effort Vs. ReturnCost ofBadRecordImproveDataQualityLunch&LearnsContactReportTemplatehttp://www.supportingadvancement.com/reporting/general/brian_dowling_metrics_strategy.htmBrian Dowling, Blackbaud International Conference | Page #57

Define Critical Success Factors• Note: These examples based on case study, your organization’ssuccess factors may be very different.• Improvements in Data Quality• Reduce Manual Efforts in Re-compiling Information• Management Metrics to Manage the Business and Make Key StrategicDecisions(The reason this is 3 rd in the list is that the first two greatly constrain theability to provide accurate metrics.)• Integrate Key Data from Other Systems• Success of Data Warehouse ProjectBrian Dowling, Blackbaud International Conference | Page #58

Governance/Committees/Partnershipshttp://www.supportingadvancement.com/systems/systems_general/dowling_is_advisory_group.htmBrian Dowling, Blackbaud International Conference | Page #59

Document Your Plan/ActivitiesWhat’s in itfor me?Brian Dowling, Blackbaud International Conference | Page #60

Start to Work Your Plan• Think Big, Start Small, Deliver Quickly• Acquire Sponsor(s)• Governance/Committees/Partnerships• Tools/Technical Requirements• Funding Approvals• Define Standards• Set up Technical Environment/Tools• Experiment/Small Initial Projects• Prioritize Metrics• Prioritize Subjects• Small Deployments• CommunicateBrian Dowling, Blackbaud International Conference | Page #61

Acquire Sponsors• # 1 reason that most projects fail is due to a lack of an appropriatesponsor.• SQL Server & Related Tools• Separate Hardware for Data Warehouse• Data Visualization Tools are Expensive• Budget for Laptop/Workstation Installation• Scheduling Software – Sometimes Hidden Cost• SQL Documentation Software• Be innovative.• Make sure you can go “Back to the Well”Brian Dowling, Blackbaud International Conference | Page #62

CommunicateWhat’s init for me?Brian Dowling, Blackbaud International Conference | Page #63

Evil Gods of Demos Permitting …• SQL Server 2008• TableauBrian Dowling, Blackbaud International Conference | Page #64

A Few ResourcesBrian Dowling, Blackbaud International Conference | Page #65

acmqueuehttp://queue.acm.org/detail.cfm?id=1805128Brian Dowling, Blackbaud International Conference | Page #66

CASE Books on Advancement ServicesBrian Dowling, Blackbaud International Conference | Page #67

CoolData Bloghttp://cooldata.wordpress.com/Brian Dowling, Blackbaud International Conference | Page #68

Data Warehousing Institutehttp://tdwi.org/Brian Dowling, Blackbaud International Conference | Page #69

FlowingDatahttp://flowingdata.com/Brian Dowling, Blackbaud International Conference | Page #70

Good Ishttp://www.good.is/departments/transparency/Brian Dowling, Blackbaud International Conference | Page #71

Listservshttp://www.supportingadvancement.com/potpourri/favorite_listservs/favorite_listservs.htmBrian Dowling, Blackbaud International Conference | Page #72

Many Eyeshttp://manyeyes.alphaworks.ibm.com/manyeyes/Brian Dowling, Blackbaud International Conference | Page #73

PASShttp://www.sqlpass.org/Brian Dowling, Blackbaud International Conference | Page #74

Periodic Table of Visualization Methodshttp://www.visual-literacy.org/periodic_table/periodic_table.htmlBrian Dowling, Blackbaud International Conference | Page #75

SupportingAdvancement.Comhttp://www.supportingadvancement.com/reporting/reports.htmBrian Dowling, Blackbaud International Conference | Page #76

Tableau Softwarehttp://www.tableausoftware.com/community/blogBrian Dowling, Blackbaud International Conference | Page #77

Other Vendors• Substantial resources, white papers, newsletters, free webinars andmuch more.Check for local chapters in your area.Brian Dowling, Blackbaud International Conference | Page #78

Questions?brian.dowling@supportingadvancement.comhttp://twitter.com/SupportAdvancehttp://youtube.com/SupportAdvancementhttp://www.facebook.com/brian.dowlingBrian Dowling, Blackbaud International Conference | Page #79