How to use your Quality and Risk Profile (QRP) - dbg

IntroductionThis guidance will help you to understand your Quality and Risk Profile (QRP).It explains:• What an NHS QRP is.• How to use your QRP.• The sources of data that we have used• How the QRP is organised• How we calculate outcome risk estimates.• How to access your QRP and get helpMain pointsThe QRP is an essential tool used for gathering together key information about yourorganisation to support how we monitor your compliance with the essential standards ofquality and safety. The QRP enables our compliance inspectors to assess where riskslie and may prompt front line regulatory activity, such as further enquiries.Compliance inspectors use the QRP alongside the Guidance about compliance:Essential standards of quality and safety (the essential standards), the JudgementFramework and your Provider Compliance Assessment record (if available).QRPs are also an important tool for providers and commissioners of services – both tosupport continuous monitoring of compliance, by ensuring that everyone is working fromthe same information, and to improve how care is provided and commissioned.Your QRP will also be useful to support how you monitor quality internally, by identifyingareas of lower than average performance and taking action to address them wherenecessary. Commissioners (including, in time, the GP commissioning consortia) shouldalso find the QRP invaluable in holding to account the providers that they commissionservices from, and in improving their commissioning for qualityQuality and Risk Profiles: Guidance for NHS providers2

ContentsNo. Question PageOverview of a Quality and Risk Profiles1. What is a Quality and Risk Profile (QRP)? 52. How will CQC use a QRP? 5Using the information presented in your QRP3. How do I navigate through the different sections of the QRP? 64. There are some ‘worse than expected’ (red) risk estimates(dials) for an outcome in my trust’s QRP? How will this affect ourregistration?5. What should I do about the ‘worse than expected’ (red) riskestimates (dials)?67Data sources used in the QRP6. What type of information does the QRP contain? 77. How does CQC get information from people who use servicesand representative groups?88. How up to date is the information presented in the QRP? 89. What level of service does the QRP relate to? 8Layout and sections of the QRP10. How is information organised in the QRP? 911. What types of information are displayed in component 1? 912. What types of information are displayed in component 2? 913. How is item level data displayed in the second component ofthe QRP?1014. What types of information are displayed in component 3? 1015. Is information displayed for all outcomes in the QRP? 11Quality and Risk Profiles: Guidance for NHS providers3

16. How will I know which site/location the item level data relates to? 1117. How is information displayed for NHS providers that have beenformed from a merger after 1 April 2008?11How we calculate risk estimates18. What is an outcome risk estimate and how is it calculated? 1219. Is all information weighted equally when producing an outcome riskestimate?1220. How does CQC measure confidence in a risk measure? 1321. Is an outcome risk estimate displayed whenever there isinformation for an outcome?1322. How are outcome risk estimates displayed in the QRP? 1423. How are the section summaries in the QRP calculated? 14Information in the third component: Contextual risk24. What are the different types of contextual risk? 1425. How will CQC use the contextual risk information within the QRP? 1526. How is contextual risk calculated? 1527. How is contextual risk displayed in the QRP? 15Accessing the QRP and getting help28. Who will have access to QRPs? 1629. How do I access the QRP? 1630. Will the QRP be published externally? 1631. Who should I contact if I have a query about the QRP? 16Appendix A: What is an information item and how is it analysed? 17Appendix B: Categorical descriptions of item level analysis results andassignment of descriptions19Appendix C Additional information on Contextual risk: 20Appendix D: Sections in the essential standards with key outcomes 21Quality and Risk Profiles: Guidance for NHS providers4

1. What is a Quality and Risk Profile (QRP)?A QRP is a tool that brings together a wide range of information about each provider ofhealth and adult social care registered with the Care Quality Commission.The QRP:1. Displays information in an easily accessible format.2. Combines both quantitative (numerical) and qualitative (textual) information.3. Provides an estimate of the risk of potential non-compliance with the essentialstandards of quality and safety.4. Is dynamic and is updated over time as new data becomes available.5. May trigger further regulatory activity by compliance inspectors to support them inmonitoring compliance, this is called a responsive review.6. Helps compliance inspectors to make judgements about an organisation’sperformance.7. Enables providers and their compliance inspectors to look at the same information.2. How will CQC use a QRP?We use QRPs to support how we monitor providers’ compliance with the essentialstandards of quality and safety. Our inspectors use a provider’s QRP:• When carrying out a planned review of compliance, to identify and prioritise potentialrisks of non-compliance.• To regularly review potential non-compliance, which may trigger a responsive reviewof compliance.Quality and Risk Profiles: Guidance for NHS providers5

3. How do I navigate through the different sectionsof the QRP?There are helpful hints at the start of each QRP to make it easier to find your wayround. These describe the bookmarks that are included and where to click to find theunderlying information.The ‘clickable’ areas include many of the dials that show the estimates about the riskof non-compliance. If you place the mouse curser over them, the curser changes to ahand symbol. By clicking on this, you will be taken to the relevant constituent levels ofdetail and information.For more information about the dials please see question 21 below.4. There are some ‘worse than expected’ (red)risk estimates (dials) for an outcome in my trust’sQRP? How will this affect our registration?The presence of ‘worse than expected’ risk estimates within your QRP will notautomatically affect your registration status. This is because the QRP does not producea judgement about your compliance with the essential standards. Instead it is designedto help our compliance inspectors to determine whether they need to target regulatoryactions and responses. Compliance inspectors will need to gather further informationto follow up ‘worse than expected’ risk estimates. For example, we may ask you tocomplete a Provider Compliance Assessment or, where necessary, we may carry out avisit. We can then make judgements about your compliance using the Guidance aboutcompliance: Essential standards of quality and safety and the Judgement Framework.Quality and Risk Profiles: Guidance for NHS providers6

5. What should I do about the ‘worse than expected’(red) risk estimates (dials)?If any of your risk estimates suggest that your performance may be worse thanexpected, you may wish to consider them in more detail to decide if you need toaddress any issues to ensure continued compliance with essential standards.You can look at the details in the QRP that describe the underlying data and theresults of the analysis to help you pinpoint the issues that have contributed to the riskestimates. We believe that the information provided in the QRP will complement theinformation that NHS trusts already use to monitor their compliance with essentialstandards.6. What type of information does the QRP contain?The QRP combines both quantitative (numerical) and qualitative (textual) information.Most quantitative data comes from existing nationally-held data sets from:• The Information Centre for Health & Social Care.• The Department of Health.• Medical royal colleges.• Other organisations with an interest in healthcare.• National assessments carried out by CQC (e.g. patient surveys, reviewsand studies).We have produced separate guidance that provides more detail about the quantitativedata used in Quality and Risk Profiles: Data Sources.Qualitative information can come from a variety of sources including:• Engagement activities by local CQC staff with providers and stakeholders.• Information from providers.• Information from people who use the services.• Our inspection reports, for example, findings from monitoring compliance withregulation on cleanliness and infection control.Quality and Risk Profiles: Guidance for NHS providers7

7. How does CQC get information from peoplewho use services and representative groups?We routinely incorporate information from people who use services within the QRP.Sources include:• Information from user representative bodies such as. Local involvement networks(LINks)• Information from statutory bodies, such as foundation trust boards of governors andoverview and scrutiny committees.• Information from people’s feedback on NHS Choices and Patient Opinion.• Findings from the national NHS patient survey programme, for example, inpatientsurvey, outpatient survey.8. How up to date is the information presentedin the QRP?We refresh the source data in each QRP on a monthly basis. However, the refreshprocess only affects a subset of the data within the QRP, as not all data sources changeeach month.9. What level of service does the QRP relate to?The QRP is presented at provider (NHS trust) level, rather than at location level. This isbecause most of the information currently held about NHS organisations is at provider level.However, where available the QRP will also show information below provider level.This information may be gathered from:• Our assessments at either each location or for a particular service.• Other external sources that use other categories to report information. For example,information from the Patient Environment Action Team (PEAT) is at hospital level,which may or may not have the same meaning as a registered location, and data aboutservice accreditation, such as accreditation of electroconvulsive therapy provision.Quality and Risk Profiles: Guidance for NHS providers8

10. How is information organised in the QRP?There are three main components of the NHS QRP:Component 1: Background information that describes the service.Component 2: Information relating to the essential standards of quality and safety.Component 3: Contextual risk measures. These refer to risks related to the type ofhealth or care service, the type of people who may be affected and risks arising fromthe organisation itself.11. What types of information are displayed incomponent 1?The first component of the QRP (background information) displays:• Descriptive information such as provider type, regulated activities and locations.• General information on regulatory performance that does not fall within the specificoutcome areas contained in the essential standards of quality and safety or thatrelates to recurrent themes (for example. equality and human rights).12. What types of information are displayed incomponent 2?The second component displays information in relation to the essential standards ofquality and safety. This is presented at three main levels:1. Level 1: The first level shows an overall summary of the risk for a group of outcomeareas (for example, Section 1: “Involvement and information” is a summary of riskfor outcomes 1 and 2 (see question 24 for more detail regarding how we derive thesummaries of risk for groups of outcomes).2. Level 2: The second level shows information for each of the regulatory outcomes (forexample “Outcome 1: Respecting and involving people who use services”). Included atthis level are the outcome risk estimates that have been produced from our statisticalmodel (see question 18 for a further explanation of outcome risk estimates).Quality and Risk Profiles: Guidance for NHS providers9

3. Level 3: This lowest level shows individual pieces or ‘items’ of information.This information is aggregated to calculate the outcome risk estimates in level 2(see appendix A for a more detailed explanation of an item of information).Question 14 of this guidance gives details about the information that you can see atthis level. For further information regarding the aggregation that has been performed,see the statistical guidance Quality and Risk Profiles: Statistical guidance.13. How is item level data displayed in the secondcomponent of the QRP?Item level information is structured around the outcomes in the essential standards.The information is displayed as a table that shows all the items for a particular outcomewith the following key pieces of information about each of the items:• An item identifier• A description of the measure• Where the information was sourced• The time period the information relates to• The outcome of the item level analysis• The actual values (numerator, denominator and expected) where these are applicable).14. What types of information are displayed incomponent 3?The third component displays contextual risk measures.This part contains information about an organisation’s risk that is not related to theRegulations. It may be about risks arising from the services that are provided, wherethey are provided from, and how they are organised. We call this the ‘environment’ or‘context’ within which the health service is provided.Quality and Risk Profiles: Guidance for NHS providers10

Contextual information is displayed in four categories:1. Inherent risk.2. Population risk.3. Situational risk.4. Uncertainty risk.You can view the underlying data that contribute to each of these categories.15. Is information displayed for all outcomesin the QRP?QRPs focus on the 16 essential standards that most closely relate to quality and safety.The ‘Suitability of Management’ section is not included as it does not contain any of theessential quality and safety outcomes (see Appendix D for details of the 16 essentialquality and safety outcomes).Over time, we will review how we use information relating to the additional outcomesand incorporate these into QRPs as appropriate.16. How will I know which site/location the itemlevel data relates to?If we have used information that relates to a specific area within an NHS provider,for example, site-specific data, the data description will include a note of which areaof the provider the item relates to.17. How is information displayed for NHS providersthat have been formed from a merger after1 April 2008?The data included in the QRP for these trusts may relate to their pre-mergerorganisations. There may be more than one instance of the same measure (item),as it was measured in all predecessor organisations.Quality and Risk Profiles: Guidance for NHS providers11

The data description will include a note of which previous provider the item relates to.Because of how we calculate contextual risk, NHS providers that were formed after1 April 2008 will not have contextual risk estimates in the early versions of the QRP.18. What is an outcome risk estimate and howis it calculated?For each quantitative (numerical) item of information, we assess whether your result isin line with what would be expected.For qualitative (textual) information, the information is converted into a numericrepresentation. Positive information indicates that you are less likely to be at risk ofnon-compliance, while negative comments suggest non-complianceWe then aggregate results for all items associated with an outcome. This produces anestimate of the risk of non-compliance with the outcome.For more information regarding item level analysis please see appendix A of thisguidance.19. Is all information weighted equally whenproducing an outcome risk estimate?The risk model has been constructed to recognise that some items of information areof greater importance than others to each of the outcomes. We give greater weight toinformation that:• Is most closely related to the outcome• Tells us about the experience of people who use services• Is of better quality (high data quality).Each of the weightings is on a 3-point scale, where 3 is the highest (‘best’) weighting.See Quality and Risk Profiles: Statistical guidance for more information.Quality and Risk Profiles: Guidance for NHS providers12

20. How does CQC measure confidence in arisk measure?As part of the analytical method used to calculate the outcome risk estimate, we alsoinvestigate how confident we are about the estimate before displaying it in the QRP.We assess how closely the items of information relate to each outcome and the qualityweighting of the items used to produce the estimate.If the information driving this estimate is not of sufficient strength, volume or data qualityto give us confidence in the risk estimate, the risk estimate is inconclusive.It is important to note:• This does not reflect the data quality of each individual item and only reflects theaggregation of items. Each item is assessed for data quality individually and onlyincluded in the QRP when it meets certain criteria.• An inconclusive outcome risk estimate is not a comment on your performance, but isa reflection of the information available to us about your trust for that outcome.Our separate guidance on the statistical methods gives more detailed informationabout this.21. Is an outcome risk estimate displayed wheneverthere is information for an outcome?No. Sometimes item level information is mapped to an outcome but an outcome riskestimate dial is not shown.This is because there is no confidence in the risk estimate produced from aggregatingall the items. Instead, the QRP will indicate that there is ‘no confidence’ at the outcomerisk estimate level, but you will still be able to view the individual item level results.If there is no information that can be mapped to the outcome, this is indicated.Quality and Risk Profiles: Guidance for NHS providers13

22. How are outcome risk estimates displayed inthe QRP?The results for the outcome risk estimates are displayed on a summary page,which groups outcomes under their relevant sections.Where it has been possible to produce a risk estimate with confidence, the results willbe displayed as a coloured dial, which has been designed to be a quick method forinterpreting risk at the outcome level. The dial represents the level of risk running from‘low’ on the left to ‘high’ on the right and the colour ranges from green (low risk) to red(high risk).23. How are the section summaries in theQRP calculated?For each section in the essential standards, the QRP provides a summary of theunderlying information. These are displayed as coloured dials which represent thehighest risk result from the outcome risk estimates within that section.24. What are the different types of contextual risk?Contextual risk has been grouped into four categories:• Inherent risk can be defined as the risk attributable to an organisation by virtue of itscare case mix i.e. high complex surgical volumes, high occupancy rates etc.• Population risk relates to those features in the local population that have beenshown to affect the outcomes of care or people’s access to care.• Situational risk is the risk attributable to the care provider by virtue of itsorganisational context i.e. continuity of staff, complaints management, effectivenessof internal process for risk management etc.• Uncertainty risk relates to a rules-based assessment of the completeness of theinherent, population and situational risks (i.e. the less information available in theother contextual risk groups, the greater the uncertainty risk).Quality and Risk Profiles: Guidance for NHS providers14

25. How will CQC use the contextual risk informationwithin the QRP?Contextual risk profiles in QRPs help to put the outcome risk estimates and informationcontained in the QRP into context.This helps us to make an informed assessment of your compliance and to evaluate theextent to which you are able to make the necessary improvements.The relationship between contextual risk and the outcomes is based on the followingassumption: if outcome risks are high and contextual risk is also high, the improvementchallenge is likely to be greater for the organisation.26. How is contextual risk calculated?The results of several pieces of information (which we call evidence-based riskmarkers) within a contextual risk category (e.g. situational risk) are combined using astatistical method called ‘bootstrapping’ to produce an overall estimate of risk for thatcategory.The information is also combined to produce an estimate of overall contextual risk(this does not include uncertainty risk).See appendix C for additional information on contextual risk.27. How is contextual risk displayed in the QRP?The overall estimate of contextual risk and the estimates of the category risks aredisplayed as coloured dials, similar to that of the outcome risk estimates. It is alsopossible to view the results of the individual risk markers.If there is no information, or the measures are not applicable, this is indicated as‘not applicable’.Quality and Risk Profiles: Guidance for NHS providers15

28. Who will have access to QRPs?Our compliance inspectors will have access to QRPs and we will share them with otherbodies:• NHS trusts will be able to view their updated QRP each month, starting fromSeptember 2010.• Lead commissioning PCTs will be able to view the updated QRPs for theirproviders, starting from October 2010.• Strategic health authorities and Monitor will be able to view the QRPs for therelevant trusts.29. How do I access the QRP?QRPs will be available through a secure website.We will issue a login and password to the nominated lead at each of the organisationsoutlined in question 28, which will give them access to the QRP that has been producedfor their organisation.30. Will the QRP be published externally?We do not currently have plans to publish the QRPs externally. However, we areplanning to publish ‘provider profiles’, which will display the judgements made by ourcompliance inspectors.31. Who should I contact if I have a query about the QRP?If you have queries regarding the information in your QRP (e.g. data inaccuracies,data sources) please email our National Contact Centre at enquiries@cqc.org.ukwith ‘NHS QRP query’ in the subject line.Or write to:Care Quality Commission National CorrespondenceCitygate, Gallowgate, Newcastle upon Tyne NE1 4PAQuality and Risk Profiles: Guidance for NHS providers16

Appendix A:What is an information item and how is it analysed?Items of information may be either qualitative or quantitative.Quantitative informationA quantitative information item contains numerical information. Different types ofnumerical information are:1. A proportion: A fraction of a total amount e.g. proportion of patients entering A&Ewho are seen within four hours.2. A ratio of counts: The ratio of two counts e.g. number of incidents reported inrelation to the number of admissions.3. A standardised ratio: The observed number of events divided by the expectednumber of events e.g. Standardised Mortality Ratios.4. Categorical outcomes: This is where information fits into a number of discretecategories e.g. the outcome of an assessment carried out by Patient EnvironmentAction Team (‘unacceptable’, ‘poor’, ‘acceptable’, ‘good’ and ‘excellent’).The result for one organisation for one item of information is called the observed resultfor that organisation.For an item, the organisation’s observed result is compared to an expected result.How far the observed result is from an expected result and the direction of thedifference (i.e. better or worse than the expected result) is analysed.This analysis produces a statistical measure, which is presented as one of sevencategories ranging from “much worse than expected” to “much better than expected”(see appendix B for further details of how these categories are assigned).The expected level can be set in two ways:1. An expected result may be set as the average of all the NHS organisationsmeasured.2. An expected result may be set based on established expectations e.g. a governmentpolicy or target, such as length of stay in A&E departments.Quality and Risk Profiles: Guidance for NHS providers17

An NHS organisation’s result may be significantly influenced by factors beyond theircontrol. In these cases, the ‘raw’ data for the provider may be standardised (forexample by age and sex) or the expected result may be set as the average for a groupof other organisations with similar local circumstances (referred to as the ‘benchmarkgroup’).None of the methods penalise (or reward) NHS organisation simply for being at thebottom (or top) of a list, they are designed to look for genuine differences from theexpected result. It is entirely possible that all NHS organisations will be performingsimilarly to expectation on a data item.Qualitative informationEach qualitative source, e.g. report, is analysed for information relevant to theregistration outcomes for a provider and is broken down into parts, which may be asentence, paragraph or more.An item of qualitative information is one of these parts. These differ from quantitativedata as one item only covers one provider and there may not be equivalent informationfor other providers.In order to be included in the QRP outcome risk estimates, the qualitative informationhas to be converted into a numeric representation.Each identified information item is determined as being either positive or negative,therefore indicating if the NHS organisation is less or more likely to be at risk of noncompliancerespectively.For more information regarding the statistical model used in the QRP please see ourstatistical guidance on Quality and Risk Profiles.Quality and Risk Profiles: Guidance for NHS providers18

Appendix B: Categorical descriptions of item levelanalysis results and assignment of descriptionsThe outcome of the analysis for each item is a statistical measure regarding how fareach provider’s result was from the expected level. The results are presented using thecategories shown in the table below:Risk estimate categoryDescriptionMuch worse thanexpectedWorse thanexpectedTending towardsworse than expectedSimilar to expectedTending towardsbetter than expectedBetter than expectedMuch better thanexpectedThe provider’s result is statistically much worse thanexpected – roughly this equates to the provider beingoutside the 97.5% confidence intervalThe provider’s result is statistically noticeably worsethan expected – roughly this equates to the providerbeing outside the 95% confidence intervalThe provider’s result is somewhat worse thanexpectation, but not at a level that would mean theobservation is notable on its own. However, a patternof items appearing in this category will increase ourestimate of the likelihood of undeclared non-complianceThe provider’s result is broadly in line with expectationThe provider’s result is somewhat better than theexpected levelThe provider’s result is statistically noticeably betterthan expectation – roughly this equates to the providerbeing outside the 95% confidence intervalThe provider’s result is statistically much better thanexpected – roughly this equates to the provider beingoutside the 97.5% confidence intervalQuality and Risk Profiles: Guidance for NHS providers19

Appendix C: Additional information on Contextual riskThe outcome of the analysis for each item of information in contextual risk section ofthe QRP is presented using the categories shown in the table below:Risk estimate categoryLikely to be riskySomewhat likely tobe riskyUnlikely to be riskyNot applicableDescriptionThe provider’s result is noticeably below theexpected levelThe provider’s result is somewhat below theexpected levelThe provider’s result is either broadly in line withexpectation, or is better than the expected levelNot applicableSituational risk has been broken down in the following way:1. Leadership- SR01: Quality of services- SR02: Financial management- SR03: Staff satisfaction2. General care- SR04: Vacancy rates: Doctors- SR05: Vacancy rates: Consultants- SR06: Vacancy rates: Nurses3. Governance and the use of information- SR07: Qualification after declaring compliant- SR08: Payment by Results- SR09: Auditors’ Local Evaluation (ALE)4. Risk management- SR10: National Health Service Local Authority (NHSLA)- SR11: Complaints: Closed- SR12: Complaints: NumberQuality and Risk Profiles: Guidance for NHS providers20

Appendix D: Sections in the essential standardswith key outcomesSection Outcome RegulationInvolvement andinformation1. Respecting and involving people whouse services172. Consent to care and treatment 18Personalisedcare, treatmentand support4. Care and welfare of people who use services 95. Meeting nutritional needs 146. Cooperating with other providers 24Safeguardingand safety7. Safeguarding people who use servicesfrom abuse118. Cleanliness and infection control 129. Management of medicines 1310. Safety and suitability of premises 1511. Safety, availability and suitability ofequipment16Suitability ofstaffing12. Requirements relating to workers 2113. Staffing 2214. Supporting staff 23Quality andmanagement16. Assessing and monitoring the quality ofservice provision1017. Complaints 1921. Records 20Section 6 of the essential standards, ‘Suitability of management’, is not included as itcontains no essential quality and safety outcomes.Quality and Risk Profiles: Guidance for NHS providers21