Blueprint and Test Specifications report - Medical Council of Canada

1DRAFTBLUEPRINT PROJECTProposed Qualifying ExaminationsBlueprint and Test SpecificationsStakeholder ConsultationsJune-November 2013

2BLUEPRINT PROJECTTHE MEDICAL COUNCIL OF CANADAstrives to achieve the highest level ofmedical care for Canadians throughexcellence in evaluation of physicians.The Council assesses over 11,000 medicalstudents and graduates every year throughits examinations.As part of the Assessment Review TaskForce recommendations, the MCCundertook a strategic review of itsassessment processes with a clear focuson their purposes and objectives, theirstructure and their alignment with MCC’smajor stakeholder requirements. The reviewencompassed current trends in medicaleducation, regulation and assessment.The review also took into account therole and purpose of MCC examinations inmeeting the current and future needs ofMRAs, the public and other stakeholders.Multi-stakeholder consultation and analysisactivities were conducted between 2009 and2011 with resulting recommendations andreport approved by Council in 2011.In addition to recommendations to focuson MCC’s reassessment and realignmentof exams, a key recommendation was tovalidate and update the blueprints for allMCC qualifying examinations.As part of its commitment to adhere tobest practices in medical education andassessment, the MCC has undertaken aBlueprint Project to review and establishan evidence-based approach for identifyingthe competencies that physicians will beexpected to demonstrate and be assessedon at two decision points: (1) entry intosupervised practice (residency), and (2)entry into unsupervised practice. Thepurpose of the MCC Blueprint Project is topromote assessments that assure the publicthat physicians licensed to practice medicinehave the required knowledge, skills andbehaviours for safe and effective patientcare.DRAFT

BLUEPRINT PROJECT3Defining the MCCAssessment BlueprintOver the past 18 months, work has been undertaken to set the stage to validate and update the MCCqualifying examination blueprints. With the establishment of a multi-stakeholder governance board, aproject steering committee, and a MCC project team, work proceeded with defining an evidence-based andconsultative approach to establish the blueprints.Gathering and Analyzing DataTo facilitate the task and inform the MCC Assessment Blueprint, evidence was provided through a series ofreports that highlight various facets of physician practice and assessment:• Report of the National Survey of thePhysicians, Pharmacists, Nurses, and Publicin Canada: 2013 - Explored the relativeimportance of physician knowledge, skillsand behaviours expected of physicians at thetwo decisions points – entry into supervisedpractice and entry into unsupervised practice.The National Survey was designed aroundthe CanMEDS roles 1 as per our presentobjectives.• Report of Incidence and Prevalence ofDiseases and Other Health Related Issuesin Canada - Highlights prevalent conditionsthat should be “covered” in medical licensingexaminations to ensure that doctors areadequately prepared to treat today’sCanadian population’s major health concernsas well as other areas of importance suchas population health, care of the elderly,aboriginal issues and patient safety.• Report on the Supervision of New PGY-1Residents: A Case Report of SupervisorsExpectations vs. Residents’ Perceptionsproject - Defined entrustable professionalactivities to offer perspectives on the levelsof supervision early in the postgraduate-yearone of residency. This report helped frameexpectations at the two decision points:entry into supervised practice and entry intounsupervised practice.• Report to the Medical Council of Canada onCurrent Issues in Health Professional andHealth Professional Trainee Assessment -Offers insight and perspectives into ways inwhich the MCC might consider adapting itscurrent assessment practices and developingadditional tools to align with modern andfuture conceptions of medical education andmedical practice.These reports, as summarized in Appendix A, offered perspectives on the importance of competencies,frequencies of clinical presentations, possible entrustable professional activities, and finally thoughts on thecurrent and future vision of assessment.[1]Frank, JR., Jabbour, M., et al. Eds. (2005). Report of the CanMEDSPhase IV Working Groups. Ottawa: The Royal College of Physiciansand Surgeons of CanadaDRAFT

BLUEPRINT PROJECT4Defining the Blueprint andSpecifications for AssessmentsIn defining the blueprint and test specifications for assessments, MCC invited twelve physician subjectmatter experts (SMEs) to provide judgments concerning the content that should be assessed at the twodecision points. The subject matter experts brought various viewpoints to the blueprint definition including:• MCC Council –with MRA and University representatives• Certification bodies –RCPSC and CFPC representatives• Internal MCC Committees –Central Examination Committee,Objectives Committee and Test Committeerepresentatives• Medical Education –UGME and PGME Deans representativesA three-day workshop was held in mid-May 2013 to seek consensus and a recommendation for a blueprintand test specifications for MCC qualifying examinations.Recommended Blueprint and Test Specifications -Themes and OutcomesThe physician subject matter expert (SME) panel were able to reach agreement in describing the candidatesand in identifying a common assessment blueprint. Test specifications were proposed for candidates at twodecision points – entry into supervised practice and entry into unsupervised practice.The proposed blueprint consists of two broad domains each broken down into four sections (Appendix B).Dimensions of Care 2Focus of care for the patient, family, community and/or population• Health Promotion and Illness PreventionThe process of enabling people to increasecontrol over their health and its determinants,and thereby improve their health. Illnessprevention covers measures not only to preventthe occurrence of illness, such as risk factorreduction, but also to arrest its progress andreduce its consequences once established 3 .This includes, but is not limited to screening,periodic health exam, health maintenance,patient education and advocacy, andcommunity and population health.[2]Modified from Wenghofer, E.F., Williams, A.P., Klass, D.J., Faulkner, D.Physician-Patient Encounters: The Structure of Performance in Family andGeneral Office Practice. J Cont Educ Health Prof 2006; 26:285-293.[3]World Health Organization, www.who.int/chp/en accessed 14/05/2013DRAFT

BLUEPRINT PROJECT5• AcuteBrief episode of illness within the time spandefined by initial presentation through totransition of care. This dimension includes butis not limited to urgent, emergent, and lifethreateningconditions, new conditions, andexacerbation of underlying conditions.• ChronicIllness of long duration that includes but isnot limited to illnesses with slow progression.• Psychosocial AspectsPresentations rooted in the social andpsychological determinants of health thatinclude but are not limited to life challenges,income, culture, and the impact of thepatient’s social and physical environment.Physician ActivitiesReflects the scope of practice and behaviors of a physician practicing in Canada• Assessment/ DiagnosisExploration of illness and disease throughgathering, interpreting and synthesizingrelevant information that includes but is notlimited to history taking, physical examinationand investigation.• ManagementProcess that includes but is not limitedto generating, planning, organizing carein collaboration with patients, families,communities, populations, and health careprofessionals (e.g., finding common ground,agreeing on problems and goals of care, timeand resource management, roles to arrive atmutual decisions for treatment).• CommunicationInteractions with patients, families, caregivers,other professionals, communities andpopulations. Elements include but arenot limited to active listening, relationshipdevelopment, education, verbal, non-verbaland written communication (e.g., patientcentered interview, disclosure of error,informed consent).• Professional BehaviorsAttitudes, knowledge, and skills basedon clinical and/or medical administrativecompetence, communication skills, ethics,societal, and legal duties resulting in the wiseapplication of behaviors that demonstrate acommitment to excellence, respect, integrity,empathy, accountability and altruism. (e.g.,self-awareness, reflection, life-long learning,scholarly habits and physician health forsustainable practice).These blueprint domains were thought to reflect practice that can be assessed not only at both qualifyingexamination decision points but also could be applied to any physician in any practice. This blueprint reflectsknowledge, skills and behaviors that all physicians practicing in Canada should demonstrate when takingcare of patients, families, communities and the population at large. Principles of patient safety underscoreeach of the dimensions. These dimensions can also be mapped to the CanMEDS roles with many rolesrepresented in many “boxes” within the 2 x 2 table.DRAFT

BLUEPRINT PROJECT6Test specificationsTest specifications are guidelines about how to sample each domain that is being assessed. The SMEshave proposed test specifications for each of the two decision points; entering supervised practice andentering unsupervised practice. Although all categories should be assessed at each decision point,differences do exist to emphasize certain areas. The test specifications propose that both the dimensionsof Communications and Professional Behaviors should have significant weight at both decision points.As well, the proposed specifications place slightly more emphasis on the Acute dimension of care andthe Assessment/Diagnosis activity for assessing physicians at the first decision point, readiness to entersupervised practice (Appendix C); whereas the emphasis is more on the Chronic dimension of care and theManagement activity for the second decision point, readiness to enter unsupervised practice (Appendix D).Approval-in-PrincipleBased on the recommendations from the physician subject matter expert panel in mid-May, the CentralExamination Committee (CEC) met on June 11, 2013 to review the proposed blueprint and test specifications.The CEC accepted a motion for approval-in-principle of the blueprint and test specifications that included abroad stakeholder consultation process.NEXT STEPSAs part of the consultation process, the MCC will reach out to engage various external stakeholderorganizations and internal stakeholder committees in a transparent manner to provide them with theopportunity to comment on the proposed MCC Blueprint and Test Specifications.If your organization wishes to have the opportunity to comment, please contact Dr. Claire Touchie atctouchie@mcc.ca. The MCC would be pleased to attend existing meetings as part of our consultationprocess.Once consultations are complete, feedback assessed and reflected upon the proposed MCC Blueprint andTest Specifications will be reviewed by the MCC Executive Board in January 2014 and proceed to Councilreview in September 2014.DRAFT

BLUEPRINT PROJECT7APPENDIX AReport HighlightsReport of the National Survey of the Physicians, Pharmacists, Nurses,and Public in Canada: 2013This report provides an overview of the survey that the MCC conducted as part of thepractice analysis of physicians who are either a) starting residency training (supervisedpractice) or b) newly licensed (entering unsupervised practice). The survey content wasbased on previously developed MCC objectives. Physician judgments of importance werecollected based on expected physician abilities at the two decision points. Pharmacists,nurses, and informed public members were asked to provide judgments of importance onappropriate competencies. The physician survey revealed that both medical expert andnon-medical expert roles are important, and that those competencies deemed importantat the first decision point continue to grow in importance as a physician transitions intoindependent practice.Report of Incidence and Prevalence of Diseases andOther Health Related Issues in CanadaThe primary purpose of this study is to review and summarize the frequency of mostcommon and important diagnoses as seen in the Canadian context by reviewing existingdata. Available data included inpatient admissions, emergency department visits andoutpatient care visits. The data are organized by age groups and by certain conditions,such as women’s health, psychiatric conditions and chronic medical conditions. Theinclusion of these prevalent conditions in the specifications for medical licensingexaminations would help to ensure that doctors are adequately prepared to treat today’sCanadian population’s major health concerns.As a secondary purpose, this report includes a review of certain areas of importancefor the practice of medicine which may not be well represented by patient presentationsand diagnoses or the MCC competencies. Examples include areas such as populationhealth, care of the elderly, patient safety issues and aboriginal health.DRAFT

BLUEPRINT PROJECT8Report on the Supervision of New PGY-1 Residents:A Case Report of Supervisors' Expectations vs. Residents’ PerceptionsprojectPostgraduate year-1 residents (PGY-1s) begin supervised practice in a setting wheresenior residents and attending staff are available to guide and review their work. Thereis an expectation that PGY-1s are supervised but that they will gradually take on moreresponsibilities and function independently as they become more experienced.The purpose of this study was to determine whether there is a discrepancy betweenthe level of supervision expected by clinical supervisors (CSs) and what PGY-1sactually report doing.Of the ten Entrustable professional activities (EPAs) identified (e.g., patienthandover, obtaining informed consent), three showed significant differences in thelevel of supervision reported between the CSs and PGY-1s. These included: (1) themanagement of intravenous fluids, (2) obtaining informed consent, and (3) obtainingadvanced directives. Three more EPAs (recognition and initiation of management ofcritically ill patients, handover of patient care, and coordination of patient discharge)were significantly different when performed at nighttime.PGY-1s reported performing EPAs with less supervision than expected by CSs,especially during nighttime. Using EPAs to develop curriculum and assessment toolsprior to residency may help better align CSs’ expectations with PGY-1 activities.Report to the Medical Council of Canada on Current Issues in HealthProfessional and Health Professional Trainee AssessmentThe fundamental purpose of this report is to offer insight into ways in which the MCCmight consider adapting its current assessment practices and develop additionaltools to align with modern and future conceptions of medical education and medicalpractice, with an ultimate goal of assuring better healthcare for all Canadians.DRAFT

BLUEPRINT PROJECT9Appendix CDecision 1 – Entry into Supervised PracticeAPPENDIX BProposed Blueprint Overview Specification 1- GridDIMENSIONS OF CAREHealth Promotionand IllnessPreventionAcuteChronicPsychosocialAspectsRow %PHYSICIAN ACTIVITIESAssessment/Diagnosis 30±5Management 20±5Communication 30±5ProfessionalBehaviours 20±5Column % 20±5 30±5 30±5 20±5 100DRAFT

BLUEPRINT PROJECT10Appendix CDecision 1 – Entry into Supervised PracticeAPPENDIX CDecision 1 – Entry into Supervised Specification Practice1- GridSpecification 1 – GridDIMENSIONS OF CAREHealth Promotionand IllnessPreventionAcuteChronicPsychosocialAspectsRow %PHYSICIAN ACTIVITIESAssessment/Diagnosis 30±5Management 20±5Communication 30±5ProfessionalBehaviours 20±5Column % 20±5 30±5 30±5 20±5 100Specification 1 – ConstraintsCONSTRAINTCATEGORYDESCRIPTIONCONDITIONComplexity Multiple morbidities at least 10%AgeGenderSpecialpopulationsSettingNeonate, infant/child, adolescent, adult,adult women of childbearing age, andthe frail elderlyMale, femaleIncluded but not limited to immigrant,LGBT, rural, disabled, and First Nationpopulations; end of life patients,refugees, inner city poor, the addictedand the homelessIncluded but not limited to rural orremote settings, long term careinstitutions and home visitssample across the age categoriesincluding adult woman of childbearingage and the frail elderlybalance evenly(minimum of 40% each)sample across categoriessample across categoriesDRAFT

Appendix DBLUEPRINT PROJECT11Decision 2 – Entry into Unsupervised PracticeSpecification 2- GridAPPENDIX DDecision 2 – Entry into Unsupervised PracticeDIMENSIONS OF CARESpecification 2 – GridHealth Promotionand IllnessPreventionAcuteChronicPsychosocialAspectsRow %PHYSICIAN ACTIVITIESAssessment/Diagnosis 25±5Management 35±5Communication 20±5ProfessionalBehaviours 20±5Column % 20±5 25±5 35±5 20±5 100Specification 2 – ConstraintsCONSTRAINTCATEGORYDESCRIPTIONCONDITIONComplexity Multiple morbidities at least 20%AgeGenderSpecialpopulationsSettingNeonate, infant/child, adolescent, adult,adult women of childbearing age, andthe frail elderlyMale, femaleIncluded but not limited to immigrant,LGBT, rural, disabled, and First Nationpopulations; end of life patients,refugees, inner city poor, the addictedand the homelessIncluded but not limited to rural orremote settings, long term careinstitutions and home visitssample across the age categoriesincluding adult woman of childbearingage and the frail elderlybalance evenly(minimum of 40% each)sample across categoriessample across categoriesDRAFT