Validity Evidence

members.aamc.org
  • No tags were found...

Validity Evidence

ConsultationsCommunication andInterpersonal SkillsCommunication withColleaguesTelephone Consultations fromthe ED


In the Emergency Departmentconsultations occurin 20%-40% of cases(Lee Emerg Med J 2008)29% of EM docs report a lack ofstandardized consultation protocolA majority believe residents areinadequately trained inconsultation (Kessler J Emerg Med 2012)


What are the characteristics of anEffective Consultation?Literature reviewQualitative study of EM consultations(Kessler et al J Emerg Med 2012)Consensus panel of 8 expertsBusiness model: “7 C’s of Consulting” byMick Cope• Client, clarify, create, change, confirm,continue, close


The Five C’s of Phone Consultation(Kessler J Emerg Med 2012)ContactCommunicateCore QuestionCollaborationClosing the Loop


Validity is an argumentIs it reasonable to use thescores from this assessmentfor a particular purpose?


Sources of Validity EvidenceNCME; Downing 2003ContentResponse ProcessInternal StructureRelationships to other VariablesConsequences


ContactIntroducing theconsulting andconsultantphysicians;building therelationship.States nameStates rank andserviceIdentifies supervisingattendingIdentifies name ofconsultant physician


CommunicateGiving a concisestory and askingfocusedquestionsPresents a concisestoryPresents an accuraterecount ofinformation/casedetailSpeaks clearly


Core QuestionPreparing aspecific questionfor theconsultant.Deciding on areasonabletimeframe forconsultationSpecifies need forconsultationSpecifies timeframefor consultation


CollaborationPlanning a courseof action basedon thediscussionIs open to andincorporatesconsultant’srecommendations


Closing the LoopEnsuring that bothparties agree tothe plan and tomaintainingcommunicationabout the patientReviews and repeatspatient care planThanks consultant forconsultation


Sources of Validity EvidenceNCME; Downing 2003ContentResponse ProcessInternal StructureRelationships to other VariablesConsequences


Data: from Prospective RCT(Kessler Acad Emerg Med 2012)47 EM ResidentsIntervention N=19 Control N=2490 min Didactic on 5Cs:•Review sample cases•Practice and feedback•Note cards90 min DidacticCommunication skillsunrelated to 5CsmodelAssessment: 2 simulatedphone consultations in the2 weeks following


The Simulated Consultations• PsychiatryConsultation: apatient with psychosis• SurgeryConsultation:worsening abdominalpainThe “StandardizedConsultant”:• EM/IM attending with10 years experience• No knowledge of 5Cs• About 1 hr trainingplus mock encounters


The checklist data• All simulated phone consults were recorded• 2 consults/resident x 43 residents• 3 trained EM attendings rated all recordingsusing the 12-item checklist


Sources of Validity EvidenceNCME; Downing 2003ContentResponse ProcessInternal StructureRelationships to other VariablesConsequences


Response process• Inter-rater reliability (intraclasscorrelations) = 0.94


Sources of Validity EvidenceNCME; Downing 2003ContentResponse ProcessInternal StructureRelationships to other VariablesConsequences


Validity evidence:Internal Structure• Internal consistency (Cronbach’s Alpha)for each rater: 0.7, 0.7, 0.8• Generalizability• Phi = 0.89• D study Phi of 0.80• 2 raters x 1 case• 1 rater x 3 cases


Sources of Validity EvidenceNCME; Downing 2003ContentResponse ProcessInternal StructureRelationships to other VariablesConsequences


Validity evidence:Relationships to other Variables• Intervention group scored higher on 9/12items• Mean checklist scores of Interventiongroup > control group (p


Validity evidence:Relationships to other Variables• Global rating scale• 7 items• Intro, case, objective, discussion, confirmation, IPS,overall• 5 pt scale (not effective extremely effective)• 3 different raters not familiar with 5C’s• Global rating by consulting specialist• 1 global item, 5-point scale (not effective –extremely effective)• 1 surgeon, 1 psychiatrist


Validity evidence:Relationships to other Variables• Correlation to global rating scale (EMraters): p


ConclusionsValidity Evidence:• Content• Response process• Internal Structure• Relationships toother variablesSupports the use of5C’s checklist toassess the phoneconsultation skills ofEM residents


Limitations• One EM program, one institution• Other instruments (GRS and GR item) notvalidated• Simulated consults and context may notrepresent behavior in real life• Need G studies in natural settings


Thank youChad KesslerDeputy Chief of Staff, Durham VAMCChad.Kessler@va.govRachel Yudkowskyrachely@uic.edu

More magazines by this user
Similar magazines