Advancing Educators and Education - AAMC's member profile

Advancing Educators and Education:Defining the Components and Evidence ofEducational ScholarshipLearnServeLeadSummary Report and Findings from the AAMC Group on EducationalAffairs Consensus Conference on Educational ScholarshipAssociation ofAmerican Medical Colleges

Advancing Educators and Education:Defining the Components and Evidenceof Educational ScholarshipSummary Report and Findings from the AAMC Group on Educational AffairsConsensus Conference on Educational ScholarshipAuthorsDeborah Simpson, Ph.D.Medical College of WisconsinRuth Marie E. Fincher, M.D.Medical College of GeorgiaJanet P. Hafler, Ed.D.Tufts University School of MedicineDavid M. Irby, Ph.D.University of California, SanFrancisco, School of MedicineBoyd F. Richards, Ph.D.Columbia UniversityGary C. Rosenfeld, Ph.D.University of TexasMedical School at HoustonThomas R. Viggiano, M.D., M.Ed.Mayo Medical School

For more information about this publication contact:M. Brownell AndersonSenior Associate Vice PresidentDivision of Medical EducationAAMC2450 N Street, N.W., Washington, D.C. 20037T 202-828-0562E mbanderson@aamc.org©2007 by the Association of American Medical Colleges. All rights reserved.

Scholarship Consensus ConferenceTable of ContentsSummary Report and FindingsBackground . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1Conference Design and Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Conceptual Framework:Quantity, Quality and Engagement with the Educational Community Educator Activity Categories, Criteria and Evidence• Teaching• Curriculum• Advising/Mentoring• Educational Leadership/Administration• Learner AssessmentNext Steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13 Crystallizing and Building on our Current Knowledge Unresolved Issues: Opportunities for Crucial Conversations about EducationSummary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15Conference Materials Pre-conference Readings [references] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Illustrative Portfolios to Trigger DiscussionSusan Masters, Ph.D. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19Karen Wendelberger-Marcdante, M.D. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 Educator Activity Category Format and Illustrative Examples . . . . . . . . . . . . . . . . 28Contributors and Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35Conference Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40iAssociation of American Medical Colleges, 2007

Scholarship Consensus ConferenceBackgroundEvolutionary Forces and Benchmarks inDefining Educational ScholarshipIn the early 1990s, the academic medicine communityrarely used the terms education, teaching, scholarship,and academic promotion in combination witheach other. Teaching was expected as part ofacademic citizenship, but not sufficient for academicpromotion. This perspective on teaching dominatedacademic medicine until the Carnegie Foundationfor the Advancement of Teaching published ErnestBoyer’s Scholarship Reconsidered: Priorities of theProfessoriate. 1 Boyer’s work reframed and expandedthe discussion regarding roles, expectations, recognition,and advancement of educators, providing aframework from which to challenge the prevailingconcept that “everyone teaches,” and replacing it withone that examined teaching as a form of scholarlywork. 2 The discussion was enriched by publicationof Scholarship Assessed, 3 which articulated commoncriteria for all forms of scholarship: clear goals,adequate preparation, appropriate methods, significantresults, effective presentation, and reflectivecritique.Concurrently, the emergence of certain externalforces—research dominance in medical schools aswell as dependence on clinical revenue for operationsbudgets—changed the academic medicineenvironment and created “crises of mission” relatedto medical school faculty roles and rewards. Schoolleaders began to recognize and respond to thecrisis, prompting Whitcomb 4 to report “widespreadagreement that those members of the faculty whoare most committed to, and involved in, the educationof medical students must be supported andrewarded, both professionally and financially.…”Medical schools are “cognizant that facultyappointment, promotion, and tenure policies mustreflect the changing roles and responsibilities ofmedical school faculty” by greater recognition ofeducation.Evidence is slowly emerging in support of educatorrecognition, including education as a viable careertrack, 5 the use of educator portfolios for academicpromotion, 6 the ongoing examination of elementsused by promotion committees, 7 expectations ofthose directing medical student clerkships fromeducation-related professional organizations, 8development of compacts between residents andtheir teachers, 9 and the proliferation of educationalacademies and societies. 10National organizations have played a key role byclarifying issues and potential solutions related toeducator recognition. In particular, the AAMC’sGroup on Educational Affairs (GEA) has longrecognized that excellence in physician education isdriven by faculty members, and that they must berecognized and rewarded as educators. Beginningin 1996, GEA members began to elucidate thecriteria for scholarship in medical education with aseries of case studies, 11 and then defined scholarship’score elements and the associated resourcesand infrastructure to support educators asscholars. 10, 12 GEA members documented thedramatic increase in the use of education portfoliosin the U.S. medical school academic promotionprocess—from 5 schools in 1990 to 76 schools in2003—which reflected increased attention toacademic promotion of educators. 6However, despite the emergence of a common setof educator activity categories, the evidencepresented in portfolios was highly variable. Thisvariability underscored the need to articulate a setof common standards for selecting, presenting, andevaluating evidence of educational contributionsfor academic promotion.In this paper, we provide background for theconsensus conference and present a set of standardsfor use in the educator academic promotionprocess. We delineate the contents of five literaturebasededucator activity categories. For each, wepresent standards for documenting quantity,1Association of American Medical Colleges, 2007

Scholarship Consensus Conferencequality, and scholarly engagement; a framing modelemerging from the conference. The appendixincludes illustrative examples for data presentationby category as part of educators’ academic promotionmaterials. The model, standards, and examplesare derived from the literature and the AAMC’sGEA Consensus Conference on EducationalScholarship, held February 9–10, 2006, in Charlotte,N.C.Need for Educational Scholarship ConsensusThe increased use of education portfolios and theemerging consensus regarding educator activitycategories reflects the growing attention to education-relatedcontributions in the academic promotionprocess. However, the variability in formatsand types of evidence presented for academicpromotion has been confusing. Therefore, the GEAheeded the call to elucidate a set of common standardsto guide educators and faculty promotioncommittees.The goal of the Consensus Conference onEducational Scholarship was to identify points ofagreement and disagreement to facilitate continuedevolution of the field around three target areas:1. Educator activity categories (e.g., curriculumdevelopment, advising)2. Appropriate forms of evidence and presentationdisplays for each category3. Areas that need further investigation specific toeducational scholarship.In June 2004, the GEA Steering Committeeapproved a concept proposal to host an AAMCGEA Consensus Conference on EducationalScholarship and convened a planning group. Theplanning group’s implementation proposal,approved in November 2004, adapted theconsensus conference model used for the 1992consensus conference on standardized patients. 132Association of American Medical Colleges, 2007

Scholarship Consensus ConferenceConference Designand OrganizationConference Structure and FormatAchieving consensus on educational scholarship,both conceptually and pragmatically, in evaluatingthe work of faculty for key decisions like academicpromotion requires interaction among key stakeholders,including deans, senior educationaladministrative leaders across the continuum,faculty affairs leaders, department chairs, andrepresentatives of professional societies.Representatives from the AAMC’s Council ofDeans, Council of Academic Societies (includingthe Society of Directors of Research in MedicalEducation), faculty affairs, and the GEA wereinvited to participate in the consensus conference.Conference announcements were circulated toboth the GEA and specialty-specific listservs.Registrants received a preconference electronicinformation packet that included three articles,which provided a common foundation to identifyand build on what is known about educationalscholarship, 2, 14, 15 along with two optional readings.6, 12 The one-and-a-half day conference beganwith a review by Deborah Simpson, Ph.D., of theconference’s background, purpose, and structure,followed by two plenary sessions to set the contextfor discussion. In the first plenary session, JanetHafler, Ed.D., and Ruth-Marie E. Fincher, M.D.,provided an overview of the GEA’s involvement inmedical education scholarship and the current stateof the art. Building on the frameworks of Boyerand Glassick, they emphasized that scholarshipproducts must be public, open to evaluation, andpresented in forms that others can build upon. 16Hafler and Fincher drew the distinction betweenexcellence, such as an outstanding teacher, and ascholarly approach in any given activity (oneinformed by the literature or other programs) aspart of a pathway to scholarship. 17Patricia Hutchings, Ph.D., vice president of theCarnegie Foundation for the Advancement ofTeaching, framed the second plenary sessionaround four key questions: (1) Is it scholarship? (2)Is it excellent scholarship? (3) Does excellence takedifferent forms at different career stages? (4) Whatneeds to be done? Using a combination of casestudies to illustrate the answers, Dr. Hutchings reiteratedthe three parameters of scholarship—that itis public, peer reviewed, and available in a platformthat others may build on 16 —and emphasized thatthe peer-review element requires a capacity forjudgment about quality by a community of expertsusing common criteria. The criteria for judgingscholarship with clear goals and adequate preparationare well established based on Glassick’s work,but the capacity and supporting organizationalinfrastructure and culture for judgment by experteducational reviewers is still evolving. 18At the conclusion of the two plenary sessions,participants attended one of five preassignedworking groups. Each was assigned an educatoractivity category (curriculum development,teaching, mentoring/advising, educational administration/leadership,learner assessment) 6 andcharged with defining the contents of a promotionportfolio for the assigned category. Three facultyportfolios provided by the planning committeetriggered the discussion. The working groupsreported their findings to the larger group byanswering three questions:1. What educator activities should be included inthe category?2. What types of evidence should be included forreview in the academic promotion process?3. How should a faculty member present the activitiesand associated evidence in a promotionportfolio?One or two members of the conference planningcommittee facilitated each group and presented itsfindings to the assembly. After the discussions, facilitatorsmet to review findings and frame the reports3Association of American Medical Colleges, 2007

Scholarship Consensus Conferencefor presentation to the entire conference. Eachworking group’s six-minute report focused on areasof consensus specific to the three questions.Following the reports and reactions from conferenceattendees, each working group reconvened forabout 90 minutes to reconsider and modify its findingsin light of the assembly presentation anddiscussion. Each group then identified commonthemes across the working groups as areas ofconsensus. These convergent themes became theconference’s core consensus findings on educationalscholarship.A panel of three discussants concluded the conferenceby providing additional perspective on theemergent themes. The panelists, each representingkey stakeholder constituencies, were• Darrell G. Kirch, M.D., then dean, College ofMedicine, and senior vice president for healthaffairs, Pennsylvania State University, chief operatingofficer, Milton S. Hershey Medical Center;he is now president, Association of AmericanMedical Colleges.• Patricia Hutchings, Ph.D., vice president,Carnegie Foundation for the Advancement ofTeaching.• LuAnn Wilkerson, Ed.D., professor of medicineand senior associate dean for medical education,David Geffen School of Medicine, University ofCalifornia at Los Angeles, and the Society ofDirectors of Research in Medical Educationrepresentative to the Council of AcademicSocieties.Conference Working Groups’ Data Records andSynthesisFollowing the conference, attendees were contactedto complete an electronic evaluation form focusingon the conference’s overall value, format (includingpreparatory readings), key elements, and take-homelessons. The evaluations were compiled by theAAMC conference planning staff and forwarded tothe working group for review and analysis.Each working group facilitator assembled findingsbased on newsprint, worksheet, and field notesfrom the sessions. The assembly presentations werestored as PowerPoint presentations and audiotaped;the discussant panel’s remarks were also audiotaped.Transcriptions of the working group findingsand the discussant panel’s remarks wereforwarded to the facilitators. Each facilitator drafteddetailed findings specific to the three conferencequestions and forwarded the draft documents totheir conference working group members forcomment. Revised working group reports werethen forwarded to the conference chair Dr.Simpson, for synthesis.Participants Reflected Key ConstituenciesThe 111 conference attendees represented keystakeholders and included 6 medical school deans,5 department chairs or vice chairs, 32 dean-levelrepresentatives from academic affairs or education(across the physician education continuum), 16dean-level faculty affairs representatives, 23 dean ordirector-level individuals involved in faculty developmentor medical education research, and 3 directorsof medical school academies or societies. Thespectrum of disciplines and medical specialtiescommon to U.S. medical schools was also represented.Seven participants from the AAMC representedmedical education; institutional, faculty, andstudent studies; organization and managementstudies; and medical school affairs.Conference Evaluation by AttendeesSixty-eight percent of conference attendees (77 of111) submitted an evaluation. Overall, they felt theconference was valuable, with 92 percent reportingthat the “meeting was worth their time away fromtheir institution.”Respondents were asked to list the three most andthree least valuable conference components; 212 ofthe former and 65 of the latter were submitted.Content analysis revealed a series of commonthemes. A frequently cited strength was the focuson a national consensus for the definition of educationalscholarship (“that it happened at all”),4Association of American Medical Colleges, 2007

Scholarship Consensus Conferencenetworking, and working with others in themedical education community on a common issue.Other commonly valued components were theconference leaders’ expertise; Patricia Hutchings’splenary presentation and insights, as well as otherdiscussants’ perspectives; attendance of key leaders,including the incoming AAMC president; thebreadth of constituencies represented (“fascinatingguest list”); and the small groups. A consistenttheme was the opportunity to interact withcolleagues on an issue of shared importance. Theconference’s overall value was best captured by oneparticipant: “Although my head is swimming and Iam exhausted, I have some very exciting new ideasthat I am working on—and hope that the AAMCwill move this agenda forward.”Most frequently cited, least-valued componentsfocused on three areas: time limitations of breakoutgroups, overlap of preconference readings with thefirst plenary content, and the limited application ofthe readings in the discussions relative to thebreakout groups. “Having to artificially separate theeducational activities (teaching, assessment, etc.)”seemed to limit the depth of discussion.5Association of American Medical Colleges, 2007

Scholarship Consensus ConferenceResultsConceptual Framework: Quantity, Quality, andEngagement with the Educational CommunityEducators add value to their institutions bycontributing to the educational mission and/or byadvancing knowledge in the field of education.Figure 1 (on page 8) frames a model for documentingthe quantity and quality of educatoractivities along with evidence of the educator’sengagement with the educational community.Engagement is documented with evidence thateducators’ work is informed by what is known inthe field—a scholarly approach—and how, overtime, educators contribute to knowledge in thefield—educational scholarship. There is bothsynergy and tension between faculty’s roles aseducators and as contributors to the broadermedical education field. For promotion and tenuredecisions, each institution must determine the relativebalance of these roles based on its mission andinfrastructure support for education.Evidence of educational excellence must documentthe quantity and quality of educational activities:• Quantity: Descriptive information regarding thetypes and frequencies of educational activitiesand roles.• Quality: Evidence that activities achieve excellenceusing comparative measures, when available.An educator must engage with the broader educationalcommunity to demonstrate a scholarlyapproach. 19 The type of engagement—scholarlyapproach and/or educational scholarship—and thebreadth of engagement—local, regional, national,or international—required for academic advancementmay vary by faculty rank and institution. 20• Scholarly Approach: Faculty take a scholarlyapproach when they systematically design, implement,assess, and redesign an educational activity,drawing from the literature and “best practices”in the field. Documentation describes how theactivity was informed by the literature and/orbest practices.• Educational Scholarship: Faculty engage ineducational scholarship by both drawing uponresources and best practices in the field and bycontributing resources to it. Documentationbegins by demonstrating that an educationalactivity product is publicly available to the educationcommunity in a form that others can buildon. 21 The product may be available at the locallevel—in the department, medical school, oruniversity—or at the regional, national, or internationallevel. Once a product is public and in a formthat others build on, peers can assess its value tothe community applying accepted criteria. 3, 15Educators seeking academic promotion maypresent evidence focused on a single educationalactivity category, such as teaching, or in multiplecategories, such as curriculum, learner assessment,and/or leadership. The types and forms of evidencemay vary by category, but documentation shouldbe both quantitative and qualitative and conciselypresented using common terminology, anddisplayed in easy-to-read formats using tables,figures, or graphs.Educator Activity Categories, Criteria, andEvidenceThe categories of teaching, curriculum development,mentoring/advising, educational administration/leadership,and learner assessment emergedfrom the literature as common formats inpresenting educational contributions for academicpromotion. 11 For each category, we present a briefdefinition and illustrative types of educator activities,followed by descriptions of types of evidenceto document quantity, quality, and engagementwith the educational community.TeachingTeaching is any activity that fosters learning,including direct teaching and creation of associatedinstructional materials. Examples of direct teachinginclude lectures, workshops, small-group facilitation,role modeling in any setting (such as wardattending), precepting, demonstration of proceduralskills, facilitation of online courses, and formativefeedback. Summary judgments that teachersprovide to learners in the form of grades are also6Association of American Medical Colleges, 2007

Scholarship Consensus Conferenceincluded in the learner assessment category.Instructional materials are included in the teachingcategory when they are developed to specificallyenhance instructors’ own presentations, such asmedia, handouts, or interactive materials.Development of a longitudinal set of educationalactivities would fall into the curriculum developmentcategory.Quantity: Documentation of the frequency andduration of teaching along with a description ofone’s role should be presented in an easy-to-read,concise format. A listing of instructional materialsauthored with a brief description of their purpose,format, and length should be included. The use oftables and figures rather than narrative facilitatesconcise presentation.Quality: Multiple sources and types of data shouldbe used to demonstrate teaching excellence. Includecomparative data of peer-group performance usingthe same source and method whenever possible.Summarize narrative comments using qualitativeanalysis methods. Data sources might include• Learners’ confidential evaluations of instructors’teaching using standardized forms with openendedcomments. In settings with small numbersof learners, consultant-facilitated discussions canyield narrative data for qualitative analysis. Datamay also be obtained at the end of teachingsessions, rotations, or years, and post-graduation,to assess short- or long-term impact.• Peer evaluation of teaching using a standardizedformat and process adds an important dimensionthat complements student evaluation.• A list of teaching awards and honors accompaniedby descriptions of their selection processand criteria are additional forms of teachingexcellence documentation.• Evidence of learning, the key outcome ofteaching, is a strong indicator of excellence. Anarray of local learner data may be availableincluding pre- and post-teaching assessments oflearner performance, self-reported learningoutcomes, ratings of educational objectives7achievement, or analysis of narrative data, such aslearning portfolios or critical incidents.• When learner performance data on standardizednational measures is used to assess teaching effectiveness—suchas in-service examinations,National Board of Medical Examiners (NBME)Subject Tests, or the United States MedicalLicensing Examination—it must be carefullyinterpreted, as connecting learner performance toindividual teachers is difficult.The methods that demonstrate and document thevalue of one’s own instructional materials aresimilar to those used for curriculum development(see next section). Multiple data sources and typesshould be provided when possible, including• Learner evaluations using standard rating scalesor narrative comments, including comparativeevaluation to peers.• Peer review by members of a teacher’s division,department, or institutional committee can helpdocument the accuracy and educational value ofthe content, with an eye toward objectives,format, organization, and innovation.Engagement with the Education Community: Ascholarly approach requires that instructors applythe principles and findings from the educationliterature (e.g., competency-based education, deliberatepractice) to their teaching, along with developmentof associated instructional materials.Evidence of engagement with the larger educationcommunity can be documented through• Descriptions of how teachers’ approaches or usesof instructional materials were informed by theliterature or “best practices.”• Graphical presentation of a comparative analysisof teachers’ own materials with “best practices” inthe field, documenting relative strengths andweaknesses.• Instructors’ reflections on their own teaching oron critiques by others, and the effect of thosereflections on subsequent teaching activities.Association of American Medical Colleges, 2007

Scholarship Consensus ConferenceEvidence of scholarship in teaching, as in all categories,requires that educators make productspublicly available for peer review so their contributionsto the educational community can be evaluated.Public presentation and peer review may beinternal through a division, department, teachingsociety, or education committee, or externalthrough such forums as the AAMC’s annual orregional meetings, AAMC’s MedEdPORTAL®, theHealth Education Assets Library, Family MedicineDigital Resource Library, or other peer-reviewedrepository. Interactive learning exercises (eitherWeb-based or face-to-face), slides sets with speakernotes, problem-based learning or other clinicalcases, and new models and strategies for teaching—all are examples of teaching products thatcontribute to the educational community.Documentation of these contributions include• Inclusion of the product in a peer-reviewedvenue or repository• Evaluations from a conference presentation,teaching awards, or recognition with annotationsregarding selection process and criteria• Data demonstrating adoption by other faculty• References or citations to the product in otherpeer-reviewed materials• Descriptions of how others have built on oradapted the product for their own use.CurriculumCurriculum is defined as a longitudinal set—that is,more than one teaching session or presentation—ofdesigned educational activities that includes evaluation.Curricular contributions may occur at anytraining level—medical student, resident, or grad-Figure 1.Documentation Frame for Educators’ Academic AdvancementBased on Contributions to the Institution and Engagement with the Education CommunityEDUCATORS’ CONTRIBUTIONS TO INSTITUTIONS’ EDUCATIONAL MISSIONSQuantityof educational activitiesQualityof the educational activitiesEDUCATORS’ ENGAGEMENT WITH EDUCATION COMMUNITYCommunity may be defined geographically a or by specialty/discipline bScholarly Approach to Educational ActivityEducators’ activities are informed bythe knowledge and resources ofthe educational communityQuantityof informed educational activitiesQualityof the educational activitiesEducational ScholarshipEducators’ activities contribute to educationalcommunity to advance knowledge in the fieldQuantityActivity is made available in a formothers may build on or useQualityContribution of the activityto the field is evaluated by peersa Local (department/division, university, community), regional, national/internationalb Examples include medicine, pediatrics, surgery, biochemistry, genetics, anatomy, pathology, educationalevaluation, and instructional technology.8Association of American Medical Colleges, 2007

Scholarship Consensus Conferenceuate student, or continuing medical education; invarious educational venues—course, clerkship,rotation, theme-threaded cross years, faculty development,or community program; and may be deliveredface-to-face or electronically.To include an activity in the curriculum category,educators must answer four questions: (1) What isthe educational purpose (i.e., goals, objectives) ofthe activity? (2) Which learning experiences aremost useful in achieving those purposes? (3) Howare those learning experiences organized and longitudinallysequenced for effective instruction? (4)How is the curriculum’s effectiveness evaluated?Quantity: For each curricular piece authored, documentationshould include a cogent description ofits purpose, intended audience, duration, design,and evaluation. If the curriculum was coauthored,each entry should document the candidate’s role,content contributed, and expertise provided, suchas curriculum, technology, or assessment.Quality: Documentation of a curriculum activityand associated evidence of outcomes and qualityshould include 24• Learner reactions and ratings• Outcomes, including the impact on learning(e.g., course examinations, NBME subject scores,in-service examination scores, or observation oflearner performance)• Graphic displays of improvement over time (e.g.,its relation to previous curriculum offerings).Engagement with the Education Community: A scholarlyapproach to curriculum development requiresdemonstration that the design was informed by theliterature and “best practices.” The curriculumauthors must note how it was influenced by relevantliterature or other educators. Positive and negativeresults should be presented to advance educationalknowledge and build on the authors’ experiences.Educational scholarship in curriculum requiresmaking it public in a form that others can use, suchas course syllabi, learner assessment tools, orinstructor guides, and includes9• Peer review by local experts, the institution’scurriculum committee, or accreditation reviewers• Invitations to present curriculum work at meetings,supplemented by documentation of thepresentation’s quality• Peer-reviewed or invited presentation at regional,national, or international meetings• Acceptance of curriculum material to a peerreviewedrepository such as AAMC’sMedEdPORTAL• List of institutions where the curriculum has beenadopted, including the author’s home institution• Invitations for curriculum consultation fromother departments or schools, including trackingof the consultations’ use• Number of citations in other instructors’ curricula.Advising and MentoringEducators frequently serve as advisors and mentorsin the professional development of learners andcolleagues. These activities can have a profoundimpact on advisees’ careers and, in turn, on theprofession. Advising and mentoring are developmentalrelationships encompassing a spectrum ofactivities, in which educators help learners orcolleagues accomplish their goals. More specifically,mentoring implies a sustained, committed relationshipfrom which both parties obtain reciprocalbenefits. Advising is a more limited relationshipthat usually occurs over a limited period, with theadvisor serving as a guide.Documentation of mentoring and advising activitiesmust effectively describe the nature of the relationshipsand their effectiveness in helping advisees meettheir goals, using quantitative and qualitative data.Quantity: Quantitative data should include thenumber of learners and colleagues mentored oradvised, and when appropriate, the names andpositions or status, and an estimate of time investedin each relationship (e.g., duration, frequency ofcontact, and total hours).Association of American Medical Colleges, 2007

Scholarship Consensus ConferenceQuality: Educators’ effectiveness as mentors andadvisors is demonstrated through advisees’ goalachievement. Evidence of productive relationshipsmay be documented by• Evaluations of advising and mentoring effectivenessfrom advisees using standardized forms withcomparative ratings• A listing of advisees’ significant accomplishments,including publications and presentations,and the development of tangible educationalproducts, recognitions, and awards• Narrative comments from advisees may alsoprovide evidence of a relationship’s effectivenessin facilitating goal achievement. When available,comparative data in the form of historical ordiscipline-based standards should be presented.Engagement with the Education Community:Mentoring and advising has an emerging body ofliterature, supported by resources, guides, andconferences. Evidence of scholarly engagement inthis category, as in all others, can be demonstrated by• Participating in professional development activitiesto enhance skills in mentoring and advising• Adopting effective mentoring strategies withdocumented links to the literature• Writing an institutional guide informed by theliterature and “best practices”• Designing an effective program guided by currentevidence• Leading initiatives that improve institutionalmentoring and advising practices.Scholarship related to mentoring and advising maybe demonstrated by• Receiving invitations to critically appraisementoring programs, and providing documentationof the results and the appraisal’s impact• Posing investigational questions aboutmentoring/advising, selecting methods to answerthem, collecting and analyzing data, making theresults public, and obtaining peer review• Securing program development funding througha peer-reviewed process• Conducting skill enhancement training sessionsat professional meetings• Publishing peer-reviewed materials in print orelectronic formats, such as institutionalmentoring guides• Convening scholarly conferences on mentoring,serving as a mentoring consultant to professionalorganizations, being invited to serve as a peerreviewer of mentoring or advising works,receiving mentoring or advising awards, andhaving success in competitive funding for innovativementoring-related projects.Educational Leadership and AdministrationExceptional educational administrators and leadersachieve results through others, transforming organizationsthrough their vigorous pursuit of excellence.Key features that educational administratorsor leaders should document to demonstrate theirwork’s value for promotion consideration include(1) active and continuous pursuit of excellence; (2)ongoing evaluation; (3) dissemination of results;and (4) maximization of resources.Quantity: The nature of leadership projects andtheir duration and quantity should be described inan easy-to-read, concise format along with the rolesleaders played.Quality: The pursuit of excellence should be thecore of all administrative and leadership actions;effective leaders challenge, advance, and transformthe field. They create a sense of urgency, developcoalitions, communicate vision, develop plans, evaluateachievements, garner resources, and inspireothers in the pursuit of common goals. Effectiveadministrators and leaders manage resources efficiently,and must collaborate with and mentorothers to achieve change.10Association of American Medical Colleges, 2007

Scholarship Consensus ConferenceDocumentation of quality in leadership includes aconcise description of projects, including• Leadership role and project dates• The context where the change occurred, as well asthe process, including problems identified, goalsestablished, and actions taken• Evaluation including delineation of outcomes• Financial and human resources, both new andexisting, as change requires leaders and administratorsto deploy resources to achieve desiredgoals.Engagement with the Education Community: Whenadministrators’ resource management or leaders’organizational transformation is informed by theliterature and best practices, they have made thetransition to active engagement with the largereducational community.A scholarly approach to leadership and administrationis demonstrated by• Making changes based on the literature and bestpractices.• Creatively designing and evaluating improvements,and making revisions based on local feedbackor in light of theoretical frameworks, priorresearch, best practices, and external peer review• Using pre- and post-assessment or other designs(e.g., cohort performance on licensing, in-servicetraining, board certification examinations,accreditation surveys) or newly developed toolsto measure outcomes• Demonstrating attainment of objectives orbenchmarks associated with successful change(e.g., AAMC Graduation Questionnaire andlearner ratings of teachers; courses/rotationenrollments and evaluations)• Documenting ongoing quality improvement,drawing from the knowledge and resources of theeducational community• Evaluating leaders’ effectiveness using 360-degreeevaluation with peer comparisons, benchmarking,or external peer review• Employing self-reflection informed by the literatureor best practices in the field.The scholarship of educational leadership isevidenced by sharing innovations with the educationalcommunity through materials, documents,or presentations, and through others’ recognition ofthe work’s value. Dissemination of findings makesinnovations visible to the community, creating apublic forum for discussing them and advancingthe field.Documentation of educational scholarship wouldinclude• List of invited and peer-reviewed presentations atlocal, regional, national, and international professionalmeetings, along with visiting professorshippresentations• Quantity and quality of publications• Awards received with annotations regardingselection criteria and process• List of institutions that have adopted aninnovation• Acceptance of a new curriculum model toAAMC’s MedEdPORTAL, with impact inferredfrom the number of hits the site received and thenumber of schools that have adopted thecurriculum• List of resources obtained by source—such asfoundations, grants, or internal awards—asevidence that others have judged the innovationworthy of investment.In summary, leaders and administrators commit toexcellence by making a significant difference,advancing the field, or demonstrating a positiveimpact on others. To demonstrate commitment toexcellence, educational leaders must innovate,garner, and maximize resources, and evaluate11Association of American Medical Colleges, 2007

Scholarship Consensus Conferenceactions. Leaders’ active engagement with the educationalcommunity distinguishes leadership excellencefrom scholarship, as leaders draw on thecommunity to inform action and then contributeback to advance the field.Learner AssessmentLearner assessment is defined as all activities associatedwith measuring learners’ knowledge, skills, andattitudes, and must include at least one of fourassessment activities:1. Development: Identifying and creating assessmentprocesses and tools2. Implementation: Collecting data using processesand tools3. Analysis: Comparing data with correct answerkey or performance standards4. Synthesis and presentation: Interpreting andreporting data to learners, faculty, andcurriculum leaders.Quantity: Documenting an assessment activity’ssize and scope should begin with a brief descriptionof the event using jargon-free language understandableto promotion and tenure committee members.This description should include information aboutfaculty’s role in each assessment component alongwith the size and nature of the learner populationbeing assessed, the size of the assessment, and theintended uses of the information.Quality and Engagement with the EducationalCommunity: Documenting quality in learner assessmentshould provide evidence that the evaluationmeets established reliability and validity standards,summarized in quantitative and narrative formats.When data from learner assessments are used in“high stakes” decisions—such as grades or promotion—theassessment must be well-grounded in theexisting knowledge base drawn from the educationalmeasurement field.Glassick’s six criteria provide a systematic frameworkfor a scholarly approach in determining thequality of assessment contributions:1. Goals: A clear statement of assessment goals andthe educator’s particular contributions to theassessment process2. Adequate preparation: Description of theauthor’s prior experience or literature uponwhich the assessment was based3. Appropriate methods: Details of how eachdesign phase’s methods match known bestpractices4. Significant results: Information about thequality of results according to reliability andvalidity standards5. Effective presentation: A succinct and effectivesummary of the results and lessons learned tostakeholder groups (e.g., learners, administrators,peers, and the assessment community)6. Reflective critique: Plans for improving similarassessments in the future.Scholarship in learner assessment must includedocumentation that activities were peer reviewedand that processes or tools involved have beenshared with the educational community to enhancebest practices. Faculty involved in any design phasemay present documentation associated with• Presentations on the assessment process oroutcomes to local audiences, such as curriculumcommittees or internal reviews in preparation foran RRC visit• Peer-reviewed presentations and workshops atprofessional meetings, or invited presentations• Acceptance of the assessment tool in a peerreviewedrepository• Assessment research presented at national meetingsor published in peer-reviewed journals.12Association of American Medical Colleges, 2007

Scholarship Consensus ConferenceNext StepsCrystallizing and Building on our CurrentKnowledgeInstitutions reward what they value. If educationalexcellence and scholarship are important to ourinstitutions, educators’ activities must be recognizedand rewarded. No longer can educators’ activitiesbe viewed as “largely private work, guided bytradition, but uninformed by shared inquiry orunderstanding of what works.” 24 If our institutionsare to demonstrate that they value educationthrough academic promotion of educators, theseactivities must become public and open to peerreview.Four tenets for making educators’ work public andavailable for peer review emerged from oursynthesis of the literature and consensus conferencediscussion.1. Educators contribute to institutions’ educationalmissions through activities in teaching,curriculum development, mentoring andadvising, leadership, and learner assessment.Contributions in these areas must be valued inacademic promotion decisions to demonstratethat institutions’ actions are aligned with theireducational missions.2. Educators’ contributions can be judged throughthe effective presentation of evidence associatedwith quantity, quality and, when relevant,evidence of engagement with the educationalcommunity. Evidence of such engagementdemonstrates that educators draw upon theliterature and best practices (scholarlyapproach) or contribute when appropriate tothe medical education field (scholarship).3. Promotion expectations should be congruentwith the activities assigned to faculty members.Judgments regarding educators’ contributions inassigned responsibility areas should play animportant role in faculty promotion decisions.4. The standards for academic promotion mustnot exceed the available educational supportinfrastructure within institutions and fromregional, national, and international professionalorganizations. Support infrastructureincludes• Forums for educators to share their work andhave it peer reviewed;• Faculty development to enhance educators’expertise and learn about new advances in thefield;• Access to educational journals and repositories;and• Physical, virtual, and technical resources.These tenets are intended to be evidence-basedguideposts as we lead our institutions through aprocess of valuing education and educators, helpingto align academic promotion standards with theroles and tasks needed to fulfill our unique educationalmissions. How do we provide this leadership?Unresolved Issues: Opportunities for CrucialConversations about EducationHutchings and Huber 20 challenge us to create a“teaching commons” that promotes dialogue,sharing, and improving upon medical education’sbest practices and innovations. In a teachingcommons, educators and other stakeholders cometogether to engage in crucial conversations,informed by the literature and guided by teachingand learning experiences. 20 Based on the dialoguebegun at the consensus conference and ongoingdiscussion among the authors, four unresolvedissues were identified. We present these issues tostimulate engagement within our educationalcommunities, resulting in conclusions based on bestpractices and scholarly inquiry.1. Infrastructure: Successful educators, likesuccessful researchers or clinical practitioners,need resources to fulfill the educational mission,such as protected time, consultation, journal13Association of American Medical Colleges, 2007

Scholarship Consensus Conferenceaccess, and physical facilities. Unansweredquestions include• What essential institutional or organizationalinfrastructure elements are needed to supporteducational excellence and scholarship?Examples include peer evaluations of teaching,psychometric analysis of learner assessmenttools, and faculty development.• How can we facilitate effective dialogue amongkey constituencies—including medical schooldeans, academic societies, and teaching hospitals—todevelop an infrastructure that valueseducators and educational scholarship?• Are academies and societies the most effectiveinfrastructure for engaging educators?2. Breadth of Engagement with the Community:• What level of engagement—internal orexternal, local or international—is needed todemonstrate meaningful involvement in thecommunity of educators, and in what contentareas?• Should expectations regarding the level ofengagement vary by faculty rank or availableinstitutional resources?3. Category Boundaries: The activities germane toeach educational category have been clarified;however, definitive boundaries between categoriesremain elusive.• Does further clarification of educationalactivity category boundaries constrain educators’needs for flexibility?4. Judging Scope and Sustained Individual vs.Group Accomplishments: Our promotioncommittees have long-standing traditions andstandards for judging accomplishments.However, educators’ activities are typicallydriven by institutional needs (e.g., newcurriculum, accreditation standards, or assessmentmandates) and are collaborative groupefforts. Questions needing further studyinclude:• Must institution-specific guidelines be developedto clarify the number of “within” and“between” category inclusions expected foracademic advancement?• What levels of sustained activity must educatorsdemonstrate to “count” in academicpromotion decisions?• How should educators present group accomplishmentsso that both team and individualcontributions are recognized? This issue is alsofaced by our research and clinical colleagues,who must develop program project awardsand work in multidisciplinary patient careteams. Educators’ performance assessment andleadership expertise may position them todevelop school-wide processes for judgingindividual and group accomplishments.• When does the scope of instructional materialsdeveloped to support a specific directteaching activity shift from the teaching tocurriculum category?• When does teaching become a longitudinalone-to-one mentor relationship?14Association of American Medical Colleges, 2007

Scholarship Consensus ConferenceSummaryWe know the educator activity categories and theassociated evidence needed to assess facultymembers’ accomplishments for academic promotion.Now, we must seize the opportunity to disseminateand build on this knowledge. Using existinglocal and national forums—and creating new venueswhen necessary—we must communicate what weknow about documentation for academic promotion,and stimulate conversations and systematicinquiry to answer new questions. If we succeed,communities of educators will emerge. Thesecommunities will be populated by individuals whosecontributions to our unique mission—improvingthe health of the public through excellence in educationof physicians 25 —are supported through astrong educational infrastructure, and valuedthrough academic promotion and recognition.15Association of American Medical Colleges, 2007

Scholarship Consensus ConferenceConference MaterialsPre-conference Readings [references]In preparation for the working conference, allattendees were asked to read the articles listedbelow to provide a common foundation fromwhich to “build on what is known” about educationalscholarship. Prior to reading the articles, weadvised all participants to read the annotatedsummaries of these articles to frame their detailedreading of the primary reading-list articles within abroader context.Preparatory Reading Summaries(Prepared by Janet Hafler, Ph.D.)Primary Readings1. Hafler JP, Blanco MA, Fincher RM, Lovejoy FH,Morzinski J. Educational Scholarship. Chapter 14. In:Fincher RM (Ed). Guidebook for Clerkship Directors,3rd edition. Alliance for Clinical Education.http://familymed.uthscsa.edu/ACE/guidebook.htmThe authors emphasize that, since the CarnegieFoundation’s and Boyer’s work, the educationalcontributions of clerkship directors and other clinicianeducators are increasingly being recognized andrewarded in medical education. Boyer’s landmarkpublication helped educators organize their workwithin a new framework he labeled “scholarship.” Inthis chapter, the authors provide a practical understandingof the scholarship of education so that theycan benefit and advance based on their work asscholars. The daily activities of clerkship directors,such as teaching, mentoring, curriculum development,or educational leadership, are now recognizedas scholarship under certain circumstances.The authors first provide an overview of educationalscholarship and then describe an archetypicalcase to illustrate the main concepts and guide clerkshipdirectors’ practical understanding of the scholarshipof education. Subsequent sections provideinformation on how to move an educationalactivity into a scholarly activity and then intoscholarship. Specific delineations between scholarlywork and scholarship are covered next, followed bya dialogue on how one’s work can be evaluated. Theauthors finish with a discussion about supportsystems at the institutional and department levels.2. Rice RE. Scholarship Reconsidered: History andContext. Chapter 1. In: O’Meara K, Rice RE. FacultyPriorities Reconsidered: Rewarding Multiple Forms ofScholarship. San Francisco: Jossey-Bass 2005. Pp. 17–31.Rice highlights that Scholarship Reconsidered becamea significant document because it came at the righttime and addressed the main strains developedaround central issues of faculty scholarly work thatshould be valued and rewarded. The primary goal ofthe report was to extend the debate across highereducation, create new conceptions of faculty workin a way that would reintegrate personal and institutionalpriorities, and bring a new kind of wholenessto what it means to be a scholar while respondingmore adequately to the shifting educational needs ofsociety. Rice describes the changing context andexternal events that affected the way we think aboutscholarship. He argues that, while most highereducation reforms begin on the margins of theinstitution, Scholarship Reconsidered targeted thecenter of the academic enterprise by beginning withthe faculty role and questioning the meaning ofscholarship and the academic reward system. Ricepoints out the influence of Boyer’s charismatic andpositional authority and that of the AmericanAssociation for Higher Education and the CarnegieFoundation that supported the report and extendedits impact. The author describes the debate generatedaround “scholarship of teaching,” including themove toward adopting the more inclusive “scholarshipof teaching and learning,” as well as the distinctionsamong good teaching, scholarly teaching, thescholarship of teaching and, most recently, “thescholarship of engagement.”3. Huber MT, Hutchings P, Shulman, LS. The Scholarshipof Teaching and Learning Today. Chapter 2. In:O’Meara K, Rice RE. Faculty Priorities Reconsidered:Rewarding Multiple Forms of Scholarship. SanFrancisco: Jossey-Bass 2005. Pp. 34–38.17Association of American Medical Colleges, 2007

Scholarship Consensus ConferenceHuber et al. review the emergent interest worldwidein the scholarship of teaching and learning—amovement, which, in turn, has promoted itscritique and development. Where the debate on itsplacement continues in today’s academy, theauthors conclude that, “genres, topics, and methods[of the scholarship of teaching and learning] arebeing invented as we speak.” They emphasize thatthe lack of boundaries welcomes intellectualexchange and collaboration across countries, institutions,and fields, which conveys the best chancesfor the future scholarship of teaching.The authors argue that Boyer’s introduction of thescholarship of teaching in 1990 positioned teachingwithin a broader vision of scholarship, and discussionon scholarship of teaching entered and helpedshape the continuous debate about higher education.The authors describe related developmentsthat helped enhance the scholarship of teachingand learning, and how, at the same time, theboundaries of the scholarship of teaching becameless clear. They argue that, though issues cut acrossall fields, the scholarship of teaching is differentlyshaped across disciplines: distinctive elements ofdiscovery, integration, and application within thescholarship of teaching are there to be discovered.By the end of the decade, two of the authorsproposed a new distinction among excellentteaching: scholarly teaching and the scholarship ofteaching. A new and emergent field, scholars ofteaching and learning must be prepared to make astrong case to receive support and acceptance oftheir work.Additional Reading as Time Allows4. Fincher RE, Simpson DE, Mennin SP, Rosenfeld GC,Rothman A, McGrew MC, Hansen PA, MazmanianPE, Turnbull JM. Scholarship in Teaching: AnImperative for the 21st Century. Acad Med2000;75:887–894The authors address scholarship in education,focusing on teaching and other learning-relatedactivities rather than on educational research, bybuilding on Boyer’s work. They apply Glassick etal.’s criteria for assessing faculty members’ educationalactivities to establish a basis for recognitionand reward of faculty scholarly work, one that isconsistent with those given for other forms ofscholarship. The authors outline the organizationalinfrastructure needed to support scholars in education.They maintain that faculty who meet thefollowing criteria are scholars and should be recognizedby promotion: creative teaching with rigorouslysubstantiated effectiveness, demonstratededucational leadership, and the use of educationalmethods that advance learners’ knowledge—all ofwhich are consistent with the traditional definitionof scholarship.5. Simpson D, Hafler JP, Brown D, Wilkerson L.Documentation Systems for Educators SeekingAcademic Promotion in U.S. Medical Schools. AcadMed 2004;79(8):783–790The authors conducted a two-phase qualitative studyto explore the state and use of teaching portfolios inpromotion and tenure in U.S. medical schools. Thefirst phase assessed the diffusion of teaching portfolio-likesystems in U.S. medical schools through aWeb-based search. The second phase explored thecurrent use of teaching portfolios in 16 U.S. medicalschools reporting their use in a previous study.Among the main findings: (a) 76 medical schoolshave Web-based access to information on documentationof educational activities for promotion; (b) all16 medical schools continued to use a portfolio-likesystem; (c) honors/awards and philosophy/personalstatements regarding education were included asdocumentation categories by six more schools thanpreviously surveyed; (d) dissemination of work tocolleagues had become a key inclusion at 15 of the16 schools; (e) the most common type of evidenceused was the learner and/or peer ratings, with areported infrequent use of outcome measure andinternal/external review. The authors conclude thatthe number of medical schools whose promotionpackets include portfolio-like documentation hadincreased by more than 400 percent in over 10 years,and, though the types of documentation categorieshad increased, students’ ratings of teaching are stillthe primary evidence used to document the qualityor outcomes of the educational efforts reported.18Association of American Medical Colleges, 2007

Scholarship Consensus ConferenceConference MaterialsIllustrative Portfolios to TriggerDiscussionSusan Masters, Ph.D.Teaching PortfolioBackground and Educational PhilosophyMy primary role at the University of California,San Francisco, is as an educator of medicalstudents. For professional students, pharmacologyoffers many opportunities for integration of thebasic and clinical science presented elsewhere in thecurriculum, including the demonstration ofexciting “bench-to-bedside” success stories. One ofmy teaching goals is to reinforce the knowledge ofcell biology, physiology, and pathophysiology thatstudents have learned previously. While this soundssimple, determining the content of preceding andconcurrent curriculum requires laborious investigativework. To accomplish this, I communicate withother course directors, read syllabi, and attendlectures in other courses.Currently, most of my time is devoted to theEssential Core preclinical medical curriculum,which consists of nine integrated block courses. Idirect one block, oversee portions of the pharmacologycontent of other blocks, and chair anEssential Core oversight committee. Co-directorshipof the new integrated blocks is a formidable task. Itrequires coordinating many faculty members, mostof whom are outside one’s primary department, andcareful attention to a myriad of administrativedetails, including oversight of syllabus and examinationpreparation, room scheduling, the electroniccurriculum, small-group logistics, and teachingevaluations. While the administrative challenges ofdirecting these new blocks are great, the work isexciting and rewarding because it provides opportunitiesto work closely with talented colleaguesdevoted to the curriculum’s success.In addition to teaching, I oversee my department’sprofessional teaching program. I assign and trackthe teaching hours for the department faculty andassist the other course directors. I help new facultymembers prepare lectures and distribute informationon new drugs and therapeutic advances to helpthem update their teaching materials.Educational Administration andLeadershipCourse DirectorshipIDS 101: ProloguePrologue is the first block taken by enteringmedical students. It includes basics in the disciplinesof gross anatomy, histology, pathology, cellbiology, pharmacology, medical genetics, culture,and behavior. It is anchored by a clinical casepresented by the emergency department at SanFrancisco General Hospital. The seven-week courseis cosponsored by the departments of cellular andmolecular pharmacology and anatomy. I servedwith Dr. Douglas Schmucker as course co-directorsin fall 2002 and 2003. In fall 2004, I served a major“behind the scenes” role in mentoring Dr. MariekeKruidering-Hall, the new course director, andorganizing the block exams.Pharmacology 100 A/B/CThe Pharmacology 100 A/B/C series was requiredfor second-year medical students until spring 2002;it contained 83 hours of lecture and 9 hours ofsmall groups, and involved about 20 lecturers and30 small-group instructors. As course director, Irecruited and scheduled lecturers, edited syllabi andexaminations, corrected essay examination questions,attended lectures, held weekly office hours,and, prior to examinations, led review sessions. Inaddition, I orchestrated and recruited facultyleaders for several small-group conferences.19Association of American Medical Colleges, 2007

Scholarship Consensus ConferenceResults of Student Evaluations of Pharmacology 100A, B, and C (Best is 5):Overall Quality of the Course 100A Rating 100B Rating 100C Rating1996–1997 4.6 NA 4.81997–1998 4.6 4.0 4.31998–1999 4.6 4.5 4.61999–2000 4.7 4.5 4.62000–2001 4.4 4.2 4.62001–2002 4.3 NA 4.5Curriculum LeadershipAdministrative ToolsI developed tools that have been adopted by otherblocks in the Essential Core. These include instructionsand templates for preparing syllabus chapters,tips for writing multiple-choice examination questions,course information statements, andMicrosoft® Excel tools for tracking grades andattendance. I wrote a set of Excel programs used byDr. Mary Banach in the Office of MedicalEducation Research and Development to analyzeexam data and track student performance acrossthe curriculum. Dr. Marieke Kruidering-Hall and Icreated a set of styles and guidelines for syllabuspreparation in Adobe® InDesign. I have alsoconsulted on the design of Ilios, the university’scurriculum database.Learner AssessmentI lead a project to improve Essential Core blockexams and the feedback on academic performancethat we provide students. The first phase of theproject involved making exams secure so questionscould be reused, and constructing a standardizedtemplate for designing exam questions thatincludes• Instructor name• Session title• Explanation for the correct answer• Discipline(s) under which the question falls• Session objective(s) under which the questionfalls• Once an exam is given, statistics on studentperformance (percentage of all students whoanswered correctly), answers selected by allstudents and also by students in the top andbottom quartiles, point biserial (a statistic thatestimates the ability of a question to discriminatebetween students who do well or poorly overallon the exam).This standardization of exam question preparationhas had a remarkable effect on moving instructorstoward writing questions to match their sessionobjectives!I used Microsoft® Excel, Microsoft® Word, andFilemaker® Pro to create the software tools neededfor this project. I worked closely with the EssentialCore block directors to standardize exam preparationand the means of reporting exam results tostudents. The goal of the project was to create fair,well-tested exams that reflect course learningobjectives.With disciplines attached to every exam question,we are able to track student performance across theentire Essential Core. We chose 17 subjects roughlybased on disciplines reported for the United StatesMedical Licensing Exam (USMLE) I and the way inwhich popular USMLE I board review books arestructured. After each Essential Core exam, wegenerate a score for every student in all 17 subjectsrepresented on the exam. We combine exam scoresin the 17 subjects as students move through theblocks. At the end of each block, we release tostudents their raw and percentage score in eachsubject. We also publish the number of total points,20Association of American Medical Colleges, 2007

Scholarship Consensus Conferenceclass average, and class standard deviation in eachsubject so students know where they stand withrespect to their peers. This subject tracking is especiallyimportant for subjects, such as pathology,pharmacology, and genetics, spread across multipleblocks. The hope is that students will use the informationto gauge their strengths and weakness inthese subjects and appropriately seek help and alsofocus their preparation for USMLE step I. At somepoint, we may require remediation for studentswho fall below 70 percent in a subject. The trackingprocess incidentally yields information about therelative degree of difficulty of the various subjectsand the number of questions asked for each.Standing CommitteesCommittee/RoleEssential Core Course Committee (ECCC)Member, 2002–presentChair, fall 2004–presentEC Steering CommitteeEx-Officio member 2004–presentCommittee for Curriculum and EducationalPolicy (CCEP)Ex-Officio member, 2004–present,1999–2001,School of Medicine Admissions CommitteeMember, 1997–2000, 2004–2005Overview of Committee ActivitiesThis committee meets monthly to discuss issues that affect multiple EC blocksand to share results of curricular and administrative innovations. I work closelywith Dr. Helen Loeser, associate dean for academic affairs, and Dr. RamuNagappan, curriculum coordinator, on the steerage of this committee.This committee meets monthly to set policy for Essential Core courses. Dr.Manny Pardo, the chair of this committee, and Drs. Loeser and Nagappanand I work together to balance the work of this committee and its sister,the ECCC committee.Parent committee for the medical curriculum. It sets policy and reviewscurricular progress. I update CCEP on ECCC progress.This committee selects the incoming medical student class. I rejoined thiscommittee and served on Panel 2, which is run by Dr. Leslie Zimmerman.[Editor’s note: Five additional committee/roles presented in actual portfolio.]Ad Hoc CommitteesCommittee/RoleThe Exam Data Banking Working GroupChair, spring and summer 2004The Faculty–Administrator SupportWorking GroupChair, spring and summer 2004Overview of Committee ActivitiesThis subcommittee of the ECCC worked on a plan to make EC block examssecure (i.e., not release them to students), designed a database to house examquestions and made plans to improve self-assessments. Key members includedDrs. Wade Smith and Manny Pardo.This subcommittee of the ECCC explored means of reducing the stressfulheavy administrative load on EC directors and administrators. Several initiativesdeveloped by this group have been adopted. Key members included Dr.Helen Loeser, Dr. Katherine Hyland, Dr. Tracy Fulton, Lloyda French, andCindy Irvine.[Editor’s note: Four additional ad hoc committees presented in actual portfolio.]21Association of American Medical Colleges, 2007

Scholarship Consensus ConferenceEducational Scholarship and Creation of Enduring MaterialsBooks and Book Chapters1. AJ Trevor, BG Katzung, SB Masters: Pharmacology: Examination and Board Review, Seventh Edition,Lange Medical Books/McGraw-Hill, New York, 2004.2. SB Masters: Agents Used in Anemias; Hematopoietic Growth Factors. In Basic and ClinicalPharmacology, Ninth Edition, BG Katzung (ed), Lange Medical Books/McGraw-Hill, New York, 2004.3. SB Masters: The Alcohols. In Basic and Clinical Pharmacology, Ninth Edition, BG Katzung (ed), LangeMedical Books/McGraw-Hill, New York, 2004.4. AJ Trevor, BG Katzung, SB Masters: Pharmacology: Examination and Board Review, Sixth Edition,Lange Medical Books/McGraw-Hill, New York, 2002.5. DV Santi and SB Masters: Agents Used in Anemias; Hematopoietic Growth Factors In Basic andClinical Pharmacology, Eighth Edition, BG Katzung (ed), Lange Medical Books/McGraw-Hill, NewYork, 2001.6. SB Masters: The Alcohols. In Basic and Clinical Pharmacology, Eighth Edition, BG Katzung (ed), LangeMedical Books/McGraw-Hill, New York, 2001.7. SB Masters and NM Lee: The Alcohols. In Basic and Clinical Pharmacology, Seventh Edition, BGKatzung (ed), Appleton & Lange, Norwalk, Connecticut, 1998.Direct TeachingUniversity of California, San Francisco Teaching Awards and Honors:2005 Commitment to Teaching Award, Class of 20072003 Academic Senate Distinction in Teaching Award2003 Outstanding Life Cycle Educator, School of Medicine, Class of 20052002 Pre-Clinical Faculty Teaching Award, School of Medicine, Class of 20022002 Kaiser Award for Excellence in Basic Science Teaching, School of Medicine2001 Elected to the Haile T. Debas Academy of Medical Educators2000 Long Award for Excellence in Teaching, School of Pharmacy, Class of 20012000 Long Prize for Excellence in Teaching, School of Pharmacy, Class of 20001999 Long Award for Excellence in Teaching, School of Pharmacy, Class of 20001999 A Major Contribution to Teaching Award, School of Medicine, Class of 20011997 Long Prize for Excellence in Teaching of the Basic Sciences, School of Pharmacy, Class of 19971997 Outstanding Lecture Series, School of Medicine, Class of 199922Association of American Medical Colleges, 2007

Scholarship Consensus ConferenceDirect Undergraduate Teaching (no student evaluations available):1999–presentUSMLE Step I pharmacology review sessions for medical students, University of California, San Francisco(2 hours/year). In these sessions, I use a Jeopardy-style format to guide the students through the majorCNS drug groups in a large-group setting. I use many of the same images used in previous pharmacologylectures to provide a direct link to their previous learning. I find that a question-and-answer format is muchmore engaging than the common strategy of racing through overhead after overhead of facts. I also show thestudents how to consider topics from a question-writer’s perspective—what information would studentschoose as topics for questions? This hopefully helps them focus their subsequent study of the topic.1995–2003 Lecturer, Joint Medical Program Medical Pharmacology, University of California, Berkeley(~ 5 hours of lecture/year)Teaching Skills Programs and WorkshopsWorkshop LeaderUniversity of California, San Francisco,Academy of Medical Education Workshop:Giving a Dynamic LectureApril 18, 2005Postdoctoral Teaching Fellowship2001–presentGrand Slam Presentations WorkshopUniversity of California, San Francisco,Department of Psychiatry Education RetreatJune 7, 2003Winter and Spring 2005, Being an Observer in the Academy of MedicalEducator Teaching Observation Program (TOP)Spring 2005, Lecturing Skills, Professional and Academic Skills (PASS)program for postdoctoral fellowsSummer 2004, Lecturing Skills, Preparing Future Faculty (PFF) programfor postdoctoral fellowsAt the invitation of Dr. Manny Pardo, I created and presented a workshopon designing and delivering effective lectures.With Dr. Kruidering-Hall from my department and Drs. Tracy Fulton andKatharine Hyland from biochemistry and biophysics, I helped create aprogram that provides postdoctoral fellows (postdocs) instruction andexperience in small-group teaching. Each year, we interview 10–20 applicantsand select 5–10 from the group. The selected postdocs lead smallgroups and assist with exam grading either in IDS 101: Prologue or IDS103: Cancer. We provide postdocs with instruction on leading small groupsand also give them feedback after observing them in the small-groupsetting. The feedback from participants has been highly favorable.At the invitation of Dr. Lowell Tong, I created and presented a workshopon designing and delivering effective lectures.[Editor’s note: Actual portfolio included longitudinal evaluation data from multiplecourses and student comments from two courses’ teaching evaluations.]23Association of American Medical Colleges, 2007

Scholarship Consensus ConferenceStudent Evaluation of Teaching Performance in Required University of California,San Francisco, Courses: Numerical ResultsEvaluations ask students to rate the educator on a 1–5 scale with:1 = Poor; 2 = Fair; 3 = Good 4 = Very good; 5 = ExcellentIDS 101: PrologueFall 2004; 4 hours lecture, 2 hours small groupQuestion N Leader Mean Leader Std Course MeanFacilitator’s level of preparation 4.77 0.6 4.32Ability to facilitate without dominating4.15 0.8 4.08the discussionOverall assessment of the facilitator’s teaching4.23 0.73 4.14effectivenessQuestion N Lecturer Mean Lecturer Std Course MeanRate the quality of the lecturer’s syllabus section 38 4.24 0.71 4.02Rate the lecturer’s availability outside of class 22 4.23 1.15 4.53Rate the lecturer’s enthusiasm 37 4.68 0.67 4.46Rate the lecturers’ ability to teach to the students’ 38 4.37 1.00 4.32level of understandingRate the lecturer’s use of audiovisuals 38 4.08 0.94 4.14Rate the overall effectiveness of the lecturer 38 4.34 0.85 4.29Fall 2003; 5 hours of lectureQuestion N Lecturer Mean Lecturer Std Course MeanRate the quality of the lecturer’s syllabus section 39 4.00 0.94 4.03Rate the lecturer’s availability outside of class 25 4.32 0.99 4.11Rate the lecturer’s enthusiasm 39 4.13 0.83 4.36Rate the lecturers’ ability to teach to the students’ 37 4.35 0.95 4.37level of understandingRate the lecturer’s use of audiovisuals 37 3.78 0.98 3.90Rate the overall effectiveness of the lecturer 39 4.00 0.97 4.0224Association of American Medical Colleges, 2007

Scholarship Consensus ConferenceAdvising and MentoringFacultyName/Department Dates Description[Student Name] Ph.D.Department of Biochemistryand Biophysics[ Student Name], Ph.D.Department of Cellular andMolecular Pharmacology[Student Name], Ph.D.Department of Biochemistryand Biophysics2000–present2002–present2002–presentDr. [name] teaches molecular biology and biochemistry. She currentlydirects several professional school courses and gives many lectures. I mentorDr. [name] in various areas of teaching.Dr. [name] was hired to help our department teach professional students.She currently directs several courses and gives a number of lectures in all theprofessional schools. I have a major role in mentoring [name].Dr. [name] teaches molecular biology and coordinated the medical geneticsteaching. She currently directs several professional school courses and givesmany lectures. I mentor Dr. [name] in various areas of teaching.Medical StudentsName and Graduation DateSupervised independent study,spring and summer 2005 for ’07students [33 Students Names] for’06 students [66 Students Names][Student Name] ‘06DescriptionThese students needed help preparing for USMLE Step I. I meet with students everyother week to offer encouragement, review progress, and answer pharmacologyquestions. Students were required to formulate a study plan and report their scoreson practice tests.Advisor for a project to design a set of first ILMs (FILM) for incoming MSI studentsand a distribution system via iROCKET that is being led by [ name].[Student Name] ‘08[Student Name] ‘06Advisor on student’s Summer Curriculum Ambassador work on a drug database forthe IDS102: Major Organs Systems block.Student worked as Student Ambassador in Summer ’02. I supervised her creation of“How to Use Semi-Log Paper” PowerPoint program for IDS 101: Prologue. She hasrecently begun a year-long Curriculum Ambassadorship and will be working with meon several Life Cycle projects.[Editor’s note: List continues with nine additional students from 2003 to present.]25Association of American Medical Colleges, 2007

Scholarship Consensus ConferenceKaren Wendelberger-Marcdante, M.D.Educator’s PortfolioEvidence Of Educational Leadership“Leaders motivate people and facilitate the developmentof projects toward achievement of goals.”Associate Dean For Curriculum BProblem: Lack of integration across the four years ofmedical school, need for specific objectivesto help continue curricular development.Project: Medical School Curricular RetreatsGoal:Role:To establish specific unified objectives foreach year, looking to integrate across years.Facilitator, ConvenerRetreat Outcomes:• M1 + M2 Objectives• M3 Global Objectives + M3Diagnosis/Assessment/Procedure (DAP)Objectives• M4 Objectives• Working Groups developed 4 longitudinalcurriculaº Communication curricular objectivesº Managed care curricular objectives• M3 Clerkship Evaluation Formº Required for use in all M3 clerkshipsº Faculty rate student performance on each ofthe 10 M3 Global Objectives (e.g., communicationwith patients and families, history,physical examination, medical programsolving, professional behavior).º Nine-point scale divided into three subclusters(below standards, at standards, above standards)includes behavioral descriptors associatedwith each objective Reliability (internal consistency) Cronbachalpha = 0.965 (N=530)• M4 Clinical Rotation Evaluation Formº Behaviorally anchored rating scale developed,pilot to occur in spring 2000.• Application/receipt of four Learning Resourcesrelated grants 1995–1998* (MCW): $56,750º 1995: Co-Investigator – “Reassessing the M3Year Curriculum: Matching actions to goals”$6,000. 1997 Co-Recipient MCW LearningResources Innovative Educational ProjectAward – selected by Executive Committee –Curriculum and Evaluation Committee[Editor’s note: All grants listed in actual portfolio.]º Genetics curricular objectivesº Geriatrics curricular objectivesB Note: At this candidate’s medical school, the Educator’s Portfolio is submitted as part of the promotion packet to highlight/provideevidence of educational excellence. This Portfolio was submitted in 1999 as part of packet for promotion to rank of professor.* Awarded following peer review by CEC Executive Committee.26Association of American Medical Colleges, 2007

Scholarship Consensus ConferenceDissemination:• M3 Global and DAP Objectives distributed to allM3 students; M4 Objectives distributed to all M4students• Genetics objectives used for successful grantproposal (HRSA Genetics in Primary Care)• Geriatrics objectives used for successful grantproposal (AAMC/Hartford Foundation) andpending proposal (Reynolds Foundation)Vice Chair, Department Of PediatricsProject: RRC AccreditationRole:Administratively responsible forassuring completion of paperwork forresidencies (Pediatrics andMedicine/Pediatrics) and all fellowships.Outcomes: Five-year accreditation for allprograms with minimal requests forresponse. No major concerns.• Two national presentations (one on method, oneon content)• One peer-reviewed publicationProject:Role:Performance-based AssessmentProgramCo-creator, evaluator[Editor’s note: Publications and Presentations listed in actualportfolio; Structure of Goal, Role, Outcomes, Dissemination usedto present seven additional projects.]Outcomes: Enhanced satisfaction of students(clerkship evaluations)Development from use for teaching touse for evaluationIdentification of specific areasrequiring improved teaching strategiesDissemination:Two national peer-reviewed posters[Editor’s note: Additional leadership roles and associated projectslisted in actual portfolio.]27Association of American Medical Colleges, 2007

Scholarship Consensus ConferenceEducator Activity Category Format andIllustrative Examples DemonstratingEvidence for Quantity, Quality, andEngagement with the EducationalCommunityTeachingEducator’s Portfolio Format1. Description of role (with reflective critique)2. Narrative or tabular display of who, what, when,where, how much, how many• Easy-to-read summary3. Evidence of quantity and quality• Narrative or tabular summary of student evaluationsand, if available, peer evaluations,including, if possible, change over time andnormative data• Short excerpts from supporting letters(complete letters should be in appendix orincluded with letters of recommendation forpromotion)• Invitations to teach outside department orschool• Repeat invitations to teach to the same groupor in the same course4. Evidence of engagement with the community ofeducators• Teaching awards, including the criteria forjudgment by peers• Invited presentations (e.g., workshop, discussiongroup) related to teaching expertisefocused on teaching method or effectiveteaching strategies• Peer review of teaching and/or instructionalmaterialº Cite where and how peer reviewed• Samples/examples of materials (or excerpts,summary)• Public dissemination and impact/useº Reference presentation in a peer-reviewed orinvited forum at regional/national meetingº Cite how product was disseminatedº Indicate adoption/adaptation of teachingstrategy, method, or instructional materialsby others (e.g., citation in publications)º Indicate inclusion in a national repository(data re: number of “hits,” adoptions)Educator’s Portfolio Documentation ExamplesEvidence of Quantity and QualityTeaching Activity/Role Year Quantity # Learners Quality* Additional details available in appendixSeminar Leader (Medical students)Family Medicine Clerkship 1997–present ~ 19 hrs/yr 8–12/yr For 2004–2005, mean rating, “Wasan effective seminar leader” = 5.41on a 7-point scale (n=181 ratings)**Comparative ratings for each year should be given and compared to peer group if possible.28Association of American Medical Colleges, 2007

Scholarship Consensus ConferenceStudent Evaluations: Individual Faculty Teaching Ratings by Year,“Overall Effectiveness as a Teacher”5.04.54.03.53.02.52.01.51.00.502003-04 2004-05 2005-06 2006-071=Ineffective5=Highly effectiveDr (Name)All FacultyEvidence of Engagement with the EducationalCommunityInstructional Material: Interactive diagnostic decision-makingcases: cough, chest pain, abdominalpainMy role: Co-developer of three clinical cases,designed to help students develop clinical problemsolvingskillsPeer review: To date, 500 people have accessed ourWeb site. Feedback regarding the materials and useis requested. At least 10 schools report adopting oradapting one or more of the cases. Representativecomments from reporting schools includeº “These cases are very realistic, and we haveadopted them for use in our required clerkshipin Medicine to supplement ‘real’ patient cases.”º “The evidence for diagnostic and therapeuticdecisions is documented using current literature,emphasizing the importance of evidencebaseddecision making.”The cases have not been formally peer reviewed forinclusion in a national repository.29Association of American Medical Colleges, 2007

Scholarship Consensus ConferenceCurriculum DevelopmentEducator’s Portfolio Format1. Name and educational activity2. Role / contributions (consultants / collaborators)3. Context (need, the change, description)4. Demonstrate meet criteria of value to institutionand scholarship• Clear goals• Appropriate methods• Effective presentation• Adequate preparation• Significant results• Reflective critique5. Dissemination6. Revenue (including grants)Educator’s Portfolio Documentation ExamplesTitle:Evidence-based Medicine (EBM) course(First-year students)Roles:Course Directorº Responsible for organizing instruction forfive-credit course that includes lectures andindividual exercises• Recruit and train small-group leaders• Oversee case development (with others)• Develop objectives to introduce basicconcepts of EBM and help studentsapply concepts while reading articles forsmall-group discussion sessionsClear Goals:Create a new EBM course for all first-year studentsthat students perceive as clinically relevant. Thepredecessor course consistently received “very poorstudent evaluations” and EBM content was “lackingin the curriculum.”Adequate Preparation:º Review of “best practices”: McMaster’scurriculum, NBME test contentº Ph.D. in public healthAppropriate Methods:º Multi-method approach including interactivelecture series, real-time clinical vignettes,abstract critique followed by article critiqueº Increased collaborative teaching as all smallgroups now co-led by basic scientist andphysicianº Assessment methods: test questions are application(not rote memorization), checklist evaluationSignificant Results (Outcomes):º Improved evaluation of didactic series—frommarkedly below institutional year average toaverageº AAMC graduate survey—from inadequateexposure to appropriate/excessiveº NBME performance from below to aboveaverageº Improved OSCE performance on ambulatorypractice moduleº Developed fourth-year integrative selectivewith basic scientists from targeted courses andother clerkship directors.º Collaborators: Statisticians in beginning,three clinicians currently30Association of American Medical Colleges, 2007

Scholarship Consensus ConferenceEffective Presentation (Dissemination):º Results presented to curriculum committeeº Internal review in progress with comparativedata over last two yearsº Content replicated in another clerkship andreplication in selected residencies plannedNext Steps for Continuing Improvement:º Developing electronic form to assist schoolsthat don’t have critical mass of faculty/learnersº Submit to AAMC MedEdPORTALAdvising and MentoringEducator’s Portfolio FormatNameof individualType andTrainee LevelAdvisor ormentorrelationshipPurpose Process CurrentStatusSpecific goalsof relationshipDates anddescriptionwith details ofmentoring oradvising relationshipof protégé oradvisee includingpositions,academic rank,and relatedacademicachievementsOutcome(s)Examples includeabstracts, publications,awards,grants, examples ofgoal attainment,resolution ofconcern or problemDocumentationreference to abstracts,presentations, publications,ongoingcollaboration,continued influence,etc.Educator’s Portfolio Example for Advising/MentoringName ofAdvisee orProtégé1. CharlesJasonType/Level• Mentor• MedicalstudentPurpose ofRelationship• Developmentof professionalidentity• Career guidancein service ofminority healthcareDuration andProcess9/2001–present• 1-on-1 meetings• Edit paper, CV• M1-2curriculumauditor via PDA• Advocate forLCME liaisonposition• Link to facultyrole modelsCurrentStatus ofProtégé• Internal medicineresident• M.D. received2006• Published essayin AcademicMedicineOutcome(s) ofRelationship• Appointment asAAMC StudentLiaison to LCME• Maturation asphysicianmatching careerchoice to valuesDocumentationof EffectivenessThank-you card atM.D. graduationstated: I wanted tothank you for the timeyou spent with me overthe years in making methe young man I amtoday. I’m not sure ifyou realize the impactyou’ve had in my life,always believing in me,helping me to questionthings, and teaching meabout life and medicaleducation.2. KimberlyMarie• Adviser/Junior Faculty• Preparation ofacademicpromotiondocuments8/05–12/05• 1-on-1 +e-mail• Revise/reframeCV and portfolio• Consult withdepartmentchair re: letterof rec• AssociateProfessor• Promoted 6/06to AssociateProfessor “Thankyou” via e-mail foracademic makeoverand lunch invitation31Association of American Medical Colleges, 2007

Scholarship Consensus ConferenceName ofAdvisee orProtégé1. RonaldAlbertType/Level• Mentor/ JuniorFacultyPurpose ofRelationship• Mentor educationresearchproject“TeachingQualityImprovement inthe EmergencyDepartment”Duration andProcessCurrentStatus ofProtégé9/04–8/06 • AssistantProfessorOutcome(s) ofRelationship• ReceivedEducationInnovation Grant• Published 1(#6 on CV)• Published 1manuscript(#4 on CV)Documentationof EffectivenessEducational Administration andLeadershipEducator’s Portfolio FormatFor each educational leadership project/initiativedescribe each relevant component1. The project/initiative and inclusive dates of theproject2. Need/problem/opportunity–rationale forchange3. Goal(s)4. Leadership role and contribution5. Actions taken and connection to literature andbest practices6. Resources garnered and utilized (humanresources, internal budgets, and grants)7. Evaluation (including external peer review ifrelevant), outcomes and/or impact8. DisseminationEducator’s Portfolio Documentation Example forLeadershipProject: Medical Student Basic Science andClinical Integration (2003–07)Need:Historically, selected basic science courses havebeen poorly rated by students, in part because ofthe perceived “lack of relevance” with clinical practiceand poor pedagogy. Students report that in theclinical years, faculty tell them to “ignore” what youlearned in a particular course.Goal:To increase integration of basic and clinical scienceacross all four yearsPreparation (roles):Clerkship Director, facilitator, author of cases andtutor training manuals, coauthor grant applications.Acquired resources:My collaborators’ and my time were covered by therespective departments. The dean’s office providedstaff support through an internal instructionalinnovation grant that I applied for and received.Methods (actions):I chaired a small working committee that surveyedthe literature, sought best practices from otherschools, and developed a four-year content map.We worked with course and clerkship directors todesign case-based discussion sessions, wrote thecases and a tutor training manual. I conducted fourfaculty development workshops on the use of thenew materials for 50 faculty members.Results/Evaluation of course:The new case-based tutorials were used for the firsttime in the course in 2004–05. Course ratingsimproved from 3.4/5.0 (good) prior to the introductionof the tutorials to 4.2/5.0 (very good) afterimplementation.32Association of American Medical Colleges, 2007

Scholarship Consensus ConferenceScale1 = Poor5 = ExcellentCourserating2003‒04 2004‒05 2005‒06Average ofcomparisoncoursesCourseratingAverage ofcomparisoncoursesCourseratingAverage ofcomparisoncoursesOverall quality 3.4 4.0 4.2 4.1 4.3 4.0Outcomes related to course integration project:• Improved evaluation pre-post introduction ofcases and faculty development workshops fortargeted courses compared to nontargetedcourses (clinical relevance statistically improvedon AAMC Graduation Questionnaire)• Produced cases and tutor training• Increased collaboration (all small groups intargeted courses now co-led by basic scientist andphysician)• Developed two collaborative research projects• One case workbook. Case book peer reviewedand accepted for AAMC MedEdPORTAL. In thepast year, it has received 150 hits with over 20schools adopting the curriculum.• Reflective Critique: Based on the student ratingsand faculty facilitator evaluations, several changeswill be implemented in the next academic year,including more emphasis on student facilitationof small groups, more emphasis on the evidencefrom literature that underlies decisions, and morefaculty development before each small-groupsession• Piloted the incorporation of basic science educationin clinical clerkships• Developed fourth-year integrative selectives withbasic scientists from targeted courses and otherclerkship directors.Presentation (Dissemination):[Citations listed under each using commonlyaccepted standards would appear in this section butare omitted for brevity]• One peer-reviewed journal publication• Three national peer-reviewed presentations• Two national/regional peer-reviewed posters• Two invited presentations33Association of American Medical Colleges, 2007

Scholarship Consensus ConferenceLearner AssessmentEducator’s Portfolio Format1. Context: A brief description of the goal, format,context, and faculty role2. Quantity of assessment activities3. Evidence about quality and engagement withthe educational community specific to:a) Methods (i.e., adherence to best practices,informed by the literature)b) Evidence about quality of results (i.e., measuresof reliability and validity appropriate to the typeof assessment)c) Evidence of contribution to the educationalcommunity, if applicable, (i.e., dissemination ofproducts, impact, etc)Educator’s Portfolio Documentation Example: Learner Assessment Miller’s Pyramid “Does”(Using Glassick’s Criteria to Frame Presentation of Evidence)Description:In response to data from the AAMC Graduation Questionnaire that many medical students perceive thatfaculty rarely observe their clinical skills, I have sought ways to cost effectively expand use of an establishedtool (ABIM mini-Clinical Evaluation Exercise or Mini-CEX) in multisite medical student clerkships.Glassick’s CriteriaEvidenceClear Goals • To determine the feasibility of implementing a personal digital assistant (PDA)-based Mini-CEX for third-year medical (M3) students.Adequate Preparation • Literature review highlights validity and reliability of Mini-CEX. The literature alsoprovided insights regarding ways to enhance cost-effectiveness of administrationprocedures. No PDA-based methods for the Mini-CEX were found.Appropriate Methods • Used conventional software development tools for the PDA to create PDA-basedMini-CEX observation/checklist tool.Significant Results • Demonstrated feasibility of a PDA-based Mini-CEX• Students and evaluators showed a high degree of satisfaction with the tool(comments available in Appendix X).Effective Presentation • Peer-reviewed platform presentation at national meeting (SGIM 2005).• Invited to be a plenary speaker at the annual national meeting of the ClerkshipDirectors of Internal Medicine (October 21, 2005).Reflective Critique • My successful experience developing the PDA Mini-CEX has motivated me to workto adapt a validated learner assessment form to a PDA-based tool that would facilitatecollection and analysis of important data about students’ supervision of clinicalskills in other venues.34Association of American Medical Colleges, 2007

Scholarship Consensus ConferenceContributors And AcknowledgmentsThe authors gratefully acknowledge the contributions of the following individuals and groups. Theycontributed to the success of the AAMC GEA Consensus Conference on Educational Scholarship, resultingin the findings reported here.• Conference presenters Patricia Hutchings, Ph.D., vice president, Carnegie Foundation for theAdvancement of Teaching; Darrell G. Kirch, M.D., president, Association of American Medical Colleges;and LuAnn Wilkerson, Ed.D., professor of medicine and senior associate dean for medical education,David Geffen School of Medicine at UCLA.• M. Brownell Anderson, M.Ed., senior associate vice president, AAMC Division of Medical Education, forongoing vision, support, and guidance to the GEA’s Project on Educational Scholarship, and for her criticalreview and constructive edits of this publication.• The 111 “Teaching Commons” members from 65 medical schools—the medical school deans, associatedeans, faculty members, and colleagues—who participated in the consensus conference.• Our faculty colleagues—Karen Marcdante, M.D., Dario Torre, M.D., M.P.H., and Deborah Simpson,Ph.D., Medical College of Wisconsin, and Susan Masters, Ph.D., University of California, San FranciscoSchool of Medicine—whose abbreviated educator portfolios served as case examples during the conferenceand for illustrative purposes in the appendix.• Staff from the AAMC Division of Medical Education, including Cynthia Woodard, and the AAMCSection for Meeting and Conference Management, including Nathalie Tavel and Rebecca Zurcher.35Association of American Medical Colleges, 2007

Scholarship Consensus ConferenceConference ParticipantsClaudia R. Adkison, J.D., Ph.D.Executive Associate DeanEmory UniversityRobert W. Baer, Ph.D.Associate Professor ofPhysiologyA. T. Still UniversityHealth SciencesConstance D. Baldwin, Ph.D., M.A.Professor of PediatricsUniversity of RochesterSchool of MedicineLisa Bensinger, M.D.Director,Institute for Medical EducationMount Sinai School of MedicineRay Bianchi, M.D.Chair of MedicineCarolinas Medical CenterSusan A. Bliss, M.D.Associate DirectorObstetrics/GynecologyCarolinas Medical CenterSteven M. Block, M.D.Associate Dean forFaculty ServicesWake Forest UniversitySchool of MedicineMartin S. Bogetz, M.D.Professor of Clinical AnesthesiaChair, Core Clerkship OperationsCommunicationUniversity of California,San FranciscoCarol Capello, Ph.D., Ed.M.Associate Director,Office of CurriculumAssistant Professor,Geriatrics EducationWeill-Cornell Medical CollegeRobert T. Card, M.D.Associate Dean, Clinical Affairs andVice DeanUniversity of SaskatchewanCollege of MedicineLatha Chandran, M.D.Associate Dean for Academic AffairsStony Brook UniversitySchool of MedicineSheila W. Chauvin, Ph.D., M.Ed.Professor and DirectorOffice of Medical Education andResearch DevelopmentLouisiana State UniversityHealth Sciences CenterClayton Connor, M.A.Program CoordinatorUniversity of ArizonaCollege of MedicineSonia J. Crandall, Ph.D., M.Sc.ProfessorWake Forest UniversitySchool of MedicineRaymond H. Curry, M.D.Executive AssociateDean for EducationNorthwestern UniversityFeinberg School of MedicineJ. Kevin Dorsey, M.D., Ph.D.Dean and ProvostSouthern Illinois UniversitySchool of MedicineJan Edwin Drutz, I, M.D.Professor of PediatricsBaylor College of MedicineTracy D. Eells, Ph.D., M.B.A.Associate Dean forFaculty AffairsUniversity of LouisvilleSchool of MedicineNehad I. El-Sawi, Ph.D., M.Sc.Senior AssociateDean of AcademicAffairs and Curriculum GovernanceKansas City Universityof Medicine and BiosciencesCarol Elam, Ed.D.Associate Dean for Admissionsand Institutional AdvancementDirector of Medical Education ResearchUniversity of KentuckyCollege of MedicineCam Enarson, M.D.Vice President of Health SciencesDean, School of MedicineCreighton UniversityMartin E. Feder, Ph.D.Faculty Dean of Academic AffairsDivision of Biological Science andPritzker School of MedicineThe University of ChicagoKristi J. Ferguson, Ph.D.Director, Office of Consultation andResearch in Medical EducationUniversity of Iowa College of MedicineRuth-Marie E. Fincher, M.D.Vice Dean for Academic AffairsProfessor of MedicineMedical College of GeorgiaSchool of MedicineMichael T. Fitch, M.D., Ph.D.Wake Forest UniversityDepartment of Emergency MedicineEmil J. Freireich, M.D.Professor of Medicine and Director,Adult Leukemia Research ProgramUniversity of Texas M. D. AndersonCancer Center36Association of American Medical Colleges, 2007

Scholarship Consensus ConferenceErica Friedman, M.D.Associate Dean for UndergraduateMedical Education and AssociateProfessor of MedicineMount Sinai School of Medicine ofNew York UniversityAnn W. Frye, Ph.D.Director of Educational DevelopmentUniversity of Texas Medical Branch atGalvestonThomas Garrett, M.D.Professor of Clinical MedicineColumbia UniversityCollege of Physicians and SurgeonsGregory Gruener, M.D., M.B.A.Professor and Vice Chairman ofNeurologyAssociate Dean for Educational AffairsLoyola University Chicago StritchSchool of MedicineJanet Palmer Hafler, Ed.D.Director of Faculty DevelopmentHarvard Medical SchoolDona Harris, Ph.D.Associate Dean for Faculty DevelopmentMercer School of MedicineRobert V. Higgins, M.D.Associate DirectorGynecology OncologyCarolinas Medical CenterIvor D. Hill, M.B.Ch.B., M.D.Professor of Pediatrics andInternal MedicineWake Forest UniversitySchool of MedicineJohn Dennis Hoban, Ed.D.Director, Education ResearchVirginia CommonwealthUniversity School of MedicineCarol Hodgson, Ph.D.Associate DeanCurriculum and EvaluationUniversity of ColoradoHealth Sciences CenterP. Wes Hofferbert, M.D.Associate Program DirectorCarolinas Medical CenterKathryn N. Huggett, Ph.D.Director, Medical EducationDevelopment and AssessmentCreighton UniversityDavid Irby, Ph.D.Vice Dean for EducationUniversity of California,San FranciscoWilliam B. Jeffries, Ph.D.Associate Dean forMedical EducationCreighton UniversityMark S. Johnson, M.D., M.P.H.ChairmanDepartment of Family MedicineUniversity of Medicineand Dentistry of New Jersey--New Jersey Medical SchoolEdward J. Keenan, D.Phil.Associate Dean for Medical EducationOregon Health and Science UniversitySchool of MedicineJill L. Keller, D.Phil., Ed.M.DirectorOffice of Educational DevelopmentUniversity of ArizonaCollege of MedicineDarrell G. Kirch, M.D.Senior Vice President for Health Affairsand DeanPennsylvania State UniversityCollege of MedicineDebra Lee Klamen, M.D., M.Ed.Associate Dean for Educationand CurriculumSouthern Illinois UniversitySchool of MedicineSharon K. Krackov, M.S., Ed.D.DirectorFaculty and Program DevelopmentAlbany Medical CollegeAnn Lambros, Ph.D.Assistant Dean for EducationWake Forest UniversitySchool of MedicineEllice Lieberman, M.D., Dr.P.H.Dean for Faculty AffairsProfessor of Obstetricsand GynecologyHarvard Medical SchoolSusan J. Lieff, M.D., M.Ed.Associate ProfessorFaculty of MedicineDirector, Education ScholarsProgramUniversity of TorontoFaculty of MedicineKeith D. Lindor, M.D.DeanMayo Medical School,Mayo Clinic College of MedicineNancy Ryan Lowitt, M.D., Ed.M.Associate DeanProfessional DevelopmentUniversity of MarylandSchool of MedicineJames W. Lynch, Jr., M.D., Ed.D.Professor, Fellowship ProgramDirectorUniversity of FloridaCollege of MedicineDavid Manthey, M.D.DirectorUndergraduate Medical EducationWake Forest UniversitySchool of MedicineKaren J. Marcdante, M.D.Director, Pediatric Critical CareFellowship ProgramMedical College of WisconsinEric Spencer Marks, M.D.Professor of MedicineAssociate Dean for Faculty AffairsUniformed Services University37Association of American Medical Colleges, 2007

Scholarship Consensus ConferenceMeridith Marks, M.D., M.Ed.Assistant Dean, Professional AffairsUniversity of Ottawa Faculty ofMedicineJulia E. McNabb, D.OAssistant Dean, Academic AffairsKirksville College of OsteopathicMedicineBonnie Miller, M.D.Associate Dean of UndergraduateMedical EducationVanderbilt UniversitySchool of MedicineWesley James Miller, M.D.Vice Chair for EducationDepartment of MedicineUniversity of Minnesota MedicalSchoolPage S. Morahan, Ph.D.Co-DirectorELAM and FAIMER InstitutesFoundation for Advancement ofInternational Medical Education andResearch (FAIMER)Mary M. Moran, M.D.Associate Dean for FacultyDevelopmentDrexel University College of MedicinePatricia B. Mullan, Ph.D.Associate ProfessorUniversity of Michigan Medical SchoolDavid W. Musick, Ph.D.Associate Dean for MedicalEducationBrody School of Medicineat East Carolina UniversityElza Mylona, Ph.D.Assistant Dean for Academic Affairsand Faculty Development and DirectorOffice of Curriculum SupportStony Brook UniversityHealth Sciences CenterSchool of MedicineCate Nicholas, M.Sc.Director StandardizedPatient ProgramUniversity of VermontCollege of MedicineJames C. Norton, Ph.D.Associate Dean forGraduate Medical Education,AHEC & Community OutreachUniversity of KentuckyCollege of MedicineKaren D. Novielli, M.D.Associate Dean, Faculty AffairsJefferson Medical College ofThomas Jefferson UniversityPatricia O’Sullivan, Ed.D.Office of Medical EducationUniversity of CaliforniaSan Francisco School of MedicineDean Parmelee, M.D.Associate Dean, Academic AffairsBoonshoft School of MedicineWright State UniversitySusan Pasquale, Ph.D.DirectorCurriculum and FacultyDevelopmentUniversity of MassachusettsMedical SchoolOffice of Medical EducationMichele Tortora Pato, M.D.Associate Dean for AcademicScholarshipKeck School of Medicine-University of SouthernCaliforniaHarold L. Paz, M.D.DeanUMDNJ-Robert Wood Johnson Medical School125 Paterson StreetSuite 1400New Brunswick, NJ 08903-0019Phone: (732) 235-6300Fax: (732) 235-6315paz@umdnj.eduSubha Ramani, M.D., M.Ed.DirectorFaculty Development in ClinicalTeachingBoston UniversityKathleen W. Rao, D.Phil.Professor of Pediatrics,Pathology, and GeneticsCo-Chair of the Second-YearMedical School CurriculumUniversity of North Carolinaat Chapel HillSchool of MedicineKyle E. Rarey, Ph.D.Associate DeanUniversity of FloridaCollege of MedicineBoyd Richards, Ph.D.DirectorOffice of CurriculumBaylor College of MedicineNancy A. Richeson, M.D.Assistant Dean forClinical AssessmentUniversity of SouthCarolina School of MedicineGary C. Rosenfeld, Ph.D.Professor and Assistant DeanUniversity of Texas MedicalSchool at HoustonLinda M. Roth, Ph.D.Director of Faculty DevelopmentWayne State UniversitySchool of MedicineMary Ruh, Ph.D.ProfessorSaint Louis UniversityW. Scott Schroth, M.D., M.P.H.Senior Associate DeanAcademic AffairsGeorge WashingtonSchool of Medicine38Association of American Medical Colleges, 2007

Scholarship Consensus ConferenceCarol Schwab, J.D., LL.M.Assistant DeanLegal/MedicalMedical College of GeorgiaSteve J. Schwab, M.D.Interim DeanMedical College of GeorgiaSchool of MedicineDean’s OfficeLaura Schweitzer, Ph.D.Vice President forAcademic AffairsVice Dean, College of MedicineState University of New YorkUpstate Medical CenterSusan Scott, M.D.Academic AffairsUniversity of New MexicoSchool of MedicineJohn H. Shatzer, Ph.D.DirectorAssociate Professor,Medical EducationVanderbilt UniversitySchool of MedicineOffice for Teaching andLearning in MedicineJudy Shea, Ph.D.Associate DeanMedical Education ResearchUniversity of PennsylvaniaSchool of MedicineBenjamin Siegel, M.D.Professor of PediatricsBoston University School of MedicineDeborah Simpson, Ph.D.Associate DeanEducational Support and EvaluationMedical College of WisconsinClint W. Snyder, Ph.D., M.A.Associate Dean, Medical EducationOffice of Health Professions EducationNortheastern Ohio UniversitiesCollege of MedicineKevin H. Souza, M.Sc.Director, EducationalTechnologyAssociate DirectorOffice of Medical EducationUniversity of California, San FranciscoSchool of MedicineNancy Roberta Stevenson, Ph.D.Professor of Physiologyand BiophysicsUniversity of Medicineand Dentistry of New Jersey-Robert Wood JohnsonMedical SchoolCurtis Stine, M.D.Professor and Associate ChairFlorida State University College ofMedicineHugh Stoddard, Ph.D., M.Ed.DirectorCurriculum DevelopmentUniversity of NebraskaCollege of MedicineHenry W. Strobel, Ph.D.Associate Dean for FacultyAffairsProfessor of Biochemistryand Molecular BiologyUniversity of Texas HealthScience Center at HoustonDavid Ethan Swee, M.D.Acting Senior Associate Deanfor EducationUniversity of Medicine andDentistry of New Jersey-Robert Wood JohnsonMedical SchoolPatricia Thomas, M.D.Associate Dean for CurriculumJohns Hopkins UniversitySchool of MedicineJoshua T. Thornhill IV, M.D.Assistant Dean for ClinicalCurriculumUniversity of South CarolinaSchool of MedicineThomas R. Viggiano, M.D., M.Ed.Associate Dean forFaculty AffairsMayo Medical SchoolAnna Walker, M.D.Associate Professorof PathologyMercer School of MedicineAnne Walling, M.D.University of KansasSchool of MedicineLuAnn Wilkerson, Ed.D.Senior Associate Dean forMedical EducationDavid GeffenSchool of Medicine at UCLAValerie N. Williams, Ph.D., M.P.A.Associate Dean for Faculty AffairsCollege of MedicineUniversity of OklahomaHealth Sciences CenterDaniel Wolpaw, M.D.Associate Professor of MedicineCase Western Reserve UniversitySchool of MedicineJeffrey G. Wong, M.D.Senior Associate Dean forMedical EducationCollege of MedicineMedical University of South Carolina39Association of American Medical Colleges, 2007

Scholarship Consensus ConferenceReferences1 Boyer, EL. Scholarship Reconsidered: Priorities of the Professoriate. Princeton, New Jersey: PrincetonUniversity Press, 1990.2 Rice, RE. “Scholarship Reconsidered” History and Context. In: O’Meara K, Rice RE (eds). Faculty PrioritiesReconsidered: Rewarding Multiple Forms of Scholarship. San Francisco: Jossey-Bass, 2005:17–38.3 Glassick CE, Huber MT, Maeroff GI. Scholarship Assessed: Evaluation of the Professoriate. San Francisco:Jossey-Bass, 1997.4 Whitcomb M. The medical school’s faculty is its most important asset. Acad Med. 2003;78:117–118.(pg 117 quote)5 Klingensmith ME, Anderson KD. Educational scholarship as a route to academic promotion: A depiction ofsurgical education scholars. Amer J Surg. 2006;191:533–537.6 Simpson D, Hafler J, Brown D, Wilkerson L. documentation systems for educators seeking academic promotionin US medical schools. Acad Med. 2004;79:783–790.7 Beasley BW, Wright SM, Confrancesco J, Babbott SF, Thomas PA, Bass EB. Promotion criteria for clinicianeducatorsin the United States and Canada: A survey of promotion committee chairpersons. JAMA.1997;278:723–728.8 Pangaro L, Fincher R, Bachicha J, Gelb D, et al. “Expectations of and for Clerkship Directors: ACollaborative Statement from the Alliance for Clinical Education” Teaching and Learning in Medicine,2003;15(3):217–222.9 AAMC. Compact Between Resident Physicians and Their Teachers. January 2006 accessed June 3, 2006www.aamc.org/meded/residentcompact/residentcompact.pdf10 Irby DM, Cooke M, Lowenstein D, Richards B. The academy movement: a structural approach to reinvigoratingthe education mission. Acad Med. 2004;79:729–736.11 Simpson DE, Fincher RM. Making a case for the teaching scholar. Acad Med. 1999;74:1296–1299.12 Fincher RM, Simpson D, Mennin SP, Rosenfeld GS, Rothman A, McGrew MC, Hansen PA, Mazamanian PE,Turnbull JM. Scholarship in teaching: an imperative from the 21st century. Acad Med. 2000;75:887–894.13 Anderson MB, Kassebaum DG. Proceedings of the AAMC’s consensus conference on the use of standardizedpatients in the teaching and evaluation of clinical skills. Acad Med. 1993;68:437–478.14 Hafler JP, Blanco MA, Fincher RM, Lovejoy FH, Morzinski J. Educational Scholarship. Chapter 14. In:Fincher RM (Ed). Guidebook for Clerkship Directors, 3rd edition. Alliance for Clinical Education.http://familymed.uthscsa.edu/ACE/guidebook.htm accessed June 4, 200615 Huber MT, Hutchings P, Shulman, LS. The Scholarship of Teaching and Learning Today. Chap. 2. In:O’Meara K, Rice RE. Faculty Priorities Reconsidered: Rewarding Multiple Forms of Scholarship. San Francisco:Jossey-Bass 2005. Pp. 34–38.16 Hutchings P, Shulman LS. The scholarship of teaching: New elaborations, new developments. Change.1999;31:10–15.17 Fincher RM, Work JA. Perspectives on the scholarship of teaching. Med Educ. 2006;40:293–295.18 Trigwell K, Shale S. Student Learning and the Scholarship of University Teaching. Studies in HigherEducation. 2004;29:523–536.40Association of American Medical Colleges, 2007

Scholarship Consensus Conference19 Shulman LS. Teaching as community property: Putting an end to pedagogical solitude. Change.1993:25(6):6–7.20 Huber MT, Hutchings P. The Advancement of Learning – Building the Teaching Commons. San Francisco:Jossey-Bass, 2005.21 Hutchings P, Shulman LS. The scholarship of teaching: New elaborations, new developments. Change.1999;31:10–15.22 Tyler RW. Basic Principles of Curriculum and Instruction. Chicago: University of Chicago Press, 194923 Kirkpatrick D. Evaluating Training Programs: The Four Levels. San Francisco: Berrett-Koehler, 2006.24 Hutchings P, Huber MT. Carnegie Perspectives: Building the Teaching Commons. The CarnegieFoundation for the Advancement of Teaching. Accessed March 25, 2006www.carnegiefoundation.org/conversations/sub.asp?key=244&subkey=32925 AAMC-Group on Educational Affairs. Mission Statement. www.aamc.org/members/gea/mission.htmAccessed September 24, 2006.41Association of American Medical Colleges, 2007

Association ofAmerican Medical Colleges2450 N Street, N.W., Washington, D.C. 20037-1127T 202 828 0400 F 202 828 1125www.aamc.org