Leadership Observation and Feedback Tool - UCSF School of ...

Part II – Project Narrative
TITLE OF PROJECT:
Title: Leadership Observation and Feedback Tool (LOFT)
PRINCIPAL INVESTIGATOR:
Read Pierce
PROJECT SUMMARY/ABSTRACT (maximum 300 words)
Clinical work environments provide numerous opportunities to work closely with multidisciplinary
teams. Yet despite the increasing emphasis on preparing clinicians, and particularly physicians, to lead
such teams, there remains a paucity of validated, generalizable assessment tools for evaluating
leadership behaviors and providing feedback to learners in the health professions. In 2009, we received
an Innovations grant from the Academy of Medical Educators to establish a Leadership Observation and
Feedback Tool (LOFT). Based on an intensive review of existing leadership evaluation tools in the
health sciences and business literatures, we undertook a careful adaptation of the best existing tools for
team leadership in health care, and are currently preparing a pilot study with a preliminary LOFT that is
qualitative in nature to help us identify a spectrum of leadership behaviors from novice to expert among
physician trainees across 5, independent and previously validated leadership domains. We are
conducting this pilot among residents in pediatrics and internal medicine during the 2010-2011
academic year. Completion of our qualitative LOFT study will inform phase 2 of our project in the
2011-2012 academic year: development of a second, quantitative LOFT that will undergo rigorous
review by health care leadership experts and then iterative and broad utilization with residents and
critical care fellows in pediatrics and internal medicine. Data obtained through application of this
quantitative LOFT will be analyzed to determine validity by looking for five sources of validity
evidence as previously described by Downing. The subsequent development of a validated and
quantitative, scale-based tool for evaluating clinical leadership behaviors will ultimately allow
application of the LOFT to a broad spectrum of health care professionals at multiple levels of training,
thereby supporting evaluation and improvement of leadership practices of those leading clinical teams.
BACKGROUND AND SIGNIFICANCE
In this era of rapidly increasing health care costs, worsening health disparities and unprecedented
health care reform, it is increasingly recognized that the United States needs physician-leaders in
all realms of practice. 3,4,5,7,18,19,27,28,29 Through emphasis on competence in communication and
interpersonal skills, the ACGME recognizes the well-described importance of teamwork and
team leadership, 1,11,16 and requires GME training programs to ensure that residents become
competent in these domains. In addition, more attention is being paid to the need for clinical
team leadership, especially in context of improving patient safety and reducing medical
errors. 4,18,22 At UCSF, the novel Pathways to Discovery program aims to develop physicianleaders,
but in most instances explicit incorporation of leadership training curricula is rare in
medical education, both at UCSF and nationally. 8 Even though residents have many
opportunities to serve as the leader of a team during training and to receive multisource feedback
that could promote skills development, 24 formal education to further develop skills in team
leadership is rarely provided, 8 despite evidence that multidisciplinary feedback and dedicated

Innovations Funding Application – Project Narrative Page 2
leadership curricula can be an effective means for improving team leadership. 12,15 In order to
formalize education in leadership skills and to ensure that the ACGME requirements regarding
competence are met, adequate tools to assess leadership skills are needed.
Assessment of team leadership is prevalent in the business literature and often based on project
teams. In particular, the Leadership Practice Inventory (LPI) is a validated and reliable
leadership assessment tool that has been used widely to provide multi-source feedback in varied
settings. The LPI consists of 30 questions that map to 5 distinct domains: modeling of behavior,
inspiring team members, challenging the process, enabling team members and recognizing
others’ efforts. 26 We have used the LPI for several years to provide multi-source feedback for
pediatric residents in the Pediatric Leadership for the Underserved (PLUS) program. Although
overall satisfaction with this tool has been adequate, residents and evaluators have consistently
commented that the LPI does not adequately address experiences on a clinical team and that
some of the behavioral markers identified in the LPI are difficult to assess in the clinical setting.
Recently, clinical team leadership tools have been developed for crisis resource management in
“high stakes” clinical settings such as real and simulated resuscitations, operations and
emergencies. 9,13,14,17,21,23 These tools are all remarkably similar and assess overlapping leadership
domains, including coordination, cooperation, situational awareness, and communication.
Although some of these tools have undergone validity testing, 20,23 they are limited in their
application to high-stakes, high-intensity clinical contexts that do not necessarily reflect daily
work on most inpatient wards and in outpatient clinics.
We are aware of only one published tool that was designed to measure leadership skills in non-crisis
settings: Orlander and colleagues have developed and validated a 7-item Resident Leadership Scale for
internal medicine residents on inpatient services. 25 This tool is narrow in scope, as it is designed to
measure leadership skills of senior internal medicine residents in an inpatient setting via evaluation by
first-year residents. The Resident Leadership Scale would therefore need significant adaptation to be
applicable in other settings or to incorporate feedback from more diverse, multidisciplinary evaluators.
In addition, the report published on this tool does not provide any evidence for construct validity that is
grounded in team leadership concepts identified in the literature.
We postulate that for non-crisis clinical settings, where well defined teams work within a circumscribed
setting under the longitudinal leadership of one clinician, the skill domains validated in the LPI
instrument are highly relevant for capturing and evaluating leadership behaviors. However, based on our
experience with the LPI, we feel that the instrument’s terminology is often ill-suited to clinical settings
and requires too much time for completion to be accepted for routine use by health care professionals.
We therefore set out to develop, test and validate a comprehensive Leadership Observation and
Feedback Tool (LOFT) that is based on existing leadership frameworks; is applicable to a diverse set of
learners, practitioners and clinical settings; and can be used for multi-source feedback (see Preliminary
Work section). With further refinement and validation, this tool has tremendous potential for application
to a broad spectrum of health care professionals at multiple levels of training, including medical
students, physician trainees, faculty, and non-physician providers. Our ultimate goal is to improve the
clinical leadership practices of those leading clinical teams, particularly physicians, regardless of
practice environment.

Innovations Funding Application – Project Narrative Page 3
PRELIMINARY WORK
With support from an AME Innovations Grant, we are currently implementing a LOFT for
qualitative assessment of leadership skills. In order to identify and understand existing leadership
evaluation methods in clinical environments—including relevance to individual leadership skills
development and limitations in generalizability—we performed an in-depth review of the
literature on teamwork, team leadership, and physician trainee leadership in clinical work
environments via PubMed and Business Source Complete searches. We subsequently
summarized the essential elements of the TeamSTEPPS program, 9 the Mayo High Performance
Teamwork Scale, 23 the Communication and Teamwork Skills (CATS) Assessment, 14 the
Anaesthetists' Non-Technical Skills (ANTS), 13 and the Ottawa Crisis Resource Management
Scale. 21 Due to limitations of these programs (as discussed above), and the strength of validity
testing for the LPI, we then adapted the LPI tool to create a 5-domain instrument that is based on
the essential domains of the LPI, but is shorter (10 questions rather than 30) and contains
modified phrasing to ensure clinical applicability and incorporation of key, relevant concepts
from the above programs identified in our literature search.
For the development and first pilot phase of our project, we constructed the LOFT in such a way
that the evaluator describes observed behaviors and gives suggestions for improvement, hence
only providing qualitative data. Rationale for this approach was two-fold: 1) we want ultimately
to create a quantitative rating tool that, like the LPI, is anchored to leadership behaviors ranging
from novice to expert, but descriptions of such spectra of leadership behaviors does not presently
exist in the medical education literature; 2) creation of a leadership behavior scale, whether
numeric or descriptive (i.e., poor-to-excellent) would currently be arbitrary and subject to raters’
tendency to give favorable scores, even for relatively poor performance. 6 We plan to use the first
version of the LOFT in a group of pediatric and internal medicine residents to obtain qualitative
data which will inform development of a quantitative tool with behavioral anchors. The first
version of the LOFT tool and this project design was reviewed by ESCape, and we have obtained
CHR approval for this project; enrollment of subjects will start early in 2011.
SPECIFIC AIMS
This grant application covers the second phase of development of a Leadership Observation and
Feedback Tool (LOFT), which involves the development, pilot and validation of the quantitative tool.
Specific Aim 1: To create a quantitative tool for assessment of leadership skills based on qualitative
data obtained during the 2010-2011 academic year from our first LOFT pilot.
Specific Aim 2: To pilot and further develop the LOFT by trialing the quantitative tool with pediatric
and internal medicine residents and critical care fellows and making iterative changes based on user
feedback.
Specific Aim 3: Use LOFT scores gathered in the trials to establish validity of the tool.

Innovations Funding Application – Project Narrative Page 4
PROJECT DESIGN AND METHODS
Specific Aim 1—Development of Quantitative Tool: In 2010-2011, we created a leadership behavior
rubric covering 5 leadership practice domains, which are derived from the validated LPI and known to
be independent of one another. Two broad, qualitative questions in each domain prompt LOFT users for
written, open-ended descriptions of the leadership behaviors witnessed in physician-trainees who are
leading clinical teams. In our current pilot, we are collecting a total of approximately 80 LOFTs
providing qualitative assessment and feedback on leadership behaviors among residents in pediatrics and
internal medicine. Eight pediatric residents and 8 internal medicine residents on ward rotations are
participating in the current pilot and identify 5 members of the interdisciplinary team they lead to
complete the LOFT during the final week of their rotations. Participants receive instruction on principles
of team leadership and on use of the LOFT prior to enrollment in the pilot. In this first phase, the LOFT
is administered as a paper instrument to maximize completion rates. With our current funding, we will
employ a research assistant to enter the data into qualitative analysis software in the spring of 2011. We
will then perform qualitative analysis of the LOFT responses until saturation has been reached to
identify themes and delineate a spectrum of physician behaviors in each of the 5 leadership practice
domains. This will inform creation of behavioral anchors on a scale from novice to expert.
In phase 2 of LOFT development (2011-2012 academic year), we will use an expert panel made up of 8-
10 local and national physician leaders to review the tool for completeness and accuracy in order to
ensure content validity. Each expert independently will rate the perceived accuracy and relevance of the
quantitative LOFT in two ways. First, experts will indicate the extent to which they agree that each
LOFT domain is a useful component of a clinical team leadership assessment tool. Second, experts will
be asked to rate how well the behavioral anchors capture the true novice-to-expert spectrum seen among
individuals leading clinical teams. In a second iteration of the review, aggregate results of independent
expert ratings will be shared with panel members so that each expert will know how the group of experts
as a whole evaluated each domain and the behavioral anchors. Each expert will evaluate the LOFT by
the same criteria, thereby allowing panel members to revise their ratings based on feedback from other
experts. Domains and behavioral anchors will be included in the final version of the LOFT instrument if
at least 80% of experts agree they are both relevant and adequately capture the novice-to-expert
spectrum. All expert evaluations will be performed using a web based survey method (e.g. survey
monkey).
Specific Aim 2—Pilot the quantitative LOFT with pediatric and internal medicine residents and
critical care fellows: We will pilot the quantitative LOFT tool for 4-months among residents on
general inpatient services and subspecialty fellows in critical care in pediatrics and internal medicine at
Moffitt-Long Hospital. Similar to the qualitative LOFT pilot, all participants will be instructed on the
use of the LOFT prior to participation, and each resident or fellow will be asked to select 5 members of
the multidisciplinary team to complete the LOFT. Given the number of pediatric and internal medicine
residents and critical care fellows rotating at Moffitt-Long Hospital during the 4-month pilot, we
anticipate that a total of 16 pediatric residents, 48 internal medicine residents, 6 pediatric critical care
fellows and 4 adult critical care fellows will participate. The format of the quantitative tool will be
paper-based, and we will elicit the help of a research assistant to enter data into an Excel spreadsheet.
Data will then be imported into SPSS or other software needed for further analysis as outlined in
Specific Aim 3. We also will ask all participants to complete a brief on-line survey to assess satisfaction
with the tool, including ease of use, applicability to the clinical work environment, and usefulness for
giving or receiving feedback.

Innovations Funding Application – Project Narrative Page 5
Specific Aim 3—Establish validity of the tool: For the following aim, we will obtain assistance from
the Office of Medical Education (OME) for analysis. We will apply the now common unitary view of
validity, 2 and look for five sources of validity evidence as described by Steven Downing (content
evidence, response process, internal structure, relationship to other variables and consequences). 10
A. Content Evidence: A panel of experts will review the quantitative LOFT to ensure content validity
as described in Specific Aim 1.
B. Response Process: We will look for response process, essentially defined as data integrity, by
reviewing completeness of the collected LOFTs. We will also survey participants as outlined in
Specific Aim 2 to ensure that all questions in the tool are clear and unambiguous.
C. Internal Structure: We will perform several statistical analyses to ensure reproducibility and
generalizability of the tool, including confirmatory factor analysis, known-groups comparisons and
generalizability study. For the latter, we plan a three facet generalizability study (G-study) using
raters, items and occasions as facets. Based on this analysis, we will be able to determine the amount
of score variance due to rater, item, occasion (and interactions) effects. We will then use this
information to perform a decision study (D-study) to determine the number of raters and items (or
items within sub-domains, depending on results of factor analysis) required to achieve a
generalizability coefficient ≥ 0.8. We anticipate that we can use this approach, but because there is
some unpredictability in how the LOFT will be used by different multidisciplinary evaluators, we
will work with OME to decide on the best methodology as the project evolves.
D. Relationship to other variables: Some of the participants in our pilot are routinely assessed with
other leadership assessment tools. Pediatric residents in the PLUS program are routinely assessed
with the LPI. 26 Pediatric intensive care fellows lead simulated resuscitations as part of their training
and are assessed with the Ottawa Crisis Resource Management Global Rating Scale. 21 For these
groups of participants, we will calculate correlation coefficients between scores on the LOFT and
either the LPI or the Ottawa GRS.
E. Consequences: The consequential aspect of validity refers to the impact assessment scores have on
the learner, and we will look for this by asking LOFT participants about the impact of the LOFT on
their learning of leadership skills as part of the survey described under Specific Aim 2.
We will obtain approval from the Committee for Human Research at UCSF for this project.
PLAN FOR MEASUREMENT AND DOCUMENTATION OF PROJECT EFFICACY AND
OUTCOME
We will survey participants in the LOFT pilot using a Likert-based agree/disagree scale about ease of
use, time required to complete the LOFT, perceived validity of measuring clinical team leadership skills,
and applicability of the LOFT in their particular clinical work environments. We will ask LOFT score
recipients about their perceptions regarding the value of the LOFT for self-reflection on and
improvement of their team leadership skills. In particular, we will ask score recipients about how

Innovations Funding Application – Project Narrative Page 6
accurately they believe the LOFT evaluation of their leadership behaviors along the novice-to-expert
scale describes their self-perceptions of leadership competence. We also will elicit comments regarding
potential improvements for the tool itself and use of the tool. Finally, we will use the procedures
described in the specific aims to measure the content and construct validity of the tool and determine
how to reliably use the tool.
PLAN FOR CONTINUATION OF PROJECT AT THE END OF FUNDING CYCLE
We have engaged residency program directors as well as current and future chief residents in pediatrics
and internal medicine to assist with LOFT development and implementation. After completion of the
project we plan to incorporate use of the LOFT in both internal medicine and pediatric residency and
critical care fellowship training at UCSF. We will describe the tool and the validity/reliability data in a
manuscript, which we will submit for publication in a medical education journal. In addition, we plan to
give presentations and workshops at local and national conferences to further disseminate the tool for
use by other programs and institutions. Future projects will explore the utility of this tool in
undergraduate medical education and a broader range of graduate medical education and other health
care professional schools.
LETTERS OF SUPPORT
1) Donna Ferriero, Interim Chair, Department of Pediatrics with Elena Fuentes, Chair,
Pediatrics, SFGH, and Jeff Fineman, Chair, Pediatric Critical Care
2) Dan West, Program Director, Department of Pediatrics
3) Talmadge King, Chair, Department of Medicine
4) Harry Hollander, Program Director, Department of Medicine
5) Christy Boscardin, Office of Medical Education
REFERENCES
1. Accreditation Council for Graduate Medical Education. Program Director Guide to the
Common Program Requirements.
http://www.acgme.org/acWebsite/navPages/nav_commonpr.asp
2. American Educational Research Association, American Psychological Association,
National Council on Measurement in Education. Standards for Educational and
Psychological Testing. Washington, DC: American Educational Research Association,
1999.
3. Association of American Medical Colleges. The AAMC Project on the Clinical
Education of Medical Students. Washington, DC; 2001.
4. Berwick DM, Nolan TW. Physicians as leaders in improving health care: a new series in
Annals of Internal Medicine. Ann Intern Med. 1998;128(4):289-292.

Innovations Funding Application – Project Narrative Page 7
5. Buchler P, Martin D, Knaebel HP, Buchler MW. Leadership characteristics and business
management in modern academic surgery. Langenbecks Arch Surg. 2006;391(2):149-
156.
6. Cacamese SM, Elnicki M, Speer AJ. Grade inflation and the internal medicine
subinternship: a national survey of clerkship directors. Teach Learn Med. 2007
Fall;19(4):343-6.
7. Cheng TL, Szilagyi PG. Leadership in academic general pediatrics. J Pediatr. 2007
May;150(5):451-2, 452.e1.
8. Cooke M, Irby DM, O’Brien BC. Educating Physicians: A Call for Reform of Medical
School and Residency. San Francisco: Jossey-Bass; 2010.
9. Clancy CM, Tornberg DN. TeamSTEPPS: assuring optimal teamwork in clinical settings.
Am J Med Qual. 2007 May-Jun;22(3):214-7.
10. Downing SM. Validity: on meaningful interpretation of assessment data. Med Educ. 2003
Sep;37(9):830-7.
11. Druskat VU, Wolff SB. Building the emotional intelligence of groups. Harvard Business
Review. 2001;79(3):80.
12. Fitchner CG, Hardy D, Patel, M, et al. A Self-Assessment Program for Multidisciplinary
Mental Health Teams. Psychiatric Services. 2001; 52:1352–1357, 2001.
13. Fletcher G, Flin R, McGeorge P, Glavin R, Maran N, Patey R. Anaesthetists' Non-
Technical Skills (ANTS): evaluation of a behavioural marker system. Br J Anaesth. 2003
May;90(5):580-8.
14. Frankel A, Gardner R, Maynard L, Kelly A. Using the Communication and Teamwork
Skills (CATS) Assessment to measure health care team performance. Jt Comm J Qual
Patient Saf. 2007 Sep;33(9):549-58.
15. Goldstein AO, Calleson D, Bearman R, Steiner BD, Frasier PY, Slatt L. Teaching
Advanced Leadership Skills in Community Service (ALSCS) to Medical Students.
Academic Medicine. 2009;84(6):754-764.
16. Goleman D. What makes a leader? Harvard Business Review. 1998;76(6):92-102.
17. Guise JM, Deering SH, Kanki BG, Osterweil P, Li H, Mori M, Lowe NK. Validation of a
tool to measure and promote clinical teamwork. Simul Healthc. 2008 Winter;3(4):217-23.
18. Institute of Medicine: Committee on Quality of Health Care in America. Crossing the
Quality Chasm: A New Health System for the 21st Century. Washington, DC: National
Academy Press; 2001.

Innovations Funding Application – Project Narrative Page 8
19. Kahn LH. State report - A prescription for change: The need for qualified physician
leadership in public health. Health Aff. 2003;22(4):241-248.
20. Kim J, Neilipovitz D, Cardinal P, Chiu M. A comparison of global rating scale and
checklist scores in the validation of an evaluation tool to assess performance in the
resuscitation of critically ill patients during simulated emergencies (abbreviated as "CRM
simulator study IB"). Simul Healthc. 2009 Spring;4(1):6-16.
21. Kim J, Neilipovitz D, Cardinal P, Chiu M, Clinch J. A pilot study using high-fidelity
simulation to formally evaluate performance in the resuscitation of critically ill patients:
The University of Ottawa Critical Care Medicine, High-Fidelity Simulation, and Crisis
Resource Management I Study. Crit Care Med. 2006 Aug;34(8):2167-74.
22. Lee TH. Turning Doctors into Leaders. Harv Bus Rev. 2010 Apr;88(4):50-8
23. Malec JF, Torsher LC, Dunn WF, Wiegmann DA, Arnold JJ, Brown DA, Phatak V. The
Mayo high performance teamwork scale: reliability and validity for evaluating key crew
resource management skills. Simul Healthc. 2007 Spring;2(1):4-10.
24. Miller A, Archer J. Impact of workplace based assessment on doctors' education and
performance: a systematic review. BMJ. 2010 Sep 24;341:c5064.
25. Orlander JD, Wipf JE, Lew RA. Development of a tool to assess the team leadership
skills of medical residents. Med Educ Online. 2006;11:27.
26. Posner, B. Leadership Practices Inventory Data Analysis. Retrieved on Oct 20, 2009
from: http://media.wiley.com/assets/2034/63/LPIAnalysisAug2009.pdf.
27. Schwartz RW, Tumblin TF. The power of servant leadership to transform health care
organizations for the 21st-century economy. Arch Surg. 2002;137(12):1419-1427;
discussion 1427.
28. Solyom AE. Viewpoint: Improving the health of the public requires changes in medical
education. Acad Med. 2005;80(12):1089-1093.
29. Williams SJ. What skills do physician leaders need now and in the future? Physician
Exec. 2001;27(3):46-48.