(OPPE) Using Automatically Captured Electronic Anesthesia Data

Performance Measures
The Joint Commission Journal on Quality and Patient Safety
Ongoing Professional Performance Evaluation (OPPE) Using
Automatically Captured Electronic Anesthesia Data
Jesse M. Ehrenfeld, MD, MPH; Justin P. Henneman, MS; Robert A. Peterfreund, MD, PhD; Tyler D. Sheehan, BS; Feng
Xue, MS; Stephen Spring, BA; Warren S. Sandberg, MD, PhD
In 2006 The Joint Commission updated the accreditation
standards for the process of credentialing and privileging practitioners
to make the process evidence based and more objective,
facilitate continuous monitoring of performance, help identify
substandard performance, and provide a basis for intervening
when safety and quality of care issues are identified. Specifically,
the revised standards require—for maintaining privileges—an
Ongoing Professional Practice Evaluation (OPPE; effective since
January 1, 2007).* 1–4
Medical specialties employ numerous methods to examine
physician performance, including evaluation of encounters with
simulated patients, observation of patient care, peer assessment,
medical record audits, and portfolio appraisals. 5 In specialties in
which data are both objective and saved, samples of data can be
used to assess competence. For example, radiologists often double-read
films and pathologists double-read slides to assess concordance
between an expert reviewer and the physician being
evaluated. Each of these methods is time consuming, labor intensive,
expensive, and difficult to perform on a continuous
basis. 6 Furthermore, methods that employ direct observation
have the potential to introduce an observation bias, partly because
individuals behave differently when they know they are
being watched. 7–9
The Massachusetts General Hospital (Boston) is a large academic
center providing anesthesia services for more than 49,000
procedures each year. In seeking to ensure compliance with the
new Joint Commission physician credentialing and privileging
standards, the size (149 faculty members) of the department of
Anesthesia, Critical Care and Pain Medicine and the volume of
cases performed annually necessitated the development of a
* According to Medical Staff (MS) Standard MS.08.01.03, “Ongoing professional
practice evaluation is factored into the decision to maintain existing privilege(s), to
revise existing privileges, or to revoke an existing privilege prior to or at the time of
renewal.” According to Element of Performance 2, “The type of data to be collected
is determined by individual departments and approved by the organized medical
staff.” MS.06.01.05, Element of Performance 9 stipulates the use of “Relevant practitioner-specific
data as compared to aggregate data, when available.”
February 2012 Volume 38 Number 2
Copyright 2012 © The Joint Commission
Article-at-a-Glance
Background: The Massachusetts General Hospital (Bos -
ton), a large academic center providing anesthesia services
for more than 49,000 procedures each year, created an Ongoing
Professional Practice Evaluation (OPPE) process that
could use readily available, automatically captured electronic
information from its vendor-provided anesthesia information
management system.
Methods: The OPPE credentialing committee selected the
following initial metrics: Blood pressure (BP) monitoring,
end tidal CO 2 monitoring, and timely documentation of
compliance statements. Baseline data on the metrics were
collected in an eight-month period (January 1, 2008–August
31, 2008). In February 2009 information on the metrics was
provided to the department’s staff members, and the ongoing
evaluation process began. On the basis of three months
of data, final reports for physicians being credentialed were
distributed. Each report included a listing for each metric of
the total number of compliant cases and noncompliant cases
and a comparison by percentage to the baseline departmental
evaluation. A summary statement indicated whether a
physician’s performance was within the group representing
95% of all department physicians. Noncompliant cases were
listed by medical record number and case date so providers
and reviewers could examine individual cases.
Conclusion: A novel, automated, and continuous reporting
system for physician credentialing that uses the existing
clinical information system infrastructure can serve as a key
element of a comprehensive clinical performance evaluation
that measures both technical and generalizable clinical skill
sets. It is not intended to provide a complete system for
measuring competence but rather to serve as a first-round
warning mechanism and metric scoring tool to identify
problems and potential performance noncompliance issues.
73

74
The Joint Commission Journal on Quality and Patient Safety
novel strategy to avoid the high resource and time costs associated
with traditional methods of clinical performance evaluation.
Even if we developed a process that required only one hour of
physician time and one hour of administrative time (both conservative
estimates), the need to perform the OPPE process more
frequently than annually would result in sequestration of several
hundreds of hours of physician and administrative time. Removing
clinicians from direct patient care to comply with the OPPE
mandate would be a significant financial burden and drain on
clinical efficiency. To avoid such a significant impact on clinical
operations, we sought to create a process that could use readily
available, automatically captured electronic information from
our vendor-provided anesthesia information management system
(AIMS) to address The Joint Commission’s OPPE requirements.
A fully implemented AIMS has been in place in each of
our 70 anesthetizing locations since 2002. The AIMS data provide
reliable and extensive documentation of clinical monitoring
and physician practice patterns.
Our primary goal in creating the OPPE process was to develop
a system that (1) requires little or no additional effort on
behalf of the clinician being evaluated and (2) minimizes or eliminates
the modified behavior effect that an observer or mock patient
might create. We also attempted to create a system that
would be unbiased in its measurement of clinical behavior, con-
Table 1. Common Methods of Performance Review
Review Method Advantages Disadvantages
Chart Review Large sample size Difficult to interpret
Randomized Charts can be “smoothed” afterward
No observer effect Retrospective bias
Direct Observation High level of detail Time consuming and expensive
Contextual Observer effect
Immediate feedback Observer bias
Difficult to review many cases
Simulated Patients Controlled and repeatable Time consuming and expensive
Observer bias
Observer effect
360-Degree Evaluation Balanced/minimized bias Time consuming and expensive
Video Review Reduces observer effect Time consuming and expensive
Difficult to interpret
Retrospective bias
Control Charting No observer effect Charts can be “smoothed” afterward
Continuous Retrospective bias
Easy to measure
tinuous, and relatively inexpensive to install and maintain. Such
a system, which could provide feedback that was both continuous
and transparent, would enable physicians to self-assess their
own performance and make adjustments to correct issues before
the actual credentialing process.
Methods
THE COMMITTEE’S CHARGE: DESIGN A SET OF
CREDENTIALING METRICS
We began by establishing the OPPE credentialing committee in
fall 2008 to design a set of meaningful credentialing metrics.
This committee consisted of six senior staff physicians [including
J.M.E., R.A.P., W.S.S. (chair)] representing a broad crosssection
of the department’s clinical activities—including
pediatric, transplant, neurosurgical, orthopedic, and vascular
anesthesia. The committee conducted a literature search and
consulted with other hospitals and departments to generate and
examine a list of existing methods of physician performance evaluation.
The committee discussed the advantages and disadvantages
of the commonly used methods to review physician
competency in its efforts to design a solution that was modeled
on the strengths of successful methods. A partial list of commonly
used methods to evaluate clinical performance is provided
in Table 1 (above).
A major goal of the committee was to ensure that metrics
February 2012 Volume 38 Number 2
Copyright 2012 © The Joint Commission

The Joint Commission Journal on Quality and Patient Safety
were consistent with national practice and centered on patient
care. Thus, the committee looked closely at the American Society
of Anesthesiologists (ASA) standards for monitoring and patient
care as put forth by the ASA Standards and Practice
Parameters Committee. 10
To minimize any administrative burden and reduce the effect
of observation and bias in our measurements, the committee ultimately
decided to focus on extraction of readily available electronic
AIMS data, thereby enabling the creation of an automatic,
reliable, cost-effective process applied on a continuous, ongoing
basis. We could easily modify the process over time to replace
metrics or add additional metrics as needed.
SELECTING THE INITIAL METRICS
By October 2008 the committee selected the initial metrics:
blood pressure (BP) monitoring, end tidal CO 2monitoring, and
timely documentation of compliance statements.
Blood Pressure Monitoring: Requires that a physician document
BP prior to induction of general anesthesia. Documentation
of BP in the AIMS occurs automatically if the BP is
measured. The induction of anesthesia was inferred from manually
entered comments or was inferred from the automatic detection
of inhalation anesthetics in the exhaled gases.
End Tidal CO 2 Monitoring: Requires that a physician monitor
the end tidal CO 2 level at least once during the provision of
general anesthesia. Documentation of end tidal CO 2 monitoring
is automatic if the monitor is functional and connected.
Timely Documentation of Compliance Statements. Requires
that a physician document all the necessary case compliance/
attestation statements that make a record billable no more than
120 minutes after the end of anesthesia care. This documentation
is part of the normal clinical documentation work flow.
BASELINE DATA COLLECTION
After designing the three metrics, in an eight-month period
(January 1, 2008–August 31, 2008), we performed a baseline
set of measurements to validate the metrics. Our goal was to be
able to set a performance threshold for passing each metric that
would distinguish acceptable from unacceptable performance
while taking into consideration any limitations or artifacts contained
within the electronic AIMS database. The committee ultimately
decided to set the threshold for passing each metric at
a level that encompassed 95% of all physicians in the baseline
dataset—that is, 95% of physicians met the metric.
Of the 149 anesthesiologists in the department, 128 (86%)
were subject to the metrics. The remaining 18 anesthesiologists
(9 pain physicians, 4 critical care physicians, and 5 physicians
February 2012 Volume 38 Number 2
Copyright 2012 © The Joint Commission
who work exclusively in our preoperative evaluation clinic) were
not subject to the metrics.
Electronic Database Queries. Using SQL Query Analyzer
(Microsoft, Redmond, Washington), we developed a set of electronic
database queries to extract electronic data from our AIMS.
For each case during the baseline data collection period, the
query returned the unique case identifier, date of service, physician
identification, operating room (OR), type of anesthesia
(general, monitored anesthesia care, or regional), and the specific
variables relevant to each of the three metrics—as now described.
■ BP Monitoring. To perform the BP monitoring query, the
following metric specific variables were obtained:
–Start of Anesthesia Care Time (manually documented)
–Anesthesia Induction Time (manually documented)
–First Inhalational Agent Name (automatically recorded from
gas analyzer)
–First Inhalational Agent Value (automatically recorded from
gas analyzer)
–First BP Measurement Time (automatically recorded from
physiologic monitor)
–First Systolic BP Value (automatically recorded from physiologic
monitor)
–First Diastolic BP Value (automatically recorded from physiologic
monitor)
The query compared the time stamp of the first BP recorded
in the chart to the time stamp associated with the anesthesia induction
time (the earlier of either the manually documented
anesthesia induction time or the time of first inhalational
agent).* The logic also excluded nonphysiologic BP values (from
zeroed but disconnected arterial lines) from consideration. We
chose BP measurement prior to induction as a quality measure
because it is frequently not adhered to. Beyond our own collective
clinical experience, the authors’ research into the documentation
of BP during anesthesia has shown significant evidence
that BP monitoring does not always meet our expectations and
that progress is needed to enhance patient care. 11 We therefore
believe that standard practices can, in many cases, make for ideal
metrics of patient care—that is, how often do we adhere to the
standard. Although we hope that standard practice is reflected in
routine care, we know that it sometimes does not. For example,
administering on-time antibiotics prior to surgical incision is a
standard operating procedure that is not always adhered to. 12,13
Thus, insofar as this task—or BP monitoring, for that
* For the purposes of our query we used the following thresholds: Isoflurane > 0.1
minimum alveolar concentration (MAC); sevoflurane > 0.2 MAC; desflurane > 0.5
MAC.
75

76
The Joint Commission Journal on Quality and Patient Safety
matter—does not actually occur in 100% of cases, it would appear
to be a valid quality measure with relevance to physician
performance and patient care.
■ End Tidal CO 2 Monitoring. The primary purpose of the
end tidal CO 2 query, was to determine whether or not it was
measured during a given general anesthesia case. Whenever it is
measured, the AIMS creates a unique database entry every 60
seconds, which includes a number of data elements (for example,
time stamp, value, data source). Because most cases contain
hundreds of end tidal CO 2measurements, we programmed our
SQL query to simply count the number of measurements per
case and associate that count with a single unique case identifier.
The ASA national standard for end tidal CO 2 monitoring is
that “continual monitoring for the presence of expired carbon
dioxide shall be performed unless invalidated by the nature of
the patient, procedure or equipment.” 10 However, there are circumstances
in which CO 2 monitoring is not applied at all when
patients undergo general anesthesia (particularly when a MAC
case is converted to a general anesthetic). We therefore targeted
use of end tidal CO 2 monitoring at any point during a case as
our first version of this metric, but we expect that a future version
might evaluate the frequency and duration of monitoring.
■ Timely Documentation of Compliance Statements. To
create the data needed for the query regarding timely documentation
of compliance, the “End of Anesthesia Care Time”—a
particular time stamp recorded by the anesthetist at case conclusion—was
obtained from the AIMS database. In addition, all
time stamps associated with each of the required compliance
statements were also obtained and compared. We chose this metric
because we believe that timely documentation facilitates communication.
Downstream care providers need complete
documentation to make the best clinical decisions for their patients.
Given this metric’s direct effect on patient care, it is important
to assess as an aspect of the quality of care provided
patients—just as, say, outstanding dictation reports is also frequently
used as a compliance metric. The time frame of two
hours was selected because the availability of complete electronic
charts to providers of downstream care (for example, postanesthesia
care unit, intensive care unit [ICU], and general floors) is
an important goal for our department. Although billable aspects
of care might not necessarily reflect quality of care, they do influence
a hospital’s ability to provide care. In addition, it is a
growing concern that a caregiver’s economic performance, which
is often influenced by billing metrics, is in fact a valid criteria to
determine credentialing or appointment of staff. 14
It is important that criteria reflect expectation of failure rates.
Any criterion that has a near-perfect passing rate might be a poor
measure of performance—and might not contribute to a potential
for quality improvement. 15
CASE EXCLUSIONS
Several case types were excluded from the baseline assessment.
Because of clinical practice patterns, pediatric cases were excluded
from the BP metric, as were all cases in which patients
were noted to arrive in the OR already intubated—such patients
are almost universally transported with sedation/general anesthesia
while on a transport monitor. Cases in which the anesthetic
delivered did not include a significant inhalational agent
concentration (see the footnote on page 75) were also excluded.
Therefore, cases in which total intravenous anesthesia were excluded
from capture. Finally, we elected to exclude cases in which
a transfer of care occurred, so as to not penalize physicians who
had performed a portion of the case but transferred the case to
another physician who failed to document compliance appropriately.
As such, a separate SQL query was written to find and
demarcate cases in which a transfer of care occurred.
BASELINE DATA ANALYSIS
Data collected by the SQL query were analyzed using a
spreadsheet program. Cases were determined to either pass or
fail each of the three metrics on the basis of our predefined standards.
For each of the cases, which were counted on a per-physician
basis, passing/nonpassing percentages were calculated. If an
individual physician performed fewer than 60 cases during the
assessment, he or she was removed from the analysis because of
low case volume and underwent a different evaluative process. A
summary was then created, ranking each physician by his or her
percentage of passing cases. As established by the OPPE credentialing
committee, the bottom 5% of the group was flagged as
“Not Passing the Metric” for each of the three parameters.
ONGOING PHYSICIAN PERFORMANCE REPORTING
In February 2009, after establishing passing thresholds for
each of our three metrics, we provided the information on the
metrics to the department’s staff members and began our ongoing
evaluation process. No major educational intervention was
performed. Three months of data for each physician were then
evaluated according to the three metrics, and a confidential
report was provided to each physician.
To present individual reports to clinicians and use our administrative
staff efficiently, an automated data reporting system
was implemented using a spreadsheet program with macros. Two
types of confidential reports were produced—(1) final reports
for physicians being credentialed within the given three-month
February 2012 Volume 38 Number 2
Copyright 2012 © The Joint Commission

The Joint Commission Journal on Quality and Patient Safety
Sample Ongoing Professional Practice Evaluation (OPPE) Report
Figure 1. A simple summary description of physician performance appears in the physician performance component of the sample report, below the summary data
tables for each metric.
period and (2) interim reports for all others. The reports were
then disseminated to the appropriate parties for review.
Each report includes a listing for each metric of the total
number of compliant cases, the total number of noncompliant
cases, and a comparison by percentage to the baseline departmental
evaluation. A summary statement indicates whether the
physician’s performance was within the group representing 95%
of all department physicians. Noncompliant cases are listed by
medical record number and case date at the bottom of the panel
so that providers and reviewers can then examine individual
cases.
Reports
INDIVIDUAL CONFIDENTIAL REPORTS
The results of each metric measurement are continuously compiled
for every attending clinician who practices in a clinical environment
where the AIMS records clinical data and generates
the anesthesia record. Individualized confidential reports include
the total number of compliant and noncompliant cases, as well
as the percentage of passing scores.
A simple summary description of physician performance appears
in the physician performance component of the sample report,
below the summary data tables for each metric, as shown
in Figure 1 (above). The presentation of aggregate perfor mance
data for the entire department is intended to provide the individual
clinician an opportunity to perform a comprehensive selfassessment
against benchmark performance.
COMPARISON OF INDIVIDUALS WITH THE GROUP
Graphs—one graph for each of the three metrics—were also
created to plot individuals against the group. These graphs,
which were not included in the confidential individual reports,
helped to show the overall trend of each metric. These data are
intended to allow those managing the credentialing process to
evaluate metrics over time and replace and/or modify them as
necessary. A sample summary plot for Timely Documentation of
Compliance Statements metric is shown in Figure 2 (page 78).
BASELINE DATA FOR THE THREE METRICS
The baseline data for the three metrics is shown in Table 2
(page 79), along with the average compliance scores for 2010
for comparison.
Discussion
We have created a novel, automated, and continuous reporting
system for physician credentialing, which uses our existing clinical
information system infrastructure, in an effort to address
The Joint Commission’s OPPE requirements for quantitative
metrics. This system is automated, continuous, objective, and
relatively inexpensive to implement. Its basic framework can be
refined and developed easily. The system can therefore serve as a
key element of a comprehensive clinical performance evaluation
that measures both technical and generalizable clinical skill sets.
We avoided any metrics (such as reintubation rates, or unplanned
ICU admissions) that might penalize practitioners in
high-risk specialties or would require risk adjustment for case
mix.
The system requires no additional effort on behalf of the clinician
being scrutinized and reduces the introduction of evaluation
artifact in situations in which the presence of an observer
February 2012 Volume 38 Number 2
Copyright 2012 © The Joint Commission
77

78
The Joint Commission Journal on Quality and Patient Safety
or a simulated patient scenario modifies behavior. It also has
minimal costs to install and maintain and is unbiased in its measurement
of clinical performance. It is not intended to provide a
complete system for measuring competence but rather to serve
as a first-round warning mechanism and metric scoring tool to
identify problems and potential performance noncompliance issues.
Numerous problems can arise in the process of delivering
anesthesia care (or any clinical practice), such as failure to perform
basic patient monitoring in a fashion that ensures the safety
of the patient. Continuous and transparent systems such as ours
enable physicians to self-assess their performance and make adjustments
to correct issues before engaging in the credentialing
process.
Although the process of credentialing professionals in health
care is not new, 16–18 there are few in-depth analyses of physician
performance evaluation, and most are focused on overtly technical
skills; some address generalized professional skill sets. Generalizable
skills and specialized knowledge are both important to
clinical care and should be included in any competence metric. 19
Surgery and endoscopy practices place a high value on quantifying
technical skills. 20–22 In anesthesia, it has been suggested that
technical clinical performance can also be readily evaluated by
simulation 23 or control chart methodology. 24
The development of our credentialing system reflects similar
work reported by others. 25 Schartel and Metro suggest that evaluation
should take place close to the time of the clinical encounter
and should be intended to not only correct mistakes but
also continuously improve performance. 26 Fried and Feldman
state that technical performance measurements should be objective
and practical for a clinical specialty. 27 Hill argues for standardized
credentialing, 28 but we believe that it is nearly
impossible to standardize the clinical behavior of an entire profession.
Credentialing parameters should be specific enough to
enhance patient care by resolving issues within a specialty. 29 In
any case, because of uneven adoption of technology—for example,
clinical data coding is not universal—it would be unrealistic
to set standards beyond the capacity of smaller hospitals. 30
The purpose of credentialing is to improve patient care, and thus
should be dictated by each specialty’s organization and the health
care institutions themselves.
It should be noted that the purpose of our work was to design
a credentialing process suitable to our own department, with its
particular work flow and clinical practice, rather than for all
medical specialties or even anesthesia as a whole. These metrics
are intended to apply to anesthesiologists in clinical practice in
the OR—who constitute the vast majority of hospital-based
practicing anesthesiologists—and will not apply to all anesthesiologists,
such as pain and ICU physicians. However, we believe
that our results as presented have broad applicability to fields in
which significant structured clinical and compliance documentation
are important components of clinical practice.
Other fields of medicine, particularly ICUs, have used a similar
methodology to enact a system of defining and measuring
metrics of patient care and physician performance. For example,
Wahl et al., without incurring the need for additional personnel,
used a computerized system to collect data on ICU core
mea sures—glucose management, head of bed angle, prophylaxis,
and ventilator weaning. 31
MIDCOURSE CORRECTIONS AND NEXT STEPS
We are in the process of developing additional metrics, coupled
with a determination of which metrics to add or remove on
the basis of their performance over time. Potential future metrics
may include those based on the total case duration or “time-inflight,”
as opposed to what we have initially used, that is; casebased
metrics (items that occur only once per case). This
adjustment reflects the fact that some of our clinicians (for example,
members of our cardiac group) perform a small number
of long cases, as opposed to the majority of clinicians, who administer
a moderate number of relatively short-duration anesthetics.
The long-term goal of the credentialing system is to
develop a comprehensive group of metrics that best represents
the scope of clinical OR anesthesiology.
February 2012 Volume 38 Number 2
Sample Summary Plot
Figure 2. This sample summary plot for the Timely Documentation of Compliance
Statements metric displays the percentage of passing cases. The mean
baseline passing values (horizontal line) are indicated, as are the failure cutoff
point (vertical line), for just over 100 physicians.
Copyright 2012 © The Joint Commission

The Joint Commission Journal on Quality and Patient Safety
Group Mean Baseline Group Mean Current
No. of Physicians Performance Performance
Metric Evaluated at Baseline (Jan 1, 2008–Mar 31, 2008) (Jan 1, 2010–Dec 31, 2010)
End Tidal CO 2 Monitoring 90 98.8% 99.2%
BP Prior to Induction 82 92.0% 96.2%
Compliance Statements within 120 minutes 103 97.9% 99.2%
* BP, blood pressure.
Whereas we started with reporting individual provider performance
at quarterly intervals, since June 2009 we transferred
to a monthly reporting system. We have taken several steps to
improve our credentialing system as each round of scores are reviewed
and assessed. The passing mark of 95% was not assigned
to declare those outside the bounds as having failed, but rather
as being “of interest” to determine why this individual has a different
clinical practice. It was determined that review by the chair
or another appropriate senior member was warranted (with direct
observation or case review) to ensure that the practice was
in fact safe. To minimize direct observation—and any associated
new bias, since January 2010 we started implementing quarterly
metrics reviews to allow clinicians to self-adjust their practice to
fit the guidelines. Accordingly, if the departmental member still
falls below the 95% passing rate for two thirds of the metrics, he
or she will meet with the chair for a Focused Practice Perfor -
mance Evaluation (FFPE), as in the case, for example, in which
an attending anesthesiologist did not meet the metric on both
end tidal CO2 and timely documentation. The formal report delivered
to the staff member contained detailed information describing
the measures and how each of the cases did not meet
the metric. A senior member of the anesthesia department, in
reviewing the cases, was unable to find any nonclinical issue that
might have contributed to the results, and determined that the
staff member did not perform according to standard operating
protocols–resulting in the staff member’s being flagged for closer
(monthly) observation. The senior member met with the staff
member, and discussed how the staff member could improve
compliance with the measures. Closer review was sustained until
scores improved. J
This work was supported by 5T32GM007592 from the National Institute of Health
and the Massachusetts General Hospital Department of Anesthesia, Critical Care
and Pain Medicine.
Table 2. Summary Results*
References
1. The Joint Commission: Revised credentialing and privileging standards. This
Month at the Joint Commission, State Hospital Association Edition, Jun 2006.
Accessed Dec 14, 2011. http://www.njha.com/qualityinstitute/pdf/
614200630015PM29.pdf.
2. The Joint Commission: Telephone Conference Call Transcript. Audio Conference
with Joint Commission President Dennis S. O’Leary, M.D: Joint Commission
Credentialing and Privileging Standard (online; no longer available).
3. The Joint Commission. 2012 Comprehensive Accreditation Manual for Hospitals:
The Official Handbook. Oak Brook, IL: Joint Commission Resources,
2011.
4. Freedman S. How 2007 Joint Commission standards expand hospital peer
review. Patient Safety & Quality Healthcare. 2007;4(5):14–16.
5. Donnelly LF, Strife JL. Performance-based assessment of radiology faculty: A
practical plan to promote improvement and meet JCAHO standards. AJR Am
J Roentgenol. 2005;184(5):1398–1401.
6. Overeem K, et al. Doctor performance assessment in daily practise: Does it
help doctors or not? A systematic review. Med Educ. 2007;41(11):1039–1049.
February 2012 Volume 38 Number 2
Copyright 2012 © The Joint Commission
Jesse M. Ehrenfeld, MD, MPH, formerly Director, Anesthesia Informatics
Fellowship, Department of Anesthesia, Critical Care and Pain
Medicine, Massachusetts General Hospital, Boston, and Assistant
Professor of Anesthesia, Harvard Medical School, Boston, is Director,
Perioperative Data Systems Research (PDSR); Assistant Professor
of Anesthesiology, Division of Multispecialty Adult
Anesthesiology; Assistant Professor of Biomedical Informatics; Director,
Center for Evidence Based Anesthesia; and Medical Director,
Perioperative Quality, Vanderbilt University School of Medicine,
Nashville, Tennessee. Justin P. Henneman, MS, formerly Research
Assistant, Department of Anesthesia Critical Care and Pain Medicine,
Massachusetts General Hospital, is a Medical Student, North
Shore LIJ School of Medicine, Hofstra University, Hempstead, New
York. Robert A. Peterfreund, MD, PhD, is Chair, Quality Assurance
Committee, and Anesthetist, Department of Anesthesia, Critical Care
and Pain Medicine, Massachusetts General Hospital; and Associate
Professor of Anesthesia, Harvard Medical School. Tyler D. Sheehan,
BS, and Feng Xue, MS, are Senior Financial Analysts and
Stephen Spring, BA, is Administrative Director of Finance and Technology,
Department of Anesthesia, Critical Care and Pain Medicine.
Warren S. Sandberg, MD, PhD, formerly Associate Professor of
Anesthesia, Harvard Medical School, is Chairman, Department of
Anesthesiology, and Professor of Anesthesiology, Division of Multispecialty
Adult Anesthesiology, Vanderbilt University School of Medicine.
Please address correspondence and requests for reprints to
Jesse M. Ehrenfeld, jesse.ehrenfeld@vanderbilt.edu.
79

80
The Joint Commission Journal on Quality and Patient Safety
7. Ary D, et al. Introduction to Research in Education, 8th ed. Belmont, CA:
Wadsworth, 2009.
8. Rose G, Barker DJ. Epidemiology for the uninitiated. Observer variation.
Br Med J. 1978 Oct 7;2(6143):1006–1007.
9. Zegiob LE, Arnold, S., Forehand, R. An examination of observer effects in
parent-child interactions. Child Development. 1975;46(2):509–512.
10. American Society of Anesthesiologists. Standards for Basic Anesthetic Monitoring.
Oct 20, 2010. Accessed Dec 14, 2011. http://www.asahq.org/
For-Members/~/media/For%20Members/documents/Standards%20
Guidelines%20Stmts/Basic%20Anesthetic%20Monitoring%202011.ashx.
11. Ehrenfeld JM, et al. Automatic notifications mediated by anesthesia information
management systems reduce the frequency of prolonged gaps in blood
pressure documentation. Anesth Analg. 2011;113(2):346–363.
12. Classen DC, et al. The timing of prophylactic administration of antibiotics
and the risk of surgical-wound infection. N Engl J Med. 1992 Jan
30;326:281–286.
13. Steinberg JP, et al. Timing of antimicrobial prophylaxis and the risk of surgical
site infections: Results from the Trial to Reduce Antimicrobial Prophylaxis
Errors. Ann Surg. 2009;250(1):10–16.
14. Riley DW. Legal forum. Economic credentialing of physicians: New criteria
and evaluation of physicians. Spine. 1996;21(1):141–146.
15. Fung V, et al. Meaningful variation in performance: A systematic literature
review. Med Care. 2010;48(2):140–148.
16. Starr P. The Social Transformation of American Medicine. New York: Basic
Books, 1982.
17. Wilson FC. Credentialing in medicine. Ann Thorac Surg. 1993;55(5):
1345–1348.
18. Hernandez AM. Trends in health care practitioner credentialing. J Health
Care Finance. 1998;24(3):66–70.
19. Wimmers PF, Fung CC. The impact of case specificity and generalisable
skills on clinical performance: A correlated traits-correlated methods approach.
Med Educ. 2008;42(6):580–588.
20. Sharma VK, Coppola AG Jr, Raufman JP. A survey of credentialing practices
of gastrointestinal endoscopy centers in the United States. J Clin Gastroenterol.
2005;39(6):501–507.
21. Dickinson I, et al. Guide to the assessment of competence and performance
in practising surgeons. ANZ J Surg. 2009;79(3):198–204.
22. Tedesco MM, et al. Simulation-based endovascular skills assessment: The future
of credentialing? J Vasc Surg. 2008;47(5):1008–1011; discussion 1014.
23. Zausig YA, et al. Simulation as an additional tool for investigating the performance
of standard operating procedures in anaesthesia. Br J Anaesth.
2007;99(5):673–678.
24. Runcie CJ. Assessing the performance of a consultant anaesthetist by control
chart methodology. Anaesthesia. 2009;64(3):293–296.
25. Bozadjian EM. Quality improvement, credentialing, and competency. Int
Anesthesiol Clin. 1999;37(4):47–57.
26. Schartel SA, Metro DG. Evaluation: Measuring performance, ensuring
competence, achieving long-term excellence. Anesthesiology. 2010;112(3):
519–520.
27. Fried GM, Feldman LS. Objective assessment of technical performance.
World J Surg. 2008;32(2):156–160.
28. Hill D. The case for standards. Uniform physician credentialing process can
save physician practices and hospitals time, money and frustration. Health
Manag Technol. 2004;25(10):48–49.
29. Hanson J. Measuring physician performance. Building an effective physician
performance system starts with transparency. Health Manag Technol.
2008;29(4):10, 2–3.
30. Rebagliati GS. Physician-specific OPPE models in academic medical centers.
Ann Surg. 2009;249(4):700–701.
31. Wahl WL, et al. Use of computerized ICU documentation to capture ICU
core measures. Surgery. 2006;140(4):684–689; discussion 690.
February 2012 Volume 38 Number 2
Copyright 2012 © The Joint Commission