Rating Of Coaching Success Factors Version4-May 2010
Rating Of Coaching Success Factors Version4-May 2010
Rating Of Coaching Success Factors Version4-May 2010
You also want an ePaper? Increase the reach of your titles
YUMPU automatically turns print PDFs into web optimized ePapers that Google loves.
The <strong>Rating</strong> of Eight <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />
- Observation Manual -<br />
Siegfried Greif<br />
Frank Schmidt<br />
André Thamm<br />
Version 4<br />
<strong>May</strong> <strong>2010</strong><br />
We are very grateful to Jennifer Boeun Lee, MA (Harvard), Research Assistant, Evidence Based Psychology,<br />
Cambridge (MA) for her editorial support of the English translation.
The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />
1 Introduction: Hypothetical success factors that predict<br />
coaching outcomes<br />
Can hypothesized success factors, determined via behavioral observation of coaching sessions, predict<br />
coaching outcomes? Behrendt (2004) first studied possible success factors in coaching by observing<br />
the behavior of coaches in coaching sessions. Using success factors and methods of behavior observation<br />
adopted from the psychotherapeutic outcome research of Grawe, Donati and Bernauer (1994),<br />
Behrendt recorded a total of 40 coaching sessions with 8 managers (5 sessions each) and had observers<br />
rate the behavior of the coaches in 10-minute intervals according to the Cubus-Analysis scales (Regli<br />
& Schmalenbach, 1994)). Significant correlations were discovered between resource activation, one<br />
of the success factors, and the satisfaction of the managers who were engaged in coaching (r = 0.56).<br />
Correlations were also found between resource activation and goal attainment rated by employees<br />
after the coached managers conducted performance interviews (r = 0.43). (1994). A summary of the<br />
success factors will be given in the following section.<br />
Behrend’s work encouraged us to first design observable success factors that relate more closely to the<br />
field of coaching and our integrative theory of result-oriented coaching (Greif, 2008), and then to begin<br />
similar coaching behavior observation studies (Schmidt & Thamm, 2008). We felt the work would<br />
be invaluable, as it would: 1) validate observable success factors that can predict coaching outcomes,<br />
2) give coaching students feedback during their coaching education, 3) assess the work of professional<br />
coaches and aide in their development, and 4) improve the quality and outcomes of professional<br />
coaching.<br />
We will begin with an overview of the success factors and then move on to the results of the initial<br />
pilot studies (English publications are in preparation). Finally, eight basic success factors in coaching<br />
will be defined together with the rating scales and their behavioral anchors.<br />
1.1 General change principles of psychotherapy<br />
In their meta-analysis of success factors in psychotherapy outcome research, Grawe et al. (1994) studied<br />
general change principles during the therapeutic process that predict the outcome of therapy independently<br />
of the therapeutic schools. Gassman and Grawe (2006) differentiated between five change<br />
principles:<br />
(1) The therapeutic bond (warm, empathetic and congruent behavior of the therapist, and openness<br />
and trust of the patient, resulting in a trusting relationship between the therapist and patient);<br />
(2) Problem actualization (imagination of the problems and emotions experienced by the patient);<br />
(3) Resource activation (recognition of the strengths and abilities of the patient, as well as the level<br />
of support in the patient’s environment);<br />
(4) Motivational clarification (reflection on the motives and goals of the client, including awareness<br />
of behavioral consequences, clarification of the client’s conscious and unconscious goals<br />
as well as the environmental goals, and a continuous reflection on the relationships between<br />
these goals); and<br />
(5) Mastery (action-oriented coping, transfer into the practice of concrete measures, and support<br />
in reaching the goals).<br />
Therapeutic bond. Based on this research, Grawe (2000, 2006) developed an integrative neuropsychological<br />
theory of change principles that relate to successful and unsuccessful therapeutic interactions.<br />
Such predictors are called “success factors”. The findings showed that resource activation is the<br />
strongest predictor of therapy outcomes (Gassmann & Grawe, 2006); as a result, it forms the central<br />
change principle in Grawe’s theory. Similar to the solution-focused approaches in the fields of therapy,<br />
coaching (Greene & Grant, 2003; O`Connell & Palmer, 2007; Shazer, 1988), and Positive Psychology<br />
(Kauffmann, 2006; Kauffmann, Bonniwell & Silberman, <strong>2010</strong>), the focus of the intervention<br />
changes from deficit-oriented intervention to the activation of the intra- and interpersonal resources of<br />
the client (e.g., their intellectual abilities, specific skills, motivational goals, or level of support from<br />
family members) (Gassmann & Grawe, 2006, p. 2). Grawe (2006) theorized that the therapeutic bond<br />
Siegfried Greif, Frank Schmidt and André Thamm 2
The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />
– the classic variable describing an understanding and supporting relationship between therapist/coach<br />
and client (Rogers, 1980, p. 22 f.; Stober, 2006) – forms the basis of resource activation, and increases<br />
the chances of a positive therapeutic outcome. The client must trust in the therapist’s understanding,<br />
support, and expertise before being able to grow their own potential and develop hope. Once trust is<br />
developed, the client is able to recognize and use their strengths and abilities, as well as the support of<br />
other people or technical resources in their environment, to pursue goals.<br />
Problem actualization refers to the re-experiencing and intensively processing of emotions during the<br />
therapeutic process. Patients must learn that they are “more than the sum of their problems.” To be<br />
able to change, patients must face emotions associated with their problems that have been previously<br />
avoided (Gassmann & Grawe, 2006, p. 2). The processes of transference and countertransference are<br />
examples in psychoanalysis and other psychodynamic approaches of problem actualization. Exposure<br />
therapy, an intervention practiced in behavior therapy, is another common example of problem activation.<br />
Motivational clarification includes any intervention that supports the client in reflecting their personal<br />
or intrinsic motives and goals. It also includes actions of the therapist that help the client to clarify<br />
external expectations and relationships between goals and consequences.<br />
Mastery focuses on the client’s ability to cope with the problem successfully. Examples of mastery<br />
include the actions necessary to solve the problem, as well as the ability to successfully perform these<br />
actions and manage the problem.<br />
Problem actualization and resource activation complement one another. While the actualization of the<br />
problem helps the client to clarify what to change, resource activation focuses on how this is possible<br />
(Grawe, 2000, p.99). These two success factors are central to the therapy and have largely to do with<br />
the changes clients undergo while in therapy (Gassmann & Grawe, 2006, p. 2). Motivational clarification<br />
and mastery are complementary as well. Clients take action that can be potentially detrimental to<br />
their future without motivational clarification.<br />
1.2 <strong>Success</strong> factors of coaching<br />
The observation study of Behrendt (2004)<br />
The success factors found by Grawe et al. (1994) are very general and applicable to other types of<br />
counseling interventions. Grawe’s success factors are also particularly applicable to the field of coaching.<br />
For his observation study, Behrendt (2004) adapted Cubus-Analysis, a rating instrument created<br />
by Grawe, Regli and Schmalenbach (1994). In his original manual, Behrendt only made a few changes<br />
to the rating instrument such as replacing the words “therapist” with “coach” and “therapy” with<br />
“coaching.” The results of Behrendt’s study show that only resource activation correlates significantly<br />
with outcome criteria. Upon closer inspection of the scale definitions in the manual, we decided that<br />
the success factors needed to be redfined and redesigned to be more applicable to the field of coaching.<br />
A significant issue with Behrendt’s early rating instrument is that the client resources and coach empathy<br />
ratings were combined into one common “resources” score; in later publications, Grawe and his<br />
colleagues distinguish the therapeutic relationship from resource activation (cf. Gassmann & Grawe,<br />
2006). We reanalyzed the data published by Behrendt (2004) and found (Greif, 2008, p. 267 f.) that<br />
the coach rating scales referring to exhibiting esteem/ appreciation and emotional support (empathy<br />
factor) towards the client consistently correlates higher with client satisfaction ratings (r= 0.51 - 0.55)<br />
than with client personal resource activation ratings (r= 0.39 – 0.46). Reciprocally, in the performance<br />
interview by the interview partner of the client, the correlations between client resource activation<br />
ratings and behavior-oriented ratings of goal attainment are higher (r= 0.63) than those between the<br />
empathy and the goal-attainment ratings (r= 0.43 – 0.44). According to our theoretical assumptions<br />
(Greif, 2008), empathy and emotional support shown by the coach result primarily in ratings of client<br />
satisfaction and similar subjective evaluations of the clients. In contrast, client resource activation is<br />
assumed to have a stronger influence on criteria related to behavior changes.<br />
General and specific factors<br />
Following a constructionist position, “success of coaching” is a subjective social construct. The definition<br />
of success depends on the perspective of the persons or institution that perform the evaluation, the<br />
Siegfried Greif, Frank Schmidt and André Thamm 3
The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />
quantitative or qualitative criteria selected, and their weighting. Different persons can evaluate the<br />
same coaching case completely differently. Client satisfaction and goal attainment are general evaluation<br />
criteria applied by both professional coaches and their clients. However, these criteria are not<br />
always reliable due to the unique characteristics (e.g., reasons, problems, goals and attributes) of each<br />
of the interacting persons. Therefore, coaching cases are often evaluated by very specific evaluation<br />
criteria. Criterion also have the potential of being inapplicable to other clients. An example of this<br />
would be a client who asks his coach to support him in creating a literature circle as a kind of balance<br />
to his stress at work. This is a very specific goal and as a result the methods used by the coach and<br />
client, as well as the process leading to a successful outcome, will also be very specific.<br />
Subjective theories of success and failure of coaching<br />
Through semi-structured expert interviews, we analyzed the subjective theories of coaches and their<br />
clients on success and success factors of concrete coaching cases (Greif, 2008, p. 239 ff.; in preparation;<br />
Schmidt & Thamm, 2008; Zöller, 2008). The Change Explorer, an instrument that uses structure<br />
analysis techniques with cards to visualize the grounded individual theories of coaching success factors<br />
and outcomes, was used in the analysis. Our findings support the theory that both clients and<br />
coaches often apply the general evaluation criteria of client satisfaction and goal attainment in their<br />
coaching sessions, and that the agreement between their ratings is high. Also, both ascribe positive<br />
changes primarily to intervention methods used in coaching and to the initiative and problem solving<br />
competences of the clients. Differences between coach and client were found in the specific outcome<br />
criteria, especially in the explanation of specific negative results. For example, coaches attributed<br />
negative results to the characteristics of the client more frequently (e.g., low change readiness/motivation<br />
and low self-reflexivity), while clients attributed them to low coach competencies,<br />
supporting expectation of self-serving bias (Miller & Ross, 1975). In contrast, positive results are<br />
credited more often to their own competencies and abilities (Zöller, 2008). Another significant finding<br />
is that the quality of the relationship between coach and client, as well as the quality of the process<br />
(e.g.,“I felt that it was a good process.”) have been mentioned more frequently by the coach. These<br />
qualities can be perceived as success factors or outcomes, depending on the case. This may indicate<br />
that both trusting relationships and good coaching processes are causes and effects that must be analyzed<br />
systemically by circular models.<br />
Limitations of behavior ratings<br />
The focus of this paper is the observation of success factors that can be consistently rated by trained<br />
observers. It is nearly impossible to reach rating consistency among observers when rating scales are<br />
complex and defined behaviors that are nearly indistinguishable. The success factors presented below,<br />
therefore, are by no means exhaustive. For a more comprehensive study, we suggest combining the<br />
rating scales used with interviews and questionnaire scales to assess both qualitative and quantitative<br />
data (e.g., applying our Change Explorer method) (Greif, in prep.). Additionally, there is the possibility<br />
of analyzing the transcripts of the interviews, or observing interactions in coaching sessions for<br />
case studies using qualitative methods.<br />
The success of coaching may always remain a mystery, at least in part, and may never be fully explained<br />
by scientifically defined factors or qualitative analyses. Some colleagues, scientists and practitioners<br />
argue that the success of coaching cannot be studied. But mysteries have always challenged not<br />
only scientists, but also curious practitioners who want to know and understand as far as possible how<br />
and why coaching works. If we reject evaluating coaching processes and outcomes by scientifically<br />
controlled qualitative and/or quantitative methods, we cannot argue against those who do not accept<br />
any kind of scientific evaluation and discussion of their performance. Process research is of prime<br />
importance for the future of coaching, as it will increase both the efficacy of coaching and the opportunity<br />
to support coaching clients.<br />
A list of eight observable success factors<br />
Inspired by Grawe and his scholars we began our research by defining observable behavior patterns of<br />
the coaches. We defined the factors and ratings that are expected to show a positive relationship to<br />
coaching outcomes. We have also begun developing assumptions and analyzig client behavior and<br />
interactions. In the future we plan to study individual and specific processes using theory-based case<br />
studies and qualitative methods.<br />
Siegfried Greif, Frank Schmidt and André Thamm 4
The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />
In our first observation study (Schmidt & Thamm, 2008) we distinguish between eight elementary<br />
success factors that describe observable patterns of coaching behavior:<br />
(1) Esteem and emotional support;<br />
(2) Result-oriented problem-reflection;<br />
(3) Result-oriented self-reflection;<br />
(4) Affect reflection and calibration;<br />
(5) Clarification of goals;<br />
(6) Activation of client resources;<br />
(7) Support of transfer into practice; and<br />
(8) Evaluation of process.<br />
(1) Esteem and emotional support (ES)<br />
Esteem and the activation of individual or external resources are separated in our observation system.<br />
Since our focus is on coach behavior, ES refers to the coach’s observable verbal and non-verbal expression<br />
of esteem and emotional support towards the client.<br />
(2) Problem-reflection (PR)<br />
Scales were designed in reference to the facilitation of result-oriented problem- and self-reflection, as<br />
these are central to our theory (Greif, 2008).<br />
According to Dörner (1979) a problem is typically defined by three components: 1) a given, undesired<br />
starting state (as-is state) or situation (S), 2) a desired target state or goal (G), and 3) an unknown<br />
means of how to reach the goal (M). A simple example of this is a manager who suffers from intense<br />
stress at work (S) and wants to reduce the stress (G), but does not know how (M). Dörner’s definition<br />
is broad and embraces all imaginable situations or states that a coaching client would want to change.<br />
Clients seek professional coaching for a number of reasons: customer losses, economic hardships,<br />
implementation of new technologies, organizational changes, stress, time management, performance<br />
improvement, mastery of management techniques, conflict management, and understanding of individual<br />
and organizational behavior (e.g., “Why did the employee react aggressively to the customer?”,<br />
“What caused the team conflict?”, “Why were the organizational changes ineffective?”). Excluded<br />
from this group of reflections is self-reflection (defined below).<br />
Result-oriented problem-reflection occurs when the coach examines the problem with the coachee and<br />
then develops new insights, clarifies how components of the problem are related (e.g., key players or<br />
key interactions), and discusses possible consequences. These reflections are considered resultoriented<br />
when the client derives conclusions for future reflections or actions. The classic technique<br />
used by coaches to encourage reflection is to ask open questions. Other techniques include moderation<br />
cards to describe the problem, and role play to simulate interactions.<br />
(3) Self-reflection (SR)<br />
In interviews conducted with 89 coaches and 74 clients, Mäthner, Jansen and Bachmann (2005) found<br />
that the majority of clients (62%) and coaches (71%) felt there was an increase in self-reflection as a<br />
result of the coaching. Examples of this include answers such as, “More awareness of myself and my<br />
behavior,” and, “Clarification of my personal roles and priorities.” Therefore, they concluded that the<br />
facilitation of self-reflection is one of the major outcomes of coaching. Other evidence of this increase<br />
in self-reflection includes reflection reportedly on personal values, particular interests, abilities and<br />
competencies, strengths and weaknesses, typical emotional reactions, habitual behavior and a comparison<br />
of the real and ideal self-concept. Open self-reflection requires strict confidentiality and a<br />
counselor who follows-up with the interests and goals of the client. In our coaching theory (Greif,<br />
2008), facilitation of intensive self-reflection is one of the core functions and competencies of coaching.<br />
It distinguishes coaching from other types of counseling that are less invasive. In this respect,<br />
coaching can be compared to psychotherapy.<br />
Siegfried Greif, Frank Schmidt and André Thamm 5
The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />
To be precise, self-reflection is a special subgroup of problem-reflection with the only difference being<br />
that self-reflection focuses on aspects very close to the identity of the client. Our coaching theory<br />
integrates assumptions and findings from self-concept research and Kuhl’s (2001) neuropsychological<br />
motivation and personality theory, which argues that the experience of a person with the self is part of<br />
a special type of intuitive or implicit memory (the so-called “extension” memory).<br />
We suspect that the phases of self-reflection do not occur frequently in coaching sessions, and are not<br />
standard to coaching in general. This may be due to the difficulty of facilitating systematic selfreflection<br />
since it is rooted in the implicit memory, organized in an associative way, and based on nonverbal<br />
representations. Managers who tend to be more action-oriented are particularly skeptical about<br />
the usefulness of self-reflection; we have found them less likely to buy into the benefits of selfreflection,<br />
especially in cases where their habitual behavior causes them to act prematurely and without<br />
regard to the negative consequences of their behavior. Professional coaches, similar to psychotherapists,<br />
know how to engage their coachees in intensive and systematic self-reflection when useful.<br />
Typical methods used are circular questions and visualization, or imagination techniques. As with<br />
problem-reflection, we have a result-oriented focus for self-reflection. Some clients, especially with a<br />
high state of orientation (Kuhl, 2001), tend to ruminate with long, associative, and straining selfreflection<br />
without any results. In these situations, it is necessary for coaches to help clients learn how<br />
to stop negative self-reflection and to switch to reflection with concrete positive results. This is a significant<br />
concept of Positive Psychology (Kauffmann, 2006; Kauffmann et al., <strong>2010</strong>).<br />
(4) Affect reflection and calibration (AR)<br />
Affect reflection and calibration relate to the activation, re-experiencing, and changing of affect and<br />
emotions in coaching sessions by techniques used in Grawe’s (2006) theory of psychotherapy (2006).<br />
It is our assumption that systematic, intensive psychotherapeutic techniques are rarely applied to alter<br />
affect and emotions in standard business coaching. In coaching sessions, emotions usually function as<br />
a means of allowing the client “to let off steam” after an emotionally laden situation, or as a way for<br />
the client to describe and analyze their own emotions and reactions or those of the environment. The<br />
coach may train the client to manage negative affects and emotions in order to improve psychological<br />
well-being.<br />
Following Kuhl’s theory (2001), strong negative and positive emotions restrict access to the self.<br />
Take, for example, a manager whose strong feelings of aggression, first experienced in a prior conflict<br />
situation, are reawakened during a coaching session. The manager is unable to reflect openly on his<br />
behavior and attitude that contributed to the conflict, as is fixated on his emotional state. Open access<br />
to the self will only be possible after the affect state has been reduced to a temperate level. This can<br />
easily be done by asking the manager to clearly describe his emotions. According to Kuhl, this exercise<br />
activates rational cognitions that are antagonistic to strong emotions. The typical observable reaction<br />
of the client is that their emotions are quickly subdued, and that they speak in a different tone.<br />
After this, it is easier for the client to explore and analyze the self. Other techniques, similar to behavior<br />
therapy interventions that result in the pacification of emotions (e.g., relaxation techniques) can be<br />
applied here. Affect reflection and calibration can be considered as a kind of mediating factor that<br />
gives greater access to the self and self-reflections.<br />
(5) Clarification of goals (CG)<br />
Goal clarification is one of the most important coaching tasks that was emphasized early in cognitive<br />
behavioral coaching theory and research (Grant, 2006). The definition of this factor is not identical to<br />
Grawe’s motivational clarification factor. It addresses more concrete definitions and explanations of<br />
the goals and intentions of the client in relation to the expectations of the social and organizational<br />
environment. The analysis of the personal motives in our system belongs to the self-reflection factor,<br />
as it is foundational to goal clarification that is congruent with the self.<br />
As Grant (2006, p. 160) ascertains, it is not always necessary or meaningful to have the client define<br />
goals very precisely (e.g., to follow the SMART criteria). It is adequate for the client, in a contemplative<br />
state, to set abstract or vague goals and to start to develop a broad vision even if unclear.<br />
(6) Activation of the client’s resources (RA)<br />
Siegfried Greif, Frank Schmidt and André Thamm 6
The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />
Resource activation is in direct reference to Grawe’s theory and findings (2006). Like Grawe, we distinguish<br />
between the internal resources of the client (motivational and personality traits, abilities,<br />
competencies, and potentials) and the external resources the client is able to activate (e.g., expert<br />
knowledge, consults, emotional support, concrete help by family and friends, or support from those in<br />
the occupational environment). In summary, we call these personal resources.<br />
(7) Support of transfer into practice (TP)<br />
In our first manual (Schmidt, Thamm & Greif, 2008), we introduced the rating of the transfer of the<br />
intended change into practice as a special, result-oriented rating of resource activation. The corresponding<br />
success factor defined by Grawe (2006) in his theory is called mastery. Mastery addresses:<br />
1) action-oriented coping with the problem and solution development, 2) transfer into the practice of<br />
concrete measures, and 3) support in reaching the goals of the client. We assumed that it is necessary<br />
to focus more quickly and systematically on mastery in shorter result-oriented coaching sessions than<br />
in psychotherapy. However, after reviewing two standard German handbooks of coaching tools by<br />
Rauen (2004; 2007), we found many techniques facilitating analyses/reflection or changes, and some<br />
techniques to evaluate the outcome of the sustained transfer, but not a single technique on how to support<br />
the transfer of the intended change into practice. In our theory and practice we focus more on<br />
procedures and tools for how to support transfer into practice.<br />
A systematic intervention method is a methodical reflection of and preparation for the possible barriers<br />
of transfer. After a trial run of the transfer is conducted, the client is prompted to reflect on the actions<br />
and reactions during the transfer, and if necessary, makes changes to the strategy used. A powerful<br />
tool used during the transfer is the shadowing of the client by the coach in the intended actions or behavior<br />
changes. The coach observes the client during the transfer and gives feedback afterward. Then<br />
the coach and client together analyze concrete transfer barriers and solutions.<br />
Our preference is for telephone shadowing, as it is a more economical tool. In telephone shadowing,<br />
the coach and client have a telephone appointment immediately following the situation where the client<br />
tries to perform the intended actions. From experience, shadowing is an essential benefit of the<br />
intervention. <strong>Of</strong>tentimes many of the client’s intended actions fail due to concrete individual and environmental<br />
barriers that are not predicted by the client or coach. When the client is unable to execute<br />
the intended actions and the trial run is unsuccessful, the client is much less motivated to try again. If a<br />
follow-up coaching meeting does not immediately follow the trial run, the client is more likely to reinterpret<br />
the experience in an unhealthy or ineffective way during the lag time, making it difficult to<br />
motivate the client for the next attempt. The reflection on and activation of resources directly after the<br />
trial coupled with support for the next trial, encourage the client to try again. Normally after an unsuccessful<br />
attempt, the second trial run is more successful. The learning of the client can be very demanding<br />
during this process, and if successful, the client can be successfully equipped with the ability necessary<br />
to overcome typical internal and external barriers without the support of the coach.<br />
The research (Kaufel, Scherer, Scherm & Sauer, 2006) coupled with our practical experience point to<br />
the support of transfer via shadowing to be a very powerful intervention. To further explore this, we<br />
have planned experimental control studies to test the assumption that this factor very strongly contributes<br />
to the behavioral outcome of coaching.<br />
(8) Feedback and evaluations in the process (FP)<br />
A basic principle of result-oriented coaching is to have the client receive useful results in each coaching<br />
session. Examples of useful results are concrete goal-oriented action plans and tasks that the client<br />
can put into practice, important insights that help the client to understand and overcome problems, and<br />
new and valuable knowledge. The more coaching sessions yield useful results, the more coaching, as a<br />
whole will be useful. It is therefore reasonable for coaches to ask their clients for feedback after sessions<br />
to make sure that the client is satisfied with the results. We define this is as a success factor since<br />
it enables the coach to change the method of coaching if the client is unsatisfied. If the client were to<br />
spontaneously give a positive evaluation of the coaching after many coaching sessions, we would also<br />
expect that this is a predictor of the overall positive success rating after the end of the coaching. Therefore,<br />
we rate the observable evaluations – both the coach’s asking for feedback and the client’s spontaneous<br />
positive evaluations.<br />
Siegfried Greif, Frank Schmidt and André Thamm 7
The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />
First results: Correlations of the success factors to outcome criteria<br />
We constructed the observation instrument described below. In our first study, trained observers rated<br />
the success factors of 44 videotapes of coaching sessions (first, middle, and last). The clients were<br />
business and law students, and the coaches were seven graduate psychology students trained with systematic<br />
coaching education and supervision (Schmidt & Thamm, 2008). The ratings were taken at 5minute<br />
intervals. Since some of the frequencies and variances of the quality ratings were low, not all<br />
observer reliabilities were satisfactory (see below for the details). Therefore, it was surprising to find<br />
high and significant correlations between the observed elementary success factors and outcome criteria<br />
that conform to our theoretical assumptions. A short summary of some of these is given in the following<br />
passages. Details of the observer reliabilities and correlations of the factors will be described in the<br />
observer manual.<br />
Significant correlations with outcome criteria:<br />
Non-verbal reinforcement correlates with an emotional clarity scale (0.74), and verbal empathy with<br />
scales assessing the client’s behavior reflection (0.69), goal reflection (0.67), and goal-attainment<br />
satisfaction (0.52). Facilitation of self-reflection is related to the reduction of helplessness (0.56). Resource<br />
activation correlates with goal satisfaction (0.49). As expected, support of transfer predicts<br />
behavior-oriented scale values of effective time management (0.55) and information management<br />
(0.46). But the correlations found between resource activation and information management (0.49),<br />
and reduction of the feeling of listlessness (r=0.50) are only tendentially significant (10% level).<br />
Since both the sample and the variance of the behavior observation scales of the coaches are small,<br />
only high correlations of 0.46 and above are statistically significant. For example, correlations between<br />
resource activation and information management (0.49) and reduction of the feeling of listlessness<br />
(0.50) are only tendentially significant (10% level). Below in the observation manual the correlations<br />
of all scales will be presented.<br />
Contrary to our expectations, non-verbal reinforcement correlates with negative affect (0.53) and facilitation<br />
of self-reflection correlates with demotivation (0.57). Since correlations do not show the<br />
causal direction of influences or time sequences, the first correlation may be a result of the coaches<br />
reinforcing reactions to the clients’ negative affect, and the second of the stimulation of self-reflection<br />
of clients after evidence of their demotivation. Analyses of the interactions between coaches and clients<br />
and their time sequence may help to clarify these results.<br />
Siegfried Greif, Frank Schmidt and André Thamm 8
The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />
Research perspectives<br />
It is promising that the correlations observed in our pilot study are higher than expected. But the sample<br />
of our first study is small and we must be careful not to generalize the results to similar samples of<br />
students. Behavior observation studies are very laborious, but replication of these studies is necessary.<br />
It would potentially be more useful to observe and analyze interactions of professional coaches and<br />
business clients.<br />
The observation of coaching sessions opens a new research field that will add to our understanding of<br />
the coaching process. Beyond using the rating scale of success factors for session observations, the<br />
narrative from each session, obtained through transcription, will prove interesting. Geißler (2009),<br />
based on the communication theory of the German philosopher Jürgen Habermas, analyzed frequencies<br />
of various fundamental types of communication acts in selected coaching cases (one of our cases<br />
was one of these). Examples of Geißler’s coach categories include: shows attention, asks, mirrors,<br />
questions/confronts/critizes, explains, gives feedback and prompts the client to act. Catogories describing<br />
the reactions of the client, initiated by the coach, include: focuses cognitively on a subject, describes<br />
concretely or reflects on the process, the causes or the meaning, evaluates and develops action<br />
intentions. Analysis of ten transcripts of coaching sessions (one of them rated as problematic) found<br />
that the most frequent communication act of the coach with a large range is to ask questions (between<br />
26 and 61% of the acts, and only 19% in the problematic coaching). The clients are frequently<br />
prompted to focus cognitively on a subject (about 24 to 52%, and 16% in the problematic coaching).<br />
Geißler’s study merely intended to describe typical communication acts, not success factors. Therefore,<br />
further information from coaching process research on factors predicting outcomes of coaching<br />
have to be obtained through studies via qualitative methods, quantitative methods (with a focus on<br />
scales), or a combination of both of these. In our future research, in addition to conducting research on<br />
the observation categories described below, we plan to conduct case-studies applying qualitative<br />
methodsand to analyze coaching narratives of highly succesful and not successful coachings (rated<br />
both by clients and coaches).<br />
2 Observation Manual<br />
2.1 Short definitions of the success factors and ratings<br />
The following manual gives the definitions of elementary hypothetical success factors. The list below<br />
gives an overview of all ratings. A description of the rating scales, examples of the different types of<br />
behavior, and the anchors belonging to the factors follow.<br />
ES: Esteem and emotional support<br />
ES1: The coach verbally shows esteem for the client.<br />
ES2: The coach non-verbally shows esteem for the client.<br />
ES3: The coach emotionally supports the client.<br />
PR: Problem-reflection<br />
PR1: The coach engages the client in result-oriented problem-reflection.<br />
PR1+: The coach guides the client to derive concrete solutions from the problem-reflection (only if a<br />
facilitation of problem-reflection is shown).<br />
SR: Self-reflection<br />
SR1: The coach engages the client in result-oriented reflection about the self (e.g., behavior, personal<br />
experiences, needs and motives, strengths and weaknesses, ideal and real self-image).<br />
SR1+: The coach guides the client to derive concrete solutions from the problem-reflection (only if a<br />
facilitation of problem-reflection is shown).<br />
AR: Affect reflection and calibration<br />
AR1: The coach engages the client in reflection about feelings.<br />
Siegfried Greif, Frank Schmidt and André Thamm 9
The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />
AR2: The coach supports pacification and calm and open thinking about the client’s feelings.<br />
CG: Clarification of goals<br />
CG1: The coach supports the client in clarifying goals and concretely defining them.<br />
RA: Resource activation<br />
RA: The coach encourages the client to explore and activate personal internal and external resources,<br />
which support goal attainment.<br />
TP: Support of transfer into practice<br />
TP: The coach supports the client to transfer the planned goal-oriented behavior and measures it into<br />
practice.<br />
FP: Feedback and evaluations in the process<br />
EP1: The coach asks the client to evaluate progress and satisfaction with the coaching, goal attainment,<br />
and/or cooperation.<br />
EP2: The client spontaneously offers feedback regarding the level of satisfaction with the session or<br />
the entire coaching engagement.<br />
2.2 Observation periods<br />
Observation periods are the periods of time where the observers are asked to stop the tape and rate the<br />
behavior in the sequence before stopping independently from rating other periods. The observers must<br />
stop the tape every five minutes and repeat the rating procedure until the end of the recording.<br />
2.3 <strong>Rating</strong> scales<br />
The attributes used to define the success factors are important for the observers to know. Following<br />
Flückinger’s behavior rating of resource activation in psychotherapy (Flückiger, Caspar, Holtforth &<br />
Willutzki, 2009; Flückiger & Holtforth, 2008), we rate the behavior of the coach by the following<br />
quality ratings:<br />
<strong>Rating</strong> Definition<br />
-1 Negative evaluation.<br />
0 Behavior not shown.<br />
1 Behavior is shown only to some extent; the quality is below standard coaching<br />
behavior.<br />
2 Standard of coaching behavior is met; improvement is possible.<br />
3 Ideal behavior; best practice; improvement seems impossible.<br />
The quality ratings of the different factors have been constructed by rules comparable to the ones<br />
above. A basic assumption here is that there is a standard coaching behavior that contributes to the<br />
effectiveness of the coaching, but that the behavior partly has to be adapted to the situation. These<br />
quality ratings are merely meant to give an orientation. The observers are instructed to use their intuitive<br />
impression to classify behavior that appears to be maladaptive or disproportional. Therefore, the<br />
examples given below used to define the rating levels are not conclusive.<br />
After the observation of each 5-minute period, the observers are expected to compare the behavior<br />
with the success factors, scale levels, and examples. The method is to choose the rating that is most<br />
similar to the descriptions in the manuals.<br />
2.4 Definitions and behavior anchors of the success factors<br />
The following definitions are used for observer training. The training begins with an introduction and<br />
description of the theoretical background. Then, observers are taken through the success factors and<br />
Siegfried Greif, Frank Schmidt and André Thamm 10
The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />
quality rating levels. These must be explained as concretely as possible through examples, (e.g., literal<br />
citations, short role-playing sequences of coach-client interactions, or demonstration of video clips<br />
(excluding the coaching sessions that are to be rated). In order to reach high agreement between raters<br />
and observer reliability, the most important phase of the training is when observers rate 5-minute periods<br />
of video or audiotape recordings of all factors and rating levels and then receive feedback on their<br />
ratings. The training should not be finished before the observer reaches a very high level of correct<br />
ratings (above 95%). Raters who do not reach this level should not be deployed – it is recommended to<br />
repeat the training and feedback phase with all observers after two days of the practice phase or after<br />
an interruption of more than two days.<br />
Important notes: With intention, the definitions of the success factors refer to rather complex theoretical<br />
concepts. To reach satisfactory agreement between the observers, it is necessary that they develop<br />
a clear understanding of the theoretical meaning of the factors, their differences and the quality<br />
rating levels. The raters need a concrete, convincing and reliable model and many concrete examples.<br />
We recommend that the best expert of the system be selected to be the trainer. She/he needs a great<br />
deal of experience, persuasive power and self-confidence, since she/he has to decide which ratings are<br />
correct. Raters who do not like to rate the behaviour as demanded by the manual and the trainer will<br />
never reach high agreement and reliability rates.<br />
Please contact us if you need support (sgreif@uos.de). We will answer your mail as soon as possible,<br />
and can arrange a Skype teleconference and help you with concrete suggestions based on our experience!<br />
The following descriptions start with a general definition of the factor and the individual ratings (between<br />
negative one and three for each factor). The next passage gives examples of observable behaviors<br />
and specifies what the observer should pay attention to, or how the factor is distinguishable from<br />
similar or related factors. Following this, a description of the quality rating levels and examples of<br />
behavior anchors are given. At the end of each desription, a table shows the intraclass coefficients<br />
(where possible), the mean and standard deviation of the quality rating, the percentages of agreement<br />
between raters, and significant correlations between the qualtity ratings or frequencies of positive observations<br />
of the factors with outcome criteria. All data below and the correlations described above are<br />
based on version 3 of the manual (Schmidt & Thamm, 2008).<br />
The first intraclass coefficients in the tables show the reliability of a “mean rater”, while the second<br />
shows the reliability of the “mean ratings” (normally higher). Most of the reliabilities compared to<br />
other studies can be classified as mean values – only some reliabilities are lower. According to Wirtz<br />
and Caspar (2002), low reliabilities may be tolerated if, as in our case, the variances of the ratings are<br />
low. It is remarkable that instead of their low frequencies and variances, some of the ratings correlate<br />
rather high with outcome criteria. Therefore they have not been completely eliminated from our study.<br />
It would be beneficial to improve the reliabilities by developing better definitions and/or observer<br />
training.<br />
The definitions of the present version 4 of the manual include minor changes in order to sharpen the<br />
definitions of the ratings with low reliabilities and high correlations. The changes will be mentioned<br />
below.<br />
ES: Esteem and emotional support<br />
This success factor describes how much the coach shows empathy and esteem for the client.<br />
ES1: The coach verbally shows esteem for the client.<br />
Verbal remarks made towards the client are observed.<br />
Observable behavior:<br />
The coach:<br />
• Produces an agreeable/warm-hearted atmosphere (e.g., pays attention, tunes in, uses small talk,<br />
asks sensitive questions).<br />
Siegfried Greif, Frank Schmidt and André Thamm 11
The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />
• Verbally expresses a positive attitude towards the client (e.g., “I am happy to see you,” “It is<br />
wonderful that you…,” “It is admirable that you…”).<br />
• Shows interest in the client beyond the subject of coaching (e.g., hobbies, important episodes<br />
in her/his life).<br />
• Takes up client remarks, mirrors or paraphrases what the client expresses (e.g. repeats important<br />
sentences), or summarizes statements to show zhat she/he listens attentively.<br />
• Positively reinforces statements through methods such as approval and accentuation (e.g.<br />
“This is a very important point that you have mentioned here!”).<br />
• Adapts language and wording to the language of the client, and avoids unnecessary technical<br />
terms. The coach speaks in a way that is easy to understand.<br />
Remarks:<br />
Esteem may not be formulated in a circumstantial or exaggerated way. The overall impression must be<br />
credible in the view of the rater.<br />
Esteem and negative feedback or criticism:<br />
Negative feedback and criticism must not be classified as negative or of showing low esteem. Positive<br />
feedback and an appreciation of the strengths of the client should be accounted for, along with how the<br />
negative feedback is communicated.<br />
Delineation: Evaluations of client feelings are not classified as esteem behavior. They must be coded<br />
as affect reflection (AR).<br />
Examples of behavioral anchors:<br />
-1: The coach expresses skepticism, a negative attitude, and/or indifference towards the client, and formulates<br />
feedback circumstantially or with exaggeration. The coach lacks sincerity or credibility.<br />
0: The coach shows none of the standard attributes or behavior examples mentioned in the description of<br />
the factor, does not follow-up with remarks made by the client, and remains reserved in speech (e.g., uses<br />
abstract formulations without warmth or emotional adjectives, or speaks impersonally without addressing<br />
the client).<br />
1: The coach shows some of the behaviors defined by the standard attributes (e.g., takes up the client’s<br />
remarks in a cursory or selective manner, only briefly expresses esteem and empathy, makes few positive<br />
remarks, and proceeds without asking questions or waiting for the client to respond (i.e., does not go into<br />
answer)).<br />
2: The coach shows the standard behavior (e.g., makes positive verbal remarks that the rater believes are<br />
credible and sincere). The coach expresses interest in client beyond the goals of coaching, listens to the<br />
client and follows-up on significant remarks, speaks with “I-messages,” addresses the client directly<br />
through positive evaluations, uses comparative forms (e.g., says "very helpful” with a friendly slant instead<br />
of "helpful"). However, there is room for improvement.<br />
3: The coach’s behavior and coaching methods are ideal (e.g. uses accurate wording, and creates a very<br />
good atmosphere and relationship with the client).<br />
The following table shows the intraclass coefficients of the raters and the ratings, the mean and standard<br />
deviation of the quality rating, and the percentages of agreement between raters. The last column<br />
of the table enumerates significant correlations between the qualtity ratings or frequencies of positive<br />
observations of the factors with outcome criteria.<br />
Observer reliability and frequency:<br />
Intraclass Coefficient Frequencies Percentages<br />
Mean rater<br />
Mean<br />
ratings<br />
Mean SD Mean SD<br />
Correlation with r<br />
.54** . 70** 1.4 0.3 74.8% 16.4 Goal satisfaction<br />
Result-oriented selfreflection<br />
.52*<br />
.68*<br />
Siegfried Greif, Frank Schmidt and André Thamm 12
The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />
* p> 0.05<br />
** p>0.01<br />
The intraclass coefficient of the mean ratings is satisfactory, but the percentage of the agreement between<br />
the raters is very unsatisfactory (50%). An inspection of this factor shows that the definition is<br />
too broad and the examples too heterogeneous. For some examples, the rating depends too much on<br />
the subjective impression of the observer (e.g. the creation of a positive atmosphere). It is difficult to<br />
distinguish the observations from emotional support (ES3). It might be better to focus on short verbal<br />
reinforcement behaviors of the coach. Surprisingly, the factor shows high and significant correlations<br />
to important outcome criteria.<br />
ES2: The coach non-verbally shows esteem for the client. (Only partly observable by audio tapes.)<br />
The scale codes approve non-verbal behavior observable by body language (e.g., friendly nodding and<br />
attentive eye contact).<br />
Observable behavior:<br />
The coach:<br />
• Has a posture that is turned towards the client.<br />
• Seeks eye contact and holds it.<br />
• Demonstrates a positive attitude towards the client through facial expressions (e.g. smiling) or<br />
gestures that show accord (e.g. a movement of the hand that emphazises a sentence of the client).<br />
• Reinforces the client with affirmative phonemes (e.g. mmhhhh) and nodding of the head.<br />
• Has a posture where the arms and legs appear open and turned towards the client indicating<br />
openness.<br />
• Speaks with a friendly and warm tone of voice.<br />
Remarks:<br />
The non-verbal behavior may not be maladjusted or exaggerated (e.g. permanent nodding). The overall<br />
impression has to be credible as evaluated by the rater.<br />
Delineation: Non-verbal reactions of the coach to feelings expressed by the client are not classified as<br />
esteem behavior. They must be coded as affect reflection (AR).<br />
Examples of behavioral anchors:<br />
-1: The coach’s posture is not turned toward the client. Eye contact is infrequently made. The coach mimics<br />
ineffectively, showing little interest, absent-mindedness, inhibition, and a lack of authenticity. The<br />
coach’s voice lacks warmth and is not adapted to the situation in volume, intonation, or speed. The<br />
coach’s behavior is intimidating and aggressive; the coach does not keep a safe space.<br />
0: The coach shows none of the standard attributes mentioned in the definition (e.g., does not use affirmative<br />
phonemes like “mmhhhh”). However, the coach’s behavior does not appear inappropriate.<br />
1: The coach shows some of the behaviors defined by the standard attributes, but in a rudimentary way<br />
(e.g., the coach’s posture is only turned slightly to the client, the voice shows a little warm, the eye contact<br />
is there but very short, etc.).<br />
2: The coach shows the standard behavior: posture is turned towards the client, voice is warm and friendly,<br />
mimic and gestures are active and appropriate, reactions of the client are supported non-verbally (e.g.,<br />
by nodding and saying, “mmmh, mmmh”), and the eye contact is good. The whole impresssion is<br />
supportive. However, there is room to improve some attributes or the whole pattern of behavior.<br />
3: The coach shows appropriate behavior and is flexibly adapted to the client. The observer does not perceive<br />
any possible improvements in behavior or in general. The coach has an optimal and relaxed posture,<br />
very warm voice (but not exaggerated), sensitive and supportive affect with smiling and nodding. The<br />
coach establishes a very safe environment and relationship with the client. The observer focus here is the<br />
flexibility and perfect adaption of the coach’s behavior to different situations.<br />
Siegfried Greif, Frank Schmidt and André Thamm 13
The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />
Observer reliability and frequency:<br />
Intraclass Coefficient Quality ratings Percentages<br />
Mean rater<br />
Mean<br />
ratings<br />
Mean SD Mean SD<br />
Correlation with r<br />
0.44** 0.61** 1.7 0.4 96.3% 3.4 Emotional Insight<br />
Helplessness<br />
* p> 0.05; ** p>0.01<br />
Angry feelings<br />
.74*<br />
-.57*<br />
.59*<br />
The low intraclass coefficients may partly reflect the low variance of the factor, but the unsatisfactory<br />
percentage of the agreement between the raters (65.7%) shows that it is more difficult than expected to<br />
observe the non-verbal behavior precisely. The difficulty may result from the high percentage of time<br />
sequences (96.3%) where raters had to rate minimal non-verbal reactions. Although three cameras and<br />
split-screens were used for observation, this seemed inadequate as raters reported that the tasks demand<br />
full attention. A possible solution to this is an increase in rest pauses. We have to be very careful<br />
not to interpret causal directions between non-verbal and emotional outcome criteria. It would be plausible<br />
that the non-verbal behavior of the coach is not a cause, but rather a result of emotional insight,<br />
lower helplessness, and openly showing angry feelings or a circular process. Here, interaction process<br />
analyses of the behavior of the coach and the client are necessary.<br />
ES3: The coach emotionally supports the client.<br />
This rating refers to the verbal supporting behavior of the coach and the methods used to help the client<br />
(e.g. to reach her/his goals).<br />
Observable behavior:<br />
The coach:<br />
• Communicates with optimism (e.g., “I am sure that you will be successful if you…”) and explains<br />
why optimism is realistic.<br />
• Helps the client to overcome negative emotional states through supporting tools (e.g., worksheets),<br />
writing down essentials, and giving information on how to solve a problem or get access<br />
to special knowledge (e.g., literature).<br />
• Is reliable (e.g., observes planned actions and agreements; is accessible).<br />
Examples of behavioral anchors:<br />
-1: The coach appears uninterested and expresses a pessimistic perspective on the client’s chances of<br />
goal-attainment. The coach apologizes for arriving late or for postponing the appointment for a long<br />
time. The coach has completely forgotten about agreements and plans made with the client.<br />
0: The coach does not show any of the behaviors in the standard definition (e.g., does not express<br />
optimism, and does not prepare supporting tools to overcome negative emotional states). However,<br />
the overall behavior of the coach remains moderate. The coach arrives slightly late for the appointment,<br />
and/or does not remember well agreements or plans made with the client.<br />
1: The coach shows standard behavior in a rudimentary way. The coach expresses some optimism,<br />
but it is not very convincing and does not explain why the optimism is realistic (e.g., uses standard<br />
flowery language such as, “Cheer up! You will be successful!”). Some signs of supportive behavior<br />
are shown but seem superficial (the assessment is based merely on the intuitive impression of the<br />
rater).<br />
2: The coach shows the standard behavior. The coach expresses optimism and explains why, helps the<br />
client to overcome emotionally straining states through the use of supporting tools, and arrives on<br />
schedule and remembers all agreements and plans without difficulty. There is room for improvement.<br />
3: The coach behaves so credibly that further improvement does not seem possible. The client acknowledges<br />
that the coach has invested much work into the coaching (e.g., has prepared a very useful<br />
worksheet or planning tool), or that the coach has organized the provision of helpful information for<br />
Siegfried Greif, Frank Schmidt and André Thamm 14
The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />
the client.<br />
Observer reliability and frequency:<br />
Intraclass Coefficient Quality ratings Percentages<br />
Mean rater<br />
Mean<br />
ratings<br />
Mean SD Mean SD<br />
Correlation with r<br />
0.53** 0.69** 1.3 0.2 37.4% 14.6 Helplessness<br />
Listlessness<br />
(pre-post-differences)<br />
* p> 0.05<br />
** p>0.01<br />
-.56*<br />
-.56*<br />
The mean value and variance of the quality rating is low. This may explain unsatisfactory intraclass<br />
coefficients. The percentage of agreement between raters remains low (61.1%) as well. Therefore, the<br />
description and examples have been sharpened for future studies.<br />
PR: Problem-reflection<br />
Problem-reflection refers to an elaborate analysis of a problem situation where the client embraces the<br />
antecedents, interactions, and processes in the situation and the perceived consequences. The reflection<br />
is classified as “result-oriented” only if the coach instructs the client on how to develop concrete solutions,<br />
which guide future actions of the client. The following ratings differ between the facilitation of<br />
problem-reflection (RP1) and a result-oriented focus of the problem-reflection (RP1+).<br />
Remarks:<br />
Following logic, the second rating depends on the first. Therefore, correlations between both ratings<br />
should not be calculated, as they will be artificially high.<br />
PR1: The coach has the client reflect on problems.<br />
The rating assesses how much the coach stimulates intensive reflection on a problem or problematic<br />
situation.<br />
Delineation:<br />
It is difficult to differentiate between problem-reflection (PR) and self-reflection (SR). Where there is<br />
doubt, the reflection should be classified as problem-reflection (the subordinate term). The reflection<br />
should only be categorized as a self-reflection if the focus is on the following client attributes:<br />
• Strengths and weaknesses;<br />
• Needs and motives;<br />
• Real and ideal self-concept or self-image;<br />
• Personal, cultural, or family background;<br />
• Peculiar abilities, competencies, potentials, and their future development;<br />
• Personal behavior, feelings, and experiences; and<br />
• Personal and intrinsic goals and plans for self-changes or self-development.<br />
Observable behavior:<br />
The coach:<br />
• Prompts the client to reflect on the problem situation (e.g., antecedents, interactions, processes,<br />
and consequences).<br />
• Asks direct questions to have the client think more deeply about the problem.<br />
• Summarizes and structures the client’s reflection on the problem situation.<br />
• Asks the client to describe examples of similar experiences.<br />
Siegfried Greif, Frank Schmidt and André Thamm 15
The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />
• Supports the client in structuring the problem situation (e.g., “What is most important? What<br />
is of minor importance?” or “How did it start? What followed next?”).<br />
• Asks the client to change their perspective of other persons in the situation (e.g., “How did<br />
person X react and how did he probably feel?” or “Imagine you were person X. How would<br />
you feel after your remark in the discussion?”), or uses techniques like role playing (e.g.,<br />
changing chairs) to facilitate thinking on different perspectives of those involved in the situation.<br />
Examples of behavioral anchors:<br />
-1: The coach gives advice without having knowledge of the situation. The coach fails to acknowledge<br />
a significant problem mentioned by the client, does not support a client who has clearly asked<br />
for help in trying to solve a problem, or does not help to structure a problem situation.<br />
0: The coach does not show elements of the standard behavior. The coach does not ask the client to<br />
describe or reflect on a problem situation.<br />
1: The coach shows some elements of the standard behavior, but the intervention remains incomplete<br />
(e.g., the coach asks the client to describe a problem, but then changes the subject before the description<br />
is complete or comprehensible). The coach does not provide support in systematically structuring<br />
a complex situation, or treats the situation in a cursory manner.<br />
2: The coach shows behavior similar to the standard definition above. The coach prompts the client to<br />
think and talk about the problem situation, and directly asks the client to describe the antecedents,<br />
interactions, processes, and consequences. The coach helps the client through systematic techniques<br />
to change client perspective. The coach also structures the descriptions and analysis. However, the<br />
rater has the impression that there is room for improvement in the coach’s behavior.<br />
3: The behavior of the coach is an example of best practice in the facilitation of client problemreflection.<br />
The coach helps the client to develop new insights, which impresses the client.<br />
Observer reliability and frequency:<br />
Intraclass Coefficient Quality ratings Percentages<br />
Mean rater<br />
Mean<br />
ratings<br />
Mean SD Mean SD<br />
0.5** 0.67** 1.1 0.1 37.2% 13.7 -<br />
* p> 0.05<br />
** p>0.01<br />
Correlation with r<br />
The mean value of 1.1 shows that the quality of the problem-reflection behaviour of the coaches is<br />
rather low. The variance is minimal. This explains why the intraclass coefficients remain low, while<br />
the absolute percentage of agreement between the raters is satisfactory (72.8%). It is plausible that the<br />
factor does not correlate substantially with outcome criteria. It is interesting that the following subcategory,<br />
which focuses on result-oriented problem-reflection, comes up with a prediction of low fear<br />
of failure after coaching.<br />
PR1+: The coach facilitates result-oriented problem reflection for the client:<br />
Remarks:<br />
Result orientation does not only refer to reflection on consequences for future actions. It also embraces<br />
concrete insights, plans, or intended actions for future analysis.<br />
PR1+ can only be coded if facilitation of problem-reflection is shown (i.e. if PR1 is coded at least by<br />
1). The behavior categories PR1 and PR1+ therefore are not independent. (In statistical analyses, artificial<br />
correlations are expected.)<br />
Observable behavior:<br />
Siegfried Greif, Frank Schmidt and André Thamm 16
The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />
In addition to facilitation of problem-reflection (see above, PR1) the coach:<br />
• Prompts the client to derive concrete consequences for future actions.<br />
• Points to the relevance of problem analysis for future action or reflection.<br />
• Trains the client to become aware of aimless rumination and trains the client on how to stop it<br />
(e.g., by self-instruction techniques and deviation of thoughts to positive experiences).<br />
Examples of behavioral anchors:<br />
-1: The coach allows the client to think aimlessly or erratically, or asks the client erratic questions.<br />
0: The coach does not show any of the standard behaviors.<br />
1: The coach shows elements of the standard behavior (e.g., begins problem-reflection and rudimentarily<br />
mentions possible action consequences without describing or explaining them precisely, or<br />
without asking the client to cogitate autonomously).<br />
2: The coach shows behavior similar to the standard behavior. Following the problem analysis, the<br />
coach asks the client to derive concrete actions or future consequences. The coach instructs the client<br />
in how to identify rumination and how to stop recurring thoughts via thought-stopping techniques or<br />
similar interventions. The rater has the impression that there is room for improvement.<br />
3: The support of result-oriented problem-reflection is ideal. The coach prompts the client in a convincing<br />
way to derive important or creative solutions from the problem analysis. The coach helps the<br />
client to stop rumination in a sensitive and effective way.<br />
Observer reliability and frequency:<br />
Intraclass Coefficient Quality ratings Percentages<br />
Mean rater<br />
Mean<br />
ratings<br />
* p> 0.05 ** p>0.01<br />
Mean SD Mean SD<br />
Correlation with r<br />
1.1 0.2 15.2% 9.5 Fear of failure -.47*<br />
Intraclass coefficients could not be calculated because of a very low quality rating and low variances.<br />
The absolute percentage of agreement between raters is satisfactory (74.8%). It is remarkable that the<br />
factor, instead of these preconditions, shows a significant correlation, indicating a reduction of fear of<br />
failure. A single significant correlation may be not replicable in follow-up studies, but additional correlations<br />
to other success factors (e.g., high negative correlations with SR1 self-reflection (-.64**) and<br />
SR1+ result-oriented self-reflection (-.51*) show a pattern that fits with our theoretical expectations<br />
that problem-reflection is a factor that is different from self-reflection.<br />
SR: Self-reflection<br />
Self-reflection is a special type of problem-reflection that specifically regards the self. As mentioned<br />
above, typical examples are reflections that focus the client’s:<br />
• Strengths and weaknesses;<br />
• Needs and motives;<br />
• Real and ideal self-concept or self-image;<br />
• Personal, cultural, or family background;<br />
• Peculiar abilities, competencies, potentials, and their future development;<br />
• Personal behavior, feelings, and experiences; and<br />
• Personal and intrinsic goals and plans for self-changes or self-development.<br />
The rater is asked to observe the behavior of the client when instructed by the coach to reflect on these<br />
problems systematically.<br />
Remarks:<br />
Siegfried Greif, Frank Schmidt and André Thamm 17
The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />
Delineation: The rating focuses on broad reflections of personal strengths and weaknesses, or potentials.<br />
Resource activation (RA) is classified as the reflection, evaluation, and activation of concrete<br />
personal resources (e.g., special abilities or competencies, see below RA) that directly support the<br />
attainment of concretely defined goals of coaching.<br />
SR1: The coach has the client engage in self-reflection (e.g,. on behavior, personal experiences,<br />
needs and motives, strength and weaknesses, ideal and real self-image).<br />
The rating assesses whether or not the coach is able to engage the client in self-reflection (e.g.,<br />
strengths and weaknesses, personal motives, attributes and typical behavior (see list above), and to<br />
what degree.<br />
Observable behavior:<br />
The coach:<br />
• Guides the client’s self-reflection through questions about the client’s personal attributes.<br />
• Reinforces descriptions of special abilities and competencies of the client by verbal communication.<br />
• Helps the client to develop new insights about the self, or between behavior and the reactions<br />
of others.<br />
• Facilitates the exploration of the relationship between the client’s actions and self-concept or<br />
personality.<br />
• Uses special techniques and tools that help the client analyze real and ideal self-concept (e.g.,<br />
by drawing a self-image).<br />
Remarks:<br />
Prompting the client to reflect on or apply her/his personal resources (e.g. abilities and competencies)<br />
or the resources of the social networks of the client to reach goal attainment should not be coded here.<br />
They belong to the factor resource activation (RA, see below).<br />
Examples of behavioral anchors:<br />
-1: The coach makes statements without respecting the client’s personal needs, goals, or special abilities<br />
and competencies. The coach generalizes, gives standard advice, and interprets the client’s behavior<br />
without making sure the client feels accepted. The coach does not seize on the self-reflection of<br />
the client or does not help structure orderless self-descriptions.<br />
0: The coach does not show any of the standard behaviors described above. The coach does not<br />
stimulate self-reflection.<br />
1: The coach shows some of the behaviors and attributes mentioned in the standard behavior description.<br />
The coach does not prompt the client systematically or intensively to think and talk about motives,<br />
special abilities and competencies, strengths, and weaknesses. The coach merely comments in a<br />
cursorily manner on the client’s self-reflection and does not support deep reflection and insight. The<br />
coach does not use systematic or adequate methods to encourage self-reflection.<br />
2: The coach shows behavior similar to the standard behavior description. The client stimulates reflection<br />
on personal motives, norms, cultural background and experiences, abilities and competencies,<br />
experiences, or strengths and weaknesses, ideal and real self-concept of the client by direct, or circular<br />
questions or other methods. The behavior of the coach conforms to the standard, but there is room<br />
for improvement.<br />
3: The performance of the coach is ideal. The coach behaves sensitively, wording questions well and<br />
explaining the importance of the client to reflect on strengths and weaknesses. The coach helps the<br />
client explain self-reflection in concrete terms. Through probing questions or other methods, the<br />
coach helps the client to obtain new insights about the self that are not easily forgotten.<br />
Siegfried Greif, Frank Schmidt and André Thamm 18
The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />
Observer reliability and frequency:<br />
Intraclass Coefficient Quality ratings Percentages<br />
Mean rater<br />
Mean<br />
ratings<br />
Mean SD Mean SD<br />
Correlation with r<br />
0.41** 0.58** 1.1 0.1 21.2% 12.4 Demotivation<br />
Helplessness (pre-postdifferences)<br />
Goal reflection<br />
Behavior reflection<br />
* p> 0.05 ; ** p>0.01<br />
.57*<br />
-.49<br />
Again the mean values and variance of the ratings of this factor are low and worsen the preconditions<br />
to reach satisfactory intraclass coefficients. The absolute percentage of agreement between raters is<br />
satisfactory (74.8%). As discussed above, the significant correlation found shows that demotivation<br />
and self-reflection may be related. The remaining correlations in our study are not significant.<br />
SR1+: Facilitation of result-oriented self-reflection: The coach guides the client to derive concrete<br />
solutions from the problem-reflection<br />
(Only if facilitation of problem-reflection is shown.)<br />
Remarks:<br />
Result orientation does not only refer to action consequences for future actions. It also embraces concrete<br />
insights for future analysis, plans, or intended actions.<br />
SR1+ can only be coded if facilitation of self-reflection is shown (SR1 is rated at least by 1). The behavior<br />
categories SR1 and SR1+ therefore are not independent. (In statistical analyses, artificial correlations<br />
are expected.)<br />
The exploration of abilities, competencies, and social networks of the client to reach goal attainment<br />
do not need to be coded. They belong to resource activation (RA).<br />
Observable behavior:<br />
In addition to engaging the client in self-reflection, the coach:<br />
• Prompts the client to derive concrete consequences for future actions from self-reflection.<br />
• Points to the relevance of the insights derived from the reflection for future plans or selfreflection.<br />
• Stimulates the client’s commitment to stick to the solution derived from self-reflection.<br />
• Trains the client in how to to be aware of rumination and trains in how to stop it (e.g., by selfinstruction<br />
techniques and deviation of thoughts to positive self-images).<br />
Examples of behavioral anchors:<br />
-1: The coach does not stop the client from engaging in aimless or erratic self-reflection or selfdescription.<br />
The coach does not stop the client from ruminating.<br />
0: The coach does not show any of the standard behavior. The coach does not ask the client to derive<br />
concrete actions or other consequences from self-reflection.<br />
1: The coach behaves rudimentarily and therefore only facilitates cursory, result-oriented selfreflection.<br />
The coach attempts to stop the client from ruminating, but does so inefficiently. The results<br />
or consequences remain abstract or without commitment.<br />
2: The coach shows behaviors similar to the standard description above. The coach encourages the<br />
client to derive concrete consequences from the self-reflection and a commitment to these consequences.<br />
The coach helps the client to stop rumination effectively. There is room for improvement in<br />
coach’s behavior.<br />
Siegfried Greif, Frank Schmidt and André Thamm 19<br />
-.49<br />
-.51
The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />
3: The coach’s facilitation of the result-oriented self-reflection is ideal. The coach systematically asks<br />
the client to derive significant concrete consequences from the self-reflection. The coach activates a<br />
strong and sustained commitment by the client to follow-through with the solutions.<br />
Observer reliability and frequency:<br />
Intraclass Coefficient Quality ratings Percentages<br />
Mean rater<br />
Mean<br />
ratings<br />
* p> 0.05 ; ** p>0.01<br />
Mean SD Mean SD<br />
Correlation with r<br />
1.1 0.2 3.2% 4.4 Goal satisfaction<br />
Irritation<br />
Helplessness<br />
The very low mean of the quality rating and the variance of the specification of result-oriented selfreflection<br />
forbid a calculation of intraclass coefficients. The absolute percentage of agreements between<br />
raters is satisfactory (74.8%). It is remarkable that under such preconditions several high but<br />
insignificant correlations can be observed that are theoretically plausible and differ from other factors.<br />
AR: Affect reflection and calibration<br />
Preliminary remark: Basic positive and negative emotions such as anxiety, anger, or happiness are<br />
often referred to as types of “affect”. In some theories, “emotion” is the superordinate term while in<br />
other theories “emotion” refers to more complex feelings that integrate cultural meaning (e.g., feelings<br />
of success or helplessness). We use the term “emotion” for simplicity and clarity.<br />
The following two rating factors are often are related. Calibrations of emotions often require prior<br />
reflections, and the reflection itself reduces strong emotion. Therefore, the observations are not logically<br />
independent, and a correlation between both results in artificially high values.<br />
The ratings aim at assessing coach behavior, which directly stimulates the description and reflection of<br />
affect or emotion by the client (AR1), as well as behavior that pacifies emotion.<br />
AR1: The coach has the clients reflect on their own feelings.<br />
The coach has the client engage in deep and intensive reflection on emotion.<br />
Observable behavior:<br />
The coach:<br />
• Prompts the client to specify and describe emotions or feelings in critical situations.<br />
• Asks the client to explain feelings in a friendly manner. If the descriptions of the client’s feelings<br />
as well as the factors that elicit those feelings are ambiguous, the coach offers possible<br />
hypotheses’ about situation elicitors or intra-individual causes, and asks the client to evaluate<br />
their relevance.<br />
• Encourages the client to analyze and understand possible situational and personal reasons for<br />
the feelings and consequences contributing to their general affect state, well-being, performance<br />
(e.g., lower quality), typical reactions of the social environment to emotional behavior of<br />
the client, and backlashes.<br />
Examples of behavioral anchors:<br />
-1: The coach responds negatively or neutrally to the client’s emotions.<br />
0: The coach does not show any standard behavior. Attempts of the client to talk about their feelings<br />
are neglected. The coach does not comment on the client’s observable emotional reactions, or on possible<br />
consequences and reactions of the environment.<br />
1: The coach shows some of the behaviors described, but in a rudimentary way. The coach asks the<br />
client to describe feelings, but does not pick up on the descriptions and does not help to specify them.<br />
Elicitors and consequences are also not specified.<br />
Siegfried Greif, Frank Schmidt and André Thamm 20<br />
.41<br />
-.46<br />
-.52
The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />
2: The coach shows behaviors similar to the standard descriptions above. The coach encourages the<br />
client to describe feelings precisely, and asks the client to give situational elicitors or causes in a personal<br />
way. The coach helps the client to analyze and understand the consequences (e.g., negative<br />
affect and low well-being), the reactions of the environment, and the possible backlash. There is room<br />
for improvement in the coach’s behavior related to the factor.<br />
3: The coach’s support of affect reflection is ideal. The coach sensitively facilitates deep reflection<br />
and specifies the client’s feelings. The client is able to develop new and lasting insights through a<br />
systematic and precise analysis of situational elicitors, personal causes and consequences, environment,<br />
and backlashes.<br />
Observer reliability and frequency:<br />
Intraclass Coefficient Quality ratings Percentages<br />
Mean rater<br />
Mean<br />
ratings<br />
Mean SD Mean SD<br />
Correlation with r<br />
0.11* 0.2* 1.1 0.3 7.7% 5.1 Feeling of anger at the<br />
time of the first and<br />
middle session<br />
* p> 0.05 ; ** p>0.01<br />
The absolute percentage of agreement between the rater is good (88.9%), but the intraclass coefficients<br />
are too low. This shows that compared to psychotherapy, affect reflection is seldom used in coaching,<br />
as seen in the first comparison study (Borsum, 2008). The plausible correlation between the affect<br />
sales measuring state feelings of aggression at the time of the first and middle sessions show that it<br />
may be premature to discard the factor from future coaching observations. We plan to conduct coaching-intervention<br />
studies with clients who work under high-stress conditions. Here, we would expect<br />
that both ratings of AR factors are of high relevance.<br />
AR2: The coach pacifies the client’s feelings and is able to facilitate calm and open thinking about<br />
the feelings.<br />
The coach intervenes in way that pacifies the emotions of the client (e.g., with calming words or relaxation<br />
techniques) during a critical situation, and facilitates open conversation about the feelings and<br />
their possible negative consequences.<br />
Observable behavior:<br />
The coach:<br />
• By verbal or non-verbal communication shows appreciation and understanding of the client’s<br />
emotions.<br />
• Gives the client time to deescalate by sharing an emotionally moving experience.<br />
• Asks the client to explain the emotionally intense reaction and to consider whether there are<br />
long-term negative consequences (e.g., stress reactions, irritation, or low personal well-being)<br />
or negative reactions of other persons in the environment (e.g., negative feelings, hidden rejection<br />
or stress reactions of team members), and if the client would prefer to reduce the intensity<br />
of the feelings in future similar situations<br />
• Uses relaxation techniques or similar systematic methods, and/or trains the client in the practice<br />
of these techniques.<br />
Examples of behavioral anchors:<br />
-1: The coach reinforces negative emotions (e.g., aggressive feelings) or tries to explain the emotions<br />
with inadequately simple or speculative causes (e.g. “The only reason of the emotional reaction of<br />
your team colleague is that he hates you.”)<br />
0: The coach does not show any of the behaviors described above.<br />
Siegfried Greif, Frank Schmidt and André Thamm 21<br />
.63*<br />
.66*
The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />
1: The coach shows some of the behaviors described in a rudimentary form. The coach accepts the<br />
client’s emotions, but unsuccessfully tries to calm them.<br />
2: The coach demonstrates behavior similar to the standard descriptions. The coach allows the client<br />
to deescalate, sympathizes with the client, helps the client to understand the emotional reaction and<br />
personal consequences, as well as reactions of other persons in the environment. The coach also takes<br />
the client through methods of reducing such intense reactions in similar situations. Relaxation techniques<br />
or similar methods are used to train the client to pacify emotions. The reduction of emotions<br />
can still be improved.<br />
3: The coach behaves in an ideal way, applying the appropriate relaxation techniques or methods.<br />
The coach trains the client to apply these techniques for self-pacification in the best possible way.<br />
The client is able to use the methods effectively.<br />
Observer reliability and frequency:<br />
Intraclass Coefficient Quality ratings Percentages<br />
Mean rater<br />
Mean<br />
ratings<br />
* p> 0.05; ** p>0.01<br />
Mean SD Mean SD<br />
1.4 0.5 1.1% 1.6<br />
Correlation with r<br />
As expected, the percentage of situations where affect calibration is observable in our coaching study<br />
is marginally low. This forbids any further analysis of the intraclass coefficients even where the percentage<br />
of agreement between raters is artificially high (98.9%). As mentioned above, the rating is<br />
assumed to be more important for the coaching of stress management where pacification of affect is<br />
used systematically to reduce post-work stress levels.<br />
CG: Clarification of goals<br />
The rating assesses whether the coach helps the client to to clarify goals in a concrete way, to what<br />
degree this is done, and if goals are prioritized.<br />
CG1: The coach supports the client’s clarification and/or defining of goals.<br />
The rating focuses on the coach’s ability to clarify and explain the client’s goals (e.g., by a concrete<br />
description, definition, or specification of criteria, and measures the rate of goal attainment).<br />
Remark:<br />
Goal clarification should be observed throughout the coaching.<br />
Observable behavior:<br />
The coach:<br />
• Asks the client to describe or define goal(s).<br />
• Uses direct questions to deepen the client’s goal descriptions.<br />
• Helps the client clarify the relationship between extrinsic and intrinsic goals.<br />
• Asks the client to describe in concrete terms what will be different once the goals have been<br />
achieved.<br />
• Helps the client understand which goals are realistic.<br />
• Helps the client detect conflict between different goals that are being pursued.<br />
• Systematically guides the client to decide which of the goals is a priority.<br />
Examples of behavioral anchors:<br />
-1: The coach constrains or disrupts the client from clarifying goals.<br />
0: The coach does not show any of the behaviors described above. The coach does not clarify the<br />
client’s goals, or help to define them concretely.<br />
Siegfried Greif, Frank Schmidt and André Thamm 22
The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />
1: The coach shows some of the behaviors of the standard description, but in a rudimentary way. The<br />
goals are noted, but their definitions remain vague – further clarification is possible and necessary.<br />
Potential conflicts with the goals remain undetected, and the client and coach do not examine the<br />
goals to see if they are realistic.<br />
2: The coach shows the standard behaviors. The coach helps the client analyze and define the goals<br />
concretely, detecting potential conflicts and ascertaining whether the goals are realistic. Where necessary,<br />
the coach encourages the client to redefine the goals. There is room for improvement in goal<br />
clarification.<br />
3: The coach helps the client to clarify the goals. All aspects described above are accounted for in an<br />
ideal way.<br />
Observer reliability and frequency:<br />
Intraclass Coefficient Quality ratings Percentages<br />
Mean rater<br />
Mean<br />
ratings<br />
Mean SD Mean SD<br />
Correlation with r<br />
0.62** 0.77** 1.2 0.2 20.3% 10.3 Overall sum of negative-affect<br />
scales<br />
Activity level at the<br />
start of the coaching<br />
* p> 0.05; ** p>0.01<br />
The intraclass coefficients and the percentage of agreement between raters (86.4%) is satisfactory.<br />
Goal clarification correlates significantly with problem-reflection (PR1 .46* and PR1+ .55*) and the<br />
activity level at the start of the coaching.<br />
RA: Resource activation<br />
The coach helps the client become aware of internal and external resources to reach concrete coaching<br />
goals.<br />
Internal resources are potentials within the individual, e.g., motivational and personality traits, abilities,<br />
competencies and experiences. External resources are potentials in the social and material environment<br />
that the client is able to use (e.g., expert knowledge, consultations, emotional support or concrete<br />
help from people in the occupational environments, family or network of friends, technical literature<br />
/knowledge of IT systems, and financial resources).<br />
RA: The coach encourages the client to explore and make use of internal and external resources<br />
that will support goal attainment.<br />
The rating assesses the level to which the coach helps the client explore and evaluate external resources,<br />
and to what extent the coach encourages the client to use the resources that will potentially<br />
support concrete coaching goals.<br />
Remarks:<br />
Delineation: The rating focuses on resources directly supporting the attainment of concrete coaching<br />
goals. Broader reflections of personal strengths and weaknesses or potentials are classified as selfreflections<br />
(SR1 or SR1+).<br />
Observable behavior:<br />
The coach:<br />
• Encourages the client to explore and specify internal strengths and potentials that may result in<br />
establishing concrete coaching goals (e.g., strong achievement motivation, special abilities and<br />
competencies, or experiences).<br />
• Encourages the client to explore and name possible external resources that may support the attainment<br />
of concrete coaching goals (e.g., knowledge of experts, consultations, emotional<br />
Siegfried Greif, Frank Schmidt and André Thamm 23<br />
-.48<br />
.61*
The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />
support, or concrete help from people in the occupational environments, family or network of<br />
friends, technical literature or IT knowledge systems, and financial resources).<br />
• Asks the client to remember former successes, the personal resources that helped in those instances,<br />
and how the same resources can be applied in the current situation.<br />
• Explains how the client can use personal resources to reach goals in the current situation.<br />
• Appreciates the client’s personal resources, explaining their value and stimulating positive<br />
self-evaluation.<br />
• Encourages the client to use personal resources in pursuing concrete coaching goals.<br />
Examples of behavioral anchors:<br />
-1: The coach gives skeptical remarks concering the client’s personal resources, or distracts the client<br />
by asking to focus on their weaknesses and problems.<br />
0: The coach does not show any of the behaviors described above. The coach does not try to activate<br />
the client’s strengths and potentials.<br />
1: The coach shows the standard behaviors described above, but only in a rudimentary way. The<br />
coach asks the client to name strengths, competencies, or external resources in the environment etc.,<br />
but the analysis is not deep or precise.<br />
2: The coach helps the client to analyze and activate internal and external resources in a similar way<br />
as described above by the standard behaviors. For example, strengths and competencies or external<br />
resources are analyzed precisely and the client identifies new ones that she/he did not think of beforehand.<br />
The client is asked to evaluate how useful the resources will be for the attainment of the coaching<br />
goals. Improvements of the behaviors of the coach are possible.<br />
3: The coach helps the client analyze and activate internal and external resources in an ideal manner.<br />
This is shown by high precision, intensity, or completeness of the analysis, and a convincing evaluation<br />
that raises the motivation of the client to use the resources intelligently.<br />
Observer reliability and frequency:<br />
Intraclass Coefficient Quality ratings Percentages<br />
Mean rater<br />
Mean<br />
ratings<br />
Mean SD Mean SD<br />
Correlation with r<br />
0.29** 0.45** 1.1 0.1 17.4% 12.3 Goal satisfaction<br />
Listlessness<br />
Information management<br />
* p> 0.05 ; ** p>0.01<br />
.49*<br />
-.50<br />
.49<br />
The intraclass coefficients of this important scale are not satisfactory, but the absolute percentage of<br />
agreements between raters is (75.76%). As the low mean value and very narrow variance show, the<br />
coaches did not realize the factor as optimally as demanded by the standard attributes. This may explain<br />
why the correlations with outcome criteria do not reach the level of significance. A problem may<br />
also result from delineation to goal clarification and support of transfer into practice. A more intensive<br />
training of the observers highlighting the differences here may be useful.<br />
TP: Support of transfer into practice<br />
TP: The coach encourages the client to put the planned goal-oriented behavior and measures into<br />
practice.<br />
The coach encourages the client to put the planned behavior and/or other measures into practice<br />
through a persistent pursuit of goal-oriented actions and measures despite difficulties and barriers.<br />
Siegfried Greif, Frank Schmidt and André Thamm 24
The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />
Observable behavior:<br />
The coach:<br />
• Helps the client develop a plan of concrete goals and subgoals comprised of small actions and<br />
measures that the client is able to perform.<br />
• Stimulates the client to explore and analyze possible internal and external difficulties or barriers<br />
to taking the intended actions and measures.<br />
• Helps the client develop a realistic plan of actions and measures despite possible difficulties<br />
and barriers.<br />
• Consults the client on how to overcome possible internal and external difficulties and barriers<br />
to trials and to act according to plan.<br />
• Encourages the client, after a seemingly insurmountable difficulty, to try again after reexamining<br />
and redefining goals, actions, or measures.<br />
• Uses systematic questioning techniques or tools to support the transfer of planned changes<br />
into practice (e.g., telephone shadowing).<br />
Examples of behavioral anchors:<br />
-1: The behavior of the coach interferes with client’s attempts to perform intentions practically. The<br />
client expresses indifference or disrespect towards the client’s attempts to transfer the intended actions<br />
and measures into practice. The coach criticizes the action plans without developing alternative<br />
plans that might be more realistic.<br />
0: The coach does not show any of the standard behaviors.<br />
1: The coach shows basic support that helps the client transfer the planned behavior into practice. The<br />
coach asks the client to develop a plan of action or a measure that will analyze some internal and external<br />
difficulties. The analysis and the measures for how to overcome them, however, remain vague<br />
or abstract.<br />
2: The coach helps the client to prepare the transfer of the planned behavior into practice. The behavior<br />
is similar as described in the examples above. The analysis and support is systematic and incorporates<br />
nearly all of the important facets of the transfer problems of the client. The coach helps the client<br />
to develop a realistic plan of small steps of actions, to analyze possible internal and external difficulties,<br />
and to measure how to overcome them. The coach encourages the client to redefine goals<br />
after setbacks. Improvement of the behavior is possible, (e.g., the analysis is not completely systematic,<br />
does not embrace all important barriers and relevant measures, or is not fully convincing).<br />
3: The coach supports the transfer into practice. The coach helps the client analyze all relevant internal<br />
and external difficulties and barriers, as well as effective measures in a convincing way. The<br />
coach uses systematic transfer tools (e.g., telephone shadowing) and other methods (e.g., simulations<br />
by role-playing). After a setback, the coach sensitively helps the client to understand inner transfer<br />
difficulties and change resistances in the environment. The coach motivates the client to draw adequate<br />
conclusions from the trials and sustainably motivates them to redesign the plans and try again<br />
until goals are reached.<br />
Observer reliability and frequency:<br />
Intraclass Coefficient Quality ratings Percentages<br />
Mean rater<br />
Mean<br />
ratings<br />
Mean SD Mean SD<br />
Correlation with r<br />
0.6** 0.75** 1.2 0.2 26.7% 18.1 Helplessness<br />
Listlessness<br />
Self-management (procrastination)<br />
Goal satisfaction<br />
* p> 0.05; ** p>0.01<br />
-.55*<br />
-.58*<br />
.55*<br />
Siegfried Greif, Frank Schmidt and André Thamm 25<br />
.44
The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />
The intraclass coefficients and the percentage of agreement between the raters (73.74%) are satisfactory.<br />
But the low mean value and variance show that the coaches did not perform well in comparison<br />
to the standard example behavior descriptions of the factor. It is surprising that relatively high and<br />
significant correlations to outcome criteria have been found here.<br />
FP: Feedback and evaluations in the process<br />
The ideal standard of result-oriented coaching demands that each coaching session create concrete<br />
results perceived by the client. This is shown by a positive evaluation at the end of the session, either<br />
after a question from the coach or by a spontaneous positive feedback from the client.<br />
FP1: The coach asks the client to evaluate progress in the coaching as well as satisfaction with the<br />
coaching, especially in goal attainment.<br />
The coach asks the client for feedback, to evaluate that specific coaching session or the coaching in<br />
general, especially in relation to the perceived progress or goal attainment. The evaluation of the client<br />
is positive.<br />
Observable behavior:<br />
The coach receives positive feedback and/or evaluation from the client, after:<br />
• Asking the client to rate satisfaction with the session or the coaching up to that point.<br />
• Asking the client to rate progress in the coaching or the goal-attainment rate.<br />
• Using rating scales to assess satisfaction, goal attainment, or other evaluation criteria of the<br />
coaching.<br />
• Applying specialized instruments or standard questionnaires.<br />
Examples of behavioral anchors:<br />
-1: The coach hinders the client from evaluating important aspects of the coaching, or gives the impression<br />
of being disinterested in the results. The client gives negative feedback or expresses disapproval<br />
of a session or of the coaching up to that point. The coach does not mention possible changes.<br />
0: The coach does not ask the client for feedback. The coach does not show any of the behaviors described<br />
above and does not apply instruments assessing the client’s satisfaction with the coaching.<br />
1: The coach briefly asks the client for feedback or evaluations. The feedback or evaluations of the<br />
client remain rudimentary or cursory. The coach does not ask the client to explain the evaluation.<br />
Also, if the evaluation is merely somewhat positive, the rating of 1 is to be coded here.<br />
2: The coach asks the client for feedback or uses rating scales and evaluation instruments, similar as<br />
described above. The overall results of the evaluation are unambiguously positive and the coach protocols<br />
them. An improvement of the behavior is possible.<br />
3: The coach sensitively asks the client to give feedback, or uses systematic scales or evaluation instruments<br />
and interprets the results together with the client. The client’s overall evaluations are very<br />
positive. The results of the evaluation are analyzed systematically and are used for future planning of<br />
the subjects and changes to the coaching or client’s plans.<br />
Observer reliability and frequency:<br />
Intraclass Coefficient Quality ratings Percentages<br />
Mean rater<br />
Mean<br />
ratings<br />
Mean SD Mean SD<br />
Correlation with r<br />
0.64** 0.78** 1.1 0.2 10.2% 8.3 <strong>Coaching</strong> start: Overall<br />
sum of negative affect<br />
* p> 0.05; ** p>0.01<br />
<strong>Coaching</strong> ending:<br />
Happiness<br />
Activity<br />
Siegfried Greif, Frank Schmidt and André Thamm 26<br />
.61*<br />
.57**<br />
.69**
The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />
The intraclass coefficients and the percentage of agreement among raters (88.4%) is satisfactory. The<br />
remaining values are based on the whole range of the ratings (-1 to -3). In the study the highest affect<br />
state measures can be found in the starting and ending session. Also, evaluations are more meaningful<br />
in the starting and ending phase of the coaching. It is an interesting observation that positive evaluations<br />
of the clients correlate with the sum of all negative-affect scales. This could mean that clients<br />
who are in a negative mood are more grateful for starting the coaching and are more likely to give<br />
positive feedback. The high correlations at the end of the coaching show that positive evaluations of<br />
the clients are associated with their happiness, but also that they might activate them.<br />
FP2: The client spontaneously remarks satisfaction or dissatisfaction with the meeting or the whole<br />
coaching engagement.<br />
The client spontaneously expresses satisfaction or dissatisfaction with the session or the whole coaching<br />
engagement.<br />
Examples of behavioral anchors:<br />
-1: The client spontaneously gives negative feedback or evaluation of the coaching.<br />
0: The client does not make spontaneous evaluations of the coaching.<br />
1: The client briefly expresses satisfaction with special aspects of the coaching without any prompting.<br />
2: The client briefly expresses satisfaction with the overall coaching engagement without any prompting.<br />
3: The client expresses great satisfaction with the coaching without any prompting.<br />
Observer reliability and frequency:<br />
Intraclass Coefficient Quality ratings Percentages<br />
Mean rater<br />
Mean<br />
ratings<br />
Mean SD Mean SD<br />
- - 1.17 0.33 0.73% 1.32 -<br />
* p> 0.05; ** p>0.01<br />
Correlation with r<br />
Due to a low quality rating and variance, intraclass coefficients could not be calculated. The percentage<br />
of agreement between raters is extremely high (97.9%). We doubt that it is useful to use this scale<br />
in future studies, since the frequency of the behavior is very low and it does not correlate as expected<br />
with overall client satisfaction.<br />
Correlations between the success factors<br />
Non-verbal empathy is an analogue communication process that is performed parallel to all other verbal<br />
behaviors. Therefore, it does not make sense to define it as a so-called disjunct behavior category.<br />
In the 5-minute observation periods, the coach is able to show up to three different behaviors. Therefore,<br />
we did not try to design disjunct behavior ratings or to force the observers to choose only one<br />
factor to be rated. (Exceptions are the PR and SR ratings, see below.) As a methodological consequence,<br />
it is necessary to examine the correlations between the ratings and to make sure that they are<br />
not too high. In addition, the correlations are theoretically very interesting.<br />
Table 1 shows the correlations between the success factors in our study. They range from r= -0.64 to<br />
r= 0.89. The majority of the correlations are low and not significantly different from zero correlations.<br />
Siegfried Greif, Frank Schmidt and André Thamm 27
The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />
Table 1: Correlations between success factor ratings based on version 3 (*p>.05; **p>.01)<br />
ES1 ES2 ES3 AR1 AC2 RP1 RP1+ RS1 RS1+ GC1 RA ST1<br />
ES1 –<br />
ES2<br />
–<br />
.56*<br />
ES3<br />
.10 –<br />
.51*<br />
AR1 .36 -.19 .41 –<br />
AC2 -.03 -.43 .09<br />
.47*<br />
–<br />
RP1 .09 .38 -.25 .06 -.53* –<br />
RP1+ .29 ,22 .02 .38 -.13 .74** –<br />
SR1 -.18 -.18 .45 .09 .40 -.64** -.32 –<br />
SR1+ .25 .03 .57* .18 .30 -.51* -.13 .81** –<br />
GC1 .30 .16 -.14 .26 -.31 .46* .55* .45 .01 –<br />
RA .33 .11 .79** .16 .28 -.56* -.23 .66** .65** -.37 –<br />
ST1 .26 .00 .71** .03 .27 -.60* -.43 .46* .51* -.34 .89*<br />
*<br />
–<br />
EP1 .41 .25 .65** .03 -.14 -.10 -.03 .17 .53* .24 .47* .53*<br />
As mentioned above, the correlations between RP1 and RP1+ and also SR1 and SR1+ are influenced<br />
by a logical dependency (the second ratings can be observed only if the first are present). Therefore<br />
they are artificially high and cannot be interpreted. In the table they are marked with grey shading.<br />
Resource activation (RA) and the support of transfer (ST1) correlate by r= 0.89. This is definitively<br />
too high. As mentioned above, in this version 4 of our manual, the differences between the ratings<br />
have been sharpened, and the recommendation has been put forth to train the raters more rigorously so<br />
that they are able to distinguish better between the activation of the client`s resources (RA) through<br />
identifying them, reflecting on them, and supporting how to overcome difficulties in the transfer of<br />
intended actions into practice by systematic interventions (ST1), e.g. shadowing.<br />
A similar problem in part is shown by the correlation of r= 0.79 between emotional support (ES3) and<br />
resource activation (RA) and also r= 0.71 with support of tranfer (ST1). Both constructs have some<br />
similarity in their definitions. Therefore we have sharpened the descriptions and examples of ES3. But<br />
here the reason for the high correlation may also be that for resource activation and support of transfer,<br />
emotional support is a necessary companion. Emotional support (ES3) correlates with ratings of these<br />
factors, which demand sensitive interventions by the coach. Future analyses of interaction may be<br />
helpful to clarify the problem.<br />
It is plausible that the esteem and emotional support items (ES1, ES2 and ES3) form a correlating<br />
cluster. Also expected is the finding that problem-reflection and self-reflection correlate negatively.<br />
They have been designed as disjunct ratings. The negative correlation shows that the raters are able to<br />
distinguish the constructs. Two further correlating pairs of factors that we hoped to find are the correlations<br />
(from r= 0.46 to 0.66) between the two self-reflection-ratings (SR1 and SR1+) with resource<br />
activation (RA) and support of transfer (ST1). This may show that self-reflection is necessary for activating<br />
strengths and abilities of the client and supporting the transfer of the planned behavior into<br />
practice. In contrast to this, reflection on problems not directlty related to the self (PR1 and PR1+)<br />
correlate negatively, but postitively with goal clarification (GC).<br />
We have to be careful not to overrate the correlations found in our first study. Future replication studies<br />
with similar and different samples are necessary, before we interprete them as reliable results and<br />
draw expansive theoretical conclusions. But since the results largely confirm our theoretical assumptions<br />
(Greif, 2008) and the correlations between the factors and the outcome criteria are astonishingly<br />
high, future research may be encouraged. Also professionals may discuss about possible practical conclusions<br />
in observing and improving the factors as a means improving coaching outcomes.<br />
References<br />
Behrendt, P. (2004). Wirkfaktoren im <strong>Coaching</strong>. Diplom thesis, Institute of Psychology, University of<br />
Freiburg, Germany.<br />
Siegfried Greif, Frank Schmidt and André Thamm 28
The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />
Borsum, F. (2008). Vergleich von <strong>Coaching</strong> und Psychotherapie – Vergleich und Abgrenzung von<br />
Wirkfaktoren und Inhalten. Diploma thesis, Work and Psychology Unit, University of Osnabrück,<br />
Germany.<br />
Dörner, D. (1979). Problemlösen als Informationsverarbeitung. Stuttgart: Kohlhammer.<br />
Flückiger, C., Caspar, F., Holtforth, M. G. & Willutzki, U. (2009). Working with patients' strengths: A<br />
microprocess approach. Psychotherapy Research, 19(2), 213-223.<br />
Flückiger, C. & Holtforth, M. G. (2008). Focusing the therapist's attention on the patient's strengths: A<br />
preliminary study to foster a mechanism of change in outpatient psychotherapy. Journal of<br />
Clinical Psychology, 64(7), 876-890.<br />
Gassmann, D. & Grawe, K. (2006). General Change Mechanisms: The Relation Between Problem<br />
Activation and Resource Activation in <strong>Success</strong>ful and Unsuccessful Therapeutic Interactions.<br />
Clinical Psychology & Psychotherapy, 13(1), 1-11.<br />
Geißler, H. (2009). Die inhaltsanalytische „Vermessung“ von <strong>Coaching</strong>prozessen. In B. Birgmeier<br />
(Hrsg.), <strong>Coaching</strong>wissen - Denn sie wissen icht, was sie tun? (pp. 93-128). Wiesbaden: Verlag<br />
für Sozialwissenschaften.<br />
Grant, A. M. (2006). An Integrative Goal-Focused Approach to Executive <strong>Coaching</strong>. In D. R. Stober,<br />
Grant, A. M. (Hrsg.), Evidence Based <strong>Coaching</strong> Handbook: Putting Best Practices to Work<br />
for Your Clients (S. 153-192). New York: John Wiley & Sons.<br />
Grawe, K. (2000). Psychologische Therapie (2. korr. Aufl.). Göttingen: Hogrefe.<br />
Grawe, K. (2006). Neuropsychotherapy: How the Neurosciences Inform Effective Psychotherapy:<br />
Erlbaum.<br />
Grawe, K., Donati, R. & Bernauer, F. (1994). Psychotherapie im Wandel: von der Konfession zur Profession.<br />
Hogrefe: Göttingen.<br />
Grawe, K., Regli, D. & Schmalbach, S. (1994). Cubus-Analyse. Bern: University of Bern.<br />
Greene, J. & Grant, A. (2003). Solution Focused <strong>Coaching</strong>. Harlow, UK: Pearson Education Limited.<br />
Greif, S. (2008). <strong>Coaching</strong> und ergebnisorientierte Selbstreflexion. Theorie, Forschung und Praxis<br />
des Einzel- und Gruppencoachings (<strong>Coaching</strong> and result-oriented self-reflection. Theory, research<br />
and practice of individual and team coaching). Göttingen: Hogrefe.<br />
Greif, S. (in prep.). Analysis of Subjektive Theories on <strong>Success</strong> and Failure of <strong>Coaching</strong> by the<br />
Change Explorer Method. Osnabrueck: University of Osnabrueck, Work and Organizational<br />
Psychology Unit.<br />
Greif, S. (in preparation). Analysis of Subjektive Theories on <strong>Success</strong> and Failure of <strong>Coaching</strong> by the<br />
Change Explorer Method. Osnabrueck: University of Osnabrueck, Work and Organizational<br />
Psychology Unit.<br />
Kaufel, S., Scherer, S., Scherm, M. & Sauer, M. (2006). Führungbegleitung in der Bundeswehr -<br />
<strong>Coaching</strong> für militärische Führungskräfte. In W. Backhausen & J.-P. Thommsen (Hrsg.),<br />
<strong>Coaching</strong>. Durch systemisches Denken zur innovativen Personalentwicklung (pp. 419-438).<br />
Wiesbaden: Gabler.<br />
Kauffmann, C. (2006). Positive Psychology: The Science of the Heart of <strong>Coaching</strong>. In D. R. Stober &<br />
A. M. Grant (Hrsg.), Evidence Based <strong>Coaching</strong> Handbook: Putting Best Practices to Work for<br />
Your Clients (pp. 219-254). New York: John Wiley & Sons.<br />
Kauffmann, C., Bonniwell, I. & Silberman, J. (<strong>2010</strong>). The Positive Psychology Approach to<br />
<strong>Coaching</strong>. In E. Cox, T. Bachkirova & D. A. Clutterbuck (Hrsg.), The Complete Handbook of <strong>Coaching</strong><br />
(pp. 158-171). London: Sage.<br />
Kuhl, J. (2001). Motivation und Persönlichkeit. Interaktionen psychischer Systeme. Göttingen: Hogrefe.<br />
Mäthner, E., Jansen, A. & Bachmann, T. (2005). Wirksamkeit und Wirkung von <strong>Coaching</strong>. In C.<br />
Rauen (Hrsg.), Handbuch <strong>Coaching</strong> (pp. 55-76). Göttingen: Hogrefe.<br />
Miller, D. T. & Ross, M. (1975). Self-Serving Bias in the Attribution of Causality. Fact or Fiction?<br />
Psychological Bulletin, 82, 213-225.<br />
O`Connell, B. & Palmer, S. (2007). Solution-focused coaching. In S. Palmer & A. Whybrow (Hrsg.),<br />
The Handbook of <strong>Coaching</strong> Psychology: A Guide for Practitioners (S. 278-292). London:<br />
Brunner-Routledge.<br />
Rauen, C. (Hrsg.). (2004). <strong>Coaching</strong>-Tools. Erfolgreiche Coaches präsentieren 60 Interventionstechniken<br />
aus ihrer <strong>Coaching</strong>-Praxis (6st ed.). Bonn: managerSeminare.<br />
Siegfried Greif, Frank Schmidt and André Thamm 29
The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />
Rauen, C. (Hrsg.). (2007). <strong>Coaching</strong>-Tools II. Erfolgreiche Coaches präsentieren Interventionstechniken<br />
aus ihrer <strong>Coaching</strong>-Praxis. Bonn: managerSeminare.<br />
Rogers, C. R. (1980). A Way of Being. Boston: Houghton Mifflin.<br />
Schmidt, F. & Thamm, A. (2008). Wirkungen und Wirkfaktoren im <strong>Coaching</strong>. – Verringerung von<br />
Prokrastination und Optimierug des Lernverhaltens bei Studierenden. Diplom thesis, Work<br />
and Organizational Psychology Unit, University of Osnabrück, Germany.<br />
Schmidt, F., Thamm, A. & Greif, S. (2008). <strong>Rating</strong>-Manual der Wirkfaktoren im <strong>Coaching</strong>. In F.<br />
Schmidt & A. Thamm (Hrsg.), Wirkungen und Wirkfaktoren im <strong>Coaching</strong>. – Verringerung<br />
von Prokrastination und Optimierug des Lernverhaltens bei Studierenden - Appendix (pp. 78-<br />
94). Osnabrück: University of Osnabrück, Diploma thesis.<br />
Shazer, S. d. (1988). Clues: Investigating Solutions in Brief Therapy. New York: Norton.<br />
Stober, D. R. (2006). <strong>Coaching</strong> from the Humanistic Perspective. In D. R. Stober & A. M. Grant<br />
(Hrsg.), Evidence Based <strong>Coaching</strong> Handbook: Putting Best Practices to Work for Your Clients<br />
(pp. 17-50). New York: Wiley and Sons.<br />
Wirtz, M. & Caspar, F. (2002). Beurteilerübereintimmung und Beurteilerreliabilität. Göttingen: Hogrefe.<br />
Zöller, R. (2008). Evaluation von Erfolgskriterien und Erfolgsfaktoren im <strong>Coaching</strong>. – Ein Vergleich<br />
der Perspektiven von Coaches und Klienten. Diplom thesis, Work and Organizational Psychology<br />
Unit, University of Osnabrück, Germany.<br />
Siegfried Greif, Frank Schmidt and André Thamm 30