24.12.2012 Views

Rating Of Coaching Success Factors Version4-May 2010

Rating Of Coaching Success Factors Version4-May 2010

Rating Of Coaching Success Factors Version4-May 2010

SHOW MORE
SHOW LESS

You also want an ePaper? Increase the reach of your titles

YUMPU automatically turns print PDFs into web optimized ePapers that Google loves.

The <strong>Rating</strong> of Eight <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />

- Observation Manual -<br />

Siegfried Greif<br />

Frank Schmidt<br />

André Thamm<br />

Version 4<br />

<strong>May</strong> <strong>2010</strong><br />

We are very grateful to Jennifer Boeun Lee, MA (Harvard), Research Assistant, Evidence Based Psychology,<br />

Cambridge (MA) for her editorial support of the English translation.


The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />

1 Introduction: Hypothetical success factors that predict<br />

coaching outcomes<br />

Can hypothesized success factors, determined via behavioral observation of coaching sessions, predict<br />

coaching outcomes? Behrendt (2004) first studied possible success factors in coaching by observing<br />

the behavior of coaches in coaching sessions. Using success factors and methods of behavior observation<br />

adopted from the psychotherapeutic outcome research of Grawe, Donati and Bernauer (1994),<br />

Behrendt recorded a total of 40 coaching sessions with 8 managers (5 sessions each) and had observers<br />

rate the behavior of the coaches in 10-minute intervals according to the Cubus-Analysis scales (Regli<br />

& Schmalenbach, 1994)). Significant correlations were discovered between resource activation, one<br />

of the success factors, and the satisfaction of the managers who were engaged in coaching (r = 0.56).<br />

Correlations were also found between resource activation and goal attainment rated by employees<br />

after the coached managers conducted performance interviews (r = 0.43). (1994). A summary of the<br />

success factors will be given in the following section.<br />

Behrend’s work encouraged us to first design observable success factors that relate more closely to the<br />

field of coaching and our integrative theory of result-oriented coaching (Greif, 2008), and then to begin<br />

similar coaching behavior observation studies (Schmidt & Thamm, 2008). We felt the work would<br />

be invaluable, as it would: 1) validate observable success factors that can predict coaching outcomes,<br />

2) give coaching students feedback during their coaching education, 3) assess the work of professional<br />

coaches and aide in their development, and 4) improve the quality and outcomes of professional<br />

coaching.<br />

We will begin with an overview of the success factors and then move on to the results of the initial<br />

pilot studies (English publications are in preparation). Finally, eight basic success factors in coaching<br />

will be defined together with the rating scales and their behavioral anchors.<br />

1.1 General change principles of psychotherapy<br />

In their meta-analysis of success factors in psychotherapy outcome research, Grawe et al. (1994) studied<br />

general change principles during the therapeutic process that predict the outcome of therapy independently<br />

of the therapeutic schools. Gassman and Grawe (2006) differentiated between five change<br />

principles:<br />

(1) The therapeutic bond (warm, empathetic and congruent behavior of the therapist, and openness<br />

and trust of the patient, resulting in a trusting relationship between the therapist and patient);<br />

(2) Problem actualization (imagination of the problems and emotions experienced by the patient);<br />

(3) Resource activation (recognition of the strengths and abilities of the patient, as well as the level<br />

of support in the patient’s environment);<br />

(4) Motivational clarification (reflection on the motives and goals of the client, including awareness<br />

of behavioral consequences, clarification of the client’s conscious and unconscious goals<br />

as well as the environmental goals, and a continuous reflection on the relationships between<br />

these goals); and<br />

(5) Mastery (action-oriented coping, transfer into the practice of concrete measures, and support<br />

in reaching the goals).<br />

Therapeutic bond. Based on this research, Grawe (2000, 2006) developed an integrative neuropsychological<br />

theory of change principles that relate to successful and unsuccessful therapeutic interactions.<br />

Such predictors are called “success factors”. The findings showed that resource activation is the<br />

strongest predictor of therapy outcomes (Gassmann & Grawe, 2006); as a result, it forms the central<br />

change principle in Grawe’s theory. Similar to the solution-focused approaches in the fields of therapy,<br />

coaching (Greene & Grant, 2003; O`Connell & Palmer, 2007; Shazer, 1988), and Positive Psychology<br />

(Kauffmann, 2006; Kauffmann, Bonniwell & Silberman, <strong>2010</strong>), the focus of the intervention<br />

changes from deficit-oriented intervention to the activation of the intra- and interpersonal resources of<br />

the client (e.g., their intellectual abilities, specific skills, motivational goals, or level of support from<br />

family members) (Gassmann & Grawe, 2006, p. 2). Grawe (2006) theorized that the therapeutic bond<br />

Siegfried Greif, Frank Schmidt and André Thamm 2


The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />

– the classic variable describing an understanding and supporting relationship between therapist/coach<br />

and client (Rogers, 1980, p. 22 f.; Stober, 2006) – forms the basis of resource activation, and increases<br />

the chances of a positive therapeutic outcome. The client must trust in the therapist’s understanding,<br />

support, and expertise before being able to grow their own potential and develop hope. Once trust is<br />

developed, the client is able to recognize and use their strengths and abilities, as well as the support of<br />

other people or technical resources in their environment, to pursue goals.<br />

Problem actualization refers to the re-experiencing and intensively processing of emotions during the<br />

therapeutic process. Patients must learn that they are “more than the sum of their problems.” To be<br />

able to change, patients must face emotions associated with their problems that have been previously<br />

avoided (Gassmann & Grawe, 2006, p. 2). The processes of transference and countertransference are<br />

examples in psychoanalysis and other psychodynamic approaches of problem actualization. Exposure<br />

therapy, an intervention practiced in behavior therapy, is another common example of problem activation.<br />

Motivational clarification includes any intervention that supports the client in reflecting their personal<br />

or intrinsic motives and goals. It also includes actions of the therapist that help the client to clarify<br />

external expectations and relationships between goals and consequences.<br />

Mastery focuses on the client’s ability to cope with the problem successfully. Examples of mastery<br />

include the actions necessary to solve the problem, as well as the ability to successfully perform these<br />

actions and manage the problem.<br />

Problem actualization and resource activation complement one another. While the actualization of the<br />

problem helps the client to clarify what to change, resource activation focuses on how this is possible<br />

(Grawe, 2000, p.99). These two success factors are central to the therapy and have largely to do with<br />

the changes clients undergo while in therapy (Gassmann & Grawe, 2006, p. 2). Motivational clarification<br />

and mastery are complementary as well. Clients take action that can be potentially detrimental to<br />

their future without motivational clarification.<br />

1.2 <strong>Success</strong> factors of coaching<br />

The observation study of Behrendt (2004)<br />

The success factors found by Grawe et al. (1994) are very general and applicable to other types of<br />

counseling interventions. Grawe’s success factors are also particularly applicable to the field of coaching.<br />

For his observation study, Behrendt (2004) adapted Cubus-Analysis, a rating instrument created<br />

by Grawe, Regli and Schmalenbach (1994). In his original manual, Behrendt only made a few changes<br />

to the rating instrument such as replacing the words “therapist” with “coach” and “therapy” with<br />

“coaching.” The results of Behrendt’s study show that only resource activation correlates significantly<br />

with outcome criteria. Upon closer inspection of the scale definitions in the manual, we decided that<br />

the success factors needed to be redfined and redesigned to be more applicable to the field of coaching.<br />

A significant issue with Behrendt’s early rating instrument is that the client resources and coach empathy<br />

ratings were combined into one common “resources” score; in later publications, Grawe and his<br />

colleagues distinguish the therapeutic relationship from resource activation (cf. Gassmann & Grawe,<br />

2006). We reanalyzed the data published by Behrendt (2004) and found (Greif, 2008, p. 267 f.) that<br />

the coach rating scales referring to exhibiting esteem/ appreciation and emotional support (empathy<br />

factor) towards the client consistently correlates higher with client satisfaction ratings (r= 0.51 - 0.55)<br />

than with client personal resource activation ratings (r= 0.39 – 0.46). Reciprocally, in the performance<br />

interview by the interview partner of the client, the correlations between client resource activation<br />

ratings and behavior-oriented ratings of goal attainment are higher (r= 0.63) than those between the<br />

empathy and the goal-attainment ratings (r= 0.43 – 0.44). According to our theoretical assumptions<br />

(Greif, 2008), empathy and emotional support shown by the coach result primarily in ratings of client<br />

satisfaction and similar subjective evaluations of the clients. In contrast, client resource activation is<br />

assumed to have a stronger influence on criteria related to behavior changes.<br />

General and specific factors<br />

Following a constructionist position, “success of coaching” is a subjective social construct. The definition<br />

of success depends on the perspective of the persons or institution that perform the evaluation, the<br />

Siegfried Greif, Frank Schmidt and André Thamm 3


The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />

quantitative or qualitative criteria selected, and their weighting. Different persons can evaluate the<br />

same coaching case completely differently. Client satisfaction and goal attainment are general evaluation<br />

criteria applied by both professional coaches and their clients. However, these criteria are not<br />

always reliable due to the unique characteristics (e.g., reasons, problems, goals and attributes) of each<br />

of the interacting persons. Therefore, coaching cases are often evaluated by very specific evaluation<br />

criteria. Criterion also have the potential of being inapplicable to other clients. An example of this<br />

would be a client who asks his coach to support him in creating a literature circle as a kind of balance<br />

to his stress at work. This is a very specific goal and as a result the methods used by the coach and<br />

client, as well as the process leading to a successful outcome, will also be very specific.<br />

Subjective theories of success and failure of coaching<br />

Through semi-structured expert interviews, we analyzed the subjective theories of coaches and their<br />

clients on success and success factors of concrete coaching cases (Greif, 2008, p. 239 ff.; in preparation;<br />

Schmidt & Thamm, 2008; Zöller, 2008). The Change Explorer, an instrument that uses structure<br />

analysis techniques with cards to visualize the grounded individual theories of coaching success factors<br />

and outcomes, was used in the analysis. Our findings support the theory that both clients and<br />

coaches often apply the general evaluation criteria of client satisfaction and goal attainment in their<br />

coaching sessions, and that the agreement between their ratings is high. Also, both ascribe positive<br />

changes primarily to intervention methods used in coaching and to the initiative and problem solving<br />

competences of the clients. Differences between coach and client were found in the specific outcome<br />

criteria, especially in the explanation of specific negative results. For example, coaches attributed<br />

negative results to the characteristics of the client more frequently (e.g., low change readiness/motivation<br />

and low self-reflexivity), while clients attributed them to low coach competencies,<br />

supporting expectation of self-serving bias (Miller & Ross, 1975). In contrast, positive results are<br />

credited more often to their own competencies and abilities (Zöller, 2008). Another significant finding<br />

is that the quality of the relationship between coach and client, as well as the quality of the process<br />

(e.g.,“I felt that it was a good process.”) have been mentioned more frequently by the coach. These<br />

qualities can be perceived as success factors or outcomes, depending on the case. This may indicate<br />

that both trusting relationships and good coaching processes are causes and effects that must be analyzed<br />

systemically by circular models.<br />

Limitations of behavior ratings<br />

The focus of this paper is the observation of success factors that can be consistently rated by trained<br />

observers. It is nearly impossible to reach rating consistency among observers when rating scales are<br />

complex and defined behaviors that are nearly indistinguishable. The success factors presented below,<br />

therefore, are by no means exhaustive. For a more comprehensive study, we suggest combining the<br />

rating scales used with interviews and questionnaire scales to assess both qualitative and quantitative<br />

data (e.g., applying our Change Explorer method) (Greif, in prep.). Additionally, there is the possibility<br />

of analyzing the transcripts of the interviews, or observing interactions in coaching sessions for<br />

case studies using qualitative methods.<br />

The success of coaching may always remain a mystery, at least in part, and may never be fully explained<br />

by scientifically defined factors or qualitative analyses. Some colleagues, scientists and practitioners<br />

argue that the success of coaching cannot be studied. But mysteries have always challenged not<br />

only scientists, but also curious practitioners who want to know and understand as far as possible how<br />

and why coaching works. If we reject evaluating coaching processes and outcomes by scientifically<br />

controlled qualitative and/or quantitative methods, we cannot argue against those who do not accept<br />

any kind of scientific evaluation and discussion of their performance. Process research is of prime<br />

importance for the future of coaching, as it will increase both the efficacy of coaching and the opportunity<br />

to support coaching clients.<br />

A list of eight observable success factors<br />

Inspired by Grawe and his scholars we began our research by defining observable behavior patterns of<br />

the coaches. We defined the factors and ratings that are expected to show a positive relationship to<br />

coaching outcomes. We have also begun developing assumptions and analyzig client behavior and<br />

interactions. In the future we plan to study individual and specific processes using theory-based case<br />

studies and qualitative methods.<br />

Siegfried Greif, Frank Schmidt and André Thamm 4


The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />

In our first observation study (Schmidt & Thamm, 2008) we distinguish between eight elementary<br />

success factors that describe observable patterns of coaching behavior:<br />

(1) Esteem and emotional support;<br />

(2) Result-oriented problem-reflection;<br />

(3) Result-oriented self-reflection;<br />

(4) Affect reflection and calibration;<br />

(5) Clarification of goals;<br />

(6) Activation of client resources;<br />

(7) Support of transfer into practice; and<br />

(8) Evaluation of process.<br />

(1) Esteem and emotional support (ES)<br />

Esteem and the activation of individual or external resources are separated in our observation system.<br />

Since our focus is on coach behavior, ES refers to the coach’s observable verbal and non-verbal expression<br />

of esteem and emotional support towards the client.<br />

(2) Problem-reflection (PR)<br />

Scales were designed in reference to the facilitation of result-oriented problem- and self-reflection, as<br />

these are central to our theory (Greif, 2008).<br />

According to Dörner (1979) a problem is typically defined by three components: 1) a given, undesired<br />

starting state (as-is state) or situation (S), 2) a desired target state or goal (G), and 3) an unknown<br />

means of how to reach the goal (M). A simple example of this is a manager who suffers from intense<br />

stress at work (S) and wants to reduce the stress (G), but does not know how (M). Dörner’s definition<br />

is broad and embraces all imaginable situations or states that a coaching client would want to change.<br />

Clients seek professional coaching for a number of reasons: customer losses, economic hardships,<br />

implementation of new technologies, organizational changes, stress, time management, performance<br />

improvement, mastery of management techniques, conflict management, and understanding of individual<br />

and organizational behavior (e.g., “Why did the employee react aggressively to the customer?”,<br />

“What caused the team conflict?”, “Why were the organizational changes ineffective?”). Excluded<br />

from this group of reflections is self-reflection (defined below).<br />

Result-oriented problem-reflection occurs when the coach examines the problem with the coachee and<br />

then develops new insights, clarifies how components of the problem are related (e.g., key players or<br />

key interactions), and discusses possible consequences. These reflections are considered resultoriented<br />

when the client derives conclusions for future reflections or actions. The classic technique<br />

used by coaches to encourage reflection is to ask open questions. Other techniques include moderation<br />

cards to describe the problem, and role play to simulate interactions.<br />

(3) Self-reflection (SR)<br />

In interviews conducted with 89 coaches and 74 clients, Mäthner, Jansen and Bachmann (2005) found<br />

that the majority of clients (62%) and coaches (71%) felt there was an increase in self-reflection as a<br />

result of the coaching. Examples of this include answers such as, “More awareness of myself and my<br />

behavior,” and, “Clarification of my personal roles and priorities.” Therefore, they concluded that the<br />

facilitation of self-reflection is one of the major outcomes of coaching. Other evidence of this increase<br />

in self-reflection includes reflection reportedly on personal values, particular interests, abilities and<br />

competencies, strengths and weaknesses, typical emotional reactions, habitual behavior and a comparison<br />

of the real and ideal self-concept. Open self-reflection requires strict confidentiality and a<br />

counselor who follows-up with the interests and goals of the client. In our coaching theory (Greif,<br />

2008), facilitation of intensive self-reflection is one of the core functions and competencies of coaching.<br />

It distinguishes coaching from other types of counseling that are less invasive. In this respect,<br />

coaching can be compared to psychotherapy.<br />

Siegfried Greif, Frank Schmidt and André Thamm 5


The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />

To be precise, self-reflection is a special subgroup of problem-reflection with the only difference being<br />

that self-reflection focuses on aspects very close to the identity of the client. Our coaching theory<br />

integrates assumptions and findings from self-concept research and Kuhl’s (2001) neuropsychological<br />

motivation and personality theory, which argues that the experience of a person with the self is part of<br />

a special type of intuitive or implicit memory (the so-called “extension” memory).<br />

We suspect that the phases of self-reflection do not occur frequently in coaching sessions, and are not<br />

standard to coaching in general. This may be due to the difficulty of facilitating systematic selfreflection<br />

since it is rooted in the implicit memory, organized in an associative way, and based on nonverbal<br />

representations. Managers who tend to be more action-oriented are particularly skeptical about<br />

the usefulness of self-reflection; we have found them less likely to buy into the benefits of selfreflection,<br />

especially in cases where their habitual behavior causes them to act prematurely and without<br />

regard to the negative consequences of their behavior. Professional coaches, similar to psychotherapists,<br />

know how to engage their coachees in intensive and systematic self-reflection when useful.<br />

Typical methods used are circular questions and visualization, or imagination techniques. As with<br />

problem-reflection, we have a result-oriented focus for self-reflection. Some clients, especially with a<br />

high state of orientation (Kuhl, 2001), tend to ruminate with long, associative, and straining selfreflection<br />

without any results. In these situations, it is necessary for coaches to help clients learn how<br />

to stop negative self-reflection and to switch to reflection with concrete positive results. This is a significant<br />

concept of Positive Psychology (Kauffmann, 2006; Kauffmann et al., <strong>2010</strong>).<br />

(4) Affect reflection and calibration (AR)<br />

Affect reflection and calibration relate to the activation, re-experiencing, and changing of affect and<br />

emotions in coaching sessions by techniques used in Grawe’s (2006) theory of psychotherapy (2006).<br />

It is our assumption that systematic, intensive psychotherapeutic techniques are rarely applied to alter<br />

affect and emotions in standard business coaching. In coaching sessions, emotions usually function as<br />

a means of allowing the client “to let off steam” after an emotionally laden situation, or as a way for<br />

the client to describe and analyze their own emotions and reactions or those of the environment. The<br />

coach may train the client to manage negative affects and emotions in order to improve psychological<br />

well-being.<br />

Following Kuhl’s theory (2001), strong negative and positive emotions restrict access to the self.<br />

Take, for example, a manager whose strong feelings of aggression, first experienced in a prior conflict<br />

situation, are reawakened during a coaching session. The manager is unable to reflect openly on his<br />

behavior and attitude that contributed to the conflict, as is fixated on his emotional state. Open access<br />

to the self will only be possible after the affect state has been reduced to a temperate level. This can<br />

easily be done by asking the manager to clearly describe his emotions. According to Kuhl, this exercise<br />

activates rational cognitions that are antagonistic to strong emotions. The typical observable reaction<br />

of the client is that their emotions are quickly subdued, and that they speak in a different tone.<br />

After this, it is easier for the client to explore and analyze the self. Other techniques, similar to behavior<br />

therapy interventions that result in the pacification of emotions (e.g., relaxation techniques) can be<br />

applied here. Affect reflection and calibration can be considered as a kind of mediating factor that<br />

gives greater access to the self and self-reflections.<br />

(5) Clarification of goals (CG)<br />

Goal clarification is one of the most important coaching tasks that was emphasized early in cognitive<br />

behavioral coaching theory and research (Grant, 2006). The definition of this factor is not identical to<br />

Grawe’s motivational clarification factor. It addresses more concrete definitions and explanations of<br />

the goals and intentions of the client in relation to the expectations of the social and organizational<br />

environment. The analysis of the personal motives in our system belongs to the self-reflection factor,<br />

as it is foundational to goal clarification that is congruent with the self.<br />

As Grant (2006, p. 160) ascertains, it is not always necessary or meaningful to have the client define<br />

goals very precisely (e.g., to follow the SMART criteria). It is adequate for the client, in a contemplative<br />

state, to set abstract or vague goals and to start to develop a broad vision even if unclear.<br />

(6) Activation of the client’s resources (RA)<br />

Siegfried Greif, Frank Schmidt and André Thamm 6


The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />

Resource activation is in direct reference to Grawe’s theory and findings (2006). Like Grawe, we distinguish<br />

between the internal resources of the client (motivational and personality traits, abilities,<br />

competencies, and potentials) and the external resources the client is able to activate (e.g., expert<br />

knowledge, consults, emotional support, concrete help by family and friends, or support from those in<br />

the occupational environment). In summary, we call these personal resources.<br />

(7) Support of transfer into practice (TP)<br />

In our first manual (Schmidt, Thamm & Greif, 2008), we introduced the rating of the transfer of the<br />

intended change into practice as a special, result-oriented rating of resource activation. The corresponding<br />

success factor defined by Grawe (2006) in his theory is called mastery. Mastery addresses:<br />

1) action-oriented coping with the problem and solution development, 2) transfer into the practice of<br />

concrete measures, and 3) support in reaching the goals of the client. We assumed that it is necessary<br />

to focus more quickly and systematically on mastery in shorter result-oriented coaching sessions than<br />

in psychotherapy. However, after reviewing two standard German handbooks of coaching tools by<br />

Rauen (2004; 2007), we found many techniques facilitating analyses/reflection or changes, and some<br />

techniques to evaluate the outcome of the sustained transfer, but not a single technique on how to support<br />

the transfer of the intended change into practice. In our theory and practice we focus more on<br />

procedures and tools for how to support transfer into practice.<br />

A systematic intervention method is a methodical reflection of and preparation for the possible barriers<br />

of transfer. After a trial run of the transfer is conducted, the client is prompted to reflect on the actions<br />

and reactions during the transfer, and if necessary, makes changes to the strategy used. A powerful<br />

tool used during the transfer is the shadowing of the client by the coach in the intended actions or behavior<br />

changes. The coach observes the client during the transfer and gives feedback afterward. Then<br />

the coach and client together analyze concrete transfer barriers and solutions.<br />

Our preference is for telephone shadowing, as it is a more economical tool. In telephone shadowing,<br />

the coach and client have a telephone appointment immediately following the situation where the client<br />

tries to perform the intended actions. From experience, shadowing is an essential benefit of the<br />

intervention. <strong>Of</strong>tentimes many of the client’s intended actions fail due to concrete individual and environmental<br />

barriers that are not predicted by the client or coach. When the client is unable to execute<br />

the intended actions and the trial run is unsuccessful, the client is much less motivated to try again. If a<br />

follow-up coaching meeting does not immediately follow the trial run, the client is more likely to reinterpret<br />

the experience in an unhealthy or ineffective way during the lag time, making it difficult to<br />

motivate the client for the next attempt. The reflection on and activation of resources directly after the<br />

trial coupled with support for the next trial, encourage the client to try again. Normally after an unsuccessful<br />

attempt, the second trial run is more successful. The learning of the client can be very demanding<br />

during this process, and if successful, the client can be successfully equipped with the ability necessary<br />

to overcome typical internal and external barriers without the support of the coach.<br />

The research (Kaufel, Scherer, Scherm & Sauer, 2006) coupled with our practical experience point to<br />

the support of transfer via shadowing to be a very powerful intervention. To further explore this, we<br />

have planned experimental control studies to test the assumption that this factor very strongly contributes<br />

to the behavioral outcome of coaching.<br />

(8) Feedback and evaluations in the process (FP)<br />

A basic principle of result-oriented coaching is to have the client receive useful results in each coaching<br />

session. Examples of useful results are concrete goal-oriented action plans and tasks that the client<br />

can put into practice, important insights that help the client to understand and overcome problems, and<br />

new and valuable knowledge. The more coaching sessions yield useful results, the more coaching, as a<br />

whole will be useful. It is therefore reasonable for coaches to ask their clients for feedback after sessions<br />

to make sure that the client is satisfied with the results. We define this is as a success factor since<br />

it enables the coach to change the method of coaching if the client is unsatisfied. If the client were to<br />

spontaneously give a positive evaluation of the coaching after many coaching sessions, we would also<br />

expect that this is a predictor of the overall positive success rating after the end of the coaching. Therefore,<br />

we rate the observable evaluations – both the coach’s asking for feedback and the client’s spontaneous<br />

positive evaluations.<br />

Siegfried Greif, Frank Schmidt and André Thamm 7


The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />

First results: Correlations of the success factors to outcome criteria<br />

We constructed the observation instrument described below. In our first study, trained observers rated<br />

the success factors of 44 videotapes of coaching sessions (first, middle, and last). The clients were<br />

business and law students, and the coaches were seven graduate psychology students trained with systematic<br />

coaching education and supervision (Schmidt & Thamm, 2008). The ratings were taken at 5minute<br />

intervals. Since some of the frequencies and variances of the quality ratings were low, not all<br />

observer reliabilities were satisfactory (see below for the details). Therefore, it was surprising to find<br />

high and significant correlations between the observed elementary success factors and outcome criteria<br />

that conform to our theoretical assumptions. A short summary of some of these is given in the following<br />

passages. Details of the observer reliabilities and correlations of the factors will be described in the<br />

observer manual.<br />

Significant correlations with outcome criteria:<br />

Non-verbal reinforcement correlates with an emotional clarity scale (0.74), and verbal empathy with<br />

scales assessing the client’s behavior reflection (0.69), goal reflection (0.67), and goal-attainment<br />

satisfaction (0.52). Facilitation of self-reflection is related to the reduction of helplessness (0.56). Resource<br />

activation correlates with goal satisfaction (0.49). As expected, support of transfer predicts<br />

behavior-oriented scale values of effective time management (0.55) and information management<br />

(0.46). But the correlations found between resource activation and information management (0.49),<br />

and reduction of the feeling of listlessness (r=0.50) are only tendentially significant (10% level).<br />

Since both the sample and the variance of the behavior observation scales of the coaches are small,<br />

only high correlations of 0.46 and above are statistically significant. For example, correlations between<br />

resource activation and information management (0.49) and reduction of the feeling of listlessness<br />

(0.50) are only tendentially significant (10% level). Below in the observation manual the correlations<br />

of all scales will be presented.<br />

Contrary to our expectations, non-verbal reinforcement correlates with negative affect (0.53) and facilitation<br />

of self-reflection correlates with demotivation (0.57). Since correlations do not show the<br />

causal direction of influences or time sequences, the first correlation may be a result of the coaches<br />

reinforcing reactions to the clients’ negative affect, and the second of the stimulation of self-reflection<br />

of clients after evidence of their demotivation. Analyses of the interactions between coaches and clients<br />

and their time sequence may help to clarify these results.<br />

Siegfried Greif, Frank Schmidt and André Thamm 8


The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />

Research perspectives<br />

It is promising that the correlations observed in our pilot study are higher than expected. But the sample<br />

of our first study is small and we must be careful not to generalize the results to similar samples of<br />

students. Behavior observation studies are very laborious, but replication of these studies is necessary.<br />

It would potentially be more useful to observe and analyze interactions of professional coaches and<br />

business clients.<br />

The observation of coaching sessions opens a new research field that will add to our understanding of<br />

the coaching process. Beyond using the rating scale of success factors for session observations, the<br />

narrative from each session, obtained through transcription, will prove interesting. Geißler (2009),<br />

based on the communication theory of the German philosopher Jürgen Habermas, analyzed frequencies<br />

of various fundamental types of communication acts in selected coaching cases (one of our cases<br />

was one of these). Examples of Geißler’s coach categories include: shows attention, asks, mirrors,<br />

questions/confronts/critizes, explains, gives feedback and prompts the client to act. Catogories describing<br />

the reactions of the client, initiated by the coach, include: focuses cognitively on a subject, describes<br />

concretely or reflects on the process, the causes or the meaning, evaluates and develops action<br />

intentions. Analysis of ten transcripts of coaching sessions (one of them rated as problematic) found<br />

that the most frequent communication act of the coach with a large range is to ask questions (between<br />

26 and 61% of the acts, and only 19% in the problematic coaching). The clients are frequently<br />

prompted to focus cognitively on a subject (about 24 to 52%, and 16% in the problematic coaching).<br />

Geißler’s study merely intended to describe typical communication acts, not success factors. Therefore,<br />

further information from coaching process research on factors predicting outcomes of coaching<br />

have to be obtained through studies via qualitative methods, quantitative methods (with a focus on<br />

scales), or a combination of both of these. In our future research, in addition to conducting research on<br />

the observation categories described below, we plan to conduct case-studies applying qualitative<br />

methodsand to analyze coaching narratives of highly succesful and not successful coachings (rated<br />

both by clients and coaches).<br />

2 Observation Manual<br />

2.1 Short definitions of the success factors and ratings<br />

The following manual gives the definitions of elementary hypothetical success factors. The list below<br />

gives an overview of all ratings. A description of the rating scales, examples of the different types of<br />

behavior, and the anchors belonging to the factors follow.<br />

ES: Esteem and emotional support<br />

ES1: The coach verbally shows esteem for the client.<br />

ES2: The coach non-verbally shows esteem for the client.<br />

ES3: The coach emotionally supports the client.<br />

PR: Problem-reflection<br />

PR1: The coach engages the client in result-oriented problem-reflection.<br />

PR1+: The coach guides the client to derive concrete solutions from the problem-reflection (only if a<br />

facilitation of problem-reflection is shown).<br />

SR: Self-reflection<br />

SR1: The coach engages the client in result-oriented reflection about the self (e.g., behavior, personal<br />

experiences, needs and motives, strengths and weaknesses, ideal and real self-image).<br />

SR1+: The coach guides the client to derive concrete solutions from the problem-reflection (only if a<br />

facilitation of problem-reflection is shown).<br />

AR: Affect reflection and calibration<br />

AR1: The coach engages the client in reflection about feelings.<br />

Siegfried Greif, Frank Schmidt and André Thamm 9


The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />

AR2: The coach supports pacification and calm and open thinking about the client’s feelings.<br />

CG: Clarification of goals<br />

CG1: The coach supports the client in clarifying goals and concretely defining them.<br />

RA: Resource activation<br />

RA: The coach encourages the client to explore and activate personal internal and external resources,<br />

which support goal attainment.<br />

TP: Support of transfer into practice<br />

TP: The coach supports the client to transfer the planned goal-oriented behavior and measures it into<br />

practice.<br />

FP: Feedback and evaluations in the process<br />

EP1: The coach asks the client to evaluate progress and satisfaction with the coaching, goal attainment,<br />

and/or cooperation.<br />

EP2: The client spontaneously offers feedback regarding the level of satisfaction with the session or<br />

the entire coaching engagement.<br />

2.2 Observation periods<br />

Observation periods are the periods of time where the observers are asked to stop the tape and rate the<br />

behavior in the sequence before stopping independently from rating other periods. The observers must<br />

stop the tape every five minutes and repeat the rating procedure until the end of the recording.<br />

2.3 <strong>Rating</strong> scales<br />

The attributes used to define the success factors are important for the observers to know. Following<br />

Flückinger’s behavior rating of resource activation in psychotherapy (Flückiger, Caspar, Holtforth &<br />

Willutzki, 2009; Flückiger & Holtforth, 2008), we rate the behavior of the coach by the following<br />

quality ratings:<br />

<strong>Rating</strong> Definition<br />

-1 Negative evaluation.<br />

0 Behavior not shown.<br />

1 Behavior is shown only to some extent; the quality is below standard coaching<br />

behavior.<br />

2 Standard of coaching behavior is met; improvement is possible.<br />

3 Ideal behavior; best practice; improvement seems impossible.<br />

The quality ratings of the different factors have been constructed by rules comparable to the ones<br />

above. A basic assumption here is that there is a standard coaching behavior that contributes to the<br />

effectiveness of the coaching, but that the behavior partly has to be adapted to the situation. These<br />

quality ratings are merely meant to give an orientation. The observers are instructed to use their intuitive<br />

impression to classify behavior that appears to be maladaptive or disproportional. Therefore, the<br />

examples given below used to define the rating levels are not conclusive.<br />

After the observation of each 5-minute period, the observers are expected to compare the behavior<br />

with the success factors, scale levels, and examples. The method is to choose the rating that is most<br />

similar to the descriptions in the manuals.<br />

2.4 Definitions and behavior anchors of the success factors<br />

The following definitions are used for observer training. The training begins with an introduction and<br />

description of the theoretical background. Then, observers are taken through the success factors and<br />

Siegfried Greif, Frank Schmidt and André Thamm 10


The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />

quality rating levels. These must be explained as concretely as possible through examples, (e.g., literal<br />

citations, short role-playing sequences of coach-client interactions, or demonstration of video clips<br />

(excluding the coaching sessions that are to be rated). In order to reach high agreement between raters<br />

and observer reliability, the most important phase of the training is when observers rate 5-minute periods<br />

of video or audiotape recordings of all factors and rating levels and then receive feedback on their<br />

ratings. The training should not be finished before the observer reaches a very high level of correct<br />

ratings (above 95%). Raters who do not reach this level should not be deployed – it is recommended to<br />

repeat the training and feedback phase with all observers after two days of the practice phase or after<br />

an interruption of more than two days.<br />

Important notes: With intention, the definitions of the success factors refer to rather complex theoretical<br />

concepts. To reach satisfactory agreement between the observers, it is necessary that they develop<br />

a clear understanding of the theoretical meaning of the factors, their differences and the quality<br />

rating levels. The raters need a concrete, convincing and reliable model and many concrete examples.<br />

We recommend that the best expert of the system be selected to be the trainer. She/he needs a great<br />

deal of experience, persuasive power and self-confidence, since she/he has to decide which ratings are<br />

correct. Raters who do not like to rate the behaviour as demanded by the manual and the trainer will<br />

never reach high agreement and reliability rates.<br />

Please contact us if you need support (sgreif@uos.de). We will answer your mail as soon as possible,<br />

and can arrange a Skype teleconference and help you with concrete suggestions based on our experience!<br />

The following descriptions start with a general definition of the factor and the individual ratings (between<br />

negative one and three for each factor). The next passage gives examples of observable behaviors<br />

and specifies what the observer should pay attention to, or how the factor is distinguishable from<br />

similar or related factors. Following this, a description of the quality rating levels and examples of<br />

behavior anchors are given. At the end of each desription, a table shows the intraclass coefficients<br />

(where possible), the mean and standard deviation of the quality rating, the percentages of agreement<br />

between raters, and significant correlations between the qualtity ratings or frequencies of positive observations<br />

of the factors with outcome criteria. All data below and the correlations described above are<br />

based on version 3 of the manual (Schmidt & Thamm, 2008).<br />

The first intraclass coefficients in the tables show the reliability of a “mean rater”, while the second<br />

shows the reliability of the “mean ratings” (normally higher). Most of the reliabilities compared to<br />

other studies can be classified as mean values – only some reliabilities are lower. According to Wirtz<br />

and Caspar (2002), low reliabilities may be tolerated if, as in our case, the variances of the ratings are<br />

low. It is remarkable that instead of their low frequencies and variances, some of the ratings correlate<br />

rather high with outcome criteria. Therefore they have not been completely eliminated from our study.<br />

It would be beneficial to improve the reliabilities by developing better definitions and/or observer<br />

training.<br />

The definitions of the present version 4 of the manual include minor changes in order to sharpen the<br />

definitions of the ratings with low reliabilities and high correlations. The changes will be mentioned<br />

below.<br />

ES: Esteem and emotional support<br />

This success factor describes how much the coach shows empathy and esteem for the client.<br />

ES1: The coach verbally shows esteem for the client.<br />

Verbal remarks made towards the client are observed.<br />

Observable behavior:<br />

The coach:<br />

• Produces an agreeable/warm-hearted atmosphere (e.g., pays attention, tunes in, uses small talk,<br />

asks sensitive questions).<br />

Siegfried Greif, Frank Schmidt and André Thamm 11


The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />

• Verbally expresses a positive attitude towards the client (e.g., “I am happy to see you,” “It is<br />

wonderful that you…,” “It is admirable that you…”).<br />

• Shows interest in the client beyond the subject of coaching (e.g., hobbies, important episodes<br />

in her/his life).<br />

• Takes up client remarks, mirrors or paraphrases what the client expresses (e.g. repeats important<br />

sentences), or summarizes statements to show zhat she/he listens attentively.<br />

• Positively reinforces statements through methods such as approval and accentuation (e.g.<br />

“This is a very important point that you have mentioned here!”).<br />

• Adapts language and wording to the language of the client, and avoids unnecessary technical<br />

terms. The coach speaks in a way that is easy to understand.<br />

Remarks:<br />

Esteem may not be formulated in a circumstantial or exaggerated way. The overall impression must be<br />

credible in the view of the rater.<br />

Esteem and negative feedback or criticism:<br />

Negative feedback and criticism must not be classified as negative or of showing low esteem. Positive<br />

feedback and an appreciation of the strengths of the client should be accounted for, along with how the<br />

negative feedback is communicated.<br />

Delineation: Evaluations of client feelings are not classified as esteem behavior. They must be coded<br />

as affect reflection (AR).<br />

Examples of behavioral anchors:<br />

-1: The coach expresses skepticism, a negative attitude, and/or indifference towards the client, and formulates<br />

feedback circumstantially or with exaggeration. The coach lacks sincerity or credibility.<br />

0: The coach shows none of the standard attributes or behavior examples mentioned in the description of<br />

the factor, does not follow-up with remarks made by the client, and remains reserved in speech (e.g., uses<br />

abstract formulations without warmth or emotional adjectives, or speaks impersonally without addressing<br />

the client).<br />

1: The coach shows some of the behaviors defined by the standard attributes (e.g., takes up the client’s<br />

remarks in a cursory or selective manner, only briefly expresses esteem and empathy, makes few positive<br />

remarks, and proceeds without asking questions or waiting for the client to respond (i.e., does not go into<br />

answer)).<br />

2: The coach shows the standard behavior (e.g., makes positive verbal remarks that the rater believes are<br />

credible and sincere). The coach expresses interest in client beyond the goals of coaching, listens to the<br />

client and follows-up on significant remarks, speaks with “I-messages,” addresses the client directly<br />

through positive evaluations, uses comparative forms (e.g., says "very helpful” with a friendly slant instead<br />

of "helpful"). However, there is room for improvement.<br />

3: The coach’s behavior and coaching methods are ideal (e.g. uses accurate wording, and creates a very<br />

good atmosphere and relationship with the client).<br />

The following table shows the intraclass coefficients of the raters and the ratings, the mean and standard<br />

deviation of the quality rating, and the percentages of agreement between raters. The last column<br />

of the table enumerates significant correlations between the qualtity ratings or frequencies of positive<br />

observations of the factors with outcome criteria.<br />

Observer reliability and frequency:<br />

Intraclass Coefficient Frequencies Percentages<br />

Mean rater<br />

Mean<br />

ratings<br />

Mean SD Mean SD<br />

Correlation with r<br />

.54** . 70** 1.4 0.3 74.8% 16.4 Goal satisfaction<br />

Result-oriented selfreflection<br />

.52*<br />

.68*<br />

Siegfried Greif, Frank Schmidt and André Thamm 12


The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />

* p> 0.05<br />

** p>0.01<br />

The intraclass coefficient of the mean ratings is satisfactory, but the percentage of the agreement between<br />

the raters is very unsatisfactory (50%). An inspection of this factor shows that the definition is<br />

too broad and the examples too heterogeneous. For some examples, the rating depends too much on<br />

the subjective impression of the observer (e.g. the creation of a positive atmosphere). It is difficult to<br />

distinguish the observations from emotional support (ES3). It might be better to focus on short verbal<br />

reinforcement behaviors of the coach. Surprisingly, the factor shows high and significant correlations<br />

to important outcome criteria.<br />

ES2: The coach non-verbally shows esteem for the client. (Only partly observable by audio tapes.)<br />

The scale codes approve non-verbal behavior observable by body language (e.g., friendly nodding and<br />

attentive eye contact).<br />

Observable behavior:<br />

The coach:<br />

• Has a posture that is turned towards the client.<br />

• Seeks eye contact and holds it.<br />

• Demonstrates a positive attitude towards the client through facial expressions (e.g. smiling) or<br />

gestures that show accord (e.g. a movement of the hand that emphazises a sentence of the client).<br />

• Reinforces the client with affirmative phonemes (e.g. mmhhhh) and nodding of the head.<br />

• Has a posture where the arms and legs appear open and turned towards the client indicating<br />

openness.<br />

• Speaks with a friendly and warm tone of voice.<br />

Remarks:<br />

The non-verbal behavior may not be maladjusted or exaggerated (e.g. permanent nodding). The overall<br />

impression has to be credible as evaluated by the rater.<br />

Delineation: Non-verbal reactions of the coach to feelings expressed by the client are not classified as<br />

esteem behavior. They must be coded as affect reflection (AR).<br />

Examples of behavioral anchors:<br />

-1: The coach’s posture is not turned toward the client. Eye contact is infrequently made. The coach mimics<br />

ineffectively, showing little interest, absent-mindedness, inhibition, and a lack of authenticity. The<br />

coach’s voice lacks warmth and is not adapted to the situation in volume, intonation, or speed. The<br />

coach’s behavior is intimidating and aggressive; the coach does not keep a safe space.<br />

0: The coach shows none of the standard attributes mentioned in the definition (e.g., does not use affirmative<br />

phonemes like “mmhhhh”). However, the coach’s behavior does not appear inappropriate.<br />

1: The coach shows some of the behaviors defined by the standard attributes, but in a rudimentary way<br />

(e.g., the coach’s posture is only turned slightly to the client, the voice shows a little warm, the eye contact<br />

is there but very short, etc.).<br />

2: The coach shows the standard behavior: posture is turned towards the client, voice is warm and friendly,<br />

mimic and gestures are active and appropriate, reactions of the client are supported non-verbally (e.g.,<br />

by nodding and saying, “mmmh, mmmh”), and the eye contact is good. The whole impresssion is<br />

supportive. However, there is room to improve some attributes or the whole pattern of behavior.<br />

3: The coach shows appropriate behavior and is flexibly adapted to the client. The observer does not perceive<br />

any possible improvements in behavior or in general. The coach has an optimal and relaxed posture,<br />

very warm voice (but not exaggerated), sensitive and supportive affect with smiling and nodding. The<br />

coach establishes a very safe environment and relationship with the client. The observer focus here is the<br />

flexibility and perfect adaption of the coach’s behavior to different situations.<br />

Siegfried Greif, Frank Schmidt and André Thamm 13


The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />

Observer reliability and frequency:<br />

Intraclass Coefficient Quality ratings Percentages<br />

Mean rater<br />

Mean<br />

ratings<br />

Mean SD Mean SD<br />

Correlation with r<br />

0.44** 0.61** 1.7 0.4 96.3% 3.4 Emotional Insight<br />

Helplessness<br />

* p> 0.05; ** p>0.01<br />

Angry feelings<br />

.74*<br />

-.57*<br />

.59*<br />

The low intraclass coefficients may partly reflect the low variance of the factor, but the unsatisfactory<br />

percentage of the agreement between the raters (65.7%) shows that it is more difficult than expected to<br />

observe the non-verbal behavior precisely. The difficulty may result from the high percentage of time<br />

sequences (96.3%) where raters had to rate minimal non-verbal reactions. Although three cameras and<br />

split-screens were used for observation, this seemed inadequate as raters reported that the tasks demand<br />

full attention. A possible solution to this is an increase in rest pauses. We have to be very careful<br />

not to interpret causal directions between non-verbal and emotional outcome criteria. It would be plausible<br />

that the non-verbal behavior of the coach is not a cause, but rather a result of emotional insight,<br />

lower helplessness, and openly showing angry feelings or a circular process. Here, interaction process<br />

analyses of the behavior of the coach and the client are necessary.<br />

ES3: The coach emotionally supports the client.<br />

This rating refers to the verbal supporting behavior of the coach and the methods used to help the client<br />

(e.g. to reach her/his goals).<br />

Observable behavior:<br />

The coach:<br />

• Communicates with optimism (e.g., “I am sure that you will be successful if you…”) and explains<br />

why optimism is realistic.<br />

• Helps the client to overcome negative emotional states through supporting tools (e.g., worksheets),<br />

writing down essentials, and giving information on how to solve a problem or get access<br />

to special knowledge (e.g., literature).<br />

• Is reliable (e.g., observes planned actions and agreements; is accessible).<br />

Examples of behavioral anchors:<br />

-1: The coach appears uninterested and expresses a pessimistic perspective on the client’s chances of<br />

goal-attainment. The coach apologizes for arriving late or for postponing the appointment for a long<br />

time. The coach has completely forgotten about agreements and plans made with the client.<br />

0: The coach does not show any of the behaviors in the standard definition (e.g., does not express<br />

optimism, and does not prepare supporting tools to overcome negative emotional states). However,<br />

the overall behavior of the coach remains moderate. The coach arrives slightly late for the appointment,<br />

and/or does not remember well agreements or plans made with the client.<br />

1: The coach shows standard behavior in a rudimentary way. The coach expresses some optimism,<br />

but it is not very convincing and does not explain why the optimism is realistic (e.g., uses standard<br />

flowery language such as, “Cheer up! You will be successful!”). Some signs of supportive behavior<br />

are shown but seem superficial (the assessment is based merely on the intuitive impression of the<br />

rater).<br />

2: The coach shows the standard behavior. The coach expresses optimism and explains why, helps the<br />

client to overcome emotionally straining states through the use of supporting tools, and arrives on<br />

schedule and remembers all agreements and plans without difficulty. There is room for improvement.<br />

3: The coach behaves so credibly that further improvement does not seem possible. The client acknowledges<br />

that the coach has invested much work into the coaching (e.g., has prepared a very useful<br />

worksheet or planning tool), or that the coach has organized the provision of helpful information for<br />

Siegfried Greif, Frank Schmidt and André Thamm 14


The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />

the client.<br />

Observer reliability and frequency:<br />

Intraclass Coefficient Quality ratings Percentages<br />

Mean rater<br />

Mean<br />

ratings<br />

Mean SD Mean SD<br />

Correlation with r<br />

0.53** 0.69** 1.3 0.2 37.4% 14.6 Helplessness<br />

Listlessness<br />

(pre-post-differences)<br />

* p> 0.05<br />

** p>0.01<br />

-.56*<br />

-.56*<br />

The mean value and variance of the quality rating is low. This may explain unsatisfactory intraclass<br />

coefficients. The percentage of agreement between raters remains low (61.1%) as well. Therefore, the<br />

description and examples have been sharpened for future studies.<br />

PR: Problem-reflection<br />

Problem-reflection refers to an elaborate analysis of a problem situation where the client embraces the<br />

antecedents, interactions, and processes in the situation and the perceived consequences. The reflection<br />

is classified as “result-oriented” only if the coach instructs the client on how to develop concrete solutions,<br />

which guide future actions of the client. The following ratings differ between the facilitation of<br />

problem-reflection (RP1) and a result-oriented focus of the problem-reflection (RP1+).<br />

Remarks:<br />

Following logic, the second rating depends on the first. Therefore, correlations between both ratings<br />

should not be calculated, as they will be artificially high.<br />

PR1: The coach has the client reflect on problems.<br />

The rating assesses how much the coach stimulates intensive reflection on a problem or problematic<br />

situation.<br />

Delineation:<br />

It is difficult to differentiate between problem-reflection (PR) and self-reflection (SR). Where there is<br />

doubt, the reflection should be classified as problem-reflection (the subordinate term). The reflection<br />

should only be categorized as a self-reflection if the focus is on the following client attributes:<br />

• Strengths and weaknesses;<br />

• Needs and motives;<br />

• Real and ideal self-concept or self-image;<br />

• Personal, cultural, or family background;<br />

• Peculiar abilities, competencies, potentials, and their future development;<br />

• Personal behavior, feelings, and experiences; and<br />

• Personal and intrinsic goals and plans for self-changes or self-development.<br />

Observable behavior:<br />

The coach:<br />

• Prompts the client to reflect on the problem situation (e.g., antecedents, interactions, processes,<br />

and consequences).<br />

• Asks direct questions to have the client think more deeply about the problem.<br />

• Summarizes and structures the client’s reflection on the problem situation.<br />

• Asks the client to describe examples of similar experiences.<br />

Siegfried Greif, Frank Schmidt and André Thamm 15


The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />

• Supports the client in structuring the problem situation (e.g., “What is most important? What<br />

is of minor importance?” or “How did it start? What followed next?”).<br />

• Asks the client to change their perspective of other persons in the situation (e.g., “How did<br />

person X react and how did he probably feel?” or “Imagine you were person X. How would<br />

you feel after your remark in the discussion?”), or uses techniques like role playing (e.g.,<br />

changing chairs) to facilitate thinking on different perspectives of those involved in the situation.<br />

Examples of behavioral anchors:<br />

-1: The coach gives advice without having knowledge of the situation. The coach fails to acknowledge<br />

a significant problem mentioned by the client, does not support a client who has clearly asked<br />

for help in trying to solve a problem, or does not help to structure a problem situation.<br />

0: The coach does not show elements of the standard behavior. The coach does not ask the client to<br />

describe or reflect on a problem situation.<br />

1: The coach shows some elements of the standard behavior, but the intervention remains incomplete<br />

(e.g., the coach asks the client to describe a problem, but then changes the subject before the description<br />

is complete or comprehensible). The coach does not provide support in systematically structuring<br />

a complex situation, or treats the situation in a cursory manner.<br />

2: The coach shows behavior similar to the standard definition above. The coach prompts the client to<br />

think and talk about the problem situation, and directly asks the client to describe the antecedents,<br />

interactions, processes, and consequences. The coach helps the client through systematic techniques<br />

to change client perspective. The coach also structures the descriptions and analysis. However, the<br />

rater has the impression that there is room for improvement in the coach’s behavior.<br />

3: The behavior of the coach is an example of best practice in the facilitation of client problemreflection.<br />

The coach helps the client to develop new insights, which impresses the client.<br />

Observer reliability and frequency:<br />

Intraclass Coefficient Quality ratings Percentages<br />

Mean rater<br />

Mean<br />

ratings<br />

Mean SD Mean SD<br />

0.5** 0.67** 1.1 0.1 37.2% 13.7 -<br />

* p> 0.05<br />

** p>0.01<br />

Correlation with r<br />

The mean value of 1.1 shows that the quality of the problem-reflection behaviour of the coaches is<br />

rather low. The variance is minimal. This explains why the intraclass coefficients remain low, while<br />

the absolute percentage of agreement between the raters is satisfactory (72.8%). It is plausible that the<br />

factor does not correlate substantially with outcome criteria. It is interesting that the following subcategory,<br />

which focuses on result-oriented problem-reflection, comes up with a prediction of low fear<br />

of failure after coaching.<br />

PR1+: The coach facilitates result-oriented problem reflection for the client:<br />

Remarks:<br />

Result orientation does not only refer to reflection on consequences for future actions. It also embraces<br />

concrete insights, plans, or intended actions for future analysis.<br />

PR1+ can only be coded if facilitation of problem-reflection is shown (i.e. if PR1 is coded at least by<br />

1). The behavior categories PR1 and PR1+ therefore are not independent. (In statistical analyses, artificial<br />

correlations are expected.)<br />

Observable behavior:<br />

Siegfried Greif, Frank Schmidt and André Thamm 16


The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />

In addition to facilitation of problem-reflection (see above, PR1) the coach:<br />

• Prompts the client to derive concrete consequences for future actions.<br />

• Points to the relevance of problem analysis for future action or reflection.<br />

• Trains the client to become aware of aimless rumination and trains the client on how to stop it<br />

(e.g., by self-instruction techniques and deviation of thoughts to positive experiences).<br />

Examples of behavioral anchors:<br />

-1: The coach allows the client to think aimlessly or erratically, or asks the client erratic questions.<br />

0: The coach does not show any of the standard behaviors.<br />

1: The coach shows elements of the standard behavior (e.g., begins problem-reflection and rudimentarily<br />

mentions possible action consequences without describing or explaining them precisely, or<br />

without asking the client to cogitate autonomously).<br />

2: The coach shows behavior similar to the standard behavior. Following the problem analysis, the<br />

coach asks the client to derive concrete actions or future consequences. The coach instructs the client<br />

in how to identify rumination and how to stop recurring thoughts via thought-stopping techniques or<br />

similar interventions. The rater has the impression that there is room for improvement.<br />

3: The support of result-oriented problem-reflection is ideal. The coach prompts the client in a convincing<br />

way to derive important or creative solutions from the problem analysis. The coach helps the<br />

client to stop rumination in a sensitive and effective way.<br />

Observer reliability and frequency:<br />

Intraclass Coefficient Quality ratings Percentages<br />

Mean rater<br />

Mean<br />

ratings<br />

* p> 0.05 ** p>0.01<br />

Mean SD Mean SD<br />

Correlation with r<br />

1.1 0.2 15.2% 9.5 Fear of failure -.47*<br />

Intraclass coefficients could not be calculated because of a very low quality rating and low variances.<br />

The absolute percentage of agreement between raters is satisfactory (74.8%). It is remarkable that the<br />

factor, instead of these preconditions, shows a significant correlation, indicating a reduction of fear of<br />

failure. A single significant correlation may be not replicable in follow-up studies, but additional correlations<br />

to other success factors (e.g., high negative correlations with SR1 self-reflection (-.64**) and<br />

SR1+ result-oriented self-reflection (-.51*) show a pattern that fits with our theoretical expectations<br />

that problem-reflection is a factor that is different from self-reflection.<br />

SR: Self-reflection<br />

Self-reflection is a special type of problem-reflection that specifically regards the self. As mentioned<br />

above, typical examples are reflections that focus the client’s:<br />

• Strengths and weaknesses;<br />

• Needs and motives;<br />

• Real and ideal self-concept or self-image;<br />

• Personal, cultural, or family background;<br />

• Peculiar abilities, competencies, potentials, and their future development;<br />

• Personal behavior, feelings, and experiences; and<br />

• Personal and intrinsic goals and plans for self-changes or self-development.<br />

The rater is asked to observe the behavior of the client when instructed by the coach to reflect on these<br />

problems systematically.<br />

Remarks:<br />

Siegfried Greif, Frank Schmidt and André Thamm 17


The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />

Delineation: The rating focuses on broad reflections of personal strengths and weaknesses, or potentials.<br />

Resource activation (RA) is classified as the reflection, evaluation, and activation of concrete<br />

personal resources (e.g., special abilities or competencies, see below RA) that directly support the<br />

attainment of concretely defined goals of coaching.<br />

SR1: The coach has the client engage in self-reflection (e.g,. on behavior, personal experiences,<br />

needs and motives, strength and weaknesses, ideal and real self-image).<br />

The rating assesses whether or not the coach is able to engage the client in self-reflection (e.g.,<br />

strengths and weaknesses, personal motives, attributes and typical behavior (see list above), and to<br />

what degree.<br />

Observable behavior:<br />

The coach:<br />

• Guides the client’s self-reflection through questions about the client’s personal attributes.<br />

• Reinforces descriptions of special abilities and competencies of the client by verbal communication.<br />

• Helps the client to develop new insights about the self, or between behavior and the reactions<br />

of others.<br />

• Facilitates the exploration of the relationship between the client’s actions and self-concept or<br />

personality.<br />

• Uses special techniques and tools that help the client analyze real and ideal self-concept (e.g.,<br />

by drawing a self-image).<br />

Remarks:<br />

Prompting the client to reflect on or apply her/his personal resources (e.g. abilities and competencies)<br />

or the resources of the social networks of the client to reach goal attainment should not be coded here.<br />

They belong to the factor resource activation (RA, see below).<br />

Examples of behavioral anchors:<br />

-1: The coach makes statements without respecting the client’s personal needs, goals, or special abilities<br />

and competencies. The coach generalizes, gives standard advice, and interprets the client’s behavior<br />

without making sure the client feels accepted. The coach does not seize on the self-reflection of<br />

the client or does not help structure orderless self-descriptions.<br />

0: The coach does not show any of the standard behaviors described above. The coach does not<br />

stimulate self-reflection.<br />

1: The coach shows some of the behaviors and attributes mentioned in the standard behavior description.<br />

The coach does not prompt the client systematically or intensively to think and talk about motives,<br />

special abilities and competencies, strengths, and weaknesses. The coach merely comments in a<br />

cursorily manner on the client’s self-reflection and does not support deep reflection and insight. The<br />

coach does not use systematic or adequate methods to encourage self-reflection.<br />

2: The coach shows behavior similar to the standard behavior description. The client stimulates reflection<br />

on personal motives, norms, cultural background and experiences, abilities and competencies,<br />

experiences, or strengths and weaknesses, ideal and real self-concept of the client by direct, or circular<br />

questions or other methods. The behavior of the coach conforms to the standard, but there is room<br />

for improvement.<br />

3: The performance of the coach is ideal. The coach behaves sensitively, wording questions well and<br />

explaining the importance of the client to reflect on strengths and weaknesses. The coach helps the<br />

client explain self-reflection in concrete terms. Through probing questions or other methods, the<br />

coach helps the client to obtain new insights about the self that are not easily forgotten.<br />

Siegfried Greif, Frank Schmidt and André Thamm 18


The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />

Observer reliability and frequency:<br />

Intraclass Coefficient Quality ratings Percentages<br />

Mean rater<br />

Mean<br />

ratings<br />

Mean SD Mean SD<br />

Correlation with r<br />

0.41** 0.58** 1.1 0.1 21.2% 12.4 Demotivation<br />

Helplessness (pre-postdifferences)<br />

Goal reflection<br />

Behavior reflection<br />

* p> 0.05 ; ** p>0.01<br />

.57*<br />

-.49<br />

Again the mean values and variance of the ratings of this factor are low and worsen the preconditions<br />

to reach satisfactory intraclass coefficients. The absolute percentage of agreement between raters is<br />

satisfactory (74.8%). As discussed above, the significant correlation found shows that demotivation<br />

and self-reflection may be related. The remaining correlations in our study are not significant.<br />

SR1+: Facilitation of result-oriented self-reflection: The coach guides the client to derive concrete<br />

solutions from the problem-reflection<br />

(Only if facilitation of problem-reflection is shown.)<br />

Remarks:<br />

Result orientation does not only refer to action consequences for future actions. It also embraces concrete<br />

insights for future analysis, plans, or intended actions.<br />

SR1+ can only be coded if facilitation of self-reflection is shown (SR1 is rated at least by 1). The behavior<br />

categories SR1 and SR1+ therefore are not independent. (In statistical analyses, artificial correlations<br />

are expected.)<br />

The exploration of abilities, competencies, and social networks of the client to reach goal attainment<br />

do not need to be coded. They belong to resource activation (RA).<br />

Observable behavior:<br />

In addition to engaging the client in self-reflection, the coach:<br />

• Prompts the client to derive concrete consequences for future actions from self-reflection.<br />

• Points to the relevance of the insights derived from the reflection for future plans or selfreflection.<br />

• Stimulates the client’s commitment to stick to the solution derived from self-reflection.<br />

• Trains the client in how to to be aware of rumination and trains in how to stop it (e.g., by selfinstruction<br />

techniques and deviation of thoughts to positive self-images).<br />

Examples of behavioral anchors:<br />

-1: The coach does not stop the client from engaging in aimless or erratic self-reflection or selfdescription.<br />

The coach does not stop the client from ruminating.<br />

0: The coach does not show any of the standard behavior. The coach does not ask the client to derive<br />

concrete actions or other consequences from self-reflection.<br />

1: The coach behaves rudimentarily and therefore only facilitates cursory, result-oriented selfreflection.<br />

The coach attempts to stop the client from ruminating, but does so inefficiently. The results<br />

or consequences remain abstract or without commitment.<br />

2: The coach shows behaviors similar to the standard description above. The coach encourages the<br />

client to derive concrete consequences from the self-reflection and a commitment to these consequences.<br />

The coach helps the client to stop rumination effectively. There is room for improvement in<br />

coach’s behavior.<br />

Siegfried Greif, Frank Schmidt and André Thamm 19<br />

-.49<br />

-.51


The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />

3: The coach’s facilitation of the result-oriented self-reflection is ideal. The coach systematically asks<br />

the client to derive significant concrete consequences from the self-reflection. The coach activates a<br />

strong and sustained commitment by the client to follow-through with the solutions.<br />

Observer reliability and frequency:<br />

Intraclass Coefficient Quality ratings Percentages<br />

Mean rater<br />

Mean<br />

ratings<br />

* p> 0.05 ; ** p>0.01<br />

Mean SD Mean SD<br />

Correlation with r<br />

1.1 0.2 3.2% 4.4 Goal satisfaction<br />

Irritation<br />

Helplessness<br />

The very low mean of the quality rating and the variance of the specification of result-oriented selfreflection<br />

forbid a calculation of intraclass coefficients. The absolute percentage of agreements between<br />

raters is satisfactory (74.8%). It is remarkable that under such preconditions several high but<br />

insignificant correlations can be observed that are theoretically plausible and differ from other factors.<br />

AR: Affect reflection and calibration<br />

Preliminary remark: Basic positive and negative emotions such as anxiety, anger, or happiness are<br />

often referred to as types of “affect”. In some theories, “emotion” is the superordinate term while in<br />

other theories “emotion” refers to more complex feelings that integrate cultural meaning (e.g., feelings<br />

of success or helplessness). We use the term “emotion” for simplicity and clarity.<br />

The following two rating factors are often are related. Calibrations of emotions often require prior<br />

reflections, and the reflection itself reduces strong emotion. Therefore, the observations are not logically<br />

independent, and a correlation between both results in artificially high values.<br />

The ratings aim at assessing coach behavior, which directly stimulates the description and reflection of<br />

affect or emotion by the client (AR1), as well as behavior that pacifies emotion.<br />

AR1: The coach has the clients reflect on their own feelings.<br />

The coach has the client engage in deep and intensive reflection on emotion.<br />

Observable behavior:<br />

The coach:<br />

• Prompts the client to specify and describe emotions or feelings in critical situations.<br />

• Asks the client to explain feelings in a friendly manner. If the descriptions of the client’s feelings<br />

as well as the factors that elicit those feelings are ambiguous, the coach offers possible<br />

hypotheses’ about situation elicitors or intra-individual causes, and asks the client to evaluate<br />

their relevance.<br />

• Encourages the client to analyze and understand possible situational and personal reasons for<br />

the feelings and consequences contributing to their general affect state, well-being, performance<br />

(e.g., lower quality), typical reactions of the social environment to emotional behavior of<br />

the client, and backlashes.<br />

Examples of behavioral anchors:<br />

-1: The coach responds negatively or neutrally to the client’s emotions.<br />

0: The coach does not show any standard behavior. Attempts of the client to talk about their feelings<br />

are neglected. The coach does not comment on the client’s observable emotional reactions, or on possible<br />

consequences and reactions of the environment.<br />

1: The coach shows some of the behaviors described, but in a rudimentary way. The coach asks the<br />

client to describe feelings, but does not pick up on the descriptions and does not help to specify them.<br />

Elicitors and consequences are also not specified.<br />

Siegfried Greif, Frank Schmidt and André Thamm 20<br />

.41<br />

-.46<br />

-.52


The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />

2: The coach shows behaviors similar to the standard descriptions above. The coach encourages the<br />

client to describe feelings precisely, and asks the client to give situational elicitors or causes in a personal<br />

way. The coach helps the client to analyze and understand the consequences (e.g., negative<br />

affect and low well-being), the reactions of the environment, and the possible backlash. There is room<br />

for improvement in the coach’s behavior related to the factor.<br />

3: The coach’s support of affect reflection is ideal. The coach sensitively facilitates deep reflection<br />

and specifies the client’s feelings. The client is able to develop new and lasting insights through a<br />

systematic and precise analysis of situational elicitors, personal causes and consequences, environment,<br />

and backlashes.<br />

Observer reliability and frequency:<br />

Intraclass Coefficient Quality ratings Percentages<br />

Mean rater<br />

Mean<br />

ratings<br />

Mean SD Mean SD<br />

Correlation with r<br />

0.11* 0.2* 1.1 0.3 7.7% 5.1 Feeling of anger at the<br />

time of the first and<br />

middle session<br />

* p> 0.05 ; ** p>0.01<br />

The absolute percentage of agreement between the rater is good (88.9%), but the intraclass coefficients<br />

are too low. This shows that compared to psychotherapy, affect reflection is seldom used in coaching,<br />

as seen in the first comparison study (Borsum, 2008). The plausible correlation between the affect<br />

sales measuring state feelings of aggression at the time of the first and middle sessions show that it<br />

may be premature to discard the factor from future coaching observations. We plan to conduct coaching-intervention<br />

studies with clients who work under high-stress conditions. Here, we would expect<br />

that both ratings of AR factors are of high relevance.<br />

AR2: The coach pacifies the client’s feelings and is able to facilitate calm and open thinking about<br />

the feelings.<br />

The coach intervenes in way that pacifies the emotions of the client (e.g., with calming words or relaxation<br />

techniques) during a critical situation, and facilitates open conversation about the feelings and<br />

their possible negative consequences.<br />

Observable behavior:<br />

The coach:<br />

• By verbal or non-verbal communication shows appreciation and understanding of the client’s<br />

emotions.<br />

• Gives the client time to deescalate by sharing an emotionally moving experience.<br />

• Asks the client to explain the emotionally intense reaction and to consider whether there are<br />

long-term negative consequences (e.g., stress reactions, irritation, or low personal well-being)<br />

or negative reactions of other persons in the environment (e.g., negative feelings, hidden rejection<br />

or stress reactions of team members), and if the client would prefer to reduce the intensity<br />

of the feelings in future similar situations<br />

• Uses relaxation techniques or similar systematic methods, and/or trains the client in the practice<br />

of these techniques.<br />

Examples of behavioral anchors:<br />

-1: The coach reinforces negative emotions (e.g., aggressive feelings) or tries to explain the emotions<br />

with inadequately simple or speculative causes (e.g. “The only reason of the emotional reaction of<br />

your team colleague is that he hates you.”)<br />

0: The coach does not show any of the behaviors described above.<br />

Siegfried Greif, Frank Schmidt and André Thamm 21<br />

.63*<br />

.66*


The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />

1: The coach shows some of the behaviors described in a rudimentary form. The coach accepts the<br />

client’s emotions, but unsuccessfully tries to calm them.<br />

2: The coach demonstrates behavior similar to the standard descriptions. The coach allows the client<br />

to deescalate, sympathizes with the client, helps the client to understand the emotional reaction and<br />

personal consequences, as well as reactions of other persons in the environment. The coach also takes<br />

the client through methods of reducing such intense reactions in similar situations. Relaxation techniques<br />

or similar methods are used to train the client to pacify emotions. The reduction of emotions<br />

can still be improved.<br />

3: The coach behaves in an ideal way, applying the appropriate relaxation techniques or methods.<br />

The coach trains the client to apply these techniques for self-pacification in the best possible way.<br />

The client is able to use the methods effectively.<br />

Observer reliability and frequency:<br />

Intraclass Coefficient Quality ratings Percentages<br />

Mean rater<br />

Mean<br />

ratings<br />

* p> 0.05; ** p>0.01<br />

Mean SD Mean SD<br />

1.4 0.5 1.1% 1.6<br />

Correlation with r<br />

As expected, the percentage of situations where affect calibration is observable in our coaching study<br />

is marginally low. This forbids any further analysis of the intraclass coefficients even where the percentage<br />

of agreement between raters is artificially high (98.9%). As mentioned above, the rating is<br />

assumed to be more important for the coaching of stress management where pacification of affect is<br />

used systematically to reduce post-work stress levels.<br />

CG: Clarification of goals<br />

The rating assesses whether the coach helps the client to to clarify goals in a concrete way, to what<br />

degree this is done, and if goals are prioritized.<br />

CG1: The coach supports the client’s clarification and/or defining of goals.<br />

The rating focuses on the coach’s ability to clarify and explain the client’s goals (e.g., by a concrete<br />

description, definition, or specification of criteria, and measures the rate of goal attainment).<br />

Remark:<br />

Goal clarification should be observed throughout the coaching.<br />

Observable behavior:<br />

The coach:<br />

• Asks the client to describe or define goal(s).<br />

• Uses direct questions to deepen the client’s goal descriptions.<br />

• Helps the client clarify the relationship between extrinsic and intrinsic goals.<br />

• Asks the client to describe in concrete terms what will be different once the goals have been<br />

achieved.<br />

• Helps the client understand which goals are realistic.<br />

• Helps the client detect conflict between different goals that are being pursued.<br />

• Systematically guides the client to decide which of the goals is a priority.<br />

Examples of behavioral anchors:<br />

-1: The coach constrains or disrupts the client from clarifying goals.<br />

0: The coach does not show any of the behaviors described above. The coach does not clarify the<br />

client’s goals, or help to define them concretely.<br />

Siegfried Greif, Frank Schmidt and André Thamm 22


The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />

1: The coach shows some of the behaviors of the standard description, but in a rudimentary way. The<br />

goals are noted, but their definitions remain vague – further clarification is possible and necessary.<br />

Potential conflicts with the goals remain undetected, and the client and coach do not examine the<br />

goals to see if they are realistic.<br />

2: The coach shows the standard behaviors. The coach helps the client analyze and define the goals<br />

concretely, detecting potential conflicts and ascertaining whether the goals are realistic. Where necessary,<br />

the coach encourages the client to redefine the goals. There is room for improvement in goal<br />

clarification.<br />

3: The coach helps the client to clarify the goals. All aspects described above are accounted for in an<br />

ideal way.<br />

Observer reliability and frequency:<br />

Intraclass Coefficient Quality ratings Percentages<br />

Mean rater<br />

Mean<br />

ratings<br />

Mean SD Mean SD<br />

Correlation with r<br />

0.62** 0.77** 1.2 0.2 20.3% 10.3 Overall sum of negative-affect<br />

scales<br />

Activity level at the<br />

start of the coaching<br />

* p> 0.05; ** p>0.01<br />

The intraclass coefficients and the percentage of agreement between raters (86.4%) is satisfactory.<br />

Goal clarification correlates significantly with problem-reflection (PR1 .46* and PR1+ .55*) and the<br />

activity level at the start of the coaching.<br />

RA: Resource activation<br />

The coach helps the client become aware of internal and external resources to reach concrete coaching<br />

goals.<br />

Internal resources are potentials within the individual, e.g., motivational and personality traits, abilities,<br />

competencies and experiences. External resources are potentials in the social and material environment<br />

that the client is able to use (e.g., expert knowledge, consultations, emotional support or concrete<br />

help from people in the occupational environments, family or network of friends, technical literature<br />

/knowledge of IT systems, and financial resources).<br />

RA: The coach encourages the client to explore and make use of internal and external resources<br />

that will support goal attainment.<br />

The rating assesses the level to which the coach helps the client explore and evaluate external resources,<br />

and to what extent the coach encourages the client to use the resources that will potentially<br />

support concrete coaching goals.<br />

Remarks:<br />

Delineation: The rating focuses on resources directly supporting the attainment of concrete coaching<br />

goals. Broader reflections of personal strengths and weaknesses or potentials are classified as selfreflections<br />

(SR1 or SR1+).<br />

Observable behavior:<br />

The coach:<br />

• Encourages the client to explore and specify internal strengths and potentials that may result in<br />

establishing concrete coaching goals (e.g., strong achievement motivation, special abilities and<br />

competencies, or experiences).<br />

• Encourages the client to explore and name possible external resources that may support the attainment<br />

of concrete coaching goals (e.g., knowledge of experts, consultations, emotional<br />

Siegfried Greif, Frank Schmidt and André Thamm 23<br />

-.48<br />

.61*


The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />

support, or concrete help from people in the occupational environments, family or network of<br />

friends, technical literature or IT knowledge systems, and financial resources).<br />

• Asks the client to remember former successes, the personal resources that helped in those instances,<br />

and how the same resources can be applied in the current situation.<br />

• Explains how the client can use personal resources to reach goals in the current situation.<br />

• Appreciates the client’s personal resources, explaining their value and stimulating positive<br />

self-evaluation.<br />

• Encourages the client to use personal resources in pursuing concrete coaching goals.<br />

Examples of behavioral anchors:<br />

-1: The coach gives skeptical remarks concering the client’s personal resources, or distracts the client<br />

by asking to focus on their weaknesses and problems.<br />

0: The coach does not show any of the behaviors described above. The coach does not try to activate<br />

the client’s strengths and potentials.<br />

1: The coach shows the standard behaviors described above, but only in a rudimentary way. The<br />

coach asks the client to name strengths, competencies, or external resources in the environment etc.,<br />

but the analysis is not deep or precise.<br />

2: The coach helps the client to analyze and activate internal and external resources in a similar way<br />

as described above by the standard behaviors. For example, strengths and competencies or external<br />

resources are analyzed precisely and the client identifies new ones that she/he did not think of beforehand.<br />

The client is asked to evaluate how useful the resources will be for the attainment of the coaching<br />

goals. Improvements of the behaviors of the coach are possible.<br />

3: The coach helps the client analyze and activate internal and external resources in an ideal manner.<br />

This is shown by high precision, intensity, or completeness of the analysis, and a convincing evaluation<br />

that raises the motivation of the client to use the resources intelligently.<br />

Observer reliability and frequency:<br />

Intraclass Coefficient Quality ratings Percentages<br />

Mean rater<br />

Mean<br />

ratings<br />

Mean SD Mean SD<br />

Correlation with r<br />

0.29** 0.45** 1.1 0.1 17.4% 12.3 Goal satisfaction<br />

Listlessness<br />

Information management<br />

* p> 0.05 ; ** p>0.01<br />

.49*<br />

-.50<br />

.49<br />

The intraclass coefficients of this important scale are not satisfactory, but the absolute percentage of<br />

agreements between raters is (75.76%). As the low mean value and very narrow variance show, the<br />

coaches did not realize the factor as optimally as demanded by the standard attributes. This may explain<br />

why the correlations with outcome criteria do not reach the level of significance. A problem may<br />

also result from delineation to goal clarification and support of transfer into practice. A more intensive<br />

training of the observers highlighting the differences here may be useful.<br />

TP: Support of transfer into practice<br />

TP: The coach encourages the client to put the planned goal-oriented behavior and measures into<br />

practice.<br />

The coach encourages the client to put the planned behavior and/or other measures into practice<br />

through a persistent pursuit of goal-oriented actions and measures despite difficulties and barriers.<br />

Siegfried Greif, Frank Schmidt and André Thamm 24


The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />

Observable behavior:<br />

The coach:<br />

• Helps the client develop a plan of concrete goals and subgoals comprised of small actions and<br />

measures that the client is able to perform.<br />

• Stimulates the client to explore and analyze possible internal and external difficulties or barriers<br />

to taking the intended actions and measures.<br />

• Helps the client develop a realistic plan of actions and measures despite possible difficulties<br />

and barriers.<br />

• Consults the client on how to overcome possible internal and external difficulties and barriers<br />

to trials and to act according to plan.<br />

• Encourages the client, after a seemingly insurmountable difficulty, to try again after reexamining<br />

and redefining goals, actions, or measures.<br />

• Uses systematic questioning techniques or tools to support the transfer of planned changes<br />

into practice (e.g., telephone shadowing).<br />

Examples of behavioral anchors:<br />

-1: The behavior of the coach interferes with client’s attempts to perform intentions practically. The<br />

client expresses indifference or disrespect towards the client’s attempts to transfer the intended actions<br />

and measures into practice. The coach criticizes the action plans without developing alternative<br />

plans that might be more realistic.<br />

0: The coach does not show any of the standard behaviors.<br />

1: The coach shows basic support that helps the client transfer the planned behavior into practice. The<br />

coach asks the client to develop a plan of action or a measure that will analyze some internal and external<br />

difficulties. The analysis and the measures for how to overcome them, however, remain vague<br />

or abstract.<br />

2: The coach helps the client to prepare the transfer of the planned behavior into practice. The behavior<br />

is similar as described in the examples above. The analysis and support is systematic and incorporates<br />

nearly all of the important facets of the transfer problems of the client. The coach helps the client<br />

to develop a realistic plan of small steps of actions, to analyze possible internal and external difficulties,<br />

and to measure how to overcome them. The coach encourages the client to redefine goals<br />

after setbacks. Improvement of the behavior is possible, (e.g., the analysis is not completely systematic,<br />

does not embrace all important barriers and relevant measures, or is not fully convincing).<br />

3: The coach supports the transfer into practice. The coach helps the client analyze all relevant internal<br />

and external difficulties and barriers, as well as effective measures in a convincing way. The<br />

coach uses systematic transfer tools (e.g., telephone shadowing) and other methods (e.g., simulations<br />

by role-playing). After a setback, the coach sensitively helps the client to understand inner transfer<br />

difficulties and change resistances in the environment. The coach motivates the client to draw adequate<br />

conclusions from the trials and sustainably motivates them to redesign the plans and try again<br />

until goals are reached.<br />

Observer reliability and frequency:<br />

Intraclass Coefficient Quality ratings Percentages<br />

Mean rater<br />

Mean<br />

ratings<br />

Mean SD Mean SD<br />

Correlation with r<br />

0.6** 0.75** 1.2 0.2 26.7% 18.1 Helplessness<br />

Listlessness<br />

Self-management (procrastination)<br />

Goal satisfaction<br />

* p> 0.05; ** p>0.01<br />

-.55*<br />

-.58*<br />

.55*<br />

Siegfried Greif, Frank Schmidt and André Thamm 25<br />

.44


The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />

The intraclass coefficients and the percentage of agreement between the raters (73.74%) are satisfactory.<br />

But the low mean value and variance show that the coaches did not perform well in comparison<br />

to the standard example behavior descriptions of the factor. It is surprising that relatively high and<br />

significant correlations to outcome criteria have been found here.<br />

FP: Feedback and evaluations in the process<br />

The ideal standard of result-oriented coaching demands that each coaching session create concrete<br />

results perceived by the client. This is shown by a positive evaluation at the end of the session, either<br />

after a question from the coach or by a spontaneous positive feedback from the client.<br />

FP1: The coach asks the client to evaluate progress in the coaching as well as satisfaction with the<br />

coaching, especially in goal attainment.<br />

The coach asks the client for feedback, to evaluate that specific coaching session or the coaching in<br />

general, especially in relation to the perceived progress or goal attainment. The evaluation of the client<br />

is positive.<br />

Observable behavior:<br />

The coach receives positive feedback and/or evaluation from the client, after:<br />

• Asking the client to rate satisfaction with the session or the coaching up to that point.<br />

• Asking the client to rate progress in the coaching or the goal-attainment rate.<br />

• Using rating scales to assess satisfaction, goal attainment, or other evaluation criteria of the<br />

coaching.<br />

• Applying specialized instruments or standard questionnaires.<br />

Examples of behavioral anchors:<br />

-1: The coach hinders the client from evaluating important aspects of the coaching, or gives the impression<br />

of being disinterested in the results. The client gives negative feedback or expresses disapproval<br />

of a session or of the coaching up to that point. The coach does not mention possible changes.<br />

0: The coach does not ask the client for feedback. The coach does not show any of the behaviors described<br />

above and does not apply instruments assessing the client’s satisfaction with the coaching.<br />

1: The coach briefly asks the client for feedback or evaluations. The feedback or evaluations of the<br />

client remain rudimentary or cursory. The coach does not ask the client to explain the evaluation.<br />

Also, if the evaluation is merely somewhat positive, the rating of 1 is to be coded here.<br />

2: The coach asks the client for feedback or uses rating scales and evaluation instruments, similar as<br />

described above. The overall results of the evaluation are unambiguously positive and the coach protocols<br />

them. An improvement of the behavior is possible.<br />

3: The coach sensitively asks the client to give feedback, or uses systematic scales or evaluation instruments<br />

and interprets the results together with the client. The client’s overall evaluations are very<br />

positive. The results of the evaluation are analyzed systematically and are used for future planning of<br />

the subjects and changes to the coaching or client’s plans.<br />

Observer reliability and frequency:<br />

Intraclass Coefficient Quality ratings Percentages<br />

Mean rater<br />

Mean<br />

ratings<br />

Mean SD Mean SD<br />

Correlation with r<br />

0.64** 0.78** 1.1 0.2 10.2% 8.3 <strong>Coaching</strong> start: Overall<br />

sum of negative affect<br />

* p> 0.05; ** p>0.01<br />

<strong>Coaching</strong> ending:<br />

Happiness<br />

Activity<br />

Siegfried Greif, Frank Schmidt and André Thamm 26<br />

.61*<br />

.57**<br />

.69**


The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />

The intraclass coefficients and the percentage of agreement among raters (88.4%) is satisfactory. The<br />

remaining values are based on the whole range of the ratings (-1 to -3). In the study the highest affect<br />

state measures can be found in the starting and ending session. Also, evaluations are more meaningful<br />

in the starting and ending phase of the coaching. It is an interesting observation that positive evaluations<br />

of the clients correlate with the sum of all negative-affect scales. This could mean that clients<br />

who are in a negative mood are more grateful for starting the coaching and are more likely to give<br />

positive feedback. The high correlations at the end of the coaching show that positive evaluations of<br />

the clients are associated with their happiness, but also that they might activate them.<br />

FP2: The client spontaneously remarks satisfaction or dissatisfaction with the meeting or the whole<br />

coaching engagement.<br />

The client spontaneously expresses satisfaction or dissatisfaction with the session or the whole coaching<br />

engagement.<br />

Examples of behavioral anchors:<br />

-1: The client spontaneously gives negative feedback or evaluation of the coaching.<br />

0: The client does not make spontaneous evaluations of the coaching.<br />

1: The client briefly expresses satisfaction with special aspects of the coaching without any prompting.<br />

2: The client briefly expresses satisfaction with the overall coaching engagement without any prompting.<br />

3: The client expresses great satisfaction with the coaching without any prompting.<br />

Observer reliability and frequency:<br />

Intraclass Coefficient Quality ratings Percentages<br />

Mean rater<br />

Mean<br />

ratings<br />

Mean SD Mean SD<br />

- - 1.17 0.33 0.73% 1.32 -<br />

* p> 0.05; ** p>0.01<br />

Correlation with r<br />

Due to a low quality rating and variance, intraclass coefficients could not be calculated. The percentage<br />

of agreement between raters is extremely high (97.9%). We doubt that it is useful to use this scale<br />

in future studies, since the frequency of the behavior is very low and it does not correlate as expected<br />

with overall client satisfaction.<br />

Correlations between the success factors<br />

Non-verbal empathy is an analogue communication process that is performed parallel to all other verbal<br />

behaviors. Therefore, it does not make sense to define it as a so-called disjunct behavior category.<br />

In the 5-minute observation periods, the coach is able to show up to three different behaviors. Therefore,<br />

we did not try to design disjunct behavior ratings or to force the observers to choose only one<br />

factor to be rated. (Exceptions are the PR and SR ratings, see below.) As a methodological consequence,<br />

it is necessary to examine the correlations between the ratings and to make sure that they are<br />

not too high. In addition, the correlations are theoretically very interesting.<br />

Table 1 shows the correlations between the success factors in our study. They range from r= -0.64 to<br />

r= 0.89. The majority of the correlations are low and not significantly different from zero correlations.<br />

Siegfried Greif, Frank Schmidt and André Thamm 27


The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />

Table 1: Correlations between success factor ratings based on version 3 (*p>.05; **p>.01)<br />

ES1 ES2 ES3 AR1 AC2 RP1 RP1+ RS1 RS1+ GC1 RA ST1<br />

ES1 –<br />

ES2<br />

–<br />

.56*<br />

ES3<br />

.10 –<br />

.51*<br />

AR1 .36 -.19 .41 –<br />

AC2 -.03 -.43 .09<br />

.47*<br />

–<br />

RP1 .09 .38 -.25 .06 -.53* –<br />

RP1+ .29 ,22 .02 .38 -.13 .74** –<br />

SR1 -.18 -.18 .45 .09 .40 -.64** -.32 –<br />

SR1+ .25 .03 .57* .18 .30 -.51* -.13 .81** –<br />

GC1 .30 .16 -.14 .26 -.31 .46* .55* .45 .01 –<br />

RA .33 .11 .79** .16 .28 -.56* -.23 .66** .65** -.37 –<br />

ST1 .26 .00 .71** .03 .27 -.60* -.43 .46* .51* -.34 .89*<br />

*<br />

–<br />

EP1 .41 .25 .65** .03 -.14 -.10 -.03 .17 .53* .24 .47* .53*<br />

As mentioned above, the correlations between RP1 and RP1+ and also SR1 and SR1+ are influenced<br />

by a logical dependency (the second ratings can be observed only if the first are present). Therefore<br />

they are artificially high and cannot be interpreted. In the table they are marked with grey shading.<br />

Resource activation (RA) and the support of transfer (ST1) correlate by r= 0.89. This is definitively<br />

too high. As mentioned above, in this version 4 of our manual, the differences between the ratings<br />

have been sharpened, and the recommendation has been put forth to train the raters more rigorously so<br />

that they are able to distinguish better between the activation of the client`s resources (RA) through<br />

identifying them, reflecting on them, and supporting how to overcome difficulties in the transfer of<br />

intended actions into practice by systematic interventions (ST1), e.g. shadowing.<br />

A similar problem in part is shown by the correlation of r= 0.79 between emotional support (ES3) and<br />

resource activation (RA) and also r= 0.71 with support of tranfer (ST1). Both constructs have some<br />

similarity in their definitions. Therefore we have sharpened the descriptions and examples of ES3. But<br />

here the reason for the high correlation may also be that for resource activation and support of transfer,<br />

emotional support is a necessary companion. Emotional support (ES3) correlates with ratings of these<br />

factors, which demand sensitive interventions by the coach. Future analyses of interaction may be<br />

helpful to clarify the problem.<br />

It is plausible that the esteem and emotional support items (ES1, ES2 and ES3) form a correlating<br />

cluster. Also expected is the finding that problem-reflection and self-reflection correlate negatively.<br />

They have been designed as disjunct ratings. The negative correlation shows that the raters are able to<br />

distinguish the constructs. Two further correlating pairs of factors that we hoped to find are the correlations<br />

(from r= 0.46 to 0.66) between the two self-reflection-ratings (SR1 and SR1+) with resource<br />

activation (RA) and support of transfer (ST1). This may show that self-reflection is necessary for activating<br />

strengths and abilities of the client and supporting the transfer of the planned behavior into<br />

practice. In contrast to this, reflection on problems not directlty related to the self (PR1 and PR1+)<br />

correlate negatively, but postitively with goal clarification (GC).<br />

We have to be careful not to overrate the correlations found in our first study. Future replication studies<br />

with similar and different samples are necessary, before we interprete them as reliable results and<br />

draw expansive theoretical conclusions. But since the results largely confirm our theoretical assumptions<br />

(Greif, 2008) and the correlations between the factors and the outcome criteria are astonishingly<br />

high, future research may be encouraged. Also professionals may discuss about possible practical conclusions<br />

in observing and improving the factors as a means improving coaching outcomes.<br />

References<br />

Behrendt, P. (2004). Wirkfaktoren im <strong>Coaching</strong>. Diplom thesis, Institute of Psychology, University of<br />

Freiburg, Germany.<br />

Siegfried Greif, Frank Schmidt and André Thamm 28


The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />

Borsum, F. (2008). Vergleich von <strong>Coaching</strong> und Psychotherapie – Vergleich und Abgrenzung von<br />

Wirkfaktoren und Inhalten. Diploma thesis, Work and Psychology Unit, University of Osnabrück,<br />

Germany.<br />

Dörner, D. (1979). Problemlösen als Informationsverarbeitung. Stuttgart: Kohlhammer.<br />

Flückiger, C., Caspar, F., Holtforth, M. G. & Willutzki, U. (2009). Working with patients' strengths: A<br />

microprocess approach. Psychotherapy Research, 19(2), 213-223.<br />

Flückiger, C. & Holtforth, M. G. (2008). Focusing the therapist's attention on the patient's strengths: A<br />

preliminary study to foster a mechanism of change in outpatient psychotherapy. Journal of<br />

Clinical Psychology, 64(7), 876-890.<br />

Gassmann, D. & Grawe, K. (2006). General Change Mechanisms: The Relation Between Problem<br />

Activation and Resource Activation in <strong>Success</strong>ful and Unsuccessful Therapeutic Interactions.<br />

Clinical Psychology & Psychotherapy, 13(1), 1-11.<br />

Geißler, H. (2009). Die inhaltsanalytische „Vermessung“ von <strong>Coaching</strong>prozessen. In B. Birgmeier<br />

(Hrsg.), <strong>Coaching</strong>wissen - Denn sie wissen icht, was sie tun? (pp. 93-128). Wiesbaden: Verlag<br />

für Sozialwissenschaften.<br />

Grant, A. M. (2006). An Integrative Goal-Focused Approach to Executive <strong>Coaching</strong>. In D. R. Stober,<br />

Grant, A. M. (Hrsg.), Evidence Based <strong>Coaching</strong> Handbook: Putting Best Practices to Work<br />

for Your Clients (S. 153-192). New York: John Wiley & Sons.<br />

Grawe, K. (2000). Psychologische Therapie (2. korr. Aufl.). Göttingen: Hogrefe.<br />

Grawe, K. (2006). Neuropsychotherapy: How the Neurosciences Inform Effective Psychotherapy:<br />

Erlbaum.<br />

Grawe, K., Donati, R. & Bernauer, F. (1994). Psychotherapie im Wandel: von der Konfession zur Profession.<br />

Hogrefe: Göttingen.<br />

Grawe, K., Regli, D. & Schmalbach, S. (1994). Cubus-Analyse. Bern: University of Bern.<br />

Greene, J. & Grant, A. (2003). Solution Focused <strong>Coaching</strong>. Harlow, UK: Pearson Education Limited.<br />

Greif, S. (2008). <strong>Coaching</strong> und ergebnisorientierte Selbstreflexion. Theorie, Forschung und Praxis<br />

des Einzel- und Gruppencoachings (<strong>Coaching</strong> and result-oriented self-reflection. Theory, research<br />

and practice of individual and team coaching). Göttingen: Hogrefe.<br />

Greif, S. (in prep.). Analysis of Subjektive Theories on <strong>Success</strong> and Failure of <strong>Coaching</strong> by the<br />

Change Explorer Method. Osnabrueck: University of Osnabrueck, Work and Organizational<br />

Psychology Unit.<br />

Greif, S. (in preparation). Analysis of Subjektive Theories on <strong>Success</strong> and Failure of <strong>Coaching</strong> by the<br />

Change Explorer Method. Osnabrueck: University of Osnabrueck, Work and Organizational<br />

Psychology Unit.<br />

Kaufel, S., Scherer, S., Scherm, M. & Sauer, M. (2006). Führungbegleitung in der Bundeswehr -<br />

<strong>Coaching</strong> für militärische Führungskräfte. In W. Backhausen & J.-P. Thommsen (Hrsg.),<br />

<strong>Coaching</strong>. Durch systemisches Denken zur innovativen Personalentwicklung (pp. 419-438).<br />

Wiesbaden: Gabler.<br />

Kauffmann, C. (2006). Positive Psychology: The Science of the Heart of <strong>Coaching</strong>. In D. R. Stober &<br />

A. M. Grant (Hrsg.), Evidence Based <strong>Coaching</strong> Handbook: Putting Best Practices to Work for<br />

Your Clients (pp. 219-254). New York: John Wiley & Sons.<br />

Kauffmann, C., Bonniwell, I. & Silberman, J. (<strong>2010</strong>). The Positive Psychology Approach to<br />

<strong>Coaching</strong>. In E. Cox, T. Bachkirova & D. A. Clutterbuck (Hrsg.), The Complete Handbook of <strong>Coaching</strong><br />

(pp. 158-171). London: Sage.<br />

Kuhl, J. (2001). Motivation und Persönlichkeit. Interaktionen psychischer Systeme. Göttingen: Hogrefe.<br />

Mäthner, E., Jansen, A. & Bachmann, T. (2005). Wirksamkeit und Wirkung von <strong>Coaching</strong>. In C.<br />

Rauen (Hrsg.), Handbuch <strong>Coaching</strong> (pp. 55-76). Göttingen: Hogrefe.<br />

Miller, D. T. & Ross, M. (1975). Self-Serving Bias in the Attribution of Causality. Fact or Fiction?<br />

Psychological Bulletin, 82, 213-225.<br />

O`Connell, B. & Palmer, S. (2007). Solution-focused coaching. In S. Palmer & A. Whybrow (Hrsg.),<br />

The Handbook of <strong>Coaching</strong> Psychology: A Guide for Practitioners (S. 278-292). London:<br />

Brunner-Routledge.<br />

Rauen, C. (Hrsg.). (2004). <strong>Coaching</strong>-Tools. Erfolgreiche Coaches präsentieren 60 Interventionstechniken<br />

aus ihrer <strong>Coaching</strong>-Praxis (6st ed.). Bonn: managerSeminare.<br />

Siegfried Greif, Frank Schmidt and André Thamm 29


The <strong>Rating</strong> of the <strong>Coaching</strong> <strong>Success</strong> <strong>Factors</strong><br />

Rauen, C. (Hrsg.). (2007). <strong>Coaching</strong>-Tools II. Erfolgreiche Coaches präsentieren Interventionstechniken<br />

aus ihrer <strong>Coaching</strong>-Praxis. Bonn: managerSeminare.<br />

Rogers, C. R. (1980). A Way of Being. Boston: Houghton Mifflin.<br />

Schmidt, F. & Thamm, A. (2008). Wirkungen und Wirkfaktoren im <strong>Coaching</strong>. – Verringerung von<br />

Prokrastination und Optimierug des Lernverhaltens bei Studierenden. Diplom thesis, Work<br />

and Organizational Psychology Unit, University of Osnabrück, Germany.<br />

Schmidt, F., Thamm, A. & Greif, S. (2008). <strong>Rating</strong>-Manual der Wirkfaktoren im <strong>Coaching</strong>. In F.<br />

Schmidt & A. Thamm (Hrsg.), Wirkungen und Wirkfaktoren im <strong>Coaching</strong>. – Verringerung<br />

von Prokrastination und Optimierug des Lernverhaltens bei Studierenden - Appendix (pp. 78-<br />

94). Osnabrück: University of Osnabrück, Diploma thesis.<br />

Shazer, S. d. (1988). Clues: Investigating Solutions in Brief Therapy. New York: Norton.<br />

Stober, D. R. (2006). <strong>Coaching</strong> from the Humanistic Perspective. In D. R. Stober & A. M. Grant<br />

(Hrsg.), Evidence Based <strong>Coaching</strong> Handbook: Putting Best Practices to Work for Your Clients<br />

(pp. 17-50). New York: Wiley and Sons.<br />

Wirtz, M. & Caspar, F. (2002). Beurteilerübereintimmung und Beurteilerreliabilität. Göttingen: Hogrefe.<br />

Zöller, R. (2008). Evaluation von Erfolgskriterien und Erfolgsfaktoren im <strong>Coaching</strong>. – Ein Vergleich<br />

der Perspektiven von Coaches und Klienten. Diplom thesis, Work and Organizational Psychology<br />

Unit, University of Osnabrück, Germany.<br />

Siegfried Greif, Frank Schmidt and André Thamm 30

Hooray! Your file is uploaded and ready to be published.

Saved successfully!

Ooh no, something went wrong!