Development of an outcome monitoring system for crisis telephone ...

DEVELOPMENT OF AN OUTCOME MONITORING SYSTEM FOR 

CRISIS TELEPHONE SERVICES OF THE DISTRESS CENTRE OF 

OTTAWA AND REGION & TEL-AIDE OUTAOUAIS: FINAL REPORT 

Prepared for: Distress Centre of Ottawa and Region 

& Tel-Aide Outaouais 

Prepared by: Tim Aubry 1 

Centre for Research on Community Services 

University of Ottawa 

Myriam Lebel & Shannon Hodgson 

Distress Centre of Ottawa and Region & Tel-Aide Outaouais 

& Danielle Bouchard 

Centre for Research on Community Services 

University of Ottawa 

December 2005 

The project was funded by the Trillium Foundation of Ontario, United Way of Ottawa, City of 

Ottawa, la Ville de Gatineau, and Helping Other People Everywhere 

1 For queries or further information about this research, contact Tim Aubry, Centre for Research on Community 

Services, University of Ottawa, 34 Stewart St., Ottawa, ON K1N 6N5, crsc@uottawa.ca.

Development of an Outcome Monitoring System 

1 

Executive Summary 

• The Distress Centre of Ottawa and Region (DCO) and Tel-Aide Ouataouais 

(TAO) worked together in collaboration with the Centre for Research on 

Community Services (CRCS) at the University of Ottawa to develop an outcome 

monitoring system of their crisis telephone services. 

• The project was started in July 2003 and ended in August 2005. Steps that were 

completed in developing the outcome monitoring system were: (1) Definition of 

program logic models (PLM’s) for telephone crisis services of services of DCO 

and TAO, (2) literature review of research on the evaluation of telephone crisis 

services, (3) identification of outcomes of highest priority from the PLMs, (4) 

development of methodology for identification and/or creation of methodology 

for outcome monitoring, and (5) pilot testing of developed methodology. 

• The current report is intended to provide a summary of the work completed in 

steps 1 & 3-5, including the results emerging from the monitoring of outcomes 

during the third pilot study. A separate technical report was completed on a 

review of the literature on research evaluating the quality and effectiveness of 

crisis telephone services. 

• PLMs of crisis telephone services for DCO and TAO were developed in advance 

of the current project. Consultations were undertaken with personnel and 

volunteers at the two participating agencies to refine the previously developed 

PLMs. 

• Overall, based on the conducted literature review, the PLMs developed for DCO 

and TAO identify many needs and outcomes for callers that are consistent with 

research findings. Consistent with the outcomes cited in the PLMs, research 

suggests that crisis telephone services increase coping abilities, decrease distress, 

diminish the severity of problems, and decrease suicidal intent when it is present. 

• Taking into account the challenging aspects of the service context and based on 

research designs used in previous studies of crisis telephone services, it was 

decided to use a design for outcome monitoring that included two sources of data: 

(1) volunteer ratings of callers, and (2) follow-up interviews or web-based surveys 

of callers. Based on our literature review of research on crisis telephone services, 

supplemented by a review of standardized rating measures, a series of measures 

were chosen or created for the outcome monitoring system. 

• Three pilot studies of the outcome monitoring methodology were conducted over 

the course of the project. The first pilot study tested the caller rating measures on 

a small number of calls at each agency. Based on the first pilot study, revisions 

were made to the rating measures. 

Centre for Research on Community Services


2 

• The second pilot study re-tested the revised caller rating measure and tested for 

the first time the follow-up interview protocol. Changes made to the follow-up 

interview were completed subsequent to the second pilot study. 

• The third pilot study implemented the outcome monitoring procedures over the 

course of a few months in order to both test the full implementation of the system 

as well as collect data on a meaningful number of callers that could be analyzed. 

• Callers from both agencies were rated in the third pilot study as experiencing the 

outcomes identified in the program logic models for crisis services offered in the 

two agencies, namely a decrease in distress, an increase in effective coping, a 

decrease in ineffective coping, and a decrease in suicide risk. There were no 

differences in outcomes experienced by male callers when compared to female 

callers at either agency. Different types of callers showed different patterns of 

improvements on outcomes. In particular, new and occasional callers were rated 

by volunteers as experiencing greater improvements on different outcomes than 

regular or repeat callers. 

• In order to review and improve the methodology developed for the outcome 

monitoring system, interviews were conducted with volunteers from both 

agencies who participated in rating callers and with volunteers who conducted 

follow-up interviews. Volunteers discussed the strengths and difficulties 

encountered in implementing the methodology of the outcome monitoring system 

and provided suggestions for improving it. 

• Based on the lessons learned in the project, a series of recommendations are 

proposed relating to caller ratings, follow-up interviews, and other aspects of the 

outcome monitoring system. 



3 

TABLE OF CONTENTS 

EXECUTIVE SUMMARY …………………………….……………………..... 1 

I. CONTEXT …………………………………………………………….… 4 

II. DEVELOPMENT OF PROGRAM LOGIC MODELS ……………………….... 5 

III. IDENTIFICATION OF OUTCOMES OF HIGHEST PRIORITY …………….... 11 

IV. DEVELOPMENT OF OUTCOME MONITORING METHODOLOGY ……..……12 

V. PILOT TESTING OF OUTCOME MONITORING METHODOLOGY …………..15 

PILOT STUDY I …………………………………………………….. 15 

PILOT STUDY II …………………………………..……….……….. 16 

PILOT STUDY III ……..…………………………………………….. 18 

VI. CONCLUSIONS AND RECOMMENDATIONS ……………………………. 31 

REFERENCES ……………………………………………………………… 35 



4 

I. CONTEXT 

Recent developments in the delivery of publicly-funded health and social services 

have included an increasing emphasis on accountability and the evaluation of services 

(McEwan & Goldner, 2001). There is also increasing interest and expectations of the 

evaluation of services offered by community agencies in the not-for-profit sector (Hall, 

Phillips, Meillat, & Pickering, 2003). 

The United Way of Ottawa (UWO), funders of both the Distress Centre of Ottawa 

and Region (DCO) and Tel-Aide Outaouais (TAO), is currently in the process of 

implementing with its community agencies an approach focusing on services outcomes. 

Building on work conducted by United Way in the United States and elsewhere in 

Canada, UWO will be requiring that its funded agencies identify their targeted outcomes 

and evaluate the extent that their services are meeting these outcomes. 

In this context, DCO and TAO have worked together in collaboration with the 

Centre for Research on Community Services (CRCS) at the University of Ottawa over the 

past two years to develop an outcome monitoring system of its telephone crisis services. 

To conduct the project, DCO and TAO received a grant from the Trillium Foundation. 

This grant was supplemented by funding from the United Way of Ottawa, the City of 

Ottawa, Ville de Gatineau, and Helping Other People Everywhere. 

The project was begun in July 2003 and completed in August 2005. In 

developing an outcome monitoring system for DCO and TAO, the process facilitated by 

CRCS mirrored that typically followed by community agencies (United Way of America, 

1996). In particular, the steps that were completed included: (1) Development of 

program logic models for telephone crisis services of DCO and TAO, (2) literature 

review of research on the evaluation of telephone crisis services, (3) identification of 

outcomes of highest priority from the PLMs, (4) development of methodology for 

identification and/or creation of methodology for outcome monitoring, and (5) pilot 

testing of developed methodology. 

In order to guide the work of developing an outcome monitoring system, an 

Advisory Committee was created with membership including representatives from DCO, 

TAO, City of Ottawa, United Way of Ottawa, and CRCS. The Advisory Committee 

met every two or three months throughout the two-year period of the project, providing 

input on the completed steps identified above. Members of the Advisory Committee 

were: 

• Charles Laframboise, Executive Director of DCO 

• Graeme Kirby, Member of the Board of Directors of DCO 

• Louise Delisle, Executive Director of TAO 

• Laurette Thibodeau, Former member of the Board of Directors of TAO 

• Myriam Lebel, Project Coordinator 

• Diane Martin, City of Ottawa 



5 

• Aline Chalifoux 

• Tim Aubry, Co-Principal Investigator, CRCS 

• Caroline Andrew, Co-Principal Investigator, CRCS 

• Fatimeh Gichevan, Research Assistant, CRCS 

The current report is intended to provide a summary of the work completed in 

steps 1 & 3-5 cited above, including the results emerging from the monitoring of 

outcomes during the last stage of the pilot testing of the methodology. A separate 

technical report was completed on a review of the literature of research evaluating the 

quality and effectiveness of crisis telephone services. This literature review was used in 

the project to validate the PLMs, assist with the identification of outcomes of highest 

priority for outcome monitoring, and inform the development of the methodology for 

outcome monitoring including identifying good outcome measures. 

II. DEVELOPMENT OF PROGRAM LOGIC MODELS 

Program logic models (PLM) represent a systematic mode of describing program 

intentions and sequencing to serve program development, implementation and evaluation. 

This development of program logic models is being used with increasing frequency in 

education, health and social services as a tool for describing program theory and for 

guiding program measurement, monitoring and management (McLaughlin & Jordan, 

1999). Program logic is generally laid out in program proposals or associated 

documentation, but a PLM adds value by assisting program practitioners and evaluators 

to operationalize (develop indicators and measures) of program components, including 

outcomes. The process can be one that helps clarify program objectives, how objectives 

are to be accomplished, and the intended client benefits as a result of the service . 

PLMs take several different forms, but common to all are basic characteristics. 

As shown in Figure 1, any social program is developed in response to an identified need 

in a particular population of individuals. Through the development and implementation 

of program services and the participation of intended program beneficiaries in program 

activities, the identified need is addressed. To the extent that the program is successful or 

effective, the identified need is reduced, minimized or otherwise ameliorated. This 

coincides with the observation of intended and valued outcomes of the program. 

Figure 1: Program Logic 

Needs Services Outcomes 

The PLMs for DCO and TAO was developed in advance of the present project by 

a group of stakeholders from the two agencies at the point of writing the proposal for the 



6 

project. Researchers from CRCS (Caroline Andrew, Tim Aubry, & Robert Flynn) 

facilitated the process of developing these PLMs with these stakeholders. The PLMs 

originally developed by DCO and TAO were reviewed and revised by members of the 

Advisory Committee after the start of the project. Table 1 presents the PLM for DCO 

and Table 2 presents the PLM for TAO. 

Program Logic Model for Distress Centre of Ottawa and Region 

Client Needs. As shown in Table 1, the client group for which the telephone 

crisis services are intended to serve are persons in the community (i..e, Ottawa, West 

Quebec, and parts of the Champlain District) who are experiencing suicidal crisis, acute 

or ongoing emotional distress, loneliness, psychiatric difficulties, or traumatic life 

situations. Telephone services by volunteers at DCO are intended to provide callers with 

immediate access to assistance as an alternative to more expensive professional services. 

The services are characterized as confidential, non-judgmental and non-directive. 

Assistance with coping is provided for emotional distress, psychiatric condition and 

traumatic life situations. Provision is also made for information and referral to services in 

the community. 

Inputs. Resources for the services include 8.0 FTE staff, 150 part-time 

volunteers, and 17 part-time volunteer leaders who train, mentor and advise the regular 

volunteers. In addition to the availability of personnel, other resources required to 

operate the service are: offices for full-time regular staff, call centre facilities and 

equipment, instructional and training resources, and office supplies. 

Service Activities. As specified in the PLM, the service activities include 

answering calls, screening for acuity and risk, determining appropriate interventions and 

referrals, providing emotional support through active listening, transferring for mobile 

crisis services when appropriate and needed, coordinating emergency response when 

necessary, managing repeat / frequent callers in a consistent manner, preventing the 

development of crises, and providing crisis intervention when needed. 

Outputs. Figure 1 specifies a variety of outputs associated with crisis telephone 

services that include the number calls, number of training sessions provided to 

volunteers, and number of volunteer hours provided to the agency. 

Shorter-term outcomes. Shorter-term outcomes (i.e., caller benefits) identified in 

the PLM were: increase in ability to cope with immediate crisis, increase in effective use 

of health services, increase in belief in self and personal abilities, increase in knowledge 

of community services and resources, decrease in distress level, decrease in suicidal 

intent, decrease in feeling of isolation, increase in ability to cope with a psychiatric 

condition, and increase in ability to cope with a family crisis. 

Longer-term outcomes. Longer-term outcomes defined in the PLM were: 

decrease of violence and abuse, increase in individual and family empowerment, increase 



7 

in number of educated and informed citizens, improvement in quality of life, and increase 

in cost savings related to the utilization of health care resources. 



8 

Table 1 - Distress Centre of Ottawa & Region - Service Logic Model (2004) 

Distress 

Line 

Needs of Population 

Clients are experiencing : 

• suicidal crisis 

• acute emotional distress 

• on-going emotional distress 

• loneliness 

• on-going psychiatric condition 

• traumatic life situations 

Clients need: 

• immediate access 

• an inexpensive, highly 

accessible and confidential 

alternative to professional 

services 

• human contact 

• confidentiality 

• non-judgemental, non-directive 

assistance 

• assistance dealing with 

emotional distress, crisis 

• assistance coping with a 

psychiatric condition 

• assistance coping with traumatic 

life situations 

• information and referral 

regarding formal and informal 

services 

• access to all of the above in their 

own language 

Clients from all age ranges and 

socio-economic status are 

English-speaking citizens from 

Ottawa, West Québec and parts of 

the Champlain District or Frenchspeaking 

citizens from parts of the 

Champlain District. 

Intended Annual Inputs 

(human, material & 

financial resources) 

• financial resources 

• 8 full-time equivalents 

• 150 volunteers 

• 17 leaders 

• volunteer training, 

including materials and 

facilities 

• leader training, 

including materials and 

facilities 

• call centre facilities and 

equipment 

• supplies 

• promotional and 

educational materials 

• technology 

Intended Activities 

(Interventions) 

• answer calls 

• screen for acuity and 

appropriate level of 

intervention 

• determine suitable 

services for each 

inquiry 

• conduct risk 

assessment 

• provide information 

on other services 

• provide emotional 

support through 

active listening 

• consult with and 

transfer to Tier II 

when appropriate 

• dispatch to 911 

emergency as 

appropriate 

• develop 

individualized care 

plans for the 

management of 

repeat/frequent 

callers 

• provide crisis 

intervention 

• provide a timely 

response to the 

presenting crisis 

Intended Annual 

Outputs 

(service products or 

« units of service ») 

• 21,000 calls 

• 150 volunteers 

• 11,340 volunteer 

hours (150 

volunteers x 75.6 

hours) 

• 8 volunteer training 

sessions 

• 17 leaders 

• 8,760 leader hours 

(365 days x 24 

hours) 

• 1 leader training 

session 

• monthly reports to 

local, regional and 

provincial partners 

Intended Shorter-Term 

Outcomes 

(client benefits) 

• increase in ability to 

cope with immediate 

on-going crisis 

• increase in effectiveness 

of use of formal health 

services 

• increase in belief in 

themselves and 

confidence in their own 

strengths and abilities 

• increase in knowledge of 

services appropriate for 

them, including services 

of the Distress Centre 

• decrease in distress 

level 

• increase in suicides 

prevented 

• increase in feeling of 

being valued as a 

human being 

• decrease in 

interpersonal conflict 

• decrease in suicidal 

intent 

• decrease in feeling of 

isolation 


cope with psychiatric 

condition 


cope with family crises 

Intended Longer- 

Term Outcomes 

(client benefits) 

• decrease in level 

of violence and 

abuse 

• increase in 

individual and 

family 

empowerment 

• increase in 

number of 

educated and 

informed citizens 

• improvement in 

quality of life 

• increase in cost 

savings 



9 

Program Logic Model for Tel-Aide Outaouais 

Client needs. As shown in Table 2 and similar to DCO, the client group for which 

the crisis telephone services of TAO are intended to serve are persons in the community who 

are experiencing suicidal crisis, acute or ongoing emotional distress, social isolation, severe 

and persistent psychiatric difficulties, or interpersonal conflict. Telephone services by 

volunteers at TAO are intended to provide them immediate access to assistance, an 

inexpensive alternative to professional services, confidentiality, non-judgmental aid, 

assistance with coping with emotional distress and/or a psychiatric condition, assistance 

coping with traumatic life situations, and information and referral to services in the 

community. 

Inputs. Human resources for the services include 4.5 FTE staff, and 58 part-time 

volunteers. In addition to personnel, other resources required to operate the service are: 

offices for full-time regular staff, call centre facilities and equipment, instructional and 

training resources, and office supplies. 

Service activities. Also similar to services offered by DCO, service activities of 

TAO include answering calls, determining appropriate referrals for callers, responding to 

requests for services, coordinating emergency response and referrals when necessary, 

documenting calls according to TAO reporting procedures, training and supervising 

volunteers, and publicizing TAO services. 

Outputs. Table 2 specifies a variety of outputs associated with crisis telephone 

services that include the number of calls, number of hours of services provided to callers by 

volunteers, number of training sessions provided to volunteers, number of referrals made to 

community agencies, and number of different public education activities provided by TAO 

staff. 

Short-term outcomes. Unlike the DCO PLM, the TAO PLM identified only shortterm 

outcomes. These short-term outcomes were: decrease in distress level, decrease in 

problem severity, decrease in suicidal intent, increase in effective use of health services, 

decrease in feeling of isolation, decrease in requests for professional services, and increase in 

savings to the health care system. 

Validation of Program Logic Models Based on Research Literature 

In the literature review examining research on the quality and effectiveness of crisis 

telephone services conducted for this project, twenty studies were identified (Lebel & Aubry, 

2004). Findings in these studies suggest that callers perceived the assistance they receive 

from crisis telephone services as being useful and as having responded to their expectations. 

They also viewed the telephone counselors as being competent and as having provided them 

with good referrals. Callers also reported being satisfied with the ease of accessibility of 

crisis telephone services and perceived these services as a worthwhile alternative to 

traditional services for addressing their problems. 



10 

Table 2 - TEL-AIDE OUTAOUAIS – MODÈLE LOGIQUE DES SERVICES À LA CLIENTÈLE 

MISSION : Offrir un service d’écoute téléphonique en français pour toute personne ayant besoin d’aide, de soutien et de référence 

Ligne 

d’écoute 

Besoins de la population 

Les clients expérimentent: 

1. crises suicidaires 

2. détresse émotionnelle aiguë 

3. détresse émotionnelle persistante 

4. isolement 

5. condition psychiatrique persistante 

6. situations de vie traumatiques 

7. difficulté à s’exprimer 

8. conflits interpersonnels 

9. confusion 

Les clients ont besoin de: 

1. accès immédiat 

2. contact humain 

3. aide pour faire face aux crises 

émotionnelles 

4. aide pour faire face à une condition 

psychiatrique 

5. aide pour faire face à des situations de 

vie traumatiques 

6. information concernant les services 

formels et informels 

7. se comprendre 

8. s’exprimer 

9. aide pour faire des changements 

10. soutient dans leur exploration des 

pistes de solutions 

11. anonymat 

12. écoute sans jugement, ouverture 

13. accueil chaleureux 

14. point d’ancrage/filet de sécurité 

15. accès à des services offerts en français 

16. alternative non coûteuse, accessible et 

anonyme aux services professionnels 

17. réseau continu de services 

Intrants annuels 

proposés 

(res. hum., mat. & fin) 

1. ressources 

financières 

2. 4.5 équivalents plein 

temps 

3. 58 bénévoles 

4. formation des 

bénévoles 

5. matériel et lieux 

pour la formation 

6. 2 salles et 

équipement d’écoute 

7. matériel de bureau 

8. matériel 

promotionnel et 

éducatif 

9. ligne 800 (Québec) 

10. Ordinateurs et 

logiciels 

Activités proposées 

(Interventions) 

1. créer des lignes 

directrices pour 

répondre aux appels 

2. répondre aux appels 

3. informer la clientèle 

sur les ressources 

communautaires 

4. référer les clients 

suicidaires aux 

services appropriés 

5. correspondance par 

courriel (service 

d’aide électronique) 

6. rapports d’écoute 

7. former, superviser & 

soutenir les 

bénévoles 

8. publicité/ 

sensibilisation du 

public au service 

Extrants annuels visés 

(Produits/ unités de service ) 

1. 4539 appels répondus 

2. 58 bénévoles 

3. 4554 h d’écoute 

4. 6 sessions de formation pour 

écoutants 

5. 4480 h pour la formation initiale 

6. 315 h pour l’accompagnement des 

aspirants 

7. 180 h de formation continue 

8. 180 h d’ateliers de discussion 

9. 356 références à différents 

services communautaires 

10. publicité dans les pages jaunes 

11. publicité quotidienne sur le Canal 

Vox 

12. publicité bimensuelle dans tous 

les hebdos transcontinentaux 

13. chronique mensuelle dans l’Envol 

des Monts 

14. chronique mensuelle dans l’Écho 

de Val de Cantley 

15. 3 présentations données dans le 

cadre de la campagne de 

Centraide 

16. 20 visites rendues aux organismes 

communautaires pour faire la 

promotion des services 

17. 2500 dépliants publicitaires 

envoyés à des organismes 

communautaires 

18. 240 affiches envoyées à des 

organismes communautaires 

Résultats visés à 

court-terme 

(bienfaits pour les clients) 

1. diminution du 

sentiment de détresse 

2. diminution de la 

sévérité du problème 

3. diminution de 

l’intention de se 

suicider 

4. augmentation de 

l’usage efficace des 

services 

professionnels 

5. diminution du 

sentiment d’isolement 

6. diminution de la 

demande de services 

professionnels de crise 

7. augmentation des 

économies p/r au 

système de santé 



11 

Results from the reviewed studies also found callers indicating that they had 

experienced positive changes following their use of crisis telephone services. In particular, 

callers described experiencing a reduction in the severity of their difficulties and a decrease 

in their distress levels following the use of theses services. As well, callers also viewed calls 

as reducing their suicidal intent. 

Our literature review also examined another 19 studies that focused on communitylevel 

perceptions and outcomes of having crisis-telephone services available. These studies 

found a high level of awareness among the general population concerning the presence of 

crisis telephone services. Moreover, the presence of crisis telephone services was found to 

have a small negative correlation with suicide rates in some studies (i.e., the larger the 

number of crisis telephone services in a region the lower the suicide rate in the general 

population. In other studies, no correlation was found. 

Overall, based on the literature review, the PLMs developed for DCO and TAO 

identify many needs, service activities, and outcomes for callers that are consistent with 

research findings. Namely, there is correspondence in the research literature regarding these 

services addressing needs relating to quick and easy access to assistance that is an alternative 

to professional services and aids callers with distress through emotional support and referral 

information. Consistent with the outcomes cited in the PLMs, research shows callers 

perceiving their receipt of crisis telephone services as increasing their coping abilities, 

decreasing their distress, diminishing the severity of their problems, and decreasing their 

suicidal intent when it is present. 

Relating to other outcomes in the PLMs, there is no empirical evidence as yet that 

shows crisis telephone services to decrease interpersonal conflict or violence and abuse in 

relationships. As well, there is no research evidence that has shown crisis telephone services 

to increase feelings of empowerment or leading to more appropriate or effective use of health 

care services thereby producing costs savings. 

III. IDENTIFICATION OF OUTCOMES OF HIGHEST PRIORITY 

Subsequent to the finalization of program logic models for the project, the Advisory 

Committee proceeded to identify a set of outcomes of highest priority from the PLMs that 

would be the focus of the outcome monitoring system. Based on the experience of CRCS 

researchers and recommended procedures for developing an outcome monitoring system 

(United Way of America, 1996), it was decided to limit the number of outcomes examined to 

three areas of functioning: (1) level of distress, (2) coping, and (3) suicidal intent. These 

three areas were chosen because they were identified as outcomes in the PLMs of the crisis 

telephone services for the two agencies and were found to be supported empirically in 

research on the effectiveness of these types of services (Lebel & Aubry, 2004). 

In addition, it was decided that the outcome monitoring system would also examine 

client satisfaction. Although it is traditionally considered an indicator of service quality 



12 

rather than of outcome, the Advisory Committee felt that it should be included in the 

monitoring system since it would provide an important source of information for improving 

the services at both DCO and TAO. In addition, our review of the literature indicated that 

callers were satisfied with crisis telephone services, perceiving them as helpful in addressing 

their problems. 

IV. DEVELOPMENT OF OUTCOME MONITORING METHODOLOGY 

Design 

The context in which crisis telephone services are delivered present a number of 

challenges in terms of developing an outcome monitoring system. In particular, the 

anonymity of the services combined with the fact that services are provided within one 

telephone contact of relatively short duration limit the ability of determining client outcomes. 

Moreover, the fact callers are frequently in a state of distress at the point of accessing 

services makes it a difficult situation in which to collect any self-report information from 

them. As well, the reliance on volunteers of varying backgrounds with no formal training in 

mental health to provide these services limits the nature of information that they can be asked 

to provide on callers. 

Taking into account the challenging aspects of the service context and based on 

research designs used in previous studies of crisis telephone services, it was decided to use a 

design for outcome monitoring that included two sources of data: (1) volunteer ratings of 

callers, and (2) follow-up interviews or surveys of callers. 

In the case of volunteer ratings of callers, data would be collected on standardized 

rating scales on callers at the beginning of the call and immediately after the completion of 

the call. The intent would be to integrate the use of the rating scales into service delivery by 

volunteers so that ratings are produced on all callers. Rating scales included measures of the 

three outcome areas (i.e., distress, coping, suicidal intent) and client satisfaction. 

In the case of the follow-up interviews of callers, callers agreeing to the interview 

would either be re-contacted by an interviewer, contact an interviewer, or complete a survey 

available at the website of the participating agencies. The follow-up interviews or survey 

would be scheduled to take place within two weeks of the call in which the caller received 

services. The intent would be to interview or survey a representative sample of callers. The 

follow-up interview and survey would include the same measures, namely a combination of 

standardized quantitative measures and some open-ended qualitative questions. 

Measures 

Based on our literature review of research on crisis telephone services supplemented 

by a review of standardized rating measures, a series of measures were chosen or created for 

the outcome monitoring system. 



13 

Caller Rating Scales 

A copy of the caller rating scales used in the outcome monitoring system are available 

from the senior author (T. Aubry, crsc@uottawa.ca). 

Distress. The only relevant rating measure that we found was the one-item 

Brasington Depression Scale (Knickerbocker, 1973). The scale required volunteers to rate 

the level of depression presented by callers on a 5-point scale that ranged from no depressive 

mood (1) to extreme depressive mood (5). Criteria for scoring a caller’s depressed mood was 

based on the number of depressive symptoms present and the intensity of these symptoms. 

Using this scale as a model, we also created a one-item rating scale for anxiety. The rating of 

callers by volunteers on their level of anxiety ranged from “no anxiety” (1) to “extreme 

anxiety” (5). Similar to the rating for depressed mood, the number of anxiety symptoms and 

the level of intensity of these symptoms were used to determine the score on the rating scale. 

The overall rating scale for distress comprised of the average of these two items with the total 

possible score ranging from 1 to 5. 

Coping. Our review of coping measures revealed that no rating scales of coping 

existed. Consequently, we created a one-item measure of the use of effective coping 

strategies and another one-item measure of the use of ineffective coping strategies. Both 

scales had five response alternatives that ranged from “not at all” (1) to ”a great deal” (5). In 

rating the use of effective coping strategies, volunteers based it on the extent that a caller 

uses one or more of the following strategies: problem solving, support seeking, information 

seeking, cognitive restructuring, emotion regulation, and negotiation. In rating the use of 

ineffective coping strategies, volunteers based it on the extent that a caller engaged in the 

following strategies: rumination, helplessness, social withdrawal, opposition, and avoidance. 

Suicide Risk. A one-item suicide scale adapted from a scale previously developed for 

a school postvention program was used in the project (Séguin et al., 2004). Volunteers rated 

the callers on a scale ranging from 1 to 8. A score of 1 was given if no suicide risk was 

present while a score of 8 was coded is a suicide attempt was occurring. 

Client satisfaction. A 9-item version of the Crisis Call Outcome Rating Scale was 

used by volunteers to assess the client satisfaction of callers (Bonneson & Hartsough, 1987). 

The items require a volunteer to rate the response of callers to the services they received 

based on what callers said or how they behaved during the call. The 13 items with the 

highest item-total correlation were initially selected from a 26-item version of the measure. 

Subsequent to the first pilot test, four items that referred to negative statements or behaviours 

by callers were removed and a 9-item measure of client satisfaction was used in the second 

and third pilot test. 

Follow-up Interview Measures 

A copy of the interview protocol that includes the follow-up interview measures are 

available from the senior author (T. Aubry, crsc@uottawa.ca). 

Distress. The 12-item version of the General Health Questionnaire (GHQ) was used 

in the follow-up telephone interview and web survey as a self-report measure of distress. 



14 

The GHQ is a self-report instrument intended to identify minor psychiatric morbidity in the 

general population (Goldberg & Williams, 2004). Questions in the GHQ ask about current or 

recent difficulties, functioning levels and/or well-being with respect to a number of areas 

(e.g., sleep, decision-making, feelings of happiness). 

Items on the GHQ inquire about the extent that difficulties or regular functioning / 

well-being in different areas are present. Response alternatives range from 0 to 3 depending 

on the extent that a difficulty or regular functioning is present. Total score can range from 0 

to 36. Response on the GHQ can also be dichotomized as to whether or not difficulties or 

regular functioning are present with responses scored as either 0 or 1. Using dichotomours 

scoring, total scores can range from 0 to 12. A score of 3 or more on a dichotomously scored 

GHQ is considered consistent with the presence of a diagnosable mental health problem. 

The GHQ has been shown to have excellent psychometric properties (Goldberg & Williams, 

2004). 

Coping. An abbreviated 16-item Ways of Coping Questionnaire was initially pilot 

tested for the study (Folkman & Lazarus, 1988). However, it was decided after the first pilot 

test not to use it because of its length. Instead, three questions were created to query callers 

about the nature of their coping response since calling the crisis telephone service. Each 

question was made up of two parts with the first part being close-ended (i.e., response 

alternatives of “yes”, “no”, or “don’t know”) and the second part open-ended (i.e., asking 

callers why they answered the first part the way they did). The first question asked if a 

caller perceived his or her problem differently since calling the service. The second question 

asked if a caller had taken actions to address his or her problem since calling the service. 

The third question asked if calling the service had helped a caller cope differently with an 

issue. 

Suicide Risk. Three items making up the Suicidal Subscale of the 28-item version of 

the GHQ (Goldberg & Williams, 2004) was used in the follow-up interview to assess suicide 

risk. The subscale was shown to have significant correlations with other suicidal intent 

scales (Watson, Goldney, Fisher, & Merritt, 2001). The items ask about the extent that 

suicidal thoughts and intent are present. Response alternatives range from 0-3 with higher 

scores reflecting higher levels of suicidal intent. Possible total score can range from 0-9. 

Client Satisfaction. The 3-item version of the Client Satisfaction Questionnaire 

(CSQ) was used in the follow-up interview to assess caller satisfaction with the services they 

received from DCO or TAO (Larsen, Attkisson, Hargreaves, & Nguyen, 1979). The three 

items ask respondents about their general satisfaction, whether or not services met their 

needs, and whether or not they would recommend the services to others. The response 

alternatives for items range from 1-4 with higher scores representing higher levels of 

satisfaction. Total scores can range from 12 to 48. The CSQ has been shown to have good 

psychometric properties (Larsen et al., 1979; Nguyen, Attkisson, & Stegner, 1983). The 

CSQ was supplemented by two open-ended questions focusing on what aspects of services 

callers liked as well as asking them for suggestions for improving services. 



15 

Procedures 

Rating of callers by volunteers 

The procedures followed by volunteers to conduct caller ratings are documented in a 

manual that is provided to volunteers at the time of their training for participating in the 

evaluation. A copy of the manual is available from the senior author (T. Aubry, 

crsc@uottawa.ca). Volunteers rated callers initially at the beginning of calls (i.e., within the 

first 10 minutes) and then again after the conclusion of the call. 

Follow-up interviews 

At the end of calls, volunteers invited callers to participate in the evaluation of crisis 

telephone services through either a telephone interview or by completing a survey available 

at DCO’s or TAO’s website. The follow-up or survey was targeted to occur within two 

weeks of callers’ contact with the crisis telephone service. Callers were informed that their 

participation was voluntary and confidential and that their refusing to participate would have 

no effect on their receipt of future services. Callers were given the option to either call back 

or be called at a specific time for the telephone interview. 

V. Pilot Testing of Outcome Monitoring Methodology 

Three pilot studies of the outcome monitoring methodology were conducted between 

June 2004 and June 2005. The results of each of these pilot tests are presented next. 

Pilot Study I 

The first pilot study was conducted during the period of June and July, 2004. The 

intent of the first pilot study was to test the caller rating measures on a small number of calls 

at each agency. Caller ratings were conducted on 24 callers by the three volunteers at DCO 

and on 33 callers also by three volunteers at TAO. The rating scales were found to be simple 

and straightforward to use. They were completed at the beginning of the call after 6 to 8 

minutes had elapsed. 

Based on feedback from the volunteers using the rating scales, a number of small 

changes were made. Specifically, the coping scales were revised from focusing on approach 

types of coping behaviours and avoidance types of coping behaviours into effective coping 

behaviours and ineffective coping behaviours. A recent review of the literature in 

Psychological Bulletin (Skinner, Edge, Altman, & Sherwood, 2003) was used to identify 

behaviours in these two latter categories. As well, clarification of the criteria for rating levels 

of depressive mood and levels of anxiety were made with the addition of intensity in addition 

to number of symptoms. Some other symptoms indicative of anxiety were also added to the 

rating criteria for the anxiety scale. Finally, the number of items on the Crisis Call Outcome 

Rating Scale was reduced from 13 items to 9 items. In particular, items that referred to 

negative outcomes were removed (e.g., “the caller said the helper was not listening”). 



16 

Pilot Study II 

Tel-Aide Outaouais 

The second pilot study was conducted at TAO during the period of August to October 

2004. The intent of the second pilot study was to re-test the revised caller rating measure and 

test for the first time the follow-up interview protocol. A total of 40 calls were rated by three 

volunteers. Of this group, 10 (25%) callers agreed to participate in a follow-up interview of 

Variable 

X 

SD 

X SD 

t 

(Beginning) (Beginning) (End) (End) 

Distress 2.32 1.01 1.91 0.81 5.61*** 

Effective Coping 2.54 1.10 2.67 1.16 - 1.53 

Ineffective Coping 2.28 1.12 2.10 1.05 2.88** 

Suicide Risk 1.15 0.43 1.05 0.22 2.08* 

which eight (20%) were completed. The Project Coordinator completed the follow-up 

interviews. Follow-up interviews lasted 50 minutes on average. It was noted that the actual 

interview portion lasted 20-25 minutes with the other time spent discussing personal issues. 

As shown in Table 3, statistical analyses of ratings of callers revealed that there was a 

significant decrease in distress from the beginning of the call to the end of the call. As well, 

there was a decrease in the use of ineffective coping strategies and a decrease in suicide risk. 

Although there was an increase in effective coping strategies over the course of the call, the 

change was not statistically significant. 

Table 3 - Comparison of Caller Ratings at Beginning and End of Calls at TAO in Pilot 

Study II (N = 30) 

*** p < .001; ** p < .01; * p < .05 

An examination of the frequency distribution of ratings on the 9-item Crisis Call 

Rating Scale was normally distributed with total scores ranging from 9 to 63 (i.e., average 

scores of 1 to 7). The average total score was 39.12 representing an average score on 

individual items of 4.35. The median of the distribution of the total scores was 38.00 

(average median score on individual items of 4.22). The items making up the Crisis Call 

Rating Scale showed a strong relationship to each other, suggesting that the items are 

measuring a similar concept 2 . 

The number of follow-up interviews (n = 8) is too small to conduct meaningful 

statistical analyses. An examination of descriptive statistics found scores on the items of the 

GHQ, Ways of Coping measure and the 4-item measure of suicide risk to have acceptable 

variability. Using the dichotomous scoring of the GHQ, 3 of 8 (38%) respondents had total 

scores that were consistent with the presence of a diagnosable mental health problem. All of 

the eight respondents expressed overall satisfaction with the services they received and 

indicated that they would contact TAO again in the future if needed. Moreover, 7 of 8 (13%) 

respondents reported that TAO had met most of their needs 

2 Crisis Call Rating Scale for TAS was found to have a Cronbach’s alpha of .94. 



17 

Distress Centre of Ottawa and Region 

The second pilot study was conducted at DCO in the period between the months of 

October and November 2004. The intent of the second pilot study was to re-test the revised 

caller rating measure and test for the first time the follow-up interview protocol. A total of 

30 calls were rated by three volunteers. Of this group, eight (27%) completed a follow-up 

interview. Not all rated callers were invited to participate in a follow-up interview since some 

volunteers felt uncomfortable asking callers who were in crisis or distressed. One caller, who 

participated in the follow-up interview, was identified as a new caller 3 , two were identified 

as occasional callers 4 , and the other five as repeat callers 5 . A volunteer at DCO completed 

the follow-up interviews. Follow-up interviews lasted 30 minutes on average. 

Variable 

X 

SD 

X 

SD t 

(Beginning) (Beginning) (End) (End) 

Distress 2.45 1.07 2.10 0.86 4.37** 

Effective Coping 2.83 0.91 3.10 0.96 - 1.98 

Ineffective Coping 1.97 0.96 1.80 0.85 2.41* 

Suicide Risk 1.50 0.94 1.20 0.41 2.52* 

As shown in Table 4, statistical analyses rating of callers revealed that there was a 

significant decrease in distress from the beginning of the call to the end of the call. As well, 

there was a decrease in the use of ineffective coping strategies and a decrease in suicide risk. 

Although there was an increase in effective coping strategies over the course of the call, the 

change was not statistically significant. 

Table 4 - Comparison of Caller Ratings at Beginning and End of Calls at DCO in Pilot 

Study II (N =30) 

** p < .001; * p < .05 

An examination of the frequency distribution of the 9-item Crisis Call Rating Scale 

showed total scores ranging from 20 to 56 (i.e., average scores of 2.22 to 6.22) with the 

median falling at 37 (i.e., average score of 4.11) . The items making up the Crisis Call 

Rating Scale showed a strong relationship to each other, suggesting that the items are 

measuring a similar concept 6 . 

The number of follow-up interviews (n = 8) is too small to conduct meaningful 

statistical analyses. An examination of descriptive statistics found scores on the items of the 

GHQ, Ways of Coping measure, the measure of suicide risk, and the Client Satisfaction 

Questionnaire to have acceptable variability. 

3 New callers at DCO are defined as callers who are identified by volunteers as never having previously called 

DCO. 

4 Occasional callers at DCO are defined as callers who are identified by volunteers as having previously called 

DCO on a small number of occasions. 

5 Repeat callers at DCO are defined as callers who are identified by volunteers as calling DCO on a regular and 

frequent basis. 

6 Crisis Call Rating Scale was found to have a Cronbach’s alpha of .85. 



18 

Using the dichotomous scoring of the GHQ, 4 of 8 (50%) respondents had total 

scores that were consistent with the presence of a diagnosable mental health problem. All of 

the respondents stated that their call to DCO had helped them cope better with their 

problem(s). Seven of 8 respondents (88%) expressed satisfaction on the three-item Client 

Satisfaction Questionnaire, indicating that phoning DCO had met most of their needs and 

they were overall satisfied with the services they received from DCO. and they would use 

the services at DCO again if they needed them. 

Based on the second pilot study, it was decided by the Advisory Committee to 

continue to use the rating measures of distress, effective coping, ineffective coping, and 

suicide risk. As well, the 9-item version of the rating scale for client satisfaction (Crisis 

Caller Outcome Rating Scale) was kept for the third pilot study. In order to shorten and 

simplify the follow-up interview, it was decided to replace the Ways of Coping Measure with 

three items created for the outcome system asking respondents if calling crisis telephone 

services had affected their ability to cope and how. 

As well, some questions were created and added to the follow-up interview that asked 

callers if and how their use of crisis telephone services had changed callers’ level of distress 

and level of suicidal thoughts and intent. Finally, three items were added to the follow-up 

survey that asked callers if they had received referrals from volunteers, if they were satisfied 

with these referrals and if they had used them. These items were in two parts with openended 

questions asking callers to explain their responses of “yes” or “no” to each of these 

issues. 

Pilot Study III 

The third pilot study was conducted during the period of March to June, 2005 for 

TAO and during the period April to June, 2005 for DCO. The intent of the third pilot study 

was to implement the outcome monitoring procedures over the course of a few months in 

order to both test the full implementation of the system as well as collect data on a 

meaningful number of callers that could be analyzed. All of the volunteers at TAO 

participated in rating the calls for the third pilot study. For logistical reasons (i.e., inability to 

bring all of volunteers together for training in short amount of time available) , 15 of the 50 

volunteers at DCO participated in rating the calls for the third pilot study. The results of the 

third pilot study are presented next for each of the agencies in separate sections beginning 

with those of TAO. 

Tel-Aide Outaouais 

Sample. 

A total of 409 callers were rated by volunteers at TAO over the course of the third 

pilot study. Of these calls, two involved hang-ups, another two calls required callers to call 

back at another time, and one call was identified as an obscene phone call. Therefore, these 

five calls were removed from the data base leaving at total of 404 calls. Men represented 

almost two-thirds of the rated callers (64%). The general content of the calls as assessed by 

volunteers showed the majority identified as involving the provision of support (69%), 



19 

followed by calling to talk to someone (23%) and seeking assistance in a crisis (7%). As 

shown in Figure 2, the largest group of callers are identified as new callers 7 representing over 

half the callers (53%). Another one-quarter of the callers (25%) are supported callers 8 while 

approximately another one-fifth (22%) are considered regular callers 9 . 

7 New callers at TAO are callers identified by volunteers as either never having previously phoned TAO or if 

they had it was only on an occasional basis. 

8 Supported callers at TAO are callers identified by volunteers who call frequently and are given a specified 

amount of time for a call or limited to a specified number of calls per day or per week. 

9 Regular callers at TAO are callers identified by volunteers as phoning TAO on a very frequent basis but are 

not limited in the amount of time given to them or number of calls allowed in a specified amount of time. 



20 

Figure 2 

Type of Caller Rated at TAO (N = 404) 

25% 

22% 

53% 

New 

Supported 

Regular 

Interpersonal problems (65%) were the most frequent problem area identified by 

volunteers in the rated calls followed closely by mental health problems (61%). Other 

problems areas identified in rated calls by volunteers in descending order of frequency were 

marital or couple relationship problems (9%), occupational / financial problems (8%), family 

problems (7%), marital separation (7%), parent-child relationship problems (6%), suicide 

risk (5%), physical health problems (4%), grief over loss of loved one (4%), substance abuse 

difficulties (3%), work / school problems (3%), experience of abuse and/or violence in 

relationships (3%) and living in poverty (3%). 

Of the group of 404 calls that were rated, 13 callers agreed to a follow-up interview or 

to complete the web survey. It is not known how many of the group rated were invited to 

participate in the follow-up interview. Of this latter group, eight asked that the interviewer 

phone them, five agreed to have the interviewer phone them, and none chose the web option. 

Follow-up interviews were completed with three of these callers, representing < 1% of those 

rated. All three callers were phoned by the interviewer. 

Respondents in the follow-up comprised of two men and one woman. All of the 

respondents were regular callers. 

Results 

Ratings of callers 10 . Table 5 presents the comparisons of the caller ratings at the 

beginning of the call with those at the end of the call. As shown in Table 5, all of the 

comparisons are significant in the direction of callers experiencing a decrease in level of 

distress, an increase in level of effective coping, a decrease in level of ineffective coping, and 

a decrease in level of suicide risk. 

10 Table A1 in Appendix A presents the psychometric properties of the measures used for rating callers in the 

third pilot study at TAO. 



21 

Table 5 - Comparison of Caller Ratings at Beginning and End of Calls at TAO in Pilot 

Study III 

Variable X SD X SD 

(Beg.) (Beg.) (End) (End) df t 

Distress 4.54 1.97 3.44 1.43 400 17.33* 

Effective Coping 1.89 0.98 2.41 1.14 397 -12.19* 

Ineffective Coping 2.12 1.23 1.69 0.93 392 11.74* 

Suicide Risk 1.33 1.02 1.15 0.63 393 5.21* 

* p < .001 

A series of analyses 11 were conducted to examine for differences in changes on 

outcomes related to sex and type of caller. Improvements in caller outcomes as rated by 

volunteers over the course of the call were found to be the same for male and female callers. 

A comparison of outcomes of different types of caller (i.e., new, supported, regular) 

found significant differences 12 with new callers rated as having significantly higher levels of 

distress than supported or regular callers. All three groups experience significant decreases 

in levels of distress over the course of the call 13 . As well, improvements in level of distress 

varied by type of caller 14 . Specifically, new callers experienced a greater decrease in their 

level of distress than regular or supported callers. Figure B1 in Appendix B provides a plot of 

the changes over time for each of the three types of callers. 

Another analysis found significant differences 15 between different types of callers and 

level of effective coping with new and regular callers rated as having higher levels of 

effective coping than supported callers. No differences were found between new and regular 

callers. All three groups of callers were rated by volunteers as experiencing an increase of 

effective coping over the course of the call 16 . New callers showed a greater increase in 

effective coping over the course of the call than regular or supported callers 17 . Figure B2 in 

Appendix B shows the differences in the pattern of changes in effective coping over time for 

different types of callers. 

All three groups of callers experienced a decrease in level of ineffective coping 18 . No 

differences emerged among the three types of callers and level of ineffective coping. 

Results showed significant differences on rated level of suicide risk. Specifically, 

supported callers were rated as having a higher level of suicide risk than regular or new 

11 Repeated measures analyses of variance 

12 F (2, 397) = 5.64, p. < .005 

13 F (1, 397) = 226.75, p < .001 

14 F (2, 397) = 4.99, p. < .01 

15 F (2, 394) = 8.56, p < .001 

16 F (1, 394) = 111.09, p < .001 

17 F (2, 394) = 5.84, p. = .005 

18 F (1, 390) = 111.36, p < .001 



22 

callers 19 . Change in level of suicide risk differed according to the different types of 

callers 20 . Supported callers showed a significant decrease in rated level of suicide risk from 

the beginning to the end of the call 21 . New callers also demonstrated a significant decrease 

in suicide risk over the course of the call 22 . However, ratings of regular callers did not 

change from the beginning to the end of the call. Figure B3 in Appendix B shows the 

different pattern of change in level of suicide risk according to different types of callers. 

Finally, another analysis 23 found significant differences among the different groups of 

callers on the ratings of perceived client satisfaction conducted by volunteers on callers 24 . 

In particular, significant differences were found between new callers and regular callers 25 

and between new callers and supported callers 26 . New callers were rated by volunteers as 

expressing higher levels of satisfaction than the other two groups of callers. There were no 

differences between ratings of client satisfaction of regular callers and ratings of supported 

callers. 

Follow-up interviews and surveys. Only three follow-up interviews were conducted 

with TAO callers. Therefore, only descriptive statistics on measures and summary of 

responses to open-ended questions in the follow-up interview will be provided. Total scores 

on the GHQ 27 suggest that all three respondents participating in the follow-up interview are 

functioning in the normal range (i.e., corresponding to the general population). 

Related to any distress they were experiencing, follow-up respondents were asked if 

their call to TAO changed the way they were feeling at the time of making the call. One 

respondent said “yes” explaining that the call helped him or her to explore a solution to the 

problem. Another respondent said “no” the call did not change how he or she felt. The third 

gave no response to the question. 

In response to a question related to whether or not calling TAO helped callers change 

their perception of their problem, all three respondents stated “yes”. Similarly, all three 

respondents indicated that they had taken some action (i.e., rehearsed with volunteer 

discussing health difficulties with family physician, received referrals to other services that 

were helpful). All three respondents reported that calling TAO had helped them overcome 

their difficulties. 

All three respondents reported receiving the kind of services at TAO that they were 

seeking. Two of three respondents also stated that they would call TAO again in the future if 

19 F (2, 390) = 3.78, p. < .05 

20 F (2, 390) = 4.21, p < .02 

21 t (95) = 3.16, p < .005 

22 t (209) = 4.25, p < .001 

23 Analysis of variance 

24 F (2, 381) = 12.89, p. < .001 

25 p = .001 

26 p. < .001 

27 Using a cut-off score of 2/3 on the 12-item version of the General Health Questionnaire (Golberg & Williams, 

2004) 



23 

needed. Similarly, two of the three respondents said that TAO had met most or almost all of 

their needs. 

In response to an open-ended question about what they liked most about TAO 

telephone services, two respondents identified characteristics of volunteers that included 

helpfulness, warmth, patience, sensitivity, and calmness. The other respondent referred to 

the assistance he or she has received. 

A question asking respondents to make suggestions for improving TAO services 

yielded only a response from one respondent. In particular, the respondent suggested that the 

agency needed an answering machine. The respondent also mentioned that he disliked when 

he or she was asked to phone back because a current call was more important than his or 

hers. 

Distress Centre of Ottawa and Region 

Sample 

A total of 373 callers were rated by 15 volunteers at DCO over the course of the third 

pilot study. Women represented over two-thirds of the rated callers (72%) while men made a 

little over one-quarter of the rated callers (28%). The general content of the calls as assessed 

by volunteers showed the majority identified as involving distress (62%), followed by crisis 

(23%) and support (15%). As shown in Figure 3, there are similar proportions for the three 

type of callers who were rated in third pilot study with 39% identified by volunteers as 

occasional callers, 32% perceived to be new callers, and 29% defined as repeat callers. 

Figure 3 

Type of Caller Rated at DCO (N = 373) 

29% 

39% 

Occasional 

New 

Repeat 

32% 

Interpersonal problems (82%) were the most frequent problem area identified by 

volunteers in the rated calls followed by mental health difficulties (76%). Other problems 

areas identified in rated calls by volunteers in descending order of frequency were physical 

health problems (33%), substance abuse difficulties (24%), experience of abuse and/or 

violence in relationships (18%), occupational and/or financial difficulties (18%), and suicidal 

risk (18%). Of those 18% of callers identified as having a suicide risk, 14% were considered 

of low risk and 4% were rated as being of a moderate to high risk. 



24 

Of the group of 373 calls that were rated, 90 (24%) were invited to participate in a 

follow-up interview or complete the web survey. Another 57 of the 373 calls (15%) had 

already been asked to participate or had already participated either through the second pilot 

test or because they had called more than once and as a result they had been rated more than 

once. Follow-up interviews or web surveys were successfully completed by 35 callers, 

representing 39% of those invited and approximately 10% of the total number of rated 

callers. As shown in Figure 4, 66% of those who participated in the follow-up chose the 

option of interviewers phoning them and 26% called the interviewers. Only 9% (n=3) of 

follow-up participants opted to complete a web survey. 

Figure 4 

Option Chosen By Callers for Follow-up 

at DCO (N = 35) 

8% 

26% 

Called 

Were Called 

Web Survey 

66% 

Of the 35 follow-ups on which there was caller data, 77% were women and 23% were 

men. Of this group, 43% were identified as new callers, another 43% as occasional callers, 

and 14% as repeat callers. 

Results 

Ratings of callers 28 . Table 6 presents the comparisons of the caller ratings at the 

beginning of the call with those at the end of the call. All of the comparisons are significant 

in the direction of callers experiencing a decrease in levels of distress, an increase in levels of 

effective coping, a decrease in levels of ineffective coping, and a decrease in levels of suicide 

risk. 

Table 6 - Comparison of Caller Ratings at Beginning and End of Calls at DCO in Pilot 

Study III 

Variable X SD X SD 

(Beg.) (Beg.) (End) (End) df t 

Distress 2.83 .99 2.18 .91 358 18.62* 

Effective Coping 2.34 .84 2.77 .84 363 - 11.16* 

Ineffective Coping 2.55 1.12 2.27 1.01 363 7.96* 

Suicide Risk 1.61 1.18 1.49 1.02 356 5.05* 

* p < .001 

28 Table A2 in Appendix A presents the psychometric properties of the measures used for rating callers at DCO. 



25 

A series of analyses 29 were conducted to examine for differences in the pattern of 

changes from the beginning to the end of the call related to sex and type of caller. Changes 

in all of the rated outcomes were found to be the same for male and female callers. 

Differences on levels of distress among the three types of callers were found 30 . 

Specifically, new callers were rated as having significantly higher levels of distress than 

repeat callers. As well, occasional callers were judged to have significantly higher levels of 

distress than repeat callers. All three groups experienced significant decreases in levels of 

distress over the course of the call 31 . Additionally, different types of callers experienced 

different patterns of change in level of distress over the course of the call. In particular, new 

callers and occasional callers experience a greater decrease in levels of distress over the 

course of the call when compared to repeat callers. 32 . Figure B4 in Appendix B provides a 

graphic representation of these different patterns of change for different types of callers.. 

Changes for the different types of callers on effective coping were also examined. All 

three groups of callers were rated by volunteers as experiencing an increase in levels of 

effective coping over the course of the call 33 . However, the pattern of change differs 

according to the type of caller 34 . Although there are no differences among the three groups 

at the beginning of the call, differences emerge at the end of the call with new callers being 

rated to have a significantly higher levels of effective coping than occasional callers 35 or 

repeat callers 36 . Figure B5 in the Appendix B shows the different patterns of change in 

effective coping according to type of caller. 

Another analysis compared differences among the three types of callers on levels of 

ineffective coping. No differences on levels of ineffective coping were found overall for the 

three types of callers. All three groups of callers experienced a decrease in levels of 

ineffective coping 37 . However, the degree of change varied by type of caller with new 

callers experiencing the greatest decrease of the three groups 38 . However, there were no 

differences among the three groups on level of ineffective coping at the end of calls. Figure 

B6 in Appendix B shows the different pattern of change in ineffective coping over time for 

the different groups. 

Another analysis examined differences related to type of caller on level of suicide 

risk. All three groups showed a significant decrease in level of suicide risk over the course of 

the call 39 . Results showed differences among the three types of callers 40 with new callers 

29 Repeated Analyses of Variance 

30 F (2, 356) = 11.24, p. < .001 

31 F (1, 356) = 335.71, p < .001 

32 F (2, 356) = 3.08, p < .001 

33 F (1, 361) = 122.63, p < .001 

34 F (3, 361) = 7.03, p. = .001 

35 t (258) = 2.22, p. < .05 

36 t (219) = 3.39, p = .001 

37 F (1,361) = 61.69, p < .001 

38 F (2, 361) = 4.85, p. < .01) 

39 F (1, 354) = 24.68, p < .001 

40 (F (2, 354) = 3.76, p. < .05) 



26 

rated as having a significantly higher level of suicide risk than repeat callers. No other 

differences emerged among the groups. 

Finally, an analysis found significant differences among the different groups of callers 

on the ratings of client satisfaction conducted on callers 41 . In particular, new callers were 

rated by volunteers as expressing higher levels of client satisfaction than repeat callers. 

Follow-up interviews and surveys. Table A3 in Appendix A presents the 

psychometric properties of the quantitative measures used in the follow-up interviews and 

surveys at DCO. 

As shown in Figure 5, total scores on the General Health Questionnaire in the followup 

interviews and surveys reveal that 67% of callers at the time of the follow-up interview or 

survey report difficulties in functioning that are consistent with a diagnosable mental health 

problem 42 . 

Figure 5 

Percentage of Follow-Up Respondents at DCO with a Diagnosable Mental Health 

Problem As Shown on the GHQ (N = 33) 

33% 

67% 

M.H. Probl. 

No M.H. Probl. 

Related to any distress they were experiencing, follow-up respondents were asked if 

their call to DCO changed the way they were feeling at the time of making the call. Of the 

35 respondents who answered the question, 91% (n = 32) said “yes” and 6% (n = 2) said no. 

The other caller said he or she didn’t know. 

In an open-ended follow-up question, callers who answered “yes” explained that 

calling DCO had produced positive feelings such as feeling more optimistic, feeling happy 

and fortunate, feeling less lonely, and feeling less frustrated and more hopeful. Callers also 

41 F (2, 363) = 3.26, p. < .05 

42 Using a cut-off score of 2/3 on the 12-item version of the General Health Questionnaire (Golberg & Williams, 

2004) 



27 

perceived calling DCO as lowering their anxiety and helping them discover new ways to 

address their problem. 

In response to a question related to whether or not calling DCO helped callers change 

their perception of their problem, 69% (n = 24) of 35 follow-up respondents stated “yes” and 

22% (n = 8) said “no”. Similarly, 68% (n = 23) indicated that they had taken some action to 

address their problem(s) since phoning DCO. As shown in Figure 6, 91% (n = 30) of followup 

respondents felt that their call to DCO helped them cope with the issue for which they 

phoned. 

Figure 6 

Percentage of Respondents Who Indicated That DCO Had Helped Them Cope 

With the Issue for Which They Phoned (N = 35) 

9% 

Helped Cope 

91% 

Did Not Help 

Cope 

In response to an open-ended question about how respondents perceived their 

problems differently as a result of phoning the DC, over half of the respondents (57%) 

indicated that they had engaged in a form of “cognitive restructuring”. For example, a 

number of respondents talked of having developed a different perspective on their problem. 

Respondents also indicated having an increased sense of hope and calmness relating to their 

problem and focusing more on the present than the past. Several respondents (17%) also 

indicated that their changed perceptions of their problem led them to engage in “support 

seeking” by establishing links to professionals who could help them. Some respondents also 

referred to taking action to address their problem as evidence of changed perceptions. 

Related to coping, another open-ended question asked respondents if they had taken 

any action about their problem since calling DCO. Almost half of respondents said they did 

with their actions including taking medication, participating in physical activities, following 

up on referrals provided by volunteers, and learning to communicate better with others. Onefifth 

of the respondents indicated engaging in support-seeking (e.g. called psychiatrist / case 

worker, talked to friends, went to a clinic) in response to their problem. 

In response to a question about whether or not calling DCO affected suicidal feelings 

or thoughts, 8 (23%) of 35 follow-up respondents said “yes” and one respondent (3%) said 



28 

“no”. The remaining 74% of respondents indicated that they were not suicidal. The majority 

of those who had been helped in this area described how these services had contributed to a 

positive change in their perspective of life (e.g., instilled hope, eased thoughts, facilitated 

consideration that change was possible). As well, a majority of these respondents indicated 

that they felt calmer after the call had ended. However, based on the suicide risk measure, 

56% of 34 follow-up respondents reported having more thoughts than usual about “life being 

not worth living” or “found themselves wishing they were dead” or “the idea of taking one’s 

life” had crossed their mind. 

As shown in Figure 7, follow-up respondents expressed high levels of satisfaction 

with the services they received at DCO. In particular, 91% of 35 respondents reported being 

either “mostly satisfied” or “very satisfied” with the services they received. Moreover, 94% 

of 34 respondents indicated that they would phone DCO again if they needed help. Finally, 

77% of 35 respondents felt that services received at DCO had met their “most” or “almost 

all” of their needs. 

Figure 7 

% of Follow-up Respondents Expressing Satisfaction Related to DCO Services 

% of participants satisfied 

100 

90 

80 

70 

60 

50 

40 

30 

20 

10 

0 

Met Needs 

Overall 

Satisfaction 

Use Services 

Again 

% Satisfied 

Question 

In addition to close-ended questions about satisfaction with services, respondents 

were asked an open-ended question about what they liked most about DCO’s telephone 

services. Appendix C presents the individual responses of callers to this question. 

A review of their responses identified four major themes as characterizing what callers liked 

about DCO services: (1) Characteristics of Services, (2) characteristics of volunteers, (3) 

quality of interventions, and (4) benefits of services to callers. Appendix C presents a list of 

these responses clustered in these themes. 



29 

Characteristics of services emerged as the category with the largest number of 

responses. Characteristics that respondents liked included anonymity, availability, ability to 

access services 24 hours per day, having someone to talk to when feeling lonely, and having 

someone there to listen. 

The second most common themes raised by respondents referred to characteristics of 

volunteers and benefits to callers. Responses about characteristics of volunteers included 

“volunteers can relate to callers’ issues”, “volunteers were caring, friendly, and helpful”, and 

“volunteers knew how to calm down callers in distress”. Benefits to callers cited by 

respondents included callers reporting being cheered up and reassured, callers finding a new 

perspective, and in one case saving a caller’s life 

Finally, a small number of respondents made comments that referred to aspects of the 

quality of services such as “(there being) a connection between volunteers and callers”, 

“(services resulting in) great experiences (to) callers”, and “(volunteers providing) good 

advice/suggestions (to callers)”. Responses not falling into any of the themes included two 

respondents reporting very positive perceptions of counselors such as saying they have 

“special gifts” and they are “wonderful”. 

A question asking respondents to make suggestions for improving DCO services was 

also asked. Appendix D presents the individual responses to this question. A review of 

responses to this question identified four broad themes: 1) Characteristics of service, 2) 

skills of volunteers, 3) knowledge of volunteers, and 4) nothing needing improvement. 

The most common theme emerging with the largest number of responses was 

characteristics of service. Suggestions for improving services in this area included allotting 

more time per call, and increasing the number of phone lines available in addition to an 

increase in the number of volunteers. Some respondents suggested that the volunteers offer 

more guidance, feedback, and more empathy during the call. 

There were also a small number of respondents whose suggestions entailed increasing 

the knowledge base of volunteers. In particular, they would like to see volunteers increase 

their awareness and understanding on specific issues such as sexual abuse, bi-polar disorder, 

and other mental health problems. Two respondents felt that the younger volunteers did not 

have enough experience to truly appreciate what the caller was experiencing. 

Feedback from Volunteers on Outcome Monitoring Methodology 

In order to review the methodology developed for the outcome monitoring system of 

crisis telephone services, interviews were conducted with volunteers from both agencies who 

participated in rating callers and with volunteers who conducted follow-up interviews. 

Ratings of callers. A total of seven volunteers from DCO and three volunteers from 

TAO were interviewed on their experience with the rating of callers. Overall, interviewed 

volunteers found the rating of callers feasible and straightforward. They noted that it became 

part of their routine to responding to callers once they became used to the rating procedures. 



30 

Some volunteers indicated that they felt their evaluations were overly subjective and 

wondered how accurate their ratings were. One volunteer did express some difficulty with 

the process stating that in cases involving regular callers no change was apparent and in cases 

of new callers it required focusing to the point that making ratings was difficult. Two 

volunteers suggested that the rating scales did not apply very well to regular callers to whom 

support is provided since these callers are not likely to experience noticeable improvements 

over the course of a call. 

Overall, most volunteers did not report having problems or difficulties in completing 

rating forms. Of those who did describe difficulties, it tended to involve the rating of client 

satisfaction using the 9-item client satisfaction scale (Crisis Call Outcome Rating Scale). 

Some volunteers found it difficult to make ratings on this scale or found that the ratings were 

not pertinent to what had transpired in a call. One respondent indicated that he or she found 

it difficult rating callers on the coping items. Another respondent noted that it was easier to 

rate longer calls (as opposed to calls lasting only 10-15 minutes). One respondent reported 

finding the ratings labour intensive, requiring him or her to “keep an inventory of behaviours 

….. and then flip back to the manual to decide how to rate it.” 

In contrast to these difficulties, a number of positive outcomes were identified by 

volunteers as resulting from their rating of callers. Some respondents indicated that rating 

callers at the beginning and end of calls helped them to focus on more intently on the caller 

and their emotions. Several volunteers found that rating calls helped them realize the 

benefits that they were bringing to callers. Some volunteers also stated that rating calls 

helped them reflect after a call on how they had responded. One respondent mentioned that 

conducting the evaluation of calls helped them better understand the “wide gradient of 

emotion and distress that is shown (by callers)”. 

Of those volunteers who commented on the training related to rating callers, all of 

them perceived it as positive and useful. One volunteer found the part related to rating 

simulated calls somewhat rushed and another volunteer thought that rating a few more calls 

would strengthen the training. The large majority of interviewed volunteers made positive 

comments about the training manual, describing it as a helpful resource. One respondent 

perceived the manual’s content as over-detailed and suggested that the terminology in it be 

changed to match that used at the Centre. 

Interviewed volunteers who rated calls were also asked about their experience with 

inviting callers to participate in a follow-up interview. A small number of volunteers 

expressed discomfort with inviting callers to participate in a follow-up interview. A couple 

of volunteers simply did not do it while other volunteers explained that they selected callers 

to invite, particularly those who were not in crisis. All of the volunteers who asked callers to 

participate in a follow-up interview reported that they always received a cooperative and 

positive response with many of them agreeing to complete an interview or survey. 

Follow-up Calls. Two volunteers who conducted interviews at DCO and one 

volunteer who conducted interviews at TAO participated in the interviews relating to followup 

calls. Overall, they described follow-up calls as having gone well. In response to a 



31 

question about positive things coming out of these calls, volunteers described how they 

highlighted the value of crisis telephone services. As well, the follow-up calls provided 

callers with a chance to thank the agency and give something back to it. 

There were concerns raised on the amount of work required to set up and conduct 

follow-up interviews relative to the number of interviews actually completed. It was noted 

that a number of callers stated they would participate yet failed to follow through by phoning 

back for an interview or completing the survey on the web. As well, some callers were not 

available at the time scheduled to phone them back. 

Difficulties encountered by volunteers in the interviews included finding themselves 

focusing on a client’s crisis or feelings rather than on the effectiveness of the crisis telephone 

services. It was indicated that negative emotions sometimes were triggered by the interview 

and some callers were suspicious of the personal nature of the interview. Interviewed 

volunteers also felt that there were too many questions in the interview on suicide and these 

made callers feel uncomfortable. Another encountered issue involved the tendency of some 

callers to depart from the interview questions in order to discuss personal issues requiring the 

volunteer to bring them back to the questions. 

Suggestions by volunteers for improving the follow-up interview included shortening 

its length including the introductory instructions, clustering questions related to each other 

together, simplifying some of the questions, relying more on open-ended questions, and 

educating future interviewers about the different scales used so that they can explain it in an 

understandable manner to callers. 

VI. Conclusions and Recommendations 

The report summarizes the work undertaken to develop an outcome monitoring 

system for crisis telephone services. The nature of the project allowed the system to be 

developed and tested at two agencies in English and French, namely the Distress Centre of 

Ottawa and Region and Tel-Aide Outaouais. The development and testing of the system in 

two agencies enabled us to evaluate the generalizeability and performance of the system in 

different settings. As well, the feedback obtained from personnel and volunteers from the 

two agencies throughout the process of developing the system helped us consider the myriad 

of issues faced with developing these systems as well as work toward developing an outcome 

monitoring system that was as pertinent as possible for crisis telephone services. 

It is important to note that the results emerging from the rating of callers were very 

similar at both agencies. In particular, the overall group of callers from both agencies were 

rated as experiencing the outcomes hypothesized from the program logic model, namely a 

decrease in distress, an increase in effective coping, a decrease in ineffective coping, and a 

decrease in suicide risk. Moreover, there were no differences in outcomes experienced by 

male callers when compared to female callers at either agency. A number of interactions 

between type of caller and improvements over time emerged in the results at both agencies. 



32 

These interactions showed some similarities across agencies with new and occasional callers 

showing greater change than repeat or regular callers. 

Follow-up interviews at DCO suggested that most callers are satisfied with the 

services they receive. A majority of callers also describe positive changes in their coping 

abilities since calling DCO 

Although results obtained in the third pilot study are indicative of callers from both 

agencies experiencing positive outcomes and DCO callers expressing satisfaction with 

received services, the collected data has a number of limitations. In particular, these 

limitations included: (1) The ratings of callers need to be interpreted in light of biases 

expected in a situation where volunteers are rating callers whom they are serving, (2) the 

evaluation of callers are based on brief one- or two-item measures used to assess fairly 

complex outcomes, and (3) follow-up interviews were conducted with a group of callers 

selected by volunteers who were not necessarily representative of the caller population. 



33 

Recommendations 

A major objective of the project was to help agencies develop a system of monitoring 

that could be integrated into the routine delivery of crisis telephone services by community 

agencies. Based on the lessons learned in the project, the following recommendations 

pertaining to caller ratings, follow-up interviews, and general procedures are proposed in 

order to accomplish this objective. 

Caller Ratings 

(1) The feedback from volunteers and the results from the third pilot study suggest that 

volunteers used the rating measures of distress (anxiety and depression), coping (effective 

and ineffective), and suicide without encountering any difficulty. Therefore, it is 

recommended that a full implementation of the outcome monitoring system use the rating 

measures of distress, coping, and suicide risk. 

(2) The feedback from volunteers indicated that they encountered some difficulties in 

implementing the caller rating scale of client satisfaction. As well, the intra-class correlation 

was found to be very low for this scale. Therefore, it is recommended that a full 

implementation of the outcome monitoring system not include the caller rating scale of client 

satisfaction. 

(3) The feedback from volunteers suggested that the training and training manual were 

successful in orienting them to using the rating scales. Therefore, it is recommended that the 

process of training volunteers and the training manual continue be used to train volunteers 

in advance of the full implementation of the outcome monitoring system. 

(4) In order to further orient volunteers and increase the inter-rater reliability of the caller 

rating scales, it is recommended that the accompaniment of volunteers include the use of the 

rating scales by volunteers and their volunteer leaders on actual calls until there is good 

correspondence between them on the caller rating scales. 

(5) Given the risk of rater drift over time without some re-calibration of volunteers’ use of 

rating scales, it is recommended that booster training sessions on the use of the scales be 

provided to volunteers on an annual basis. 

(6) Given the discomfort of some volunteers with inviting callers for a follow-up interview, 

it is recommended that training related to the evaluation include some simulated practice 

using the script developed for the project. 

Follow-up Interviews / Surveys 

(7) Based on feedback provided by volunteers on their use of the follow-up interview 

protocol, it is recommended that the mix of close-ended and open-ended questions be kept in 

the follow-up interview. 



34 

(8) In order to shorten the interview and respond to the discomfort experienced by some 

volunteers in asking about suicide, it is recommended that the three questions related to 

suicide by taken out of the questionnaire. 

(9) Given the paucity of callers who opted to use the web option, it is recommended that 

future follow-up surveys use only the two options of phoning back callers or having callers 

phone back a volunteer. 

(10) Given the amount of time required to set up and complete follow-up interviews with 

callers, it is recommended that these be planned over a set period of time on a biennial basis 

(i.e., every two years). 

(11) As well, it is recommended that a special team of 6-8 volunteers be trained to conduct 

these follow-up interviews at an agency with the objective of completing 90 interviews with 

equal representation of the different types of callers using the crisis telephone services. 

General 

(12) Both agencies participating in the current project had a computerized information 

system for coding call information. It is recommended that the caller ratings be integrated 

into this information system so that data will be collected routinely on calls by volunteers. 

(13) In order to coordinate program evaluation efforts, many community agencies have 

initiated a special committee made up of personnel and volunteers. This type of committee 

ensures that program evaluation activities are being properly conducted and that results 

emerging from these activities are used to improve services. It is recommended that agencies 

strike a program evaluation committee to manage the implementation of the outcome 

monitoring system and the reporting of results produced from it. 

(14) Given the investment of time and other resources in the evaluation of services, it is 

recommended that results emerging from an outcome monitoring system be shared with 

agency personnel and volunteers on an annual basis. 

(15) Unless the capacity for implementing the outcome monitoring system and analyzing 

data is present internally among agency personnel or volunteers, it is recommended that 

external technical assistance be purchased to assist with the full implementation of the 

system and with the analyses and summary of data. 



35 

References 

Bonneson, M. E. & Hartsough, D. M. (1987). Development of the crisis call outcome 

rating scale. Journal of Consulting and Clinical Psychology, 55, 612-614. 

Folkman, S. & Lazarus, R.S. (1988). Manual for the Ways of Coping Questionnaire. 

Palo Alto: Consulting Psychologists Press. 

Goldberg, D., & Williams, P. (2004). A User’s Guide to the General Health 

Questionnaire. London, U.K.: NFER Nelson. 

Hall, M. H., Phillips, S. D., Meillat, C., & Pickering, D. (2003). Assessing 

performance: evaluation practices & perspectives in Canada’s voluntary sector. Canadian 

Centre for Philanthropy. 

Knickerbocker, D. A. (1973). Lay volunteer and professional trainee therapeutic 

functioning and outcomes in a suicide and crisis intervention service. Dissertation Abstracts 

International, 34, 416B. (University Microfilms No. 73-15, 510). 

Larsen, D. L., Atkisson, C. C., Hargreaves, W. A., & Nguyen, T. D. (1979). 

Assessment of client / patient satisfaction : Development of a general scale. Evaluation and 

program planning, 2, 197-207. 

McEwan, K. & Goldner, E. M. (2002). Indicateurs de rendement et de reddition de 

comptes pour les services de soins et de soutien en santé mentale. Santé Canada. 

McLaughlin, J. A. & Jordan, G. B. (1999). Logic models: A tool for telling your 

program’s performance story. Evaluation and Program Planning, 22, 65-72. 

Nguyen, T. D., Attkisson, C. C., & Stegner, B. L. (1983). Assessment of patient 

satisfaction : Development and refinement of a service evaluation questionnaire. Evaluation 

and Program Planning, 6, 299-314. 

Séguin, M., Roy, F., Bouchard, M., Gallagher, R., Raymond, S., Grave, C. I.. Boyer, 

R. (2004). Programme de postvention en milieu scolaire. Association québecoise de la 

prévention de suicide. 

Skinner, Edge, Altman, & Sherwood (2003). Searching for the structure of coping: A 

review and critique of category systems for classifying ways of coping. Psychological 

Bulletin, 129, 216-269. 

United Way of America (1996). Measuring Program Outcomes : A Practical 

Approach. Authors: Alexandria, VI. 

Watson, D., Goldney, R., Fisher, L., & Merritt, M. (2001). The measurement of 

suicidal ideation. Crisis, 22, 12-14. 



36 

Appendix A 

Table A1 

Psychometric Properties of Measures Used for Rating Callers at TAO 

Variable Cronbach’s alpha Inter-rater reliability 1 

Caller Ratings at Beginning of Call 

Distress .67 .54 

Effective Coping NA .81 

Ineffective Coping NA .57 

Suicide Risk NA 1.00 

Caller Ratings at End of Call 

Distress .69 .26 

Effective Coping NA .60 

Ineffective Coping NA .49 

Suicide Risk NA .97 

Client Satisfaction .93 - .05 

1 

Inter-rater reliability was calculated by having two volunteers rate 15 calls independently. One of the 

volunteers provided the service while the other volunteer listened in on the call. 

Table A2 

Psychometric Properties of Measures Used for Rating Callers at DCO 

Variable 

Cronbach’s 

alpha 

Intra-class 

correlation 1 

Correlation of caller ratings 

to corresponding follow-up 

variables 

Caller Ratings at Beginning of Call 

Distress .57 .56 2 

.50*** (N=33) 

Effective Coping NA .83 3 NA 

Ineffective Coping NA NA 

Suicide Risk 

NA 

.40** (N=33) 

Caller Ratings at End of Call 

Distress .69 .80 2 

.31* (N=33) 

Effective Coping NA .64 3 NA 

Ineffective Coping NA NA 

Suicide Risk 

NA 

.15 (N=33) 

Client Satisfaction .95 .11 (N=34) 

1 

Intra-class correlation from training session of 15 volunteers who rated two different audio-taped simulated 

calls. Intra-correlation is calculated together for the five items measuring distress (i.e., anxiety and depression), 

effective coping, ineffective coping, and suicide risk. 

*** p < .005; **p < .05; * p < .10 



37 

Table A3 

Psychometric Properties of Measures In Follow-Up Interviews and Surveys at DCO 

Variable 

Cronbach’s 

alpha 

Correlation with caller 

ratings at beginning of 

call 

Correlation with 

caller ratings at end 

of call 

Distress .87 .50*** (N=33) .30* (N=33) 

Suicide Risk .89 .40** (N=33) .15 (N=33) 

Client Satisfaction .83 NA .11 (N=34) 

** p < .005; ** p < .05; * p < .10 



38 

Appendix B 

Figure B1 

Changes in Level of Distress by Type of Caller at TAO 

L 

e 

v 

e 

l 

o 

f 

D 

i 

s 

t 

r 

e 

s 

s 

5,00 

4,50 

4,00 

3,50 

Type of Caller 

New 

Regular 

Supported 

3,00 

Beginning of Call 

Time 

End of Call 

Figure B2 

Changes in Level of Effective Coping by Type of Caller at TAO 

2,6 

2,4 


New 

Regular 

Supported 

Level of Effective Coping 

2,2 

2 

1,8 

1,6 


Time 




39 

Figure B3 

Changes in Level of Suicide Risk by Type of Caller at TAO 

1,6 

1,5 


New 

Regular 

Supported 

Level of Suicide Risk 

1,4 

1,3 

1,2 

1,1 


Time 


Figure B4 

Change in Level of Distress by Type of Caller at DCO 

3,20 

3,00 


New 

Occasional 

Repeat 

2,80 

Level of Distress 

2,60 

2,40 

2,20 

2,00 

1,80 


Time 




40 

Figure B5 

Changes in Level of Effective Coping by Type of Caller at DCO 

3 

2,9 

Caller Status 

New 

Occasional 

Repeat 

Level of Effective Coping 

2,8 

2,7 

2,6 

2,5 

2,4 

2,3 


Time 


Figure B6 

Change in Level of Ineffective Coping by Type of Caller at DCO 

2,7 

2,6 


New 

Occasional 

Repeat 

Level of Ineffective Coping 

2,5 

2,4 

2,3 

2,2 

2,1 


Time 




41 

Appendix C 

Could you tell me in your own words what you like about the Distress Centre? 

Characteristics of Service: (N=35; n=23; 65.7%) 

- anonymity 

- Someone is there 24 hours a day. 

- someone there to listen 

- It's there in case you need it, like when your doctor/psychiatrist is not available. 

- Talk to someone in off-hours when no one else is available. Someone to bounce ideas off 

- Someone there to listen 

- Just to have someone to talk to when feeling lonely. 

- Volunteers are always there for her. 

- Available at all times 

- Someone to talk to. Available 24 hours. 

- 24 hour availability 

- They're there in case you need them 

- Someone to listen 

- Someone is there to listen when necessary 

- Talk to someone understanding of problems (who expects it) other than friends who might 

not expect such difficult topics 

- Call 24 hour/day. Knowing it is there 

- Always available 

- Available 24 hours 

- Free support. Able to do it your home (comfortable setting). 

- Very proactive. Took care of issue immediately 

- Always available 

- Someone there to talk to who will listen when you are alone 

- find appropriate referrals. 

Characteristics of volunteers: (N=35; n=16; 45.7%) 

- relate to issues 

- usually friendly, usually caring, usually genuinely wanting to help 

- friendly and helpful and understanding 

- Calming voice of volunteer. They know what you're going through and how to calm you 

down 

– Encouraging. Feeling that they care and understand her progress, make very good 

suggestions and encouragement 

- Nice, helpful people 

- Talk to people who are nice. 

- Maturity of volunteer even though only a young girl 

- People are supportive 

- Volunteer was very canny 

- Volunteers put a lot of effort into call. 

- Likes that they're non-judgemental. Likes empathy 

- Very caring and helpful 



42 

– Supportive. Good listeners 

- Non-judgemental 

- Volunteer tried hard to understand needs 

Quality of Intervention: (N=35; n=9; 25.7%) 

- Good listening 

- Great experience, good advice/suggestions 

- Good to have a different outlook 

- Got to talk to someone nice and understanding 

- Really felt heard by volunteer 

- Good sounding board. Can tell them anything 

- very satisfying results. 

- Give him the opportunity to vent his frustrations. 

- Talk to caring and patient people. A safe place to vent 

Benefits of Services to Caller: (N=35; n=14; 40%) 

- Help cheer up and reassure, focus on plans and steps. 

- Feels that volunteers pulled her out of darkness 

- Help find new perspective. 

- The connection with volunteers is a life saver. At one point was very suicidal and Distress 

Centre reached her and saved her life. 

- Different perspective on situation.. 

- Saved his life a few times. 

- Retreat from feeling alone 

- Get reassurance 

- Make her feel better. Give her self-worth and confidence 

- Felt reassured and calmed. 

- Feels heard. Help to build him up. Helps to pull him out of suicidal moods 

- Knows that call can change the way she feels (e.g. feels less suicidal after call, helps her stay off 

crack) 

- Can talk things out 

- Feels safe and trusts opinions. Feels understood She is taking medication, journaling and 

doing art work - all of these and being able to call Distress Centre help her cope with her life. 

Other: (N=35; n=7; 20%) 

- Wonderful counselors. Volunteers have special gifts 

- Likes volunteers 

- She (the volunteer) was excellent. Connected with Cumberland Resource 

Centre 

- Better to talk to a stranger than someone you know 

- Recommends to other people. 

- warm and fuzzy 

- Has had very brief contact - was transferred to tier II. Doesn't know if follow-up call will 

help situation. Contact with DC volunteers was fine. Good to have a link to the hospital as 

caller phones from Montreal regarding her sister in Ottawa. 



43 

No Response: (N=35; n=1; 2.9%) 



44 

Appendix D 

Could you tell me in your own words what could be improved at the Distress Centre? 

Characteristics of Service: (N=35; n=12; 34.3%) 

- Some are clearly reading from notes on how to handle a caller and reveal the reason they 

are asking the questions they ask. These are helping from the book which leaves the heart out 

and usually is no help at all. Fortunately these are in the minority 

- More hours with service? 

- Message feels discouraging on Distress Centre phone line. The phone message 

on the Mental Health Crisis Line tells callers to stay on line while DC tells caller to hang up 

and try again. She feels that this message is discouraging. 

- Need for more volunteers. Often busy signals. Wishes she could talk longer. 

- Nothing really. Maybe more time per person. 

- More frequent time. More time per person (not just 10 minutes). 

- Make it easier to get through to someone because it gets frustrating when you can't talk to 

someone. 

- Wishes there was a Distress Centre one could visit. 

- Doesn't connect with all the volunteers. 

- Better availability for getting through 

- Hard to get through in middle of night when having a panic attack. 

- Wishes that it were easier to get through - often busy- which is difficult when one is having 

a panic attack. 

Skills of Volunteers: (N=35; n=5; 14.3%) 

- When the volunteers do not understand the problem, they frequently just end the call even if 

the caller is in tears and very upset. Many do not know when and how to apply the time limit 

rules. 

- Sometimes more empathy, sometimes more aggressive if the person is suicidal which I 

have been several times. 

- More advice needed. Feels that pours out heart and Distress Centre does nothing to help 

him or change his situation. 

- Some men are not totally comfortable with his homosexuality. 

- Some volunteers are too agreeable and don't give enough suggestions. 

Knowledge of Volunteers: (N=35; n=5; 14.3%) 

- Younger volunteers can't relate to issues. They don't have the breadth to understand what 

older people are experiencing 

- 'solutions to helping others 

- More specific information about bipolar disorder and other mental illnesses. 

- Wishes there were more older volunteers 

- More training around sexual assault and abuse 



45 

No Improvement or No Response: (N=35; n=17; 48.5%)

Development of an outcome monitoring system for crisis telephone ...

Create successful ePaper yourself

Delete template?

Save as template?