Housing Counseling Process Evaluation and Design of ... - HUD User

The specific questions asked of respondents varied somewhat based on the types of education and
counseling services offered. Some questions were designed to gather information on each type of
education and counseling service identified on HUD’s 9902 form. The survey attempted to reduce
the response burden by only having agencies answer questions related to specific types of counseling
services if they had at least 25 clients for this type of service in the previous year (FY 2006).
Information on past client volumes was identified from the agencies’ 9902 data submitted to HUD so
this was not asked in the survey, but agencies that did not report 9902 data in FY 2006 were asked in
the survey whether they exceeded this cutoff. Based on the FY 2006 9902 data, agencies that served
fewer than 25 clients annually account for only 1 to 5 percent of total client volumes in each type of
education or counseling service. So those asked these questions still accounted for the vast majority
of the industry’s counseling activity. Nonetheless, since many agencies have small client volumes,
this cutoff resulted in about a 40 percent reduction in the number of agencies that were asked to
answer questions about specific counseling service types.
Implementation of the Survey
The survey was conducted as a web-based survey, although respondents had the option of completing
a paper version of the survey. The web-based survey allowed for multiple persons to participate in
completing the survey in the event that different members of the organization were needed to
complete different sections of the survey. As a result, the survey was designed to allow respondents
to save their answers but allow for re-entry to the survey at a later time to respond to additional
questions. A survey was not considered completed until the respondent chose to submit the survey.
The survey was conducted by Matrix Performance Consulting (Matrix), under contract to Abt
Associates. The survey was pretested in three phases on a small sample of agencies to evaluate
whether the questions were clear, the options offered complete, and whether the effort needed to
complete the survey was excessive. The first two rounds of testing relied on paper copies of the
instrument, while the final round of testing (conducted after OMB clearance was obtained) tested the
web-based survey instrument. The survey field period began on November 7, 2007, with the sample
58
released in three waves of roughly equal size between November 7 th and November 21 st . A letter
was sent by U.S. Mail to each target respondent a week before the survey was fielded. The survey
instrument itself was sent via email to the agency point of contact identified in HUD’s administrative
data. Follow-up occurred via email and telephone throughout the field period, along with a final
reminder letter by U.S. Mail three weeks before the close of the survey.
58
Of note, there was a mistake with 6 percent of the sample (94 agencies) which was inadvertently left out of
the initial wave of email releases. This mistake was not discovered until January, when there were 10 days
left in the field period. Agencies were notified immediately by telephone to encourage a quick response.
As noted below, a 36 percent response rate was achieved with this group despite the short field period.
168
Appendix B. Survey of HUD-Approved Counseling Agencies