Richard de Neufville's TPP SM Thesis Manual - Title Page - MIT

instruments, or between observers, it is inappropriate to have faith in the results.An essential part of the development of instrumentation is the determination ofthe conditions that lead to reliable results. Whether we are observingtemperature or unemployment rates, we require means of observation, that isinstruments, which are reliable. (Note that the word instrument denotes not onlydevices for measuring physical phenomena, such as a spectrometer, but alsodevices for measuring social phenomena, such as questionnaires.) Reliability isthe factor that enables us to perceive real changes in the phenomenon observed.As emphasized in basic physical laboratories, standard conditions (as fortemperature and pressure) are required to achieve reliable measurements. Thestandard conditions must of course be relevant to the nature of the phenomenon.Questionnaires or interviews are notoriously unreliable unless controls areapplied rigorously. The personal relationship between the persons involved mayeasily distort the responses and lead to quite unreliable results. Differentobservers, or the same observer in different situations, may phrase the questionsdifferently or with a different nuance. The respondent may sense the questiondifferently or may take it more or less seriously.People who collect data by interview must therefore pay particular attention tostandardizing their questions. A good way to do this is by using computergenerated or written questionnaires so that the same questions are asked, in thesame way and the same sequence, each time. Even if the questioner cannot getthe respondent to sit at a keyboard or provide written answers, it is useful to usea predefined questionnaire as the basis for the oral questions.Validity: Reliability does not guarantee that observations are correct. It isentirely possible to replicate erroneous readings consistently. Instruments areoften biased. A scale, for example, may easily be loaded to give inflatedreadings; a person may be color-blind and systematically fail to observe somecolors.Validity refers to the degree to which a measurement provides a correct measureof a phenomenon. It is a concept quite different from reliability. Just as onemeasurement may be reliable yet not valid, another may be valid but unreliable.The question of validity is especially problematic for measurements doneconcerning individuals. People have all kinds of respectable reasons for notrevealing or for distorting their feelings. They may begin by thinking that thequestions are an invasion of privacy. They may not trust the interviewer not touse the information against them. They may feel embarrassed by their answers(will people think my income too low?). They may not wish to contradict someofficial position expressed by their superior, or to express some idea that mightseem disloyal (can one normally expects many employees to speak ill of a majornew product just introduced by a company?) Finally, it is presumptuous toThesis Definition and Preparation - Richard de Neufville 32