think-cell technical report TC2003/01 A GUI-based Interaction ...

2.2 Quantitative Aspects FIELD STUDY
The intention of the quantitative study is to clarify which PowerPoint-related
problems have the most impact on the over-all working process. At the same time,
data emerging from a quantitative evaluation of current PowerPoint use provides a
baseline for comparative follow-up studies. A case study that evaluates the impact
of my work is presented in chapter 6.
2.2.2 Methods and Data Validity
While data collected on-site and during the actual process is most authentic, it
takes too much time to collect enough data for a valid statistic evaluation this
way. Therefore, mass data was collected by indirect methods off-site, while on-site
observation was used to check the results for plausibility.
Again, data was largely collected from a single consultancy office. Comparison
with sample presentations from other consultancies showed that – with little devi-
ation of the absolute numbers – the general layout and the use of visual elements
and chart types is sufficiently similar across different consultancies to generalize the
findings of the study.
Live Timing. As a first approach to quantitative data, I did on-site user observa-
tion and live timing of PowerPoint related activities. For each item, some specifics
including the type of activity and the skill level of the performing user were recorded
along with the actual time required to perform the task. Moreover, for each tuple
of data the affected slides were kept to allow a comparative study later.
Expert Ratings. To collect a statistically relevant amount of data in reasonable
time, I decided to rely on off-site expert ratings. Together with an experienced
visual, I went through 6 entire presentations that were selected as a representative
sample, with a total of 371 slides. For each slide, I had the expert estimate the total
time required to create the slide as well as times for text input and single elements
like drawings and charts.
Data collected from expert ratings was backed up with a random sample of other
experts’ ratings and with live timings. The verification confirmed the estimated
times to be plausible and accurate.
Analysis and Counting. As another off-site method to efficiently collect larger
amounts of data, I went through 9 presentations (567 slides) that were rated to be
representative. I categorized graphical elements and chart types and counted the
occurrences.
Moreover, I put aside those slides that appeared especially difficult for computer
support. The sample of difficult slides helps to validate new layout concepts and to
clearly define the requirements for a better solution.
Interviews. Just as for qualitative data, interviews are an appropriate method to
access hard-to-measure quantitative data as well. In this case, personal interviews
were the only way for me to collect data concerning the consultants’ point of view.
21