think aloud protocol – hi fi #1

ELNthink aloud protocol – hi fi #1Anasassia Drofa | Peter Jones | Ben Koh | Rachel Shipman | Ray SuOverview of activitiesThe goal of this Think-Aloud is to evaluate the validity and usability of the 1 st iteration ofour high fidelity prototype of the MEDRAD ELN. Specifically, we will evaluate:1. The navigational flow of our ELN prototype as a partially functioning system thatallows users to accomplish actual tasks associated with ideation anddocumentation2. The usefulness and usability of the prototype at a finer level: layout, buttons,widgets, and associated terminology3. The look and feel of the prototype: color choices, logos, typographical hierarchy4. To receive feedback from users about how the interface supports the user’s abilityto document and collaborate on their ideasThere are limitations of what this Think-Aloud session is meant to discover. Because thetest will be accomplished with a system interface that is only partially functional, someuser interactions may be limited to the set of functionality that is currently provided.Since the user’s ability to explore the interface will be limited in some cases, this must betaken into account during the evaluation.At the end of this session, we expect to deliver Usability Aspect Reports (UARs) that willidentify critical events in the use of our Hi-Fi prototype. These reports will help informand justify our design changes in the next iteration.Protocol for activitiesThe session will comprise of an introduction to the ELN system, an introduction to thepurpose of the study, a brief warm-up exercise to familiarize the participant on how toperform a Think-Aloud, 2 separate Think-Aloud tasks, and finally a post observationalwrap-up discussion.

Target participants: MEDRAD Research Scientists and EngineersRecording device: video cameraUser Session Kit:MEDRAD ELN system charterTask descriptionsPrototype systemModerator Kit:Criteria for establishing Critical Incidents ListThis protocolTimeframe: 2 sessions with 4 different individuals are scheduled for July 19, 20061:00 PM - 2:00 PM: Marty Uram1:00 PM - 2:00 PM: George Misic2:00 PM - 3:00 PM: Jeff Hicks2:00 PM - 3:00 PM: Mike YanielloThe Session:Introduction to MEDRAD ELN: (5 mins)Prior to introducing the Think-Aloud session, the moderator will present the MEDRADELN system charter to ensure that the participant is familiar with the components andcontext the system will be used in.Introduction to the purpose of the study (5 mins)Briefly explain the external goal of the study. Explain that we’re interested in testing thesystem that we’re about to present and that we’re not testing the user specifically. Ensurethe participant that there are no wrong actions and participation is completely voluntary.Think-Aloud Warm-up exercise (5 mins)Explain the concept of Think-Aloud. Ask the participant to tell us everything they’rethinking about from the moment they read the task and when they complete it. Tell themthat they don’t need to plan/think out what they want to say. Just act as if they’re bythemselves, talking to themselves. Emphasize that the important thing is to keep talking.Briefly explain that if you notice moments of silence for a long time, that you mayinterrupt and ask them to keep talking.To help the user get a feel for actually performing a Think-Aloud, perform a sampleThink-Aloud. For example:

Think aloud as you count how many windows are in your house.Now, ask the user to try just as you did. Another example follows.“Please think aloud as you name how many doors are in your house?”or“Please re-count your actions in your morning routine before you came to work.”Next, establish some rules during the session.1. You will not be able to answer any questions during the observation2. If you have questions, go ahead and ask them, but you won’t respond until afterthe session is complete.3. Remind them that if they’re silent for more than 5-10 seconds, you will ask themto “Please keep talking”.Finally, emphasize that the think aloud session will involve the use oflimited prototype to simulate a computer application. Ask them to interact withthe interface as if they were working with an actual production system.The Think-Aloud Tasks (30 mins)To begin the Think-Aloud session, introduce the task to the participant by giving them atask description sheet to read and to follow as they perform the task.When they’re finished, remember to tell them to “Please keep talking”. Introduce thefirst screen from the ‘Task and screen order script’. As you observe for critical incidents,look for the appropriate moments to assist the user in cases where the prototype’sfunctionality is incomplete. Continue until the task is completed.During observation, please keep the following in mind:o "Please Keep Talking"o "Please don't try to explain to me what you're doing. Just ask as if you are alone,speaking to yourself as you solve the problem"o If the user gets frustrated, take a quick break.Continue the same process through the other tasks.Post Observation Wrap-up Discussion (5 mins)Following the completion of the tasks, ask the user if they have any opinions/suggestionsabout the product they just tested.Outcome of research: From this Think-Aloud session, we will have gained a betterunderstanding of the mental model our users employ when executing the tasks we seekto demonstrate in our ELN prototype. We will have evaluated the screen flows andusability of the interface elements. This will enable us to make more informed design

decisions as we finalize our prototype design. We will have also validated our conceptwith users and gained end-user buy-in as a result of involving them in the design of oursolution. This will play an important role in determining the adoption rate of the systemwhen it is eventually deployed and prototyped at MEDRAD.Think-Aloud – Establishing Criteria for Critical IncidentsPossible Criteria for ProblemsThe user articulates a goal and cannot succeed in attaining that goal within threeminutes (then the experimenter steps in and shows the user what to do).The user articulates a goal, tries several things, and then explicitly gives up.The user articulates a goal and has to try more than three things to find thesolution.The user does not succeed in a task. That is, there is a difference between whatyou asked the user to do in the task and what they actually did.The user expresses distressed surprise.The user describes something as a negative effect or says that something is aproblem.The user makes a design suggestion.Possible Criteria for Good FeaturesThe user describes something as a positive effect or says that something is reallyeasy.The user expresses happy surprise.Some previous analysis (e.g., a Heuristic Evaluation) has predicted a usabilityproblem, but this user has no difficulty with that aspect of the system.

Task #1 version 1Capture a new ideaPlease take a moment and to think about an idea you had or worked on recently. Imaginethis idea just came to you and you needed to jot it down. Please use the prototype quickcapture tool to record that idea.Please think aloud as you go through the task. This is a prototype system, so please befrank about things that are unclear. If you have suggestions/comments, please keepthem in mind until the end of the task.

Task #1 version 2Capture a new ideaYou were at home one evening when you slipped in the bathroom and hit your head onthe toilet. When you came to, you had a brilliant idea for a time machine. Right after thishappened, you called Ned Uber to share your excitement, and he gave you a suggestionfor using incandescent lamps. You want to capture this idea and Ned’s contributionquickly so that you don’t forget the details. Please use the “quick capture tool” prototypeto record this idea.The idea description is:It consists of a box with three small, flashing incandescent lamps arrangedas a "Y", located above and behind the passenger's seat of the De Loreantime machine. As the car nears 88 miles per hour, the light of the fluxcapacitor pulses faster until it has a steady stream of light.On the desktop of your computer you will find a sketch of the flux capacitor. Pleaseinclude this with the description of the idea.Although the idea is kind of wild, you are very curious to see what people think. Youmight want to get feedback from one or two colleagues.Please think aloud as you go through the task. This is a prototype system, so please befrank about things that are unclear. If you have suggestions/comments, please keepthem in mind until the end of the task.

Task #2 version 1Update your ideaNow imagine that you have some time to work on the idea. Using the Incubatorprototype, find the idea and pretend to develop it. For example, you might addinformation about a prototype you created and tested. You could also add the results of aprior art search or anything else you would do to build on the idea.Summary:1. Find the idea that you want to update.2. Add to the idea3. SavePlease think aloud as you go through the task. This is a preliminary prototype, so pleasebe frank about things that are unclear. If you have suggestions/comments, please keepthem in mind until the end of the task.THANK YOU!

Task #2 version 2Update your ideaLater that day you have a chance to build a prototype and test it. The tests for theprototype and are in the Excel file on your desktop. The data suggests that the prototypeshould be changed. Here is the description of the change:In order to travel through time it requires 1.21 gigawatts (GW) of electricityoriginally supplied by some kind of a plutonium powered nuclear reactor.You want to document the test results and necessary change.Summary:4. Find the idea that you want to update.5. Document your test data, including the Excel file6. Add your suggested changes to the idea7. SavePlease think aloud as you go through the task. This is a preliminary prototype, so pleasebe frank about things that are unclear. If you have suggestions/comments, please keepthem in mind until the end of the task.THANK YOU!