Phrasing Techniques for Multi-Stroke Selection ... - Cornell University

Phrasing Techniques for Multi-Stroke Selection GesturesKen Hinckley 1 , Francois Guimbretiere 2 , Maneesh Agrawala 1 , Georg Apitz 2 , Nicholas Chen 21 Microsoft Research, One Microsoft Way, Redmond, WA 980522 University of Maryland, A.V. Williams Building, College Park, MD 20742kenh@microsoft.com, francois@cs.umd.edu, maneesh@cs.berkeley.edu, geapi@cs.umd.edu, nchen@cs.umd.eduABSTRACTPen gesture interfaces have difficulty supporting arbitrarymultiple-stroke selections because lifting the pen introducesambiguity as to whether the next stroke should add to the existingselection, or begin a new one. We explore and evaluate techniquesthat use a non-preferred-hand button or touchpad to phrasetogether one or more independent pen strokes into a unitary multistrokegesture. We then illustrate how such phrasing techniquescan support multiple-stroke selection gestures with tapping,crossing, lassoing, disjoint selection, circles of exclusion,selection decorations, and implicit grouping operations. Thesecapabilities extend the expressiveness of pen gesture interfacesand suggest new directions for multiple-stroke pen inputtechniques.to provide the option to articulate complex gestures in a singlestroke, pen interfaces should not necessarily require users to doso. In our Move example, if the selection is complex, or if thetarget for the Move operation is on another page, a unistrokesyntax may be cumbersome or infeasible. Furthermore, it isdifficult to support a mix of selection techniques includingtapping, crossing, and lassoing disjoint sets of objects within aunistroke syntax. Another problem with a strictly unistrokestrategy is that users sometimes want to pull the pen away, forexample while forming a complex selection, so that they can viewthe screen to verify progress without their hand occluding it.CR Categories: H.5.2 [Information Interfaces and Presentation]:InputKeywords: phrasing, tablets, gestures, pen input, multiple strokes1 INTRODUCTIONA number of pen gesture interfaces reported in the literature havesought to phrase together multiple subtasks within a single penstroke [2,5,6,8,9,11,22]. For example, a single pen gesture canintegrate the verb, direct object, and indirect object for a Movecommand by lassoing the objects to move, extending the penstroke outside of the lasso, and then lifting the pen to indicatewhere to move the selected objects [11]. This unistroke strategy isextremely tempting for designers of pen-operated devices. Pencontact with the screen phrases multiple subtasks together into asingle cognitive chunk [5], and also may help avoid theintroduction of persistent modes. The unistroke strategy alsoamortizes any overhead cost required to initiate a gesture acrossmultiple subtasks.However, recent results suggest a muliple-stroke strategy canoffer advantages. Zhao & Balakrishnan [26] have shown that“simple marking menus” that use multiple straight pen strokes tonavigate through a hierarchical marking menu are fast and resultin lower error rates than traditional compound marking menus[13] that require a single stroke (with pauses or inflection pointsdelimiting the hierarchy). Guimbretiere and Apitz [2] proposecrossing-based widgets in CrossY. The user can draw a singlestroke that crosses multiple widgets, but Guimbretiere and Apitzalso report that “novice users will perform one command at atime,” so CrossY permits the user to draw multiple strokes thatindividually cross each widget in a dialog box. If CrossY requiredeverything to be done in a single continuous stroke, then a mistakelate in the process would force the user to start over. The userexits a CrossY dialog box by crossing the border, which serves asa cue to the system that the multiple-stroke input is complete.Thus, while it is sometimes desirable (particularly for experts)Fig. 1. Multiple-stroke alignment. Top: The user crosses thebottom of each word, then a third stroke with a pigtail activates amarking menu, which contains the Align command. The numbersindicate the starting position of each subsequent stroke.Bottom: Vertical crossing strokes allow the user to employ thesame command gesture to perform a different type of alignment.We advocate multiple-stroke selection gestures and demonstratehow some operations that are difficult to express with traditionalgraphical interfaces, such as alignment of specific edges ofmultiple objects (Fig. 1), become quite natural to articulate withpen gestures. However, previous gestural interfaces have haddifficulty supporting arbitrary multiple-stroke selections becauselifting the pen introduces ambiguity as to whether the next strokeshould add to the existing selection, or begin a new one. Forexample, Kurtenbach & Buxton’s GEdit prototype [11] supportscircles of exclusion by allowing the user to draw a lasso-within-alasso,but does not allow the user to add to the selection nor issuea single command that acts on two or more disjoint lassos. GEditalso cannot support crossing as a selection gesture because astraight-line stroke always triggers a marking menu command.Tivoli [20] supports bump and bite gestures that add or removeobjects from a lasso selection, but these gestures “must begin andend on the loop” [20], which makes them difficult to draw andprecludes disjoint selection regions. Thus, Tivoli and GEdit canonly support multiple-stroke selection gestures in special caseswhere a pen stroke starts on or within a prior lasso selection.We first explore techniques that use a nonpreferred-hand buttonor touchpad to phrase together one or more independent penstrokes into a unitary multi-stroke gesture. Our experimentalresults suggest that users prefer a button and that a design that

equires muscle tension from the nonpreferred hand for the fullcourse of a phrase results in a very low error rate (2.5%). We thenillustrate how this design can support multiple-stroke selectiongestures with tapping, crossing, lassoing, disjoint selection, andcircles of exclusion. We also propose novel selection techniquesincluding selection decorations and implicit groups. Together,these contributions extend the expressiveness of pen gestureinterfaces and may suggest new directions for multiple-stroke peninteraction techniques.2 RELATED WORKBuxton [5] discusses phrasing as a way to organize subtasks intocognitive subroutines. He suggests that phrasing accelerates theproceduralization of knowledge into routine skills. Buxton raisesthe example of a pull-down menu, which uses tension on themouse button to “glue together” the subtasks of invoking themenu, navigating to an item, and validating the commandselection. This perspective motivated Sellen et al. [25] to conducta study which shows that switching modes by pressing andholding a footpedal reduces mode errors in a text editor.Li et al. [15] demonstrate that a nonpreferred-hand button offersa quick and effective mechanism for mode switching betweenplain ink strokes and unistroke gestures. Scriboli [9] uses such abutton primarily to support a unistroke syntax that delimits thecommand from the selection by utilizing a self-crossing pigtailwithin the stroke. In this paper, we adopt the pigtail, but seek touse it as a way to transition to command selection in multiplestrokephrases rather than as a delimiter within unistroke gestures.Other techniques, such as the selection handle proposed byKurtenbach [11], could play a similar role but we chose not to usethem because handles on multiple selection strokes becomeconfusing and clutter the display. Scriboli [9] and PapierCraft [16]suggest that users might hold the gesture mode across multiplestrokes, but neither evaluates this approach.Some pen systems rely on recognition-based approaches toavoid explicit ink/gesture switching [14,23]. Even if we assumerecognition is always correct, which of course it is not, afundamental problem with automatic approaches is that thesystem cannot classify a set of strokes as a gesture or as ink untilafter the user has finished drawing the entire command phrase.This makes it difficult to provide incremental feedback or toprompt the user with available commands before the user commitsto an operation. Our study establishes that a simple button can beused for robust articulation of multi-stroke phrases that may bedifficult to support with purely recognition-based approaches.3 PHRASING TECHNIQUESA phrasing technique is the interaction technique used by a systemto define the boundaries of a single command phrase in terms ofelemental input actions. For example, if a system supports disjointselections, then how does the user indicate that an individualselection region should be added to a selection, or start a newselection? A phrasing technique must consider both the timing ofthe input (when can the user start or end the phrase?) and thenature of the input (what does the user do to start, maintain, or enda phrase?). The design also must consider that the user mightchange his mind or make a mistake at any point in the process, sothe technique must be flexible and allow the user to abandon anincomplete phrase.Previous work [5,25] suggests that the degree of tensionrequired of the user may be an important aspect of phrasing.Buxton [5] argues that “we can use tension and closure to developa phrase structure [that] reinforces the chunking that we are tryingto establish,” but we are not aware of any study that hasinvestigated the effectiveness of phrasing techniques for multiplestrokepen gestures. We use the term phrase tension to mean theuse of physical tension to ‘chunk’ actions, and we adopt this as aguiding principle to devise several possible phrasing techniquesthat combine a nonpreferred hand button with a pen held in thepreferred hand, as follows:Full-tension: The user must maintain muscular tensionthroughout the phrase. The user presses and holds thenonpreferred hand button at the beginning of the phrase, and mustcontinue holding it until the end of the last pen stroke (Fig. 2).That is, the user must lift the button to end the command phrase.Half-tension: The user presses and holds the mode switchbutton while perfoming one or more strokes to indicate theselection, and must continue to do so until a pigtail terminates theselection, and activates a marking menu. The user appliesmuscular tension at the start of a phrase, and maintains it throughthe climax of a phrase, but can relax after that (Fig. 3). This is thetechnique proposed by Scriboli: “the user can combine variousscopes into a single phrase that is terminated by a special operator(the pigtail)” [9]. One virtue of the half-tension technique is that itallows the user to issue several commands in a row whilecontinuing to hold the button, since a pigtail (rather than a buttonupevent) separates commands from one another. However, ourexperiment does not investigate this task scenario.Low-tension: Muscular tension of the nonpreferred handintroduces a new phrase, and the phrase is maintined by holdingthe pen in proximity of the screen. That is, the user clicks (pressesand releases) the nonpreferred-hand button to start the phrase,keeps the pen on or near the screen to maintain the phrase, andfinally pulls the pen away to end the phrase. We found that it wasnecessary to ignore brief unintentional gaps where the pen goesout-of-range before returning to proximity; we used a 600mstime-out to bridge these gaps (Fig. 4). We explored this techniquefor two reasons: (1) it was unclear if users were willing to holddown the button for extended periods; (2) if proximity of the pencould be used to phrase strokes together, this would permit aninitial click of either a graphical button or the pen barrel button asalternatives to a nonpreferred-hand button (which may not alwaysbe available). For our experiment, however, to make a rigorouscomparison between techniques, we evaluated a nonpreferredhand mechanism for all three phrasing techniques.Button vs. Touchpad: Our phrasing techniques can beimplemented with a physical button, or with a touch-sensitivesurface such as a touchpad. A touch-sensitive surface may offeradvantages since it could easily provide a large area or longtouch-sensitive strip for mode switching, thus reducing the needfor a small button at a specific location. It also may require lessstrain than holding down a button, and potentially could supportsecondary tasks such as bimanual scrolling [7].Holding a pen’s barrel button might offer another way to phrasestrokes together, but previous work has demonstrated that anonpreferred-hand button outperforms a barrel button [15]. Also,current tablets cannot sense a press or release of the barrel buttonwhile the pen is out-of-range, making the technique impractical.penbutton211inkingcross menu boundaryS 1 S 2 S 3 C 1D 1D iselectioncmddragextra strokesallowedRelease buttonto end trial0Fig. 2. Full-tension technique timing diagram. The user drawsthree ink circles, then three selections (S 1, S 2, S 3). The user issuesa Move command (C 1) and drags (D 1) the selected objects to adesired location. The user may lift the pen while dragging (D i).Lifting the button ends the phrase.

penbuttonpenbutton211021010inkingcross menu boundaryS 1 S 2 S 3 C 1D 1selectioncmddraglift pen to end trialRelease buttonany time afterpigtailFig. 3. Half-tension technique timing diagram.inkingPhrase ends oncepen stays out-of-S 1 S 2 S 3C 1D 1range for 600msBrief gaps with pen outof-rangeare permittedselection cmddragFig. 4. Low-tension technique timing diagram.Move command. The trials alternated between selecting anddragging the items from left-to-right, and then right-to-left.The user issued a Move command by drawing a pigtail andheading south in the marking menu (Fig. 2). The user dragged thenumbers to their final position by extending the marking menustroke beyond the outer boundary of the menu [8,9,22] and liftingthe pen, which ended the trial for the half-tension and low-tensiontechniques. In the full-tension condition, lifting the button endedthe trial, so it was possible for users to lift the pen while dragging;however users rarely employed this capability since the task didnot require multiple dragging strokes. This is a property of thefull-tension technique that should be further explored in futureexperiments.4 EXPERIMENTAL TASKWe conducted an experiment to assess the speed and error rates ofthe full-tension, half-tension, and low-tension phrasing techniquesas realized using both a nonpreferred-hand button and anonpreffered-hand touchpad as the mode switch control (Fig. 5).Fig. 5.Set-up. Left: touchpad. Right: Ctrl key device.We decided to investigate the effectiveness of the abovephrasing techniques for a Move command similar to that discussedin the introduction. To assess the phrasing techniques, however,our experimental task employs a Move command with multipledirect objects (i.e. multiple disjoint selections). Thus, our task hadusers switch to gesture mode and form a disjoint selection. Usersthen chose the verb (the Move command) from a 4-way markingmenu activated by making a pigtail (a cursive e) gesture [9].Finally, users dragged the pen to indicate the location to Move theobjects to (i.e. the indirect object), and lifted the pen. Since thenonpreferred-hand device serves as a mode switch between inkingand gesturing, our experimental task requires users to enter a mixof ink strokes and gesture strokes. The inking subtask consists ofcircling objects, rather than free-form writing, to reducevariablity. This task is suitable for experimental study while alsobeing representative of a realistic task.The task presented two sets numbers arranged in a 3x3 grid,with one set on each side of the screen (Fig. 6). To start, threenumbers in the left set were highlighted in blue. The user circledthe blue numbers in ink mode; each number turned dark red whenit was successfully circled. Three different numbers in the gridthen turned orange, and the user switched to gesture mode andselected these numbers using three separate lasso selections withauto-completion [18].The user’s task was to drag these selected numbers to thecorresponding empty positions in the right grid (Fig. 6) using theFig. 6. Task screen: The user has just circled (1,5,7) with ink(left grid). The user then switches to gesture mode, selects thenumbers highlighted in orange (2,6,8), and uses the Movecommand to drag them to the empty slots (at right).For each trial, the starting 3x3 grid alternated between the leftand right sides of the screen. The experiment provided auditorycues [19] when the user initiated gesture mode or selected theMove command. Any errors committed by the user produced anerror sound. Moving the wrong numbers, moving the numbers tothe wrong position, or drawing ink during the selection phase(indicating failure to switch to gesture mode) were all recorded aserrors. Selecting the wrong command from the pigtail-activatedmarking menu also was counted as an error, but users neveractually committed this error during any experimental trial.Exiting gesture mode at the wrong time also was counted as anerror. For the low-tension condition, this error occurred if the penmoved out-of-range for longer than 600ms before the Movecommand was issued. For the half-tension condition, this erroroccurred if the user released the button prior to drawing the pigtailto issue the command. For the full-tension condition, releasing thebutton too early sometimes could not be distinguished from amove-wrong-position error. Because it was sometimes difficult todetermine the type of error, and because a single mistake often ledto a cascade of multiple errors within a trial, we treated errors as apass/fail outcome per trial, rather than summing the total numberof errors.4.1 Mode Switching DevicesFor our nonpreferred-hand mode switching button we used theCtrl key on a keyboard placed next to the Tablet, because ourTablet PC lacked a suitable bezel button. For the touchpad, weused the contact area sensed by a Synaptics touchpad to estimatethe pressure exerted on the pad. Our system enters gesture modewhen the contact area exceeds a threshold. At the start of theexperiment, we ran a brief calibration sequence to set thethreshold for each user to provide a good compromise betweenrequiring heavy pressure, versus accidental activation on lighttouches.4.2 Participants and ApparatusWe recruited 12 undergraduate and graduate university students asparticipants in the experiment. Of the 12 participants, 11 were

ight-handed and 7 were male. Each participant received $20 fortheir participation in the experiment. None of the participants hadpreviously used a Tablet PC.Each participant used a Toshiba Protégé Tablet PC (1.5 GHz,512 MB) at 1024x768 (307mm diagonal) a desk in slate mode.The tablet had an external keyboard and a Synaptics touchpad toserve as the mode switching devices (Fig. 5).4.3 Experiment DesignWe used a 3x2 within-subjects design, so each user saw allconditions. Independent variables were Phrasing Technique (lowtension,half-tension, full-tension) and Device (touchpad, button).Dependent variables were time to complete the entire gesturephrase, and error rate. Order of presentation of the phrasingtechniques was fully counterbalanced. For each of the 6 possibleorders, one participant used the Button (Ctrl key) first and anotherused the Touchpad first. The 12 participants each performed:3 Phrasing Techniques (Low, Half, Full) x2 Devices (Button, Touchpad) x40 practice trials + 30 timed trials= 420 trials per subject (180 timed trials)= 5,040 total trials (2880 practice, 2160 timed)5 RESULTSWe discarded the first two experimental trials to avoid any startupeffects, and used trials 3-30 for analysis. All outliers more than3 standard deviations from the mean were removed (a total of1.63% of all trials).A 3 x 2 within-subject ANOVA on error rate for the factors ofPhrasing Technique and Device (Fig. 7) revealed a significantmain effect of Phrasing Technique (F 2,22 ,=4.914, p

operation after a mistake incurred a 600ms delay (for the penproximitytimeout to expire and exit gesture mode).For the half-tension technique, we instructed participants toremove their finger as soon as they felt it was convenient to do so,yet 6/12 participants tended to keep the button pressed until theend of the phrase, thus revealing a strong tendency to hold tensionfor an entire chunk [5].The full-tension technique was very reliable: users exhibited noerrors at all in 14 of the 24 sessions. But many users disliked therequirement to coordinate both hands at the end of the task. Basedon this feedback, we altered our implementation to be forgivingwith respect to whether the pen lifts first or the button lifts first atthe end of the phrase. The dotted rising and falling edges for thebutton (Fig. 10) show how our design iteration relaxes the relativetiming of the button press/release events with respect to thecorresponding pen-up/pen-down events. In informal testing, thisappears to foster the same usage model without requiring handsynchronization at the end of a phrase.penbutton211S 1S 2selectioncross menu boundaryC 1 D 1D icmddragging0Fig. 10. Final full-tension technique. The dotted rising and fallingedges for the button show how relative ordering of pen/buttonevents is flexible in our revised implementation.6 MULTIPLE-STROKE SELECTION TECHNIQUESOur study shows that the full-tension technique in particular is apractical and robust way to articulate multiple-stroke gestures. Toillustrate the expressive power of multiple-stroke gestures, thissection demonstrates how pen gesture commands can benefit fromselections specified using multiple strokes.We implemented a prototype sketching application thatsupports selection of objects by lassoing, tapping on an object,crossing (drawing a straight line that crosses at least one edge ofan object), or drawing a carat that points to the object (Fig. 11).We implemented several commands, including Align, Rotate, andFlip, to illustrate how the spatial properties indicated by theseselection operators can be leveraged and composed to createpowerful, general-purpose commands.Fig. 11. Selection gestures. (a) lasso (b) crossing (c) tapping (d)carat. Crossing edges: (e) top (f) horizontal center (g) bottom (h) left(i) vertical center (j) right.As seen in Fig. 12 our prototype implements complex selectionfeatures of previous work, namely circles of exclusion [11] andbump and bite gestures [20]. Unlike previous approaches [11,20]the additional strokes are not constrained to start and end withinthe initial lasso or on the initial lasso stroke, which makes themeasier for the user to draw casually. This also makes thetechniques easy to implement: any partial or complete lassotoggles the selection bit of the objects it contains; we then add theselected objects to the selection list, and remove any deselectedobjects. Thus the same code can implement both circles ofexclusion and bump/bite.Fig. 12. Left: Circle of exclusion; the user lassos hello+bye thenrecircles bye to exclude it from the selection. Right: Adding toselection with a bump and removing with a bite.6.1 Selection DecorationA selection decoration is a gesture that selects an object andindicates a spatial property of the object. Drawing a decorationgesture on or near an object selects some spatial property of theobject. As a convenience, if the object is not already selected, it isselected as well. However, a decoration never excludes an objectfrom a selection: adding a decoration to an already selected objectdoes not toggle its selection bit, it just decorates it. This allowsmultiple decorations to be composed on objects in a selection.Our prototype currently supports three types of selectiondecoration:Crossing. Highlights the closest principle edge of an objectwith a dotted line, as seen in Fig. 11 e-j.Carat. Used to specify an optional center of rotation, at theinflection point of the carat, for the Rotate command (Fig. 14).Note that we also could have used the tap selection gesture in thisway, but since users seem to expect tap to select (or deselect), andnothing else, overloading a tap to also indicate a center of rotationseemed problematic.Lasso. Instead of using lassos just for circles of exclusion, wehave experimented with an implementation that automaticallycreates a group object, known as an implicit group, when the userlassos multiple objects. This implicit group can then be actedupon by subsequent gestures, but is automatically ungroupedwhen the phrase ends. Implicit groups are one of the mostpowerful features in our prototype; we will discuss some uses forimplicit groups shortly. Using lassos to form implicit groups iscurrently an option in our system; we do not support implicitgroups and circles of exclusion (Fig. 12, left) at the same time.However, this is a limitation of our implementation, rather than afundamental conflict in functionality. When the implicit groupsoption is enabled, currently users can only exclude objects from aselection by tapping on them (which toggles their selection bit).Decorations add feedback to an object beyond the normalselection feedback. For example, the crossing selection decorationadds a dotted line that identifies the snap edge that is nearest tothe crossing stroke (Fig. 11e-j). The carat shows a blue box at theselected point (Fig. 14c). Decorations do not necessarily have tobe attached to a specific object. For example, the user can placethe carat over white space.6.2 Example: Multiple Edge AlignmentThe user can use the crossing selection operator (Fig. 11e-j) toindicate edges of objects for alignment (Fig. 1). The final edgecrossed is used as the reference edge (the edge that is aligned to).If the user misses the intended edge, he can cross the object againto override the previous stroke. This approach offers a nice

economy of design, since the system includes just a singlegeneral-purpose Align command, rather than multiple versions(Align Left, Align Center, Align Middle, etc.) as seen in MicrosoftPowerPoint, for example. Even with all these commands, someoperations are still difficult to express with traditional approaches(such as aligning the left edge of one object to the right edge ofanother, Fig. 13). During informal tests, users commented thatusing our crossing selection operators with a single generalpurpose Align command was natural and seemed obvious.rotation; the user can draw a vertical cross to instead flip about avertical axis.Fig. 15.Simple Flip command on a single object.The user can also flip several objects in one operation byindividually crossing each object along the desired axis of rotation(Fig. 16). Here, the user crosses the horizontal center of eachobject, and then chooses Flip to rotate each object about its ownlocal axis of rotation.Fig. 13. Aligning opposite edges. Left: The user crosses the leftedge of one photo, then indicates the right edge of another photo toalign to. Right: Result after the alignment.6.3 Example: Rotation Using the CaratThe carat selection operator selects a point on an object or onempty whitespace (Fig. 11d, Fig. 14a-e). The Rotate commanduses the inflection point of the carat as a center of rotation. Theuser first draws the carat, then draws a pigtail to activate amarking menu, and then chooses the Rotate command. Draggingpast the outer boundary of the menu then rotates the object aroundthe center of rotation via direct manipulation. We actuallyimplemented two different versions of our prototype sketchingapplication to more quickly explore various design options; onlyone version (with a different menu visualization) currentlysupports the Rotate command, and that version is shown in Fig.14a-e.Fig. 16.Flipping objects about their local horizontal center.But what if the user instead wanted to preserve the relativespatial relationship between the objects (a, b, c, d) and instead flipthe entire group around one of the objects? Our implicit groupingfeature can be used to specify these types of operations.6.5 Complex Operations via Implicit GroupingIn pen interfaces, drawing a lasso often allows commands to acton objects as a unit [6,11,20]. We take this concept to its logicalconclusion by using the lasso as a way to specify an implicitgroup. With implicit groups, when the user draws a lasso, this actsexactly as if the user had selected the objects and applied theGroup command. However, our prototype flags implicit groupsinternally, so that when they become deselected, the Ungroupcommand is automatically applied. Thus the “group” only existsduring the articulation of the command, and the user does nothave to explicitly invoke the Group and Ungroup commands. Weprovide feedback of the implicit group by drawing a dotted orangerectangle around the lassoed objects (Fig. 17).Fig. 14. Rotation using the carat. (a) before, (b) draw carat (c)feedback for selection decoration (d) choose rotate command (e)dragging with the carat as center-of-rotation.6.4 Example: The Flip CommandThe Flip command in our prototype is an example of a generalpurpose command that can span the full range from simple tosophisticated operations. The most elementary way to use Flip isto cross a single object, and then pigtail to choose Flip from themarking menu; in fact this can be done as a unistroke gesture ifdesired (Fig. 15). The Flip command uses the crossing axis, herethe horizontal center of the handwritten word “ink,” as the axis ofFig. 17. Flip a, b, c, and d around bottom of d. First the usercrosses the bottom of the “d.” Then user lassos a, b, c, and d. Theuser then applies the Flip command.

For most commands, such as lassoing objects and moving themas tested in our experiment, the creation of the implicit group isnot important and need not concern the user. However,knowledgeable users can specify more sophisticated operations byusing selection decorations and implicit groups together.Fig. 17 shows the Flip command with an implicit group. Here,we return to the example raised above, where a user wants to Flipthe objects (a,b,c,d) around the d. To do this, the user just crossesthe d, then lassos all four objects, and then pigtails to choose Flip.This causes the Flip command to rotate the implicit group aboutthe bottom of d. Once the command is finished, the selection isautomatically cleared, which results in our software ungroupingthe implicit group.Fig. 18. Alignment to center of implicit group. (a) Lasso items toform implicit group. (b) Cross center of implicit group. (c) The centerof the implicit group is aligned with the bottom of the triangle. (d)Final result at completion of command; the implicit group isautomatically unpacked.The user can apply selection decorations to an implicit group,allowing composition of functionality. For example, the user canuse this capability to align the center of a group of objects toanother object without any explicit use of the “Group” command(Fig. 18). As another example, to instead flip (a,b,c,d) aroundtheir collective right edge, the user would first lasso the objects(thus creating the implict group), then cross the right edge of theimplicit group, and then choose Flip. This pivots the objects aboutthe rightmost edge. Implicit groups can also be nested by drawinga lasso that encompasses one or more previous lassos. Both ofthese interactions are illustrated in the accompanying video.Nested implicit groups obviously are a feature for expert users.Nonetheless, it seems compelling that a basic lasso selectionoperation in a pen interface, which test users immediatelyunderstood, can be used in more sophisticated ways byexperienced users.In practice, we would not expect a novice user to immediatelygrasp the full complexity that can be expressed with our selectionoperators. Yet these examples show how composing multipleselection operators allows experienced users to specify complexoperations without having to repeatedly Group, act on, and thenUngroup the objects. Because the commands in our prototypehave such general functionality, the command set as experiencedby the user can remain simple (Flip, Align, etc.). The spatialproperties needed to give specific meaning to a command comefrom the multiple strokes that together specify the selection.6.6 Selection RecallDuring early testing, users would become frustrated if theydrew one or more selection strokes, activated a command, butthen made a mistake or wanted to re-use the selection for anothercommand. It was frustrating for users to have to redraw the sameselection again. To our knowledge, existing graphical interfacesdo not include mechanisms to undo or redo selections, but itappears that one consequence of making selections moreexpressive in our system is that it may become necessary toinclude a feature to “recall” a prior selection.A naive full-tension design precludes the repeated invocation ofdifferent commands that act upon on the same selection. Toaddress this problem, we allow the user to tap twice on the modeswitch button to recall the prior selection. The user then holdsdown the button, and can modify the prior selection (if desired)before proceeding with a new command as usual. This works, butusers have to be told to tap the gesture key twice to invoke thisfeature. Users suggested a “recall selection” icon or a double tapof the pen as alternatives, but we have not yet tried these.7 CONCLUSIONS AND FUTURE WORKWe have conducted the first experimental study of interactiontechniques for phrasing of multiple-stroke gestures. We havedemonstrated how our prototype uses multiple-stroke phrasingtechniques to elegantly support selection techniques explored byprevious systems, including disjoint selection regions, circles ofexclusion, and bump/bite gestures. But our approach also enablesnew techniques, such as selection decorations and implicit groups,which allow for sophisticated functionality from a few generalpurposecommands. However, while we have conducted someinformal user tests, these advanced features have not yet beensubjected to careful experimental studies.Our experimental results are consistent with the conclusion thatthe full-tension phrasing technique provides the strongestreinforcement of the chunking of multiple-stroke selection,command, and parameter phrases that our designs are trying toestablish. However, the low- and half-tension techniques workedwell enough that they could be considered as alternative multiplestrokephrasing techniques. For example, our full-tensiontechnique might not be practical for an electronic whiteboard thatlacks any button for the nonpreferred hand, or in some cases thespecific details of an application’s interaction techniques might beincompatible with the full-tension technique.We have described several benefits and extensions to existingtechniques that follow from supporting a multiple-stroke syntaxfor pen gesture interactions:• The ability to select objects using multiple strokes allowsadditional functionality, including more precise control overmultiple object selections, easier editing of selections, andnesting of multiple operations within a selection. Theresulting polymorphic commands also offer a nice economyof design, because the system can offer a single gesturecommand that acts on a wide variety of selections (e.g. Alignto right, Align to bottom, Align to horizontal center, etc).

• The use of nonpreferred-hand physical tension as a phrasingtechnique for multiple-stroke interactive dialogs offers aconcrete realization of ideas in the literature [5,10,12,15,24]that can be applied to realize new gesture-based interactiontechniques such as selection decorations.• Our implicit group selection mechanism can streamline someoperations by removing the necessity to explicitly group andungroup objects, potentially improving the cognitive andphysical workflow of such operations. This would be aninteresting issue to explore further in future studies.• Our multiple-stroke approach to selection extends orsubsumes a number of individual techniques (bump, bite,crossing, lasso selection, tapping) that have appeared in longhistory of gesture-driven systems [1,2,4,6,9,11,18,21], butnow can be offered in a single integrated technique thatoffers these as variants.• Our experimental validation confirms that multiple-strokephrases can be supported without introducing any ambiguityof which strokes “belong” to a command. In particular, the2.5% error rate exhibited by the full-tension phrasingtechnique suggests that it can offer particularly robustperformance. However, a more sophisticated experimentaldesign would be required to determine if there are any subtledifferences between the techniques in terms of modeswitching times [10,15]. Future studies of phrasingtechniques should also investigate alternative sequences ofcommands [3,10,17]. For example, if the user must issuethree successive commands, does the need to “remember” torelease the button between commands pose any problem forthe full-tension technique?Many of these capabilities would be difficult or impossible tosupport within a unistroke command syntax. Thus, holding downa button to explicitly signal a mode transition between inking andgesture input not only offers a fast and reliable means to switchmodes [15], it also offers an effective way to phrase togethermultiple-stroke pen gesture commands. We hope that ourdiscussion of the multi-stroke design strategy will help pengesture interaction designers to think about and explore new ideasin the design space of multi-stroke gestural techniques, withoutthe limitations and ambiguity that multi-stroke gestures haveraised in previous systems.ACKNOWLEDGEMENTSThis work has been supported in part by the Microsoft Center forInteraction Design and Visualization at the University ofMaryland and NSF under grant IIS-0414699. We also would liketo thank Corinna Löckenhoff for assistance with the data analysis.REFERERNCES[1] Accot, J., Zhai, S. More than dotting the i's-- Foundations forcrossing-based interfaces. ACM CHI 2002, 73-80.[2] Apitz, G., Guimbretiere, F. CrossY: A crossing based drawingapplication. UIST 2004, 3-12.[3] Appert, C., Beaudouin-Lafon, M., Mackay, W. Context matters:Evaluating interaction techniques with the CIS model. Proc. HCI2004, Springer Verlag, 279-295.[4] Buxton, W. An Informal Study of Selection-Positioning Tasks.Proceedings of Graphics Interface '82: 8th Conference of theCanadian Man-Computer Communications Society, 323-328.[5] Buxton, W. Chunking and Phrasing and the Design of Human-Computer Dialogues. Information Processing `86, Proc. IFIP 10thWorld Computer Congress, Amsterdam: North Holland, 475-480.[6] Buxton, W., Fiume, E., Hill, R., Lee, A., Woo, C. Continuous handgesturedriven input. Proceedings of Graphics Interface '83, 191-195.[7] Buxton, W., Myers, B. A Study in Two-Handed Input. Proceedingsof ACM CHI'86:, 321-326.[8] Guimbretiere, F., Winograd, T. FlowMenu: Combining Command,Text, and Data Entry. ACM UIST 2000, 213-216.[9] Hinckley, K., Baudisch, P., Ramos, G., Guimbretiere, F. Design andAnalysis of Delimiters for Selection-Action Pen Gesture Phrases inScriboli. CHI 2005, 451-460.[10] Hinckley, K., Guimbretiere, F., Baudisch, P., Sarin, R., Agrawala,M., Cutrell, E. The Springboard: Multiple Modes in One Spring-Loaded Control. CHI 2006, to appear.[11] Kurtenbach, G., Buxton, W. Issues in Combining Marking andDirect Manipulation Techniques. UIST'91, 137-144.[12] Kurtenbach, G., Fitzmaurice, G., Owen, R., Baudel, T. The Hotbox:Efficient Access to a Large Number of Menu-items. CHI'99, 231-237.[13] Kurtenbach, G., Sellen, A., Buxton, W., An emprical evaluation ofsome articulatory and cognitive aspects of 'marking menus'. J.Human Computer Interaction, 1993. 8(1).[14] LaViola, J. a. Z., R., MathPad2: A System for the Creation andExploration of Mathematical Sketches. ACM Transactions onGraphics, 2004. 23(3): p. 432-440.[15] Li, Y., Hinckley, K., Guan, Z., Landay, J. A. Experimental Analysisof Mode Switching Techniques in Pen-based User Interfaces. CHI2005, 461-470.[16] Liao, C., Guimbretiere, F., Hinckley, K. PapierCraft: A CommandSystem for Interactive Paper. UIST 2005, 241-244.[17] Mackay, W. E. Which Interaction Technique Works When? FloatingPalettes, Marking Menus and Toolglasses Support Different TaskStrategies. Proc. AVI 2002 International Conference on AdvancedVisual Interfaces, ACM, 203-208.[18] Mizobuchi, S., Yasumura, M. Tapping vs. Circling Selections onPen-based Devices: Evidence for Different Performance-ShapingFactors. CHI 2004, 607-614.[19] Monk, A., Mode Errors: A User-centered Analysis and somePreventative Measures Using Keying-contingent Sound.International J. Man-Machine Studies, 1986. 24: p. 313-327.[20] Moran, T., Chiu, P., van Melle, W. Pen-Based InteractionTechniques for Organizing Material on an Electronic Whiteboard.UIST'97, 45-54.[21] Pier, K., Landay, J. A., Issues for Location-Independent Interfaces.1992: Technical Report ISTL92-4, Xerox Palo Alto Research Center.[22] Pook, S., Lecolinet, E., Vaysseix, G., Barillot, E. Control Menus:Execution and Control in a Single Interactor. CHI 2000 ExtendedAbstracts, 263-264.[23] Saund, E., Lank, E. Stylus Input and Editing Without Prior Selectionof Mode. UIST'03, 213-216.[24] Sellen, A., Kurtenbach, G., Buxton, W. The Role of Visual andKinesthetic Feedback in the Prevention of Mode Errors. Proc. IFIPINTERACT'90: Human-Computer Interaction, 667-673.[25] Sellen, A., Kurtenbach, G., Buxton, W., The Prevention of ModeErrors through Sensory Feedback. J. Human Computer Interaction,1992. 7(2): p. 141-164.[26] Zhao, S., Balakrishnan, R. Simple vs. Compound Mark HierarchicalMarking Menus. UIST 2004, 33-42.