Metrics for Multi-Touch Input Technologies
MMetrics for Multi-Touch Input Technologies
Ahmed Sabbir Arif
Department of Computer Science and EngineeringYork UniversityToronto, Ontario, [email protected]
Figure 1: Multi-touch technologies are emerging as a major medium of high-degree of freedom interaction. Picture courtesyof Synlab.
ABSTRACT
Multi-touch input technologies are becoming popular with the in-creased interest in touchscreen- and touchpad-based devices. Agreat deal of work has been done on different multi-touch tech-nologies, and researchers and practitioners are frequently comingup with new ones. However, it is almost impossible to comparesuch technologies due to the absence of multi-touch performancemetrics. Designers usually use their own methods to report theirtechniques’ performances. Moreover, multi-touch interaction wasnever modeled. That makes it impossible for designers to predictthe performance of a new technology before developing it, costingthem valuable time, effort, and money. This article discusses thenecessity of having dedicated performance metrics and predictionmodel for multi-touch technologies, and ways of approaching that.
Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.For all other uses, contact the owner/author(s).
Computer Science & Engineering, York University, Toronto, 2010
KEYWORDS
Performance metrics, multi-touch input technologies, error correc-tion, prediction, mathematical model.
Touchscreen- and touchpad-based multi-touch technologies areemerging as a major medium of high-degree of freedom interaction(Figure 1). A great deal of work has been done on different multi-touch technologies, recognition algorithms, applications (e.g., [2–5, 7, 8, 10, 12–14, 16–18, 20]), and researchers and practitioners areconstantly coming up with new ones. But, it is almost impossibleto compare these technologies due to the absence of multi-touchperformance metrics. Designers usually use their own methods,which are typically modified versions of existing metrics, to reporttheir techniques’ performances. These metrics, however, fail toprovide a clear picture of how the technologies work because of the direct interaction strategies. Multi-touch techniques input directlyto the device, under the points of contact such as, fingers, makingit notably different from most other interaction technologies.As most multi-touch metrics were coined by the designers toshow how well their technologies perform, rather than to offera good set of metrics, they are usually very straightforward and a r X i v : . [ c s . H C ] S e p omputer Science & Engineering, York University, Toronto, 2010 Ahmed Sabbir Arif domain-specific. In other words, metrics used on a specific multi-touch technology cannot be used with any other technology. Inaddition to that, many assumptions were made to keep the metricssimple. For instance, some assume that all errors are exclusivelydue to the user, while other metrics assume it is the system thatmakes errors. In general, there is no differentiation between thehuman and system errors. Moreover, important human factors, suchas finger and hand movements, cognitive processing and decisionmaking times, and so on, as well as system factors, such as inputprocessing time, flexibility, etc. are frequently overlooked.To date, no attempt has been made on modeling the task ofmulti-touch interaction, either. Hence, it is impossible to predicta system’s performance before developing it, causing designers towaste time, effort, and money. Multi-touch technologies have thepotential of becoming one of the primary interaction techniques innear future, as touchscreen- and touchpad-based devices, such astabletops, smartphones, tablets, etc. are rapidly emerging. At thisstage it is imperative to have dedicated performance metrics andprediction model for such technologies. It will help us not only tocompare novel techniques but also to predict the performance ofnew ones before implementing them.This article starts with a brief discussion on current multitouchmeasurement techniques. It then, discusses the human and systemfactors that are likely indispensable when developing high-levelmulti-touch performance metrics and prediction model. Finally, itpresents an outline of a potential future research. Almost all recent multi-touch empirical experiments report errorrates along with other performance measures. In most cases, bothperformance and errors were classified by a straightforward hit-or-miss strategy.Participants were asked to perform specific tasks on a screen ora pad, and the experiment software kept a record of their actions.If the tasks were carried out successfully in a single attempt thenit was considered a hit or success , if not then a miss or an error .Since different systems have different ways of interpreting userinteraction hits and misses were counted differently based on thesystem design. One reason for this is that no research has beendone on multi-touch metrics and error classification techniques.Hence, designers report performance in different ways. None ofthe methods differentiate between the human and system factorsand overlook important factors, as pointed out previously. It is true that different interaction techniques have different waysof handling similar tasks. Yet, it is possible to develop domain inde-pendent performance metrics and prediction models by identifyinghigh-level tasks that are common to all technologies. For instance,tasks such as selecting, moving, or rotating an object, are commonto almost all multi-touch techniques. These high-level tasks, then,can be broken down into low-level domain-specific tasks. This strat-egy has been proven effective while modeling other interactiontechnologies [1].The question, what high- and low-level parameters need to beconsidered in new metrics and prediction model requires careful study of current multi-touch technologies and a better understand-ing of the real-life user interactions. However, at this point we caninclude at least the following human and system factors.
The two parameters below are cognitive processing times that canbe recorded via empirical studies. Alternatively, these values canbe collected from existing work, as it is safe to assume that thesecognitive pauses are fairly uniform in lengths [9]. □ T hpreparation or the preparation time is the average time ittakes to make the decision to perform a task. □ T hverif y or the verification time is the average time it takesto verify correct completion of a performed task.The parameter below is the physical movement time that can becalculated using Hick–Hyman and/or Fitts’ law [15]. The first lawcan be used to measure the choice reaction time and the latter tomeasure the rapid aimed movements. □ T hmove or the movement time is the average time it takes tomove fingers or hands from one location to another.The parameter below is the probability of making an error whileperforming a task, which can be determined based on the averageerror rate measured in empirical studies. □ R herror or the human error rate is the average probability ofmaking an error while performing a task. Although it is not possible to be definite about the behavior of firsttwo parameters below without conducting empirical experiments,it can be assumed that the values of these will be the sum of agrowing and a decaying series, respectively. □ R slearn or the learning rate is the average asymmetric learn-ing effect for a specific technology that represents how fastusers learn, or get used to, a system’s interface, functional-ities, or even bugs. This parameter can prove useful whencomparing performance between expert and novice users.However, the value for R slearn can be considered zero whenparticipants are well-trained or had lots of prior experiencewith the system. □ R suse or the usability rate expresses how user performancedecreases over time due to the system’s complexity or er-gonomic discomforts. This factor may be necessary as mostdirect input technologies are known to cause physical dis-comfort, such as fatigue, stress, occlusions from the user’shand, and so forth, during long term usage or instabilities[6, 19]. □ T sprocess or the input processing time is the average timeit takes to process a low-level task, such as a drag, pinch,display output, etc., by a specific technology. □ R serror or the system error rate is the average probability ofa system error, such as a misrecognition or an interpretationerror, for a specific technology. etrics for Multi-Touch Input Technologies Computer Science & Engineering, York University, Toronto, 2010 Prior sections provided a partial list of potential human and systemfactors. Here, we present two potential compound factors. □ R error or the error rate is the average of the compoundof the human and system error rates, in other words therelationship between R herror and R serror . □ T task or task completion time is the average of the compoundof the human and system times, in other words the relation-ship between T hpreparation , T hmove , T sprocess , and T hverif y ,to perform a task in a single attempt.These two compound parameters can be used as new multi-touchperformance metrics: R error for measuring error rates and T task for measuring the overall performance of a specific technology. Todetermine how to calculate these parameters also require furtherresearch.However, more research and studies are necessary to compilea complete list of parameters. For example, it may be necessaryto find answers to questions such as the effect of the presence orabsence of tactile feedback, which tasks are hard due to humanlimitations, the effect of constraints of human hands, and how thesize and proximity of the display affects performance. It is alsoessential to find more precise relationships between the human andsystem factors to create high-level metrics and a predictive model. A high-level goal can be “move object” that is actually the combina-tion of small operations: select object , drag object , and release object .These operations, too, are combinations of smaller operations: pre-pare to perform a task that is T hpreparation , perform the task that isthe relationship between R error and T task , and verify the task thatis T hverif y . This is how all major operations can be broken downinto smaller and basic operations. This also makes it possible topresent a predictive model. This section presents a plan for developing new metrics to measureand predict the performance of multi-touch input and interactionmethods.
Stage-1: Studying Multi-Touch Technologies.
At this stage, onemust study existing technologies from the literature, as well asexamine some academic and commercial devices in real-life sce-narios, for a better understanding of the technologies. The mainpurpose will be to identify common goals, tasks, trends, patterns,discomforts, mistakes, and confusions. This will help identifyingvarious low- and high-level human and system factors, and therelationships between them.
Stage-2: Preliminary Metrics and Model.
Here the target is to de-fine a set of preliminary performance metrics and create a predictivemodel based on the findings of the first stage.
Stage-3: Pilot Studies.
In this stage, a series of pilot studies willbe conducted to determine if the proposed metrics give the rightkind of results. If not, the metrics must be fine-tuned based on the study results. This will eventually lead the research to a final set ofmetrics and a model.
Stage-4: Empirical Studies.
After deriving the final metrics andprediction model, a full-length empirical study is needed for furtherverification. At this stage it is also a good idea to examine if the newmetrics and model can be extended to related input technologiessuch as bimanual interaction [11].
Well-defined performance metrics and prediction models are im-portant for the continued development of any maturing technology.Multi-touch is maturing rapidly, with promising trends. Researchersand practitioners are coming up with new multi-touch techniquesin regular basis. But it is almost impossible to compare, evaluate, orpredict the performance of systems, as, to date, there is no standardmetrics or model for multi-touch. This research plan presents onepotential avenue to identify such metrics and models and also anoutline of work that can build on it.
REFERENCES [1] Ahmed Sabbir Arif and Wolfgang Stuerzlinger. 2010. Predicting the Cost ofError Correction in Character-Based Text Entry Technologies. In
Proceedings ofthe SIGCHI Conference on Human Factors in Computing Systems (CHI ’10) . ACM,New York, NY, USA, 5–14. https://doi.org/10.1145/1753326.1753329 event-place:Atlanta, Georgia, USA.[2] Hrvoje Benko, Andrew D. Wilson, and Ravin Balakrishnan. 2008. Sphere: Multi-Touch Interactions on a Spherical Display. In
Proceedings of the 21st Annual ACMSymposium on User Interface Software and Technology (UIST ’08) . Associationfor Computing Machinery, New York, NY, USA, 77–86. https://doi.org/10.1145/1449715.1449729[3] Philip L. Davidson and Jefferson Y. Han. 2006. Synthesis and Control on LargeScale Multi-Touch Sensing Displays. In
Proceedings of the 2006 Conference on NewInterfaces for Musical Expression (NIME ’06) . IRCAM: Centre Pompidou, Paris,France, 216–219.[4] Paul Dietz and Darren Leigh. 2001. Diamondtouch: A Multi-User Touch Technol-ogy. In
Proceedings of the 14th Annual ACM Symposium on User Interface Softwareand Technology (UIST ’01) . Association for Computing Machinery, New York, NY,USA, 219–226. https://doi.org/10.1145/502348.502389[5] Florian Echtler, Manuel Huber, and Gudrun Klinker. 2008. Shadow Tracking onMulti-Touch Tables. In
Proceedings of the Working Conference on Advanced VisualInterfaces (AVI ’08) . Association for Computing Machinery, New York, NY, USA,388–391. https://doi.org/10.1145/1385569.1385640[6] V. Ha, K.M. Inkpen, R.L. Mandryk, and T. Whalen. 2006. Direct Intentions: TheEffects of Input Devices on Collaboration Around a Tabletop Display. In
FirstIEEE International Workshop on Horizontal Interactive Human-Computer Systems(TABLETOP ’06) . 8 pp.–. https://doi.org/10.1109/TABLETOP.2006.10[7] Jefferson Y. Han. 2005. Low-Cost Multi-Touch Sensing Through Frustrated TotalInternal Reflection. In
Proceedings of the 18th Annual ACM Symposium on User In-terface Software and Technology (UIST ’05) . Association for Computing Machinery,New York, NY, USA, 115–118. https://doi.org/10.1145/1095034.1095054[8] Steve Hodges, Shahram Izadi, Alex Butler, Alban Rrustemi, and Bill Buxton.2007. Thinsight: Versatile Multi-Touch Sensing for Thin Form-Factor Displays.In
Proceedings of the 20th Annual ACM Symposium on User Interface Softwareand Technology (UIST ’07) . Association for Computing Machinery, New York, NY,USA, 259–268. https://doi.org/10.1145/1294211.1294258[9] David Kieras. 1993.
Using the Keystroke-Level Model to Estimate Execution Times .Technical Report. University of Michigan, Michigan, MI, USA. 11 pages.[10] SK Lee, William Buxton, and K. C. Smith. 1985. A Multi-Touch Three DimensionalTouch-Sensitive Tablet.
ACM SIGCHI Bulletin
16, 4 (April 1985), 21–25. https://doi.org/10.1145/1165385.317461[11] Andrea Leganchuk, Shumin Zhai, and William Buxton. 1998. Manual and Cogni-tive Benefits of Two-Handed Input: An Experimental Study.
ACM Transactionson Computer-Human Interaction
5, 4 (Dec. 1998), 326–359. https://doi.org/10.1145/300520.300522[12] Julien Letessier and FranÃğois BÃľrard. 2004. Visual Tracking of Bare Fingers forInteractive Surfaces. In
Proceedings of the 17th Annual ACM Symposium on User In-terface Software and Technology (UIST ’04) . Association for Computing Machinery,New York, NY, USA, 119–122. https://doi.org/10.1145/1029632.1029652 omputer Science & Engineering, York University, Toronto, 2010 Ahmed Sabbir Arif [13] Shahzad Malik, Abhishek Ranjan, and Ravin Balakrishnan. 2005. Interacting withLarge Displays from a Distance with Vision-Tracked Multi-Finger Gestural Input.In
Proceedings of the 18th Annual ACM Symposium on User Interface Softwareand Technology (UIST ’05) . Association for Computing Machinery, New York, NY,USA, 43–52. https://doi.org/10.1145/1095034.1095042[14] Justin Matejka, Tovi Grossman, Jessica Lo, and George Fitzmaurice. 2009. TheDesign and Evaluation of Multi-Finger Mouse Emulation Techniques. In
Pro-ceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’09) . Association for Computing Machinery, New York, NY, USA, 1073–1082.https://doi.org/10.1145/1518701.1518865[15] Steven C. Seow. 2005. Information Theoretic Models of HCI: A Comparison ofthe Hick-Hyman Law and Fitts’ Law.
HumanâĂŞComputer Interaction
20, 3 (Sept.2005), 315–352. https://doi.org/10.1207/s15327051hci2003_3[16] Chia Shen. 2006. Multi-User Interface and Interactions on Direct-Touch Horizon-tal Surfaces: Collaborative Tabletop Research at MERL. In
First IEEE InternationalWorkshop on Horizontal Interactive Human-Computer Systems (TABLETOP ’06) . 4pp.–. https://doi.org/10.1109/TABLETOP.2006.22 [17] Christian von Hardenberg and FranÃğois BÃľrard. 2001. Bare-Hand Human-Computer Interaction. In
Proceedings of the 2001 Workshop on Perceptive UserInterfaces (PUI ’01) . Association for Computing Machinery, New York, NY, USA,1–8. https://doi.org/10.1145/971478.971513[18] Wayne Westerman, John G. Elias, and Alan Hedge. 2001. Multi-Touch: A NewTactile 2-D Gesture Interface for Human-Computer Interaction.
Proceedings of theHuman Factors and Ergonomics Society Annual Meeting
45, 6 (Oct. 2001), 632–636.https://doi.org/10.1177/154193120104500612 Publisher: SAGE Publications Inc.[19] Daniel Wigdor, Gerald Penn, Kathy Ryall, Alan Esenther, and Chia Shen. 2007.Living with a Tabletop: Analysis and Observations of Long Term Office Use of aMulti-Touch Table. In
Second Annual IEEE International Workshop on HorizontalInteractive Human-Computer Systems (TABLETOP’07) . 60–67. https://doi.org/10.1109/TABLETOP.2007.33[20] Zhengyou Zhang, Ying Wu, Ying Shan, and Steven Shafer. 2001. Visual Panel:Virtual Mouse, Keyboard and 3D Controller with an Ordinary Piece of Pa-per. In