Identifying Usability Issues of Software Analytics Applications in Immersive Augmented Reality
David Baum, Stefan Bechert, Ulrich Eisenecker, Isabelle Meichsner, Richard Müller
IIdentifying Usability Issues of Software AnalyticsApplications in Immersive Augmented Reality
David Baum ∗ , Stefan Bechert † , Ulrich Eisenecker ‡ , Isabelle Meichsner § , and Richard M¨uller ¶ Information Systems InstituteLeipzig University, Leipzig, Germany ∗ Email: [email protected] † ORCID: 0000-0002-6727-7106 ‡ Email: [email protected] § ORCID: 0000-0002-9474-382X ¶ ORCID: 0000-0001-6730-4082
Abstract —Software analytics in augmented reality (AR) issaid to have great potential. One reason why this potential isnot yet fully exploited may be usability problems of the ARuser interfaces. We present an iterative and qualitative usabilityevaluation with 15 subjects of a state-of-the-art applicationfor software analytics in AR. We could identify and resolvenumerous usability issues. Most of them were caused by applyingconventional user interface elements, such as dialog windows,buttons, and scrollbars. The used city visualization, however,did not cause any usability issues. Therefore, we argue thatfuture work should focus on making conventional user interfaceelements in AR obsolete by integrating their functionality intothe immersive visualization.
Index Terms —usability evaluation, software analytics, softwarevisualization, augmented reality, mixed reality
I. I
NTRODUCTION
Software analytics aims to obtain insightful and action-able information about software systems, projects, and usersbased on the analysis and visualization of data from softwareartifacts [1]. Three-dimensional (3D) software visualizationshave existed for many years, which also applies for theiruse in software analytics. However, they often fall short ofexpectations. This is attributed in particular to the areas of nav-igation, selection, occlusion, and text readability [2]. Merino etal. [3] showed in a controlled experiment that these issues canbe minimized by displaying 3D visualizations in immersiveaugmented reality (AR). Navigation and occlusion in particularpose fewer problems in AR. The authors also found thatsoftware comprehension tasks with 3D software visualizationscould be solved more effective with AR devices than standardcomputer screens and identified great potential of AR forsoftware engineering [4]. Nevertheless, software analytics ishardly ever done in AR and in our own experience manysoftware developers approach this topic with reservations.With this work, we contribute to the understanding of prob-lems in augmented software analytics. Based on the findings ofMerino et al. [3], we deal with the research question: “Whichparts of the user interface of software analytics applicationsin augmented reality cause usability problems?”
To answer this question, we developed an AR applica-tion for software analytics, which corresponds to the current state-of-the-art. It reimplements the city metaphor, originallyintroduced in [5], which is a one of the most popular 3Dsoftware visualizations. We conducted a qualitative studywith this application to identify usability problems in AR.We could show that most problems are not caused by thecity visualization itself, but by using inappropriate interactiveuser interface elements such as dialog boxes, and scrollbars.The main contributions of the paper are the identification ofusability issues in AR and the discussion how to solve them.We can also confirm many of the findings of Merino et al. [3],for example, that the usability problems are mainly caused byselection and not by navigation and occlusion.II. R
ELATED W ORK
There are some approaches using AR to support the analysisand visualization of data from software repositories. Souzaet al. [6] present SkyscrapAR, one of the first tools usingthe city metaphor to investigate the evolution of a softwaresystem in AR. Kapec/Brndiarov [7] introduce a graph-basedvisualization of software systems on a see-through display.Sharma et al. [8] describe an AR tool for managers to displaydeveloper information, such as monthly developer time-line,code commit statistics, and recently owed technical debts. Afurther approach of these authors [9] outlines the concept ofsmart immersive software engineering workspaces where datafrom different sources are integrated and relevant informationis presented via AR devices to developers. Reipschl¨ager etal. [10] present DebugAR, a tool to debug software systemsin AR. It has also been studied, how to control the explorationof software architectures in AR through speech [11]–[13].Mehra et al. [14] present XRaSE, a tool to create immersiverepresentations of software systems to support comprehension.Although they sketch a planned evaluation study design, wecould actually find one approach that systematically inves-tigates usability issues in AR. Merino et al. [3] conducteda controlled experiment and a user study. They found thatimmersive AR facilitates navigation, reduces occlusion, andimproves user experience without affecting the performance.But selection and text readability still remain open issues.Here, we continue with our research. a r X i v : . [ c s . H C ] A ug ig. 1. City visualization and app bar superimposed on a table III. S
TUDY D ESIGN
We use the goal definition framework [15] to summarizethe scope of our experiment:Analyze augmented reality applications for softwareanalytics for the purpose of evaluation with respect to usability from the point of view of software developers in the context of software developers performing compre-hension tasks .We used a Microsoft HoloLens headset with a stereo1268 x 720 resolution, 60 Hz content refresh rate, and 30°Hand 17.5°V field of view. The application uses the MixedReality Toolkit (MRTK), which is based on Unity 3D. Thisis the de facto standard for HoloLens applications. To focuson the user interface, we have integrated our application intothe Getaviz framework, which can create city visualizationsfor Java, C .Additionally, its usage is demonstrated in a screen-cast . Theuser interface consists of the city itself and an app bar withbuttons as shown in Fig. 1. Class names are displayed astooltips by hovering over the building. The source code ofa class can be viewed in a separate dialog window. Userscan navigate freely, this means, they can rotate and move thevisualizations or zoom in and out. Similarly, they can navigatethe system by walking and approaching buildings in the city.To be able to study more complex interactions, we have addeda filter dialog. The user can navigate through the packagecontainment tree and select individual packages. In the future, districts can thus be hidden. Currently, this window is only amock-up and does not affect the city visualization. A. Tasks
Each subject had to perform a number of tasks. These werechosen to cover the complete functionality of the applica-tion. This includes recognizing visual patterns and identifyingoutliers. However, we have taken care to ensure that non-analytical aspects of use are also covered by the tasks. Thisincludes support tasks such as setting up the visualization,starting dialog windows, and using voice commands. Weincluded these tasks because they are essential for practicaluse and may also cause usability problems. Table I gives anoverview of all tasks. Each task belongs to one of the followingcategories. • Help & Setup:
This category contains all support tasks.This includes starting and setting up the application andhelp menus. Problems with these tasks make further useconsiderably more difficult. • Visualization:
For these tasks the subject has to workwith the visualization in its current state. This includesabove all the reading of information and the recognitionof patterns. Only minimal interaction with the applicationis necessary, since neither information has to be filterednor additional information has to be displayed. • Dialog:
For these tasks, the subject must change the stateof the visualization by hiding information or showing ad-ditional information. This requires complex interactionsand additional dialog windows.
B. Procedure
We applied an iterative usability evaluation approach [18]with a total of three iterations. After each iteration, thefeedback of the subjects regarding usability was implemented.Each subject was given a short introduction about the citymetaphor and software visualization in general. To ensure thateach subject had the same prerequisites, this introduction wasgiven in written form. After the introduction, the subjectsshould familiarize themselves with the HoloLens. The
LearnGestures app, which is installed on the HoloLens by default,was used for this purpose. Thus the subjects were familiar withthe gestures and handling of AR applications. The subjectsthen worked through the tasks one after the other. Feedbackfrom the subjects was obtained in the form of a semi-structuredinterview. After each task they were asked if they experiencedany usability issues. We recorded the answers of the subjects tothe task set and to the supplementary questions about problemsas well as conspicuous interactions with the application, forexample, where the subjects positioned windows in the roomor whether they used voice or gesture control. Each sessiontook place at the same location and was conducted by the sameexperimenter. In every iteration we used the same visualizationof JUnit 4.1.Based on user feedback new dialogs were added in eachiteration. For example, in the first two iterations the visualiza-tion metaphor was described on a piece of paper. For the third
ABLE IO
VERVIEW OF ALL TASKS
Category ID Task Evaluation I Evaluation II Evaluation IIIHelp & Setup T1 Place the visualization on the table in front of you. (cid:88) (cid:88) -T2 Complete the tutorial. - - (cid:88)
T3 Change the size, orientation and position of the visualization until you feel you have an optimal view of the city. (cid:88) (cid:88) (cid:88)
Visualization T4 Find the class with the most methods. (cid:88) (cid:88) (cid:88)
T5 Find the class with the fewest methods. (cid:88) (cid:88) (cid:88)
T6 Find the class with the most attributes. (cid:88) (cid:88) (cid:88)
T7 Find the class with the fewest attributes. (cid:88) (cid:88) (cid:88)
T8 Is there a God class? If so, what is its name? (cid:88) (cid:88) (cid:88)
Dialog T9 Open the filter menu with the voice command
Filter . - (cid:88) (cid:88)
T10 Filter for the package org.junit.validator . (cid:88) (cid:88) (cid:88) T11 Close the filter menu with the voice command
Close . - (cid:88) (cid:88)
T12 Look at the source code of any class. (cid:88) - -T13 Open the source code dialog by focusing on a class and using the voice command
Select . - (cid:88) (cid:88)
T14 Use the voice commands Up and Down to navigate in the source code display. - (cid:88) (cid:88)
T15 Now use the click gesture to navigate in the source code display. - (cid:88) (cid:88)
T16 Close the source code dialog by applying the click gesture to the button in the upper right corner. - (cid:88) (cid:88)
T17 Open the About dialog using the voice command
About and read the dialog. - - (cid:88)
T18 Open and read the legend dialog. - - (cid:88) evaluation, a help dialog was added, which contains the sameinformation. New tasks were also added cover these additionaldialogs. The help dialog was evaluated in the third evaluationby the new task
T18 . C. Subjects
Five subjects participated in each evaluation. None of themparticipated more than once. Three subjects of the secondevaluation are computer science students, the other 13 subjectsare professional software developers. They participated in theevaluation voluntarily and did not receive any compensation.Two subjects have already had experience with AR, but notin the context of software analytics.In the first evaluation, five male subjects with an averageage of 30.4 5.9 years participated. They have an averageof 9.0 6.4 years experience in software development. Inthe second evaluation, two female and three male subjectswith an average age of 26.4 3.00 years participated. Theprofessional software developers have an average of 5.0 3.0years experience in software development. The students haveexperience with software development only within the contextof their studies. In the third evaluation, five male subjectswith an average age of 30.6 4.88 years participated. Theyhave an average of 8.8 3.0 years of experience in softwaredevelopment. IV. R
ESULTS
A. Evaluation I
Help & Setup:
All subjects could successfully complete T1 and T2 . However, the subjects were not aware that theycould adjust the visualizations position and size at any time.It was noticeable that all users turned the visualization untilthe edges of the city were parallel to the table. Also the sizewas usually adjusted so that the city fitted completely on thetable. Some subjects reported that they would have preferredto adjust the city to a larger size, but would have needed alarger table for this. Visualization:
All tasks were solved correctly. However, thesubjects made a number of suggestions for improvement. The app bar was described several times as annoying because it ob-structed viewing the visualization. All subjects had problemsaiming at thin and small buildings. None of them enlarged thecity afterwards to make targeting easier. The tooltips with classnames were occasionally obscured by other buildings, so thatthey could not be read. This is an implementation error, sincethis is not the desired behavior. We fixed this by implementinga custom shader that always displayed the tooltip in front ofany building.
Dialog:
The source code dialog was often not found afteropening. The test subjects felt that the text field for thesource code was too small. In addition, they had difficulties inoperating the scrollbar. They often moved beyond the targetedpoint. As a consequence, we completely redesigned the dialog.We increased its size and added an opening animation so thatthe window is easier to find for the subjects. Additionally,we replaced the scrollbar with buttons for scrolling up anddown, similar to the keys page up and page down on akeyboard. All subjects had problems with operating the filterfunction. Especially the use of the virtual on-screen keyboardcaused problems. Therefore, we also turned them into buttons.Some subjects found the use of the HoloLens through gesturesto be very strenuous. Hence, for the next evaluation, weimplemented an additional voice control to reduce strain onthe arms.
B. Evaluation II
In the second evaluation, some tasks were added becausethe application could now be controlled by voice commands.Furthermore, the tasks for displaying the source code havebeen extended to achieve a higher coverage of the controlelements. Overall, the feedback of the subjects after the secondevaluation was noticeably more positive.
Help & Setup:
Some problems, which were also apparentin the first evaluation, reappeared. Some subjects had prob-lems using the navigation mode intuitively. They could notrecognize the translation by themselves. Other subjects didnot recognize what the handles on the bounding box couldbe used for. To solve this problem, an interactive tutorial wasmplemented for evaluation III. It uses spoken text to ask theuser tasks that will guide him through the functions of the app.When the user completes the task, the next task is set untilthe user has used all functions once.
Visualization:
The subjects were still able to solve thetechnical tasks correctly. When reading the class and packagenames there was no negative feedback this time. The targetingof thin and small buildings was again difficult. One of thesubjects asked for a tolerance range. This should ensure thatan already targeted building is still highlighted, even if theuser looks slightly to the side of the building. This is onlypossible when the cursor does not focus another building ordistrict. Some of the subjects were disturbed again by the appbar because it blocked their view.
Dialog:
With the revised filter dialog, all subjects were ableto solve the corresponding task
T10 . Users rated the handlingof the source code dialog as good. They remarked that linenumbers would be practical for orientation in the source code.The problem that subjects did not find the dialogs occurredagain. When they were made aware of the animation, they saidthat they did not notice it the first time. Some said that theyhad to concentrate too much on the gestures. Since users hadproblems with the correct execution of gestures, an acousticsignal should also be played as further feedback when a buttonis successfully clicked.
C. Evaluation III
For the last evaluation, the tasks were extended once againto be able to examine the newly added in-app tutorial and thenew help menu.
Help & Setup:
The tutorial made it much easier to getstarted with the application. The tutorial tasks were success-fully solved by all subjects without further questions. All usersfound the tutorial well structured and easy to understand. Itwas clearly visible how the tutorial helped the subjects to solvethe follow-up tasks.
Visualization:
The targeting of thin buildings was simpli-fied by a tolerance range. If a user sees a building and thenlooks next to it without hitting another building or district, thefirst building is still highlighted. The legend helps to avoidsome questions with regard to the metrics in the professionaltasks. Due to the tolerance range it was easier for subjectsto target buildings that are both thin and tall. However, theyreported that targeting small, thin buildings is still difficult.
Dialog:
After subjects in the second evaluation had prob-lems finding the dialog with the open animation, the logic wasrevised again. The dialog is now opened at the height of thecity instead of the height of the HoloLens. This means that thedialog often overlaps with the city, but there is always a partof the dialog in the user’s field of view. The tutorial explainsto the user how to move dialogs. In all dialog windows anindicator has been added in the upper left corner to show thatthe dialog can be moved. Users can now use this knowledgeto position the dialog according to their wishes. Some of thesubjects criticized the long delay of more than two secondsin voice control. This was particularly noticeable with the
TABLE II
CATEGORIZATION OF IDENTIFIED USABILITY ISSUES
Category Evaluation I Evaluation II Evaluation IIIText readability 1 - -Occlusion 2 1 -Navigation - - -Selection 3 1 2User guidance 1 2 - voice command down . All subjects stated that the voice controlworked, but that they would rather not use it. This was notrelated to the delay but to a general rejection of voice control.V. D
ISCUSSION
It is striking that we did not have to make any changes tothe city metaphor itself. Right from the start, all tasks of thevisualization category could be solved completely. None of thesubjects suggested changes to the visualization.Table II gives an overview of the identified usability issues.Most issues belong to the category
Selection . This includesproblems with the selection of buildings and conventional userinterface elements. Although we were able to mitigate theseissues, they still occurred in the final evaluation. Occlusionwas not a problem within the city. Tooltips and app barcaused occlusion, but these issues could be solved. In the firstevaluation a too small text size was criticized which couldeasily be fixed. Apart from that, the subjects did not haveany problems reading texts. At the beginning, only a few helpfunctions were implemented in the application. Due to thefeedback of the users, their number was increased with eachiteration. This made the subjects feel much more confident inusing the application. This finding is not surprising. However,it shows that previous studies have concentrated too much onthe implementation of the metaphor and neglected the userguidance. In our study, the identified usability issues withrespect to missing user guidance could be completely solved.The solution consisted of a tutorial and help functions that canbe easily implemented.Most usability problems were caused by conventional userinterface elements, such as dialog windows, buttons, andscrollbars. In the first evaluation we used the standard com-ponents of MRTK without further customization to implementthe app bar and dialog windows. This solution was found to beunsuitable by the subjects, so we replaced the standard com-ponents with our own implementations. This was necessarybecause, for example, MRTK’s app bar cannot be adapted tothe user’s position as suggested by the subjects. The subjectsliked our adapted implementations much more, so that nofurther suggestions for improvement were given.Our application uses MRTK 2017.4.3.0, which was thelatest release at the time of the evaluations in 2019. The MRTKhas now been fundamentally revised. However, the problemswith the standard components are still there, for example, theposition of the app bar still cannot be adjusted to the positionof the user. With better support for the WebAR interface, ig. 2. Source code dialog window superimposed on a physical whiteboard
WebGL-based frameworks such as A-Frame could possiblybe more suitable in the future. Existing web-based softwareanalytics tools such as Getaviz already rely on A-Frame [16].Our study suggests that voice control is only partiallysuitable for facilitating the work with dialog windows. Thevoice control was rejected by our subjects, although it workedwell from a technical point of view. The virtual on-screenkeyboard has also proved to be unsuitable.In previous studies the city was simply placed in spacewithout interacting with real world objects. In our study,however, all subjects placed the city on a real table. Thistable was used as a reference point when the subjects movedaround the room. They always knew where the city was andcould walk around the table to view the city from all sides.Some subjects also placed the source code in a way that it wasdisplayed directly on a real whiteboard as shown in Fig. 2. Thiswas not intended by us, it was a coincidence that the room inwhich the study was conducted had a whiteboard. This madeit more natural for the subjects. We believe that this mixedreality aspect, that means, the interaction of virtual elementswith real objects, is very promising. Greater inclusion of thereal world could improve usability by requiring fewer dialogwindows and buttons. This can be implemented within theframework of the city metaphor. But it is also conceivableto find a new metaphor that fits better into a typical officeenvironment. VI. C
ONCLUSION
By means of three qualitative evaluations we could showthat the usability of software analytics applications for ARglasses can be improved significantly. There is a lot ofpotential for improvement especially in the area of dialogdesign. Unfortunately, this causes a high implementation effortbecause available standard components are only condition-ally appropriate. But usability can also be quickly improvedthrough better user guidance. We also found indications that astronger emphasis on mixed reality could further improve theapplications. We will explore these possibilities in our futurework. A
CKNOWLEDGMENT
We would like to thank Stefan Mutke and the LogisticsLiving Lab at Leipzig University, who provided us with theused Microsoft HoloLens.R
EFERENCES[1] D. Zhang, S. Han, Y. Dang, J. G. Lou, H. Zhang, and T. Xie, “Softwareanalytics in practice,”
IEEE Software , vol. 30, no. 5, pp. 30–37, 2013.[2] R. Brath, “3D InfoVis is here to stay: Deal with it,” in . IEEE, 2015, pp. 25–31.[Online]. Available: http://ieeexplore.ieee.org/document/7160096/[3] L. Merino, A. Bergel, and O. Nierstrasz, “Overcoming Issues of3D Software Visualization through Immersive Augmented Reality,”in . IEEE, 2018, pp. 54–64. [Online]. Available:https://ieeexplore.ieee.org/document/8530131/[4] L. Merino, M. Lungu, and C. Seidl, “Unleashing the Potentials ofImmersive Augmented Reality for Software Engineering,” in . IEEE, 2020, pp. 517–521. [Online].Available: https://ieeexplore.ieee.org/document/9054812/[5] R. Wettel and M. Lanza, “Code City,”
Proceedings of WASDeTT2008 (1st International Workshop on Advanced Software DevelopmentTools and Techniques)
Proc. of 2ndBrazilian Workshop on Software Visualization , 2012.[7] P. Kapec and G. Brndiarov, “Visual Analysis of Software Systems inVirtual and Augmented Reality,” pp. 307–312, 2015.[8] V. S. Sharma, R. Mehra, V. Kaulgud, and S. Podder, “An Immersive Fu-ture for Software Engineering - Avenues and Approaches,”
Proceedings- ICSE , pp. 105–108, 2018.[9] ——, “An extended reality approach for creating immersive softwareproject workspaces,”
Proceedings - 2019 IEEE/ACM 12th InternationalWorkshop on Cooperative and Human Aspects of Software Engineering,CHASE 2019 , pp. 27–30, 2019.[10] P. Reipschl¨ager, B. K. Ozkan, A. S. Mathur, S. Gumhold, R. Majumdar,and R. Dachselt, “DebugAR: Mixed Dimensional Displays for Immer-sive Debugging of Distributed Systems,”
Conference on Human Factorsin Computing Systems - Proceedings , vol. 2018-April, pp. 1–6, 2018.[11] P. Seipel, A. Stock, S. Santhanam, A. Baranowski, N. Hochgeschwen-der, and A. Schreiber, “Speak to your software visualization-exploringcomponent-based software architectures in augmented reality with aconversational interface,”
Proceedings - 7th IEEE Working Conferenceon Software Visualization, IEEE VISSOFT 2019 , pp. 78–82, 2019.[12] ——, “Adopting conversational interfaces for exploring OSGi-Basedsoftware architectures in augmented reality,” in
Proceedings - 2019IEEE/ACM 1st International Workshop on Bots in Software Engineering,BotSE 2019 . IEEE, 2019, pp. 20–21.[13] A. Schreiber, L. Nafeie, A. Baranowski, P. Seipel, and M. Misiak,“Visualization of Software Architectures in Virtual Reality andAugmented Reality,” in
IEEE Aerospace Conference Proceedings ,vol. 2019-March. IEEE, 2019, pp. 1–12. [Online]. Available:https://ieeexplore.ieee.org/document/8742198/[14] R. Mehra, V. S. Sharma, V. Kaulgud, and S. Podder, “XRaSE: Towardsvirtually tangible software using augmented reality,”
Proceedings -2019 34th IEEE/ACM International Conference on Automated SoftwareEngineering, ASE 2019 , pp. 1194–1197, 2019.[15] V. R. Basili and H. Dieter Rombach, “The TAME Project: TowardsImprovement-Oriented Software Environments,”
IEEE Transactions onSoftware Engineering , vol. 14, no. 6, pp. 758–773, 1988.[16] D. Baum, J. Schilbach, P. Kovacs, U. Eisenecker, and R. M¨uller,“GETAVIZ: Generating Structural, Behavioral, and Evolutionary Viewsof Software Systems for Empirical Evaluation,” in
Proceedings - 2017IEEE Working Conference on Software Visualization, IEEE VISSOFT2017 , 2017, pp. 114–118.[17] R. Wettel, M. Lanza, and R. Robbes, “Software systems as cities: acontrolled experiment,” , pp. 551–560, 2011.[18] J. Nielsen, “Iterative User-Interface Design,”