Motion Browser: Visualizing and Understanding Complex Upper Limb Movement Under Obstetrical Brachial Plexus Injuries
Gromit Yeuk-Yin Chan, Luis Gustavo Nonato, Alice Chu, Preeti Raghavan, Viswanath Aluru, Claudio T. Silva
MMotion Browser: Visualizing and Understanding Complex UpperLimb Movement Under Obstetrical Brachial Plexus Injuries
Gromit Yeuk-Yin Chan, Luis Gustavo Nonato,
Member, IEEE , Alice Chu,Preeti Raghavan, Viswanath Aluru, Cl ´audio T. Silva,
Fellow, IEEE
Fig. 1. M
OTION B ROWSER interface showing how to analyze patients’ limb muscles and movement with data collected from musclesensors, motion sensors, and video recordings. A (cid:13) Muscle Bundle Comparison View displays the muscle signals of affected andunaffected limbs side by side. Statistics from motion sensors ( a ) and stacked muscle activities ( a ) are shown. Visual highlightingtechnique allows the extraction of the relatively stronger muscle activities on both sides ( a ). B (cid:13) Time Series View on raw muscleEMG signals. Each view visualizes the signals from an individual motion and users can align the x- and y-axis of all views ( b ). C (cid:13) Video Inspection View displays the cut scenes and filtered signals from A (cid:13) and allows the export to the presentation slide show ( c ). Abstract —The brachial plexus is a complex network of peripheral nerves that enables sensing from and control of the movementsof the arms and hand. Nowadays, the coordination between the muscles to generate simple movements is still not well understood,hindering the knowledge of how to best treat patients with this type of peripheral nerve injury. To acquire enough information for medicaldata analysis, physicians conduct motion analysis assessments with patients to produce a rich dataset of electromyographic signalsfrom multiple muscles recorded with joint movements during real-world tasks. However, tools for the analysis and visualization of thedata in a succinct and interpretable manner are currently not available. Without the ability to integrate, compare, and compute multipledata sources in one platform, physicians can only compute simple statistical values to describe patient’s behavior vaguely, which limitsthe possibility to answer clinical questions and generate hypotheses for research. To address this challenge, we have developedM
OTION B ROWSER , an interactive visual analytics system which provides an efficient framework to extract and compare muscle activitypatterns from the patient’s limbs and coordinated views to help users analyze muscle signals, motion data, and video information toaddress different tasks. The system was developed as a result of a collaborative endeavor between computer scientists and orthopedicsurgery and rehabilitation physicians. We present case studies showing physicians can utilize the information displayed to understandhow individuals coordinate their muscles to initiate appropriate treatment and generate new hypotheses for future research.
Index Terms —Medical Data Visualization, Visual Analytics Application, Time Series Data, Multimodal Data, Brachial Plexus Injuries
Gromit Yeuk-Yin Chan and Cl´audio T. Silva are with New York University.E-mail: gromit.chan,[email protected]. Luis Gustavo Nonato is withUniversity of S˜ao Paulo. E-mail: [email protected]. Alice Chu is withRutgers New Jersey Medical School. E-mail: [email protected] Raghavan and Viswanath Aluru are with NYU Langone MedicalCenter. E-mail: Viswanath.Aluru,[email protected] received xx xxx. 201x; accepted xx xxx. 201x. Date of Publicationxx xxx. 201x; date of current version xx xxx. 201x. For information onobtaining reprints of this article, please send e-mail to: [email protected] Object Identifier: xx.xxxx/TVCG.201x.xxxxxxx
NTRODUCTION
Hands consist of an incredible number of muscles to perform compli-cated and delicate tasks such as picking flowers without crushing them,stringing beads without scattering them, and drinking gracefully froma wine glass without breaking it or spilling its contents. The importantstructures that connect a brain to limb muscles are brachial plexus ,a complex network of nerves, that can be divided into roots, trunks,divisions, and cords [33]. These nerves allow brain’s control and sens-ing from the arms to our palms. Even though brachial plexus injurieswill lead to the disabilities of several muscles, the brain will still try tocoordinate the remaining functioning parts of the limb to compensate a r X i v : . [ c s . G R ] J u l or hand functions. As a result, studying the mechanisms of impairmentof brachial plexus after injuries and comparing the motions throughoutthe recovery stages can help physicians develop strategies to restore thephysical, cognitive, and emotive aspects of hand functions.To study the behavior of muscles from patients under different medi-cal situations, physicians need to carry out motion analysis assessmentson them with sophisticated setups and protocols. They need to attachmultiple muscle sensors on patient’s hands, forearms and shoulders,and place a camera in front of the patient so that he or she can berecorded for the appearances and muscles activities while carrying outseveral motions. As a result, physicians collect a set of heterogeneousdatasets that record patients’ different kinds of activities, such as elbowextensions, shoulder flexions, and wrist rotations, with different sidesof limbs completed in different duration.Currently, physicians rely on visual results from Spike2 [39] andExcel to conduct analysis. Examples can be seen in Fig. 2. Spike2is a commercial software that provides signal inspection and analysistechniques such as peak detection and signal decomposition. Physi-cians mainly use it for video editing so that the patients’ recordingsare aligned with the data. For Excel, they load the signal data into thecells and plot bar charts with summary statistics such as maximum andminimum values or maximum range of motions. Analyzing the data inthis way has certain limitations. To begin with, the muscles’ activitiesare hard to be compared among patients. Each limb is attached witheight sensors so that physicians have to conduct comparative analysisof multiple multivatiate time series. Moreover, the muscles can onlybe quantitatively compared between patient’s limbs but not amongdifferent patients since they depend on the physical strength of patients.Without a standardized metric to normalize the signals among patients,it is hard to preprocess the data for automated comparison. Physicianscannot conclude unless they inspect and compare all patients’ behav-ior one by one in terms of muscle behavior, physical outcome, andappearance in the video. Furthermore, diagnosis is based on differentcriteria, such as the differences in muscle coordination between af-fected and unaffected limbs within the same patient, the comparisons ofsuch differences among different patients, or simply the compositionsof muscle activities within a single limb. These require integrationsof different attributes and different data abstractions in the dataset onthe same platform. Besides, analyzing multiple time series of muscleactivities from each patient, considered as “muscle bundles” , is also atedious task since physicians have to inspect, analyze, and align mul-tiple muscle signals, motion sensors, and videos. Without integratingthe multimodal data into one platform, physicians have to conduct thetasks manually with different softwares.In this circumstance, using a tailored visual analytics approach toaddress the problems has several advantages. First, users can extractpatterns in muscle coordination through visual sensemaking actions.The muscle signals, which are collected by attaching sensors on thepatient’s skin surface, are noisy and ambiguous. Muscle contractionscome with different shapes of waveforms depending on patients’ habitsso that signal processing techniques cannot produce standardized fea-tures for comparisons. Therefore, providing computed features thathighlight the stronger muscles on each limb while allowing users tosteer the final results can balance the validity and efficiency at thesame time. Moreover, provided the complex interplay between a largenumber of muscle activities, effective visualization aids the evaluation,reasoning, and communication of the physicians’ diagnoses. Last butnot least, a visual analytics system integrating all data sources facilitatesa more holistic analysis for the physicians.To achieve the above-mentioned objectives, we propose M OTION B ROWSER , which consists of a novel analytic workflow to compareheterogeneous muscle bundles and extract significant muscle activitiesthrough semi-automated comparisons with comprehensive visualiza-tion techniques, as well as an interactive user interface to fulfill theneeds physicians to carry out the whole data processing, analytics andcommunication processes. In short, our contributions are as follows: Without loss of generality, we refer to multiple time-varying muscle activitysignals within one limb as a muscle bundle.
Fig. 2. Examples of current tools for analyzing EMG data of muscleactivities. (a) Signals of muscle activities and different correspondingphysical motions are put together to be inspected in Spike2. (b) ExcelSpreadsheet displaying the statistics of muscle activities such as maxand min value and maximum range of motions. • An analytics pipeline to visually compare multiple multivariatetemporal muscle signals between different limbs.• Introduce an interactive visual analytics system to streamline theprocesses of inspecting, comparing and analyzing the multimodalmotion assessment data in an integrated platform.• Present two case studies of physicians using the system to discoverseveral key symptoms behind how the human brain operatesand compromises with impaired nerves, in which they result instrategies to speed up patients’ recovery processes.
ELATED W ORK
Since our work deals with the challenges of comparing a series oftemporal muscle data and analyzing human motion, we believe thatour work can be categorized in the area of motion data visualization,comparative visualization of temporal data and signal visualization.
Many techniques and applications have been proposed for motion datavisualization. Readers can refer to a more detailed survey by Bernard etal. [5]. Mainly, the motions studied ranged from whole body motionsof different entities to motions from a part of the body. For visualizingmotions of the entire body, FuryExplorer [44] compared the motionsof horses by visualizing the trajectories projected with PCA frommotion sensors attached to different parts of the horses. MotionEx-plorer [6] visualized motions with hierarchical clustering techniquesto distinguish different motions and allow experts to cluster motionsin a semi-supervised manner. Krekel et al. visualized the positionsof different joints to understand kinematic experiments [25]. Thesystem incorporated a 3D skeleton model to help users filter differentmotions based on the arrangement of the joints. Keefe et al. proposeda system that used a set of 2D visualizations (i.e. parallel coordinatesand line charts) to filter different kinds of 3D motions [25]. Nguyen et al. abstracted various motion statistics from the knee’s volumetricdata for interactive analysis [30]. GestureAnalyzer [17] clustered thesequential patterns of human motion data to help users understanddifferent motions, while MotionFlow [18] was a continued work thataddressed the same challenges through comparative visualization.Despite having a similar goal of understanding body behavior,M
OTION B ROWSER differs from the current literature by analyzingthe complex muscle coordination with the help of motion data. Ourmain goal is to uncover the relationships of muscles when a limb isinjured, with the help of visible motions that act as verification as wellas an indicator for physicians to leverage their domain knowledge toenhance the analysis.
Time series visualization and comparative visualization are areas thatearn great attention in the realm of visual analytics. Readers can referto [1, 2, 11] for an in-depth survey of how time series is visualized orprocessed for different kinds of goals and abstractions, and [13, 21] forhow comparative visualization is considered and designed. Both visual-ization and comparison are tightly integrated for temporal data analysis.For example, the initial motivation of Playfair’s invention of line chartwas to illustrate comparisons [40]. When considering the design oftemporal data comparison, literature mainly studied different layoutsnd graphs to satisfy specific tasks [19, 20]. The layouts could either bejuxtaposition, superposition, or explicit encoding, while graphs couldbe either stacked, small multiples or overlaid simple graphs.Various techniques and systems were proposed to compare temporaldata. Kehrer et al. applied focus+context techniques on climatedataset to present different abstractions at different levels to facilitatecomparisons [22]. SimilarityExplorer [31] compared spatiotemporalclimate models by separating the models’ space and time attributeswith multiple views for comparisons. Temporal Summaries [42] usedcategorical sequences as the abstraction target to facilitate comparisontasks. Li et al. used multimodal visualization to filter time series withadditional attributes [27].Electronic signals require much interactivity when it comes toexploratory data analysis (EDA). Chronolenses [47], Kronominer [46],SignalLens [23], and TimeSlice [48] proposed interactive systemsto address the challenges on volume and variety of visualizing thesetemporal data in general. There were also other data abstractiontechniques used on multivariate signals. Dal Col et al. [10] andValdivia et al. [41] abstracted multivariate signals as connected graphbased on correlations, so that wavelet theory could be applied to depictsimilarity of groups of time segments to generate abstractions forvisualization. Ward et al. extracted signals as N-grams to project theabstraction with dimensional reduction techniques (i.e. PCA) [43].Our work’s novelty compared with the above work is driven byour physicians’ needs of sequentially scanning multiple comparisons,meaning that we aim at facilitating the comparisons of comparisons.Our comparative visual analytics components have a focus on how tohelp physicians quickly compare patients’ own healthy and affectedlimbs, and then use the results to enable comparisons across differentpatients. Also, our work focuses on the visualization of comparisonsbetween different groups of multivariate time series, in which ourtechniques emphasize the clarity on comparisons, their extensibility onanalysis, and interactivity.
OMAIN P ROBLEM C HARACTERIZATION
The brachial plexus is a set of nerves supply to the upper limb.
Obstetri-cal Brachial Plexus Birth Palsy (OBPBP) refers to injury noted in theperinatal period, which is around the time of birth, to all or a portion ofthe brachial plexus. There are many competing theories as to the sourceof dysfunction in OBPBP and as many different modalities of treatment.The most often cited theories for continuing dysfunction are residualweakness, muscle co-contraction, glenohumeral dysplasia, joint or mus-cle contracture, and ineffective compensatory technique [4, 35]. Thechallenge is that the treatment for each of these etiologies is differentand sometimes contradictory. To obtain a logical algorithm of solutions,physicians need a more detailed understanding of motion in obstetricalbrachial plexus patients, especially how various muscles coordinateunder different levels of severity or symptoms.In general, there are score systems such as
Narakas Classification [3]to assess possible outcomes of patients. Yet, they are based on clinicalobservations and the scores may vary among different clinicians. Tomake treatment plans that do not only base on physical evidence butalso muscle performances, our physicians initiated the
Pediatric UpperExtremity Motion Analysis Program in 2013, which modified theequipment for adults motion analysis to facilitate pediatric motionanalysis. In this
Active Range Of Motion assessment, the patientswere connected with 8 sensors that measured activities in terms of16-channel electromyography (EMG) signals from 8 muscles. Theywere also monitored with 3-dimensional motion analysis and videorecordings. Using the data, our physicians attempted to acquire insightsand generate data facts that helped them explain the rationale behindtheir concluded treatment plans to other clinicians. This eventuallyresulted in the need for a holistic system to improve muscle analysisand data communication.While our study ultimately aims at helping clinicians address medicalchallenges and improve clinical workflow, the process of creating M O - TION B ROWSER is an example that helps provide evidence in creatinga meaningful application that applies interactive visualization to solvea medical problem. To provide a design study that balances both the values of the medical domain and visual analytics, we shaped our devel-opment under the guidelines of Multi-dimensional In-depth Long-termCase studies (MILCs) [38]. In short, we emphasize two important as-pects to maximize the understanding of the visualization design process.
Breaking Domain Experts’ Goals to Hierarchies.
Our physicianexperts hope to understand how muscles coordinate in patients withbrachial plexus injuries, and they applied a set of baseline procedures toconduct the analysis. Therefore, it is important to get both the high-levelgoals and low-level details to address experts’ key needs of interactivevisualization for our application. To do this, we paid clinical visits toconduct contextual inquiry [7] interviews to understand how they con-duct hands-on diagnoses on a total of 8 patients using the data they col-lected from the motion analysis. We formulate the whole process exclu-sively and exhaustively such that it becomes a Hierarchical Task Analy-sis (HTA) [36, 45]. In HTA, actions from the domain experts become aset of tasks and subtasks. Each task has a goal and a plan. HTA allowsus to identify the tasks that can potentially benefit the most from inter-active visualization, such that it helps justify our system requirement.
Iterative Process of Design.
Overall the project took about 12months. Our visualization researchers worked tightly with two Ortho-pedics and Rehabilitation physicians in shaping the scope, identifyingthe objectives and iterating different system designs. One physicianhad regular clinical checkups with the 8 patients and another physician-scientist was responsible for conducting the motion assessments. Wefirst replicated some baseline features such as time series plots of EMGsignals, and then the physicians requested more features for generat-ing results and different forms of communications from the data. We,therefore, held regular (bi-weekly) meetings to evaluate the availablefeatures and then introduce different designs and algorithms for thephysicians to evaluate. Our physicians also occasionally introduced thetool to other researchers and collected their feedback.
ATA A BSTRACTION
The
Active Range Of Motion assessment mentioned in Sect. 3 generatesthe AROM dataset to initiate our analysis. Each motion consists ofEMG signals of 8 muscles from each side of the upper limb, allowingphysicians to acquire muscle activities throughout patient’s forearms,elbows, hands, and fingers. Alongside this, a patient’s limb motion isalso tracked with motion trackers and video recordings. Overall, theEMG signals play the main role in comparing the patients’ muscle coor-dination, while the motion tracking data provides the context of patients’performances. The videos act as verification and direct inspection ofthe whole motion. The whole dataset is summarized in Table 1.
Table 1. Data Abstraction Summary
Key Attributes Data Abstraction ObjectivesMuscle Activity Quantitative (EMG Signals) ComparisonsMotion Tracking Quantitative (Positions) ContextVideo Monitoring Images Verification
Muscle Activity Signals.
The EMG signals play a central role in help-ing physicians clarify the coordination of muscles throughout patients’motions. While the signals capture the muscle activity with their am-plitudes, they are hard to read and compare in the raw format. To firstfacilitate the inspection of EMG signals, they are usually transformedwith root-mean-square (RMS) envelope, which is calculated using atime-windowed RMS function:
RMS = ( S S ∑ f ( s )) (1)The RMS value represents the power of the signal, which correlateswith the degree of muscle activity and is always positive. As the rawEMG signals are oscillating and produce more clutters for visualization,turning them into RMS values produces a polarized waveform that ismore easily analyzable.However, there are two challenges from the EMG signals that cannotbe solved solely by automatic computation. Firstly, the EMG signalsare only normalized within the same person. It means that the amplitude ig. 3. Hierarchical task abstraction of the clinical workflow. Each box represents a task or subtask and each level of hierarchy has a plan. Thehorizontal line at the bottom of the box means a termination. The highlighted purple text represents the task abstraction derived from the tasks. of muscle signals can only be compared between a patient’s left andright limb but not among different patients. Physicians thus use themfirst to find out which muscles in the affected limb are stronger than theunaffected limb to deduce the compensatory muscles, then compare thepresence of stronger muscles among different patients. We address thecomparisons of patients by the comparisons within their limbs as the scalability challenges of our visual analysis. Secondly, we have to dealwith the noisiness of muscle signals. As the EMG signals are collectedfrom the skin, there might be an ambiguity of the resulting power. We,therefore, have to ensure human-in-the-loop throughout the analysis. Data from motion tracking.
The 3-dimensional motion analysistracks the patient’s limbs’ coordinates in x, y and z directions. These at-tributes enable the derivation of speed and acceleration of limb motions,which allow physicians to locate a finer scope of motion and comparethe overall performance without inspecting the video.
Video Monitoring of Motions.
The video monitoring of the whole mo-tion provides a full picture of the patient’s performance. Each cut scenelets physicians verify their findings from the analysis of muscle signals.
ASK AND R EQUIREMENT A NALYSIS
Mentioned in Sect. 3, we break down the procedures of conductingan analytic workflow from the domain perspective in a HierarchicalTask Analysis. In the clinical diagnosis, the physician experts want todecide which muscles, joints or nerves should be targeted for treatment ( Task 0 ). To prepare for the analysis, they first modify the equipment toconduct a series of motion analysis (
Task 1 ). The motion data, includingmuscle activities, motion data, and video recordings, are recorded fromboth the affected and unaffected limbs so that the muscle EMG signalscan be normalized and compared between both sides (
Task 1.1 ). Then,they use video editors to trim the videos such that all the data attributesare temporally aligned (
Task 1.2 ).After the physicians acquire the data, they inspect each patient’slimb motion (
Task 2 ). Throughout the motion analysis, there may existirrelevant information related to the specific motion. For example,physicians do not need to know the fingers’ muscle activities whenthey analyze a patient’s shoulder rotation, or the patient only spend 5seconds to finish the action throughout the 15 seconds recording. As aresult, the objective of inspecting everything is to extract the relevantinformation from the motion assessment (
Task 2.1 ). When physicianslocate the useful portion of data (
Task 2.1.1 ) and remove the irrelevantparts (
Task 2.1.2 ), they will be able to acquire useful overview (
Task2.2 ). Browsing all the information regarding muscle activities, motion and videos together, physicians try to acquire a general impression ofthe performance such as which muscle(s) play(s) the main role (
Task2.2.1 ) or which of them is/are injured (
Task 2.2.2 ).At this stage, physicians already grasp a basic understanding of howthe patient behaves, so now they can analyze how is the coordinationof muscles compromised for the affected limb (
Task 3 ). This is doneby comparing the difference between the patient’s limbs. For example,physicians inspect which muscles in the affected limb react stronger,or vice versa (
Task 3.1 ). Once physicians identify which musclesplay important roles in each limb, they can separate and highlightthem (
Task 3.2 ). For example, if the affected limb is identified tohave higher activities of biceps and triceps compared with the healthylimb, then the physicians understand these muscles play an active rolein compensating the movement. Such facts allow them to comparethe results among different patients. By comparing the comparisons,physicians will be able to answers questions like “Does trapeziusmuscle play an active for patients with brachial plexus injuries?”Therefore, when the physicians finish analyzing each performance,they need to put all the patients’ results on a single page to answer surgi-cal concerns (
Task 4 ). Usually, they first classify patients into differentgroups based on their performances, such as “high functioning” and“low functioning” groups, and then summarize the muscle activities ineach group (
Task 4.1 ). The summaries are in the form of proportions tolet physicians understand which muscles are prevalent as compensatorymuscles among the patients. For example, if biceps and triceps are thetwo muscles that behave stronger in the affected limb on one patient,the physicians would like to know their ratios among other patients tosee if these muscles are common priorities to compensate the motions.Based on these, they can generate hypotheses and verify their reasoningwith video recordings at the same time (
Task 4.2 ). Note that a diagnosismade with such analysis is empirical and based on observations sothat physicians need to pack all of the findings in a presentable formatthat includes charts and video screenshots and send the report to othercolleagues. The presentation mainly acts as evidence that helps physi-cians explain more in-depth medical knowledge phenomena. Once theconsensus is reached among several physicians, then they will land amore formal diagnosis and treatment plan (
Task 5 ). Based on the hierarchical task analysis, we translate the domain tasksinto abstract forms to generate tasks in the visual analytics domain.Based on our interview with the domain experts, we noticed that whilethe task abstractions of many design studies aim at exploratory datanalysis [26], our main challenge is to combine domain knowledgeand computation power to make the analysis more streamlined andefficient . Instead of using multiple software to glue the heterogeneousdata together and just solely put the signals side by side for comparison,our experts want to have minimal interactions so that they can gothrough each patient one by one faster. Therefore, we follow severaliterations in the nested model [28] and loop through the four nestedlayers to refine our requirements from the task abstractions (Fig.3). Thehigh-level task abstractions apart from the operation contexts (i.e.
Task1 and 4 ) that our physicians want to achieve from the data analysis aresummarized as
T1-4 in Fig.3.
We identified design requirements based on the informal qualitativeinterview with physicians and task abstractions discussed above.
R1 Align heterogeneous data sources.
Physicians need to analyzeinformation from muscle and motion sensors and video recordingsto compare the muscles within a patient’s limb or between hisaffected and unaffected limb (
T.1 ). To speed up the inspectionand exploration, the display should align the information by time,patient, and muscles.
R2 Display and analyze individual patients’ performance in oneview grouped by each patient.
Our experts indicate that it is aclinical practice that they need to analyze patient’s data one byone before summarizing their behavior as a whole. Therefore, thesystem should display information and analysis of each patientin an individual view, while showing all information in the samewindow, so that physicians can explore, scroll, and analyze thepatients effectively (
T.2 ). R3 Enable efficient comparative analysis.
It will be time-consuming and cognitively overwhelming to extract stronger mus-cles on each limb by manual selection (
T.3 ). Physicians needan efficient mechanism to compare muscle coordination betweeneach patient’s limbs, then use the comparisons to understand thedifference among all patients. While the definition of strongermuscles is subtle and depends much on observation, some similarmuscle activities, such as similar waveforms and magnitudes,should be filtered away at the beginning.
R4 Export the clinical analysis for presentation.
After finishingthe data analysis (
T.2 ), experts need to report their findings on thecomparison among patients for discussion and verification (
T.4 ).An easily communicated visualization is preferred to showcasetheir results to other colleagues.
ISUAL A NALYSIS OF M USCLE A CTIVITY C OMPARISON
In this section, we present the main visual analytics component ofthe system, the muscle bundle comparison chart, for comparing themuscle coordination between a patient’s affected and unaffected limb.This component contains the main analytics pipeline in analyzing themuscle signals, while the other views in the next section are responsiblefor information and presentation. Because there exists a tradeoff be-tween visual clarity and interactivity, we introduce a visual highlightingtechnique using an entropy computational metric to quickly extractmeaningful comparisons.
To begin with, we first address the requirement of displaying all sensorsof each patient in one view ( R2 ). A bundle comparison chart displaysthe motion assessment information of patient’s both limbs (Fig. 1 A (cid:13) ,Fig. 4) side by side. The left-hand side shows the data from the affectedlimb and the right-hand side shows the data from the unaffected limb.The line chart in the view (Fig. 4) displays the quantitative outcomeof the motion (i.e. speed or displacement). When users brush on theline chart, the proportion of muscle activities in the brushed regionwill be shown as a donut chart (Fig. 1, Fig. 10). The bar charts showthe muscle activity signals. The signals are either stacked or arrangedin small multiples vertically (Fig. 1( a ), Fig. 4). Color encodes themuscles and the same muscles from both limbs contain the same coloras well. We use the eight qualitative color scale from ColorBrewer [14] Fig. 4. Entropy-based visual highlighting: (a) Transform the signalsof two limbs to histograms of value, then (b) compare the ordinaldistributions of the histogram. (c) The similar and smaller muscle signalsare reduced so that significant signals eventually stand out. in a way that two bluish colors correspond to two pushing muscles,greenish colors correspond to forearm muscles, reddish colorscorrespond to back muscles and yellowish colors correspondto finger muscles. In an aligned way, the activities from the samemuscles in both limbs are also plotted together vertically. Lastly, everyinformation is also aligned with time horizontally.
We experiment with various visual techniques by iterating several lay-outs and approaches with the physicians. Visual comparison of a seriesordered data can be accomplished by juxtaposition, superposition orexplicit encoding [13]. Explicit encoding does not apply to our prob-lem because the relationships between different muscles are hard tocalculate, which is one of the reasons for us to develop a comprehen-sive system to address the challenge. Superposition, either by placing16 muscles together or 2 compared muscles in each small multiples,causes the alignment of motion information and muscle data to becomeobscure. As the physicians want to identify which muscles contributeto rapid movement or limb positioning, they prefer inspecting everyinformation in a separate chart for better clarity ( R2 ).The next consideration is whether we should stack each muscleactivity on one chart instead of placing them in small multiples. Wefirst tried stacking the muscle activities from one limb together tosqueeze more muscle comparison views on one screen. While thephysicians appreciated the efficiency of scrolling through patients insuch layout, they raised a problem of perceiving small but importantmuscle signals in a limb. Sometimes if a muscle emits a relativelyweaker signal compared with muscles inside the limb but is indeedstronger than the same muscle across the other limb, it is still treated asimportant. Such inconsistency of scaling makes small multiples a betterchoice in comparing the difference [20]. Thus, in our current version,we include both layouts for the physicians to inspect both forms.Moreover, we consider the alternatives of donut chart when brushingthe line chart. While the physicians were keen on seeing the percentageand using it as a button to play the video at the same time, our firstversion encoded the donut chart’s radius with the total amplitude ofmuscle signals. It turned out not to be a good idea since when thechart became too small, the physicians could not inspect anythingmeaningful. Nonetheless, the amplitude information can already beperceived in the stacked charts. Therefore, we do not encode any moredetails on the donut chart except the proportion of muscle signals. ig. 5. Illustration of how users can discover the significant muscles onboth limbs. (cid:13) When users apply the highlighting option, the highlight ofmuscle activities in Fig. 1 A (cid:13) change based on their relative significance.(a) The original power is encoded with unfilled lines. (cid:13) Sliding throughthe threshold filter, the charts without any values left will collapse,reducing the number of muscles shown at the end.
The separation of all data in the view aims at helping the physiciansexamine all sources of quantitative data in a consistent and alignedtime scale ( R1 ). However, now if everything is separated, there exists achallenge in interactivity, as users need to click on each of the bar chartsone by one to remove the muscles that are similar between the limbs. Ifnone of the charts are removed, the number of visuals will accumulatewhen users scan each patient’s bundle comparison views sequentially.Thus, either remove or not remove will overload users’ cognitive abilityeasily. Therefore, we investigate the possibilities for physicians toinspect the information clearly while efficiently extracting the moresignificant muscle activities on both sides of the limbs ( R3 ). We intro-duce a method that can offset similar signals and reduce the presence ofsmaller signals so that the significant signals will eventually stand out.Fig. 4 illustrates the method. First, for each muscle activity, weput all the values into a histogram. Then we calculate the differencebetween the histograms of the same muscles in both limbs, usingKullback-Leibler divergence: D KL ( Q || P ) = ∑ i Q ( i ) ln ( Q ( i ) P ( i ) ) (2) P and Q are the distributions of values in each muscle’s activity. Thisfunction measures how one distribution deviates from another referencedistribution and is used as highlights in several situations of visualiza-tion [9, 16]. In the example here, when two distributions are similar toeach other, each log value in the summation will become very close tozero, leading to a low divergence. On the other hand, there are greatdifferences between each bucket, the divergence becomes high. We alsocalibrate the signals with the skewness [24], such that if the distributionis more left-skewed (i.e. more small values) the value will decrease.The histogram is constructed by K Means clustering. For simplicity,we use the elbow method to decide the value of K. In this way, we canvisualize a larger number of muscle signals and still quickly identifythe significant muscle activities. We introduce a mechanism of interaction to efficiently identify thesignificant muscle signals with the help of our visual highlightingmethod. When users click to apply the muscle highlighting, they can
Fig. 6. When users remove a muscle, the corresponding bar charts willcollapse and the remaining ones may rescale to become more visible. inspect the highlighted results illustrated in Fig. 5 (cid:13) . The results areencoded in colored area charts and the strokes in the chart encode theoriginal power (Fig. 5(a)) so that users do not need to switch back andforth to recall the original muscle signals. Our method of grasping thesignificant muscle signals can be summarized as sliding and collaps-ing . When users slide and filter by highlighted values, the bar chartsthat become empty will disappear. If bar charts on both sides bothdisappear, the whole space will collapse. As a result, the final displayof muscle activities will be distilled to only the muscle signals thatare significantly greater than the opposite limb (Fig. 5 (cid:13) ). On the otherhand, users can manually remove the bar charts by checking the legend.In both ways, the remaining bar charts will resize such that the musclesmay become more visible due to the rescaling of the y-axis (Fig. 6).The sliding mechanism engages users to opt for a clear comparisonwith a few amounts of interactions needed so that they can obtainminimal operations to analyze each patient sequentially ( R3 ). HE M OTION B ROWSER S YSTEM
To support users analyzing each’s behavior, our system consists of aquery panel to select the dataset, the muscle bundle comparison view,and a video inspection view. These displays follow a hierarchical rela-tionship in a way that follows the visual analytics mantra “overview first,zoom and filter, then details-on-demand” [37], from the perspective ofanalyzing an individual patient. A general overview of the workflowis as follows: first, the user obtains and analyzes each patient’s data inthe bundle comparison view in Sect. 6, then the processed details areexported to the video view for comparison among all patients. Afterthat, the findings are shown in the presentation view.
Time Series View
Users can inspect raw muscle EMG signals inthe time series view (Fig. 1 B (cid:13) ). They can query with different filetypes to select the motion of a patient’s limb. Also, the scales acrossdifferent time series views can be aligned so that different time seriespanels can be compared with each other. This view acts as a time serieseditor when users encounter an erroneous motion (i.e. unreasonablylong recordings or abnormal muscle signals). Users can visually refinethe timeline and muscle selection ( R1 ) and import the refined resultsto the bundle comparison view. Video View
After users click the play button in the selected timeintervals (Fig. 1 a ), a new window will appear to show the selectedmuscle activities under the video. All of the information is alignedwith a line to synchronize the video time frame and charts (Fig. 1 C (cid:13) ).Such encoding allows a compact integration of all information andinsights obtained aligned with video evidence ( R4 ). Users thus canverify their findings and derive reasons between muscle coordinationand physical outcome. After users finish inspecting the results they obtained in the video view,they can export these video snippets to the presentation view (Fig. 7).This view is a grid layout that allows users to align the insights fromthe analysis to compare different patients and add annotations toexplain the findings ( R4 ). Each patient’s muscle activities are shownas a percentage since the percentage of muscles used can be comparedacross different patients. For example, experts can understand a patientuses more biceps to compensate for his shoulder movement than otherpatients by showing the proportion of bicep activities. In this way,these findings can be communicated to different parties that helpfacilitate a more useful discussion. ig. 7. Presentation View. Users can create a presentation for their ana-lyzes by a (cid:13) annotating with titles and subtitles; b (cid:13) arranging each insightfrom video views in the presentation spreadsheet, and; c (cid:13) presentingthe muscle activity insights on each limb in percentages. Our system provides various interactions besides the ones in Sect. 6.
Filtering.
Filtering exists in the file selection menu in Time SeriesView. Given more than 200 motion assessments, we provide anexclusive drop-down menu, that each drop-down only shows theavailable options filtered by the user’s selection on other drop-downs.
Brushing.
The brushing interaction in the line chart illustrated inFig. 1 a plays an important role in drilling down the final presentationand outcome from the analysis so that users can achieve detail-on-demand in different stages of required actions ( R4 ). Fig. 8 illustrates the system architecture. M
OTION B ROWSER is a web-based application developed under Flask framework. The front-end wid-get functionality and plotting are achieved by Gridster and D3.js. Thedataset is stored as Pandas Dataframe indexed with files and time steps,and we store all the users’ saved files in MongoDB. We use NumPy forall data handling tasks. An important benefit of this approach is thatwe can vectorize all computations by treating the dataset as a matrix,allowing computations to be optimized in the low-level architecture.We deployed the back-end part into our server with 2GHz Intel XeonE7-2850 CPU and 32GB memory. We achieve interactive speed in allcalculations without precomputing any statistics and caching, while ourusers from North America and Europe can work in their local machinesand create, load and save their work without any installation.
ASE S TUDIES
Our physicians found many interesting patterns and insights addressingclinical problems and furthermore challenges in the medical researchdomain. To better illustrate how they generated insights and collectedevidence with M
OTION B ROWSER , we present one case study focusingon addressing clinical research challenges, and one case study abouthelping physicians land clinical findings.
Our first case study summarizes how the physicians used the system toaddress the following much discussed clinical challenge [8, 12, 15]:
Do the upper trapezius (UT) and lower trapezius (LT) muscles haveuseful activities in the affected limb for shoulder motions?
It is an interesting question because the role of trapezius muscles innormal conditions is poorly understood, and physicians may considerthe options to denerve these muscles, which they disconnect the nervesconnecting to the muscles and reconnect the healthy nerves to theaffected region, to provide a better distal ability. However, the effectof the loss of the trapezius muscle, while known to be debilitating innormal children, has to the best of our knowledge never been studied inchildren and adolescents with chronic obstetrical brachial plexus palsy.Therefore, using M
OTION B ROWSER , our physicians addressed thisproblem with the comparative analysis of shoulder motions.
Fig. 8. System architecture for data storage, modeling, and visualization.
Import, Inspect and Refine.
Our physicians imported the compar-isons of the shoulder flexion motions between the same patients’ limbsby selecting “Shoulder Flexion” in the muscle bundle comparison view.The first task before data analysis was to make sure all the data shownwas clean (
T.1 ). She discovered one of the motions’ durations differedgreatly between the limbs, therefore she opened the video by brushingthe line chart to inspect the video to see what happened to the surpris-ingly long motion. As the muscle activities of the affected limb in theextended duration did not seem to contribute to the task in the video,she use time series view to truncate the duration of the patient’s motion.Eventually, our physician acquired cleansed muscle bundle comparisonresults of all shoulder flexions for further inspection (
T.2 ). Classify Patient Behavior Through Comparisons.
After a set ofmuscle bundle comparisons was generated, our physicians usedthe visual highlighting filter to quickly remove similar and moreinsignificant muscle activties between the affected and unaffected limbs(
T.3 ). Given the stronger muscles between the limbs, they discoveredthat patients indeed behaved differently under three circumstances.Therefore they exported the analyzes of affected limbs from thecomparison charts to video views (Fig. 9) to generate a summaryof comparison (
T.4 ). To begin with, two of them (Fig. 9 A ) emittedgreater signals in almost every muscle. Physicians discovered it byinspecting the analyzed comparisons that resulted in the video viewsof highlighted muscle activities in Fig. 9 i (cid:13) . The appearance of allmuscles in the video views meant that the affected limbs fired nearlyevery muscle in greater magnitudes than the unaffected side. In thiscase, the patients’ problems were only overshooting their muscles,indicating less severity. Furthermore, four patients clearly showed alack of trapezius muscle activities in all shoulder motions (Fig. 9 B ). Itcould be easily seen that only the pronator muscle activities remained(green muscles). However, our physicians were skeptical about theirimportance for the analysis, since they mainly served forearm’s motionsbut not shoulder’s. Last but not least, two patients were classified ashaving more activities on their trapezius muscles in their affected limbs,for which the low trapezius muscles could be significantly seen on theaffected sides when highlighted muscles were inspected (Fig. 9 iii (cid:13) ). Collect Visual Evidence to Support Stance.
At this stage, our physi-cians would like to know how do patients behave with trapezius mus-cles? and how do patients behave without trapezius muscles?
They,therefore, inspected the videos of patients’ motions in the compari-son view by brushing the time intervals with high limb displacementshown in the line chart to verify their hypotheses (
T.4 ). From thecut-scenes, our physicians were able to conclude the detected muscleactivities quickly. For the patient group without more significant trapez-ius muscle activities, all of them performed the motions effectively(Fig. 9 i (cid:13) ). Nonetheless, our physicians could see little debilitation onthe patients when they perform shoulder activities. On the contrary,interestingly, our physicians could observe profound debilitation forsome patients with stronger trapezius muscle activities on both affectedand unaffected limbs. When they inspected the video cut scenes, theycould see that the patient could not flex her shoulder or even raisedher arm greater than around 45 degrees (Fig. 9 iv (cid:13) ). Therefore theyunderstood that the reason why the patient had a vigorous activity onher biceps was that she kept bending her forearm during flexing hershoulder. These visual clues, plus the facts from the physical outcomes,suggested that decreased trapezius muscle activities did not necessarilyprohibit shoulder motions while having significant activities of them ig. 9. After using the analytic workflow shown in Fig. 5 on every patient to compare each affected and unaffected limbs, our experts summarizedthree groups of patients to evaluate the usefulness of trapezius muscles (pink and red) on shoulder motions. A : Patients using extra powers onalmost every muscle on the affected limbs shown in i (cid:13) ; B : Patients with significantly stronger trapezius muscle activities on his unaffected limbshown in ii (cid:13) ; C : Patients using more trapezius muscles shown in iii (cid:13) . Further inspections in ii (cid:13) suggested that profound debilitation was not significantamong all patients in group B , while it could be observed among several patients in group C . These provide evidence to support spinal accessorydenervation on patients with Obstetrical Brachial Plexus Palsy. did not provide evidence that shoulder motions could be completedefficiently. Thus our physicians concluded that, with particular regardto obstetrical brachial plexus cases, nerve transfer could be a seeminglyattractive option compared to the alternative of exploring and graftingthe C5 nerve root. The approach for nerve transfer from these musclesis more superficial and does not require the use of autologous nervegraft, as well as being a shorter distance to the muscle. They eventuallyarranged the results in the presentation view in three columns to conveysuch findings to other colleagues. Clinical Findings Besides Research Challenges.
Besides collectingevidence for answering research questions, our physicians found ithelpful to classify patients into different categories, using the insightsacquired by the bundle comparison chart. In identifying the three pat-terns, we can make conjectures about treatment options. For the firstcase where patients nearly overshot all of their muscles in their affectedlimb (Fig. 9 A ), it demonstrated that they had a functioning cerebralpattern of activity. Our physicians believed that this subset may bebetter served by modulating duration, activation, deactivation, and co-ordination of muscle activity. For the second pattern where there werelimited trapezius muscle activities (Fig. 9 B ), patients still had adequateshoulder motion proofed by the evidence mentioned above. Such ob-servation raised a question of how the motions had compensated usingother muscles not attached by the sensors, thus further analysis could re-veal what can be improved to go for a better function. In the last pattern(Fig. 9 C ), having greater activations of trapezius muscle activities wasnot necessary to lead to good shoulder motion. Thus, our physiciansconclude that other sources were responsible for the poor shoulder mo-tion. In the future other shoulder muscles, such as deltoid, infraspinatusor supraspinatus muscles, should be added to the assessment. The second case study mainly focuses on applying M
OTION B ROWSER to aid our physicians in addressing more general assessments when con-ducting clinical consultations. Based on observation, our physician ex-perts need to give a
Narakas Classification [3], a grading system basedon clinical observation to assess possible outcomes of children with ob-stetric brachial plexus palsy. Our physicians often had a question: “
Canwe provide better anticipation before conducting clinical observation? ” Import Motions with Clinical Concerns.
Before going for a consulta-tion, our expert first loaded the motions of shoulder abduction from thepatient’s affected and unaffected limbs to the time series view. Then shediscovered that the chart was distorted because the recording of one of the patient’s motion was much longer, but there was no sign of vigorousmovement after the middle of the assessment. Also, the unaffected limbhad a strong pronator (PQ and PT) and flexor digitorum superficialis(FDS) muscle activities. She concluded that they were not helpful inshoulder motions because PQ and PT muscles were responsible forforearm movement and FDS was for fingers. Therefore, she shortenedboth activities to around 16 seconds and removed the inspection ofother muscles except biceps, triceps and trapezius muscles for a morefocused inspection on the clinical purpose (
T.1 ). . Now, the results inthe muscle comparison view from importing the cleansed time seriesbecame much clearer since there were only four muscles in the bundlecomparison chart (Fig. 10(a)). This allowed the physician to conductindividual limb analysis (
T.2 ). Acquire Facts Before Consultation.
Our experts then inspected thecoordination of muscles within one limb. After filtering some irrelevantportion with the aid of the displacement function line chart (
T.3 ), itappeared that all muscles were active when the patient conducted shoul-der abduction with similar total distribution among the limbs. However,to verify the findings using information apart from the muscle activities(
T.4 ), our physician inspected the physical outcome of the patient in thevideo clip (Fig. 10(b)) and observed that the patient did not extend hershoulder as high as her unaffected limb in Fig. 10(c). It could be seenthat the affected limb had a similar output of muscle activities withouta desirable outcome. While whether the functions of trapezius musclewere important was still unclear, a lack of effectiveness of biceps ac-tivity with a limited degree of abduction shown in the video indicateda Narakas I characteristics. While our physicians could make sucha conclusion after inspecting the data, whether the patient would fallinto Narakas II depended on her wrist movement. Nevertheless, theyacquired a brief and speedy inspection of muscle behavior with musclebundle comparisons and video inspection in M
OTION B ROWSER . Hypotheses to Current Assessment Methods.
Our physicianschecked their previous documentation, and found that the patient be-longed to the “low functioning” group, meaning the patient had “ arange of active motion in shoulder flexion and abduction <
90 degreeson the affected limb ”. Beforehand, patients were first classified intolow and high functioning groups according to their degrees of shouldermovement, then physicians tried to see if there were any differences inmuscle activity between the two groups. Yet, after checking the muscleactivities and physical outcome of the patient, our physicians were notconvinced that the patient was low functioning since there were stillsome biceps activities and the movement did not look that minimal. ig. 10. Usage scenario of using M
OTION B ROWSER to anticipate clinicalevaluation. The Muscle Bundle Comparison View in (a) displays themuscle activities from shoulder abduction conducted with the affectedlimb and the unaffected limb. The visual clues of important muscles’activities combined with the difference of visible outcomes in (b) and(c) provide explanations for medical classifications.
Our physicians thus suspected the classification of patients by suchmethodology. With M
OTION B ROWSER , they tended to rely more ona visual inspection and used the patterns mentioned with the previouscase to classify patients, as it looked more convincing and the visualfacts were more holistic and presentable.
Our physicians were impressed with the designs. They commentedthat the arrangement of placing every patients’ performance one byone with the same design made the analysis extremely convenient andefficient. Also, the ability to offset every similar muscle signals madethe analysis easy to proceed. They emphasized that these convenienceswere crucial in the diagnosis process. Physicians often did not want tospend too much time on acquiring information since they went throughdifferent kinds of diagnoses besides data analysis. Therefore, theyregarded visual analytics demonstrated in the system as an efficient andexplainable data analysis tool.We also identify a usability issue in the system. Our physiciansraised an issue of the occasional confusion of color encodings on themuscles. Thus, the system always provides labeling besides the musclesin all the views, so that when users encounter different color bars, theycan always refer to the names of the muscles shown beside the charts.
ISCUSSION
Lessons Learned.
Working with physicians, we learned a new per-spective of valuing the importance of visual analytics when users needto work on the data. The visual analytics mantra “overview first, zoomand filter, then details-on-demand” was built on the general goal of generating insights [34]. Yet, similar to many clinical workflows, oursituation was about insights through iterated inspections , that requiredlots of inevitable trial and error processes. Therefore, our physiciansemphasized the greatest benefit of our system was about shorteningtheir cycles of analyzing each patient from clicking multiple musclesto using a slider to filter the most insignificant muscles within one drag.Users might not receive any insights during the analysis of an individ-ual patient, but what makes the analysis useful is the ability to quickly extract the useful facts in a single iteration. As a result, users couldacquire an overview of useful information that eventually produces aninsightful conclusion.Another lesson learned was an enforced impression on the impor-tance of verifiable visualization techniques. Our validity of using sliderswith the visual highlighting method to highlight important muscles re-lied much on the presence of original muscle signals encoded withstrokes to keep the original data (Fig. 5(d)). When it came to clinicaldecisions that are prone to false information, the effectiveness of vi-sualization lied in the ability to verify but not solely on the ability togenerate insights. It is crucial to prevent ourselves from falling one ofthe abstraction threats mentioned in the nested models for visualizationdesign and validation: “operations and data types do not solve thecharacterized problems”. In our situation, our mistakes at the beginningwere that we tried to aggregate the muscle coordination for more intu-itive visuals like projections or clustering, but it was some iterationslater that we figured out the importance of avoiding any automatedfeature extraction when the data itself was ambiguous. Evaluation.
The effectiveness of our system is mainly evaluated bythe clinical impacts and medical findings made by our domain experts.Our limitation lies in a lack of concrete numerical values such as taskcompletion time or A/B testing to demonstrate our effectiveness. Weare currently working on using our system to conduct more in-depthmedical related user experiments.
Scalability.
Our system can handle feature extraction and comparisonsof multivariate temporal data in an interactive speed, but it does notsupport visual comparisons of a large number of signals since we usecolor to distinguish them. Though it is unrealistic to attach hundreds ofsensors to the patient, the perception of noticeable differences betweencolored signals will diminish when there are more than 12 lines [29]. Ifthere is a need for comparing a muscle bundle of more than 12 temporalattributes, we will investigate more on prioritizing time series data forvisualization. Rong et al. have investigated the topic of prioritizingdeviation of univariate temporal data [32], and we believe that establish-ing an attention aware strategy of prioritizing multi-attribute temporaldata will be a promising direction.
Application Domain.
Although our work is primarily designed forEMG signal bundle comparison, it can be easily adapted for othersimilar problems, such as adult motion analysis or sports injury analysis.In these problems, we can apply similar techniques to compare thedifferent behavior between different body-parts in respective motions.As we broadcast M
OTION B ROWSER to a greater amount of physiciansof the Pediatric Upper Extremity Motion Analysis Program, usabilitybecomes an important aspect of the software development in the future,and we need a formal user study to test its usability. We hope tointroduce our system to a wider audience in the domain of motorrecovery research so that the system will be more generalized forvarious kinds of recovery studies.
10 C
ONCLUSION
In this paper, we presented M
OTION B ROWSER , a visual analyticssystem that allows users to interactively compare muscle bundles’activity from patients under obstetrical brachial plexus injuries.M
OTION B ROWSER proposes techniques for efficient analysis ofmuscle signal bundles and integrates different sources of heterogeneousdata into consistent and coordinated views, thus aids physicians tounderstand nerve coordinations under obstetrical brachial plexus palsy. A CKNOWLEDGMENTS
This work was supported in part by: the Moore-Sloan Data Sci-ence Environment at NYU; NASA; NSF awards CNS-1229185,CCF-1533564, CNS-1544753, CNS-1626098, CNS-1730396, CNS-1828576; 302643/2013-3 CNPq-Brazil and 2016/04391-2 S˜ao PauloResearch Foundation (FAPESP) - Brazil. C. T. Silva is partially sup-ported by the DARPA MEMEX and D3M programs. Any opinions,findings, and conclusions or recommendations expressed in this mate-rial are those of the authors and do not necessarily reflect the views ofDARPA and S˜ao Paulo Research Foundation.
EFERENCES [1] W. Aigner, S. Miksch, H. Schumann, and C. Tominski.
Visualization oftime-oriented data . Springer Science & Business Media, 2011.[2] W. Aigner, A. Rind, and S. Hoffmann. Comparative evaluation of aninteractive time-series visualization that combines quantitative data withqualitative abstractions. In
Computer Graphics Forum , vol. 31, pp. 995–1004. Wiley Online Library, 2012.[3] M. Al-Qattan, A. El-Sayed, A. Al-Zahrani, S. Al-Mutairi, M. Al-Harbi,A. Al-Mutairi, and F. Al-Kahtani. Narakas classification of obstetricbrachial plexus palsy revisited.
Journal of Hand Surgery (EuropeanVolume) , 34(6):788–791, 2009.[4] G. V. Anguelova, M. J. Malessy, E. W. Zwet, and J. G. Dijk. Extensivemotor axonal misrouting after conservative treatment of obstetric brachialplexus lesions.
Developmental Medicine & Child Neurology , 56(10):984–989, 2014.[5] J. Bernard, A. V¨ogele, R. Klein, and D. W. Fellner. Approaches andchallenges in the visual-interactive comparison of human motion data. In
VISIGRAPP (3: IVAPP) , pp. 217–224, 2017.[6] J. Bernard, N. Wilhelm, B. Kr¨uger, T. May, T. Schreck, and J. Kohlhammer.Motionexplorer: Exploratory search in human motion capture data basedon hierarchical aggregation.
IEEE transactions on visualization andcomputer graphics , 19(12):2257–2266, 2013.[7] H. R. Beyer and K. Holtzblatt. Apprenticing with the customer.
Commu-nications of the ACM , 38(5):45–52, 1995.[8] J. B. Bhatt, R. Glaser, A. Chavez, and E. Yung. Middle and lower trapeziusstrengthening for the management of lateral epicondylalgia: a case report. journal of orthopaedic & sports physical therapy , 43(11):841–847, 2013.[9] M. Correll and J. Heer. Surprise! Bayesian weighting for de-biasingthematic maps.
IEEE transactions on visualization and computer graphics ,23(1):651–660, 2017.[10] A. Dal Col, P. Valdivia, F. Petronetto, F. Dias, C. T. Silva, and L. G. Nonato.Wavelet-based visual analysis of dynamic networks.
IEEE transactionson visualization and computer graphics , 24(8):2456–2469, 2017.[11] F. Du, B. Shneiderman, C. Plaisant, S. Malik, and A. Perer. Coping withvolume and variety in temporal event sequences: Strategies for sharpeninganalytic focus.
IEEE Trans. Vis. Comput. Graph. , 23(6):1636–1649, 2017.[12] B. T. Elhassan, E. R. Wagner, and J.-D. Werthel. Outcome of lowertrapezius transfer to reconstruct massive irreparable posterior-superiorrotator cuff tear.
Journal of shoulder and elbow surgery , 25(8):1346–1353,2016.[13] M. Gleicher. Considerations for visualizing comparison.
IEEE transac-tions on visualization and computer graphics , 24(1):413–423, 2018.[14] M. Harrower and C. A. Brewer. Colorbrewer. org: an online tool forselecting colour schemes for maps.
The Cartographic Journal , 40(1):27–37, 2003.[15] P. Heuberer, A. Kranzl, B. Laky, W. Anderl, and C. Wurnig. Elec-tromyographic analysis: shoulder muscle activity revisited.
Archivesof orthopaedic and trauma surgery , 135(4):549–563, 2015.[16] J. Hullman, N. Diakopoulos, and E. Adar. Contextifier: automatic gen-eration of annotated stock visualizations. In
Proceedings of the SIGCHIConference on Human Factors in Computing Systems , pp. 2707–2716.ACM, 2013.[17] S. Jang, N. Elmqvist, and K. Ramani. Gestureanalyzer: visual analyticsfor pattern analysis of mid-air hand gestures. In
Proceedings of the 2ndACM symposium on Spatial user interaction , pp. 30–39. ACM, 2014.[18] S. Jang, N. Elmqvist, and K. Ramani. Motionflow: Visual abstraction andaggregation of sequential patterns in human motion tracking data.
IEEEtransactions on visualization and computer graphics , 22(1):21–30, 2016.[19] W. Javed and N. Elmqvist. Exploring the design space of compositevisualization. In
Visualization Symposium, IEEE Pacific(PACIFICVIS) ,vol. 00, pp. 1–8, 02 2012. doi: [20] W. Javed, B. McDonnel, and N. Elmqvist. Graphical perception of multipletime series.
IEEE transactions on visualization and computer graphics ,16(6):927–934, 2010.[21] J. Kehrer and H. Hauser. Visualization and visual analysis of multifacetedscientific data: A survey.
IEEE transactions on visualization and computergraphics , 19(3):495–513, 2013.[22] J. Kehrer, F. Ladst¨adter, P. Muigg, H. Doleisch, A. Steiner, and H. Hauser.Hypothesis generation in climate research with interactive visual dataexploration.
IEEE Transactions on Visualization and Computer Graphics ,14(6):1579–1586, 2008.[23] R. Kincaid. Signallens: Focus+ context applied to electronic time series.
IEEE Transactions on Visualization and Computer Graphics , 16(6):900–907, 2010.[24] S. Kokoska and D. Zwillinger.
CRC standard probability and statisticstables and formulae . Crc Press, 1999.[25] P. R. Krekel, E. R. Valstar, J. De Groot, F. H. Post, R. G. Nelissen, and C. P. Botha. Visual analysis of multi-joint kinematic data. In
ComputerGraphics Forum , vol. 29, pp. 1123–1132. Wiley Online Library, 2010.[26] H. Lam, M. Tory, and T. Munzner. Bridging from goals to tasks withdesign study analysis reports.
IEEE transactions on visualization andcomputer graphics , 24(1):435–445, 2018.[27] Q. Li, P. Xu, Y. Y. Chan, Y. Wang, Z. Wang, H. Qu, and X. Ma. Avisual analytics approach for understanding reasons behind snowballingand comeback in moba games.
IEEE transactions on visualization andcomputer graphics , 23(1):211–220, 2016.[28] T. Munzner. A nested process model for visualization design and valida-tion.
IEEE Transactions on Visualization & Computer Graphics , (6):921–928, 2009.[29] T. Munzner.
Visualization analysis and design . CRC press, 2014.[30] K. T. Nguyen, H. Gauffin, A. Ynnerman, and T. Ropinski. Quantitativeanalysis of knee movement patterns through comparative visualization. In
Visualization in Medicine and Life Sciences III , pp. 265–284. Springer,2016.[31] J. Poco, A. Dasgupta, Y. Wei, W. Hargrove, C. Schwalm, R. Cook,E. Bertini, and C. Silva. Similarityexplorer: A visual inter-comparisontool for multifaceted climate data. In
Computer Graphics Forum , vol. 33,pp. 341–350. Wiley Online Library, 2014.[32] K. Rong and P. Bailis. Asap: prioritizing attention via time series smooth-ing.
Proceedings of the VLDB Endowment , 10(11):1358–1369, 2017.[33] V. I. Sakellariou, N. K. Badilas, G. A. Mazis, N. A. Stavropoulos, H. K.Kotoulas, S. Kyriakopoulos, I. Tagkalegkas, and I. P. Sofianos. Brachialplexus injuries in adults: evaluation and diagnostic approach.
ISRN ortho-pedics , 2014, 2014.[34] P. Saraiya, C. North, V. Lam, and K. A. Duca. An insight-based longitu-dinal study of visual analytics.
IEEE Transactions on Visualization andComputer Graphics , 12(6):1511–1522, 2006.[35] L. C. Sheffler, L. Lattanza, M. Sison-Williamson, and M. A. James. Bicepsbrachii long head overactivity associated with elbow flexion contracturein brachial plexus birth palsy.
The Journal of Bone and Joint Surgery.American volume. , 94(4):289, 2012.[36] A. Shepherd. Hta as a framework for task analysis.
Ergonomics ,41(11):1537–1552, 1998.[37] B. Shneiderman. The eyes have it: A task by data type taxonomy forinformation visualizations. In
Visual Languages, 1996. Proceedings.,IEEE Symposium on , pp. 336–343. IEEE, 1996.[38] B. Shneiderman and C. Plaisant. Strategies for evaluating informationvisualization tools: multi-dimensional in-depth long-term case studies. In
Proceedings of the 2006 AVI workshop on BEyond time and errors: novelevaluation methods for information visualization , pp. 1–7. ACM, 2006.[39] G. Smith. Spike2 for windows, version 5.
Cambridge Electronic DesignLimited, Cambridge , 2003.[40] E. R. Tufte.
The visual display of quantitative information , vol. 2. Graphicspress Cheshire, CT, 2001.[41] P. Valdivia, F. Dias, F. Petronetto, C. T. Silva, and L. G. Nonato. Wavelet-based visualization of time-varying data on graphs. In
Visual AnalyticsScience and Technology (VAST), 2015 IEEE Conference on , pp. 1–8. IEEE,2015.[42] T. D. Wang, C. Plaisant, B. Shneiderman, N. Spring, D. Roseman, G. Marc-hand, V. Mukherjee, and M. Smith. Temporal summaries: Supportingtemporal categorical searching, aggregation and comparison.
IEEE trans-actions on visualization and computer graphics , 15(6), 2009.[43] M. O. Ward and Z. Guo. Visual exploration of time-series data with shapespace projections. In
Computer Graphics Forum , vol. 30, pp. 701–710.Wiley Online Library, 2011.[44] N. Wilhelm, A. V¨ogele, R. Zsoldos, T. Licka, B. Kr¨uger, and J. Bernard.Furyexplorer: visual-interactive exploration of horse motion capture data.In
Visualization and Data Analysis 2015 , vol. 9397, p. 93970F. Interna-tional Society for Optics and Photonics, 2015.[45] Y. Zhang, K. Chanana, and C. Dunne. IDMVis: Temporal event sequencevisualization for type 1 diabetes treatment decision support.
IEEE trans-actions on visualization and computer graphics , 25(1):512–522, 2018.[46] J. Zhao, F. Chevalier, and R. Balakrishnan. Kronominer: using multi-focinavigation for the visual exploration of time-series data. In
Proceedingsof the SIGCHI Conference on Human Factors in Computing Systems , pp.1737–1746. ACM, 2011.[47] J. Zhao, F. Chevalier, E. Pietriga, and R. Balakrishnan. Exploratory analy-sis of time-series with chronolenses.
IEEE Transactions on Visualizationand Computer Graphics , 17(12):2422–2431, 2011.[48] J. Zhao, S. M. Drucker, D. Fisher, and D. Brinkman. Timeslice: Interactivefaceted browsing of timeline data. In