A Tale of Two Countries: A Longitudinal Cross-Country Study of Mobile Users' Reactions to the COVID-19 Pandemic Through the Lens of App Popularity
AA Tale of Two Countries: A Longitudinal Cross-CountryStudy of Mobile Users’ Reactions to the COVID-19 PandemicThrough the Lens of App Popularity
LIU WANG,
Beijing University of Posts and Telecommunications, China
HAOYU WANG ∗ , Beijing University of Posts and Telecommunications, China
YI WANG,
Beijing University of Posts and Telecommunications, China
GARETH TYSON,
Queen Mary University of London, United KingdomThe ongoing COVID-19 pandemic has profoundly impacted people’s life around the world, including how theyinteract with mobile technologies. In this paper, we seek to develop an understanding of how the dynamictrajectory of a pandemic shapes mobile phone users’ experiences. Through the lens of app popularity, weapproach this goal from a cross-country perspective. We compile a dataset consisting of six-month dailysnapshots of the most popular apps in the iOS App Store in China and the US, where the pandemic hasexhibited distinct trajectories. Using this longitudinal dataset, our analysis provides detailed patterns ofapp ranking during the pandemic at both category and individual app levels. We reveal that app categories’rankings are correlated with the pandemic, contingent upon country-specific development trajectories. Ourwork offers rich insights into how the COVID-19, a typical global public health crisis, has influence people’sday-to-day interaction with the Internet and mobile technologies.
The ongoing COVID-19 pandemic has impacted almost every aspect of human life. The pandemicintroduced changes in people’s perceptions and attitudes, as well as how they work, pursue leisure,interact with others, and so on [21, 26, 33]. Among these changes, interacting with internet andinformation technologies could be very significant. Compared with prior years, technologies haveplayed a critical role in the pandemic, as an essential infrastructure for supporting the responseto the pandemic. For instance, to help curb its spread, governments and public health authoritiesaround the world have launched contact-tracing apps [18, 23]. Given that mobile phones havebeen the most popular and pervasive interfaces for people to interact with the Internet, mosttechnological experience has occurred via mobile apps. We therefore posit that the popularity ofmobile apps, as well as their dynamics, may provide a lens for understanding people’s experiences withtechnologies during the pandemic.
Due to the different strategies employed by countries in fighting COVID-19, as well as thedegree of public compliance, the regional development of the pandemic has exhibited distinctpatterns. For example, China introduced a strict nation-wide lockdown during the initial outbreakand achieved strong public cooperation, making the pandemic largely under control in most ofareas from April [19]. In contrast, the United States, although implementing multiple measures atdifferent levels, failed to contain the pandemic, resulting in chaos spanning the entire year [11].We conjecture that such differences may be reflected in the app popularity in different countries.
Thus, understanding people’s experiences with mobile apps must consider the country-specific patternsof the pandemic.
This Work.
In this paper, we seek to understand, characterize, and compare the dynamics ofapp popularity during the pandemic, under the circumstance of different country-specific pandemic ∗ Corresponding Author: Haoyu Wang ([email protected]).Authors’ addresses: Liu Wang, Beijing University of Posts and Telecommunications, Beijing, China; Haoyu Wang, BeijingUniversity of Posts and Telecommunications, Beijing, China, [email protected]; Yi Wang, Beijing University ofPosts and Telecommunications, Beijing, China; Gareth Tyson, Queen Mary University of London, London, United Kingdom., Vol. 1, No. 1, Article . Publication date: February 2020. a r X i v : . [ c s . S I] F e b Liu Wang, Haoyu Wang, Yi Wang, and Gareth Tyson development patterns. We are particularly keen to explore the relationship between app utilizationand the infection trajectories experienced across countries. We first create the daily snapshots ofthe ranking of popular apps in the iOS App Store in both China and the United States (US), fromthe January 2020 to the end of June 2020 (
See Section 3 ). This longitudinal dataset enables us tostudy app usage behavior evolution across billions of mobile users in both China and US. We nextcharacterize the dynamics of app popularity from both a Macro- (i.e., app category) and Micro- (i.e.,individual app) lens. We investigate which app categories are positively or negatively correlatedwith the coronavirus pandemic and try to formulate general laws in their ranking evolution (
SeeSection 4 ), especially when comparing the differing situations and cultures in China and the US.We further perform fine-grained per-app analysis (
See Section 5 ), to summarize the dynamics ofapp popularity. Finally, we investigate whether the COVID-19 pandemic has introduced side-effectson the app maintenance behaviors (e.g., performance) by analyzing millions of app reviews (
SeeSection 6 ). We highlight a number of key findings: • COVID-19 has played a key role in changing the popularity of some categories of app . During theoutbreak, several app categories experienced significant ups and downs in popularity in bothChina and the US, particularly
Business , Education , Navigation , and
Travel . This changereflects the reactions and efforts people made in response to the outbreak, and highlightsdifferences between China and the US. • The impact of COVID-19 on app rankings are diverse and can vary from country to country .There are many apps in our dataset whose rankings are strongly correlated, either positivelyor negatively, with the pandemic case rates. Their ranking variations can be classified intoseveral groups. Most apps within the same group share some similarities in their adaptationto the pandemic situation. • The rapid popularity of some apps may be accompanied by the side effect of declining ratings,especially for Chinese apps . We explore the reasons for this from the app reviews and founda massive increase in the number of negative reviews compared to the pre-popularity. Wereveal some potential challenges for apps.To the best of our knowledge, this is the first longitudinal study to characterize the behaviordynamics of mobile users through the lens of mobile app popularity. We have released our datasetand the experimental results to the research community at Github:https://app-popularity-covid19.github.io/
We start by describing the development of COVID-19 pandemic and contrasting the responses ofChina and the US.
The first case of COVID-19 was identified in Wuhan, China, in December2019. In the face of a previously unknown virus, China rolled out perhaps the most ambitious,agile, and aggressive disease containment effort in history [5]. The central and local governmentslaunched the national emergency response and various prevention and control measures have beenimplemented rapidly. Investigations began quickly and traced the outbreak to a seafood market,which was immediately closed by Chinese authorities as an initial method to terminate all meattrades.In response to the rapid spread of SARS-Cov-2 within Hubei province, the Chinese governmentexpanded its precautionary measures, announcing a complete lockdown of Wuhan and Hubeiprovince cities by closing airports, railway stations, highway entrances, and suspending all local , Vol. 1, No. 1, Article . Publication date: February 2020.
Tale of Two Countries 3 public transportation to prevent anyone from entering or leaving [67]. The government alsoimplemented aggressive quarantine and social distancing in the entire country including cancellingactivities with large crowds, wearing a mask when going out, postponing the reopening of schools,enabling home office, etc. After all the measures taken and people’s full commitment, a decline inthe number of new cases and deaths was clearly observed by the end of February. Later on, withall the situation improvements happening, China announced the lifting of the embargo and travelrestrictions on Wuhan and restarting of the economy in April [1].
The first case of COVID-19 in the US was reported on January 20, 2020 [30]and the first known American deaths occurred in February [7]. President Donald Trump declaredthe US outbreak a public health emergency on January 31. Subsequently, restrictions were placedon flights arriving from China [12, 14]. However, the initial US response to the outbreak wasotherwise slow in terms of preparing the medical system, halting other travel, and testing [6, 8, 13].Meanwhile, Trump downplayed the threat posed by the virus and claimed the outbreak was undercontrol [10]. In addition, each state that had imposed a stay-at-home order or shelter in place hadbegun lifting the restrictions of businesses and public spaces in May [9]. Despite its considerableadvantages — abundant resources, biomedical infrastructure, and scientific expertise — the US hasbeen severely hit by COVID-19. More than 21 million confirmed cases have been reported by thetime of writing, resulting in more than 368K deaths, which is the most of any country and the 14thhighest on a per capita basis [2, 3].
App ranking refers to the position of an app in a store or marketplace. In general, app rankings canreference not only how an app ranks in a search, but also the overall app rankings from the store.These rankings are country specific, and can be viewed at an overall or category level. The AppleApp Store, one of the most common app marketplaces, provides the real-time app ranking lists bothoverall and by category. In the world of apps, ranking is of utmost importance. A minor change inrank can make a significant difference in traffic and revenue. Apptentive’s recent mobile consumersurvey [15] reveals that nearly half of all mobile app users identified browsing the app store chartsand search results (the placement on either of which depends on rankings) as a preferred methodfor finding new apps. Simply put, higher rankings mean more downloads, and to some degree,more money for app developers.The Apple iOS App Store has a complex and highly guarded algorithm for determining rankingsfor both keyword-based app store searches and composite top charts. Although the exact rankingalgorithms are not publicly available, there are several known factors that influence the rankings,including downloads (how many people have downloaded the app), rating (how many stars peoplegive the app), rating count (how many ratings the app has) and trends (how quickly the app isgrowing), etc [4].
Since it is the combination of several other factors (average rating, number of ratings,installs, etc.), this ranking speaks to the overall awareness, usefulness, and satisfaction of an app.Consequently, app ranking is a particularly useful metric for mobile app analysis, particularly, itspopularity.
Since its outbreak, COVID-19 has attracted much attention from various research communities.A large number of studies are focused on the medical domain. Many medical researchers havemade tremendous contributions to pathology study, epidemiology study, treatment study and soon [20, 42, 47, 70]. Also, a number of computer scientists have adopted computing techniques likemachine learning to help medical practitioners cope with the disease [34]. For example, Zhang et , Vol. 1, No. 1, Article . Publication date: February 2020.
Liu Wang, Haoyu Wang, Yi Wang, and Gareth Tyson al. [66] proposed the confidence-aware anomaly detection (CAAD) model to screen viral pneumoniaon chest X-ray images. Wang et al. [61] proposed COVID-Net, a deep convolutional neural networkdesign tailored for the detection of COVID-19 cases from chest X-ray (CXR) images.There are a growing number of mobile app studies relevant to COVID-19. Ahmed et al. [17]provided the first comprehensive review of contact tracing apps and discussed the concerns usershave reported regarding their usage. Oliver et al. [41] described how mobile phone data can guidegovernment and public health authorities in determining the best course of action to control theCOVID-19 pandemic and in assessing the effectiveness of control measures. He et al. [29] presenteda systematic analysis of coronavirus-themed mobile malware and found these apps aim to stealusers’ private information or to make a profit by phishing and extortion. There is also research intosecurity and privacy issues. For example, Sun et al. [48] proposed an automated assessment tool todetermine security and privacy weaknesses for apps, and undertook a user study to investigateconcerns regarding contact tracing apps. Sharma et al. [46] assessed privacy controls offered inCOVID-19 apps and users’ preferences if they were to adopt a COVID-19 app. Wang et al. [60]reveals various issues related to contact tracing apps from the users’ perspective by analyzing alarge number of user reviews. Although a few previous studies have looked at COVID-19 throughthe lens of mobile apps, to the best of our knowledge, our work is the first comprehensive study toinvestigate mobile users’ reactions to the COVID-19 pandemic through the lens of app ranking.
A large number of research studies have characterized the mobile app ecosystem from differentperspectives. Wang et al. [53] characterized the evolution of Google Play based on 5.3 millionapp records collected from three snapshots of Google Play over three years. They observed thatalthough the overall ecosystem shows promising progress, there still exists a considerable numberof unsolved security issues. Besides, some studies were focused on app developers [56, 59, 62], appmarket maintenance behaviors [54, 57, 58], app clone detection [35, 36, 52], third-party libraries [25,40, 50, 64, 65], security [24, 31, 32, 38, 39, 49, 55, 68, 69] and privacy issues [37, 44, 45, 50, 51, 63] ofthe ecosystem, based on app binary analysis, UI analysis, app traffic analysis, and app metadataanalysis, etc. This paper, investigates the mobile app ecosystem from a novel perspective, tounderstanding how the dynamic trajectory of the COVID-19 pandemic shapes mobile smartphoneusers’ experiences.
Our study is driven by the following research questions:RQ1
General Trends.
Across different regions, what are the characteristics of app popularity duringCOVID-19, and how are they impacted by regional measures?
RQ2
Behavior Patterns.
What are the different patterns in the change of app popularity overthe course of the pandemic?
We attempt to classify the app ranking variations according torelevance to the pandemic, then examine the distribution of the number of apps in each classand explore their characteristics.RQ3
Side-effect.
Considering that app popularity may change greatly during the COVID-19 pan-demic, does it introduce any side-effect on app maintenance behaviors?
We first harvest a dataset of app popularity. Given that the most straightforward impact of COVID-19 on the mobile app ecosystem will be mirrored in a number of popular apps, we leverage the , Vol. 1, No. 1, Article . Publication date: February 2020.
Tale of Two Countries 5
App Store’s ranking list and focus on the top-ranked apps. However, it is non-trivial to obtainsuch a dataset as it requires effort to continuously monitor apps in the app market to check theirrankings. To this end, we implement a crawler to retrieve the daily app ranking list from iOS appstore. Considering that the pandemic in different countries can evolve in different ways, we taketwo representative countries (China and the US), as case studies for comparison.To be specific, we crawl the ranking list with top 1500 apps each day from January 1 to June 30,2020, to keep track of the apps and their daily rankings, containing over 543K records in total forthe two countries. We then take the top 100 apps per day as the target of this study. Overall, thereare 586 unique apps in China and 590 apps in the US covering 22 categories, each of which rankedin the top 100 for at least one day during a given half year. Besides, we collect ratings for each appon a daily basis, as well as the reviews for some of these apps.Table 1 summarizes our dataset.
Table 1. Overview of our dataset (from Jan 1 to June 30, 2020).
Country
We briefly present the methodologies used in the subsequent sections.
To understand which app categories are more competitiveand explore general laws in ranking evolution, we begin our analysis by investigating the mostintuitive metric of app-category popularity, i.e., the number of apps ranked in the top 100 for eachcategory. Specifically, we measure the popularity of category by app volume, i.e., the number ofapps belonging to that app category in the top-100. The larger the number, the more popular thecategory is. We examine the number of apps in each category on a daily basis to gain insight intohow their popularity trends are changing. In more detail, we also zoom in on each category tocheck the ranking variations of all apps in that category.
To understand if any apps have been significantly affected by thepandemic, and whether the impact has been positive or negative, we also study the correlationbetween the pandemic evolution and the app ranking changes, which could reflect the popularitychanges of apps, for each app. In our case, we measure the development of the pandemic situationover the half year through two metrics, i.e., the number of daily new confirmed cases and dailyactive cases, respectively. Moreover, we present a correlation comparison strategy based on PearsonCorrelation Coefficient [16]. The Pearson Correlation Coefficient is a number that summarizes thedirection and closeness of linear relations between two variables, taking a value between -1 and 1.The closer the correlation coefficient is to 1 (-1), the stronger the positive (negative) correlation is.For each app, the daily ranking is viewed as a vector in Pearson Correlation Coefficient formula,then the correlation degree is obtained separately from the daily number of newly diagnosedcases and active cases in the corresponding countries. Note that we also take P-value into accountbecause it tells us whether the result of an experiment is statistically significant. Typically if theP-value is lower than the conventional 5% (P<0.05), the correlation coefficient is called statisticallysignificant. Thus, results with P-value less than 0.05 are of concern to us, otherwise are ignored asinsignificantly correlated. , Vol. 1, No. 1, Article . Publication date: February 2020.
Liu Wang, Haoyu Wang, Yi Wang, and Gareth Tyson
Finally, we calculate the correlation coefficient for each app and obtained 358(61.1%) apps inChina and 497(84.2%) apps in the US with statistically significant correlation with the number ofdaily new cases in respective countries, as well as 425(72.5%) apps in China and 502(85.1%) apps inthe US with statistically significant correlation with daily active cases. The results illustrate thatthe majority of apps have a statistically significant correlation between their rankings and theCOVID-19 case rates. This encourages and motivates the following analysis, i.e., the distributionsof these correlation coefficients at the overall and category level. Due to the greater number ofapps with statistically significant correlation between rankings and active cases compared to newlyconfirmed cases, we take the active cases as the pandemic marker for the subsequent study. Andnote that all the correlation coefficients provided below are statistically significant. A pp s (a) China A pp s (b) US Fig. 1. The distribution of number of apps in each category among top-100 apps per day
Figure 1 visualizes the dynamics of the popularity of each appcategory. We use stacked bar charts to show the number of apps in each category among eachday’s top 100 apps in the app stores of China and the Unite States, respectively. As aforementioned,there are 22 categories (see the legend in Figure 1). In a specific day, the change in the numbers ofapps in each category over time is represented by the fluctuations of the vertical lengths of thecorresponding color. , Vol. 1, No. 1, Article . Publication date: February 2020.
Tale of Two Countries 7
In general, most category’s popularity does not exhibit significant fluctuations in either Chinaor the US. Some categories has been constantly popular over the long run, such as
Games , SocialNetworking , Photo & Video and
Entertainment , as well as some being consistently less popular,such as
Reference , Medical and
Weather . However, there are four categories that are worth notingin both countries, i.e.,
Business , Education , Travel , and
Navigation , whose popularity exhibitsremarkable dynamics along with the outbreak in the respective countries. We further examine thedetails of the dynamics of their popularity.
For each of the four categories, we plot the overlay of scatter andline charts for both countries in Figure 2. The scatter represents the ranking of each app in thatcategory and the lines show the number of newly confirmed and active COVID-19 cases per day.First, we observe a number of
Business apps (e.g., Tencent Meetings and Zoom Cloud Meetings)and
Education apps (e.g., Tencent Classroom and Google Classroom) jumped to the top of the list.This started in early February in China and mid-March in the US, just as the COVID-19 began tospread widely in the corresponding countries. This observation is quite straightforward since theoutbreak had quickly led to a large-scale shift to remote work for employees and online learning forstudents. Besides, the
Travel apps (e.g.,Ctrip and Airbnb) and
Navigation apps (e.g., Baidu Mapsand Google Maps) show the opposite trend during the same time, with almost all of them fallingout of the top 100. This is also reasonable because the mandatory social distancing and quarantinesare required for containing virus transmission, which left such apps less desired. It is importantto note that the ranking changes in these four categories starts almost simultaneously with theoutbreak, which demonstrates the pandemic’s immediate impact. In sum, the findings indicate thatmobile app ecosystems have had immediate reactions to the COVID-19 outbreak. Regarding appcategories, COVID-19 has an effect of enhancing or decreasing the popularity of certain categoriesof apps.Comparing the two countries, it is also interesting to note that
Navigation and
Travel apps inthe US rebounded in mid-April and mid-May, respectively, while the US was still in the midst of anincreasingly severe pandemic situation. In contrast, the rankings for these two categories in Chinaimproved only after the conditions began to improve. To some extent, this may reflect the differentattitudes of the two countries in dealing with COVID-19.
As mentioned in the above, we employ Pearson Correlation Coefficient to explore the correlationsbetween app popularity and the pandemic situation. In this section, we provide correlation analysisresults at both the overall and category levels, respectively.
Figure 3 presents the Cumulative Distribution Function (CDF) of thecorrelation coefficients between app ranking and the number of active confirmed cases in thetwo countries. The trends are very similar in the US and China, with a roughly 50:50 number ofpositively and negatively correlated apps. Besides, there are indeed a number of apps whose rankingchanges have a clear correlation with the pandemic evolution in both countries. Specifically, 27% ofthe apps in China have a correlation coefficient less than -0.4 and nearly 40% in the US. In addition,over 20% of the apps show a correlation coefficient greater than 0.4 both in China and the US,which manifests many positive correlations. More notably, 23% of the apps in the US and 10% inChina have correlation coefficients less than -0.6, and nearly 10% in both countries are greater than0.6, which can be considered as strong or very strong correlations (see Table 2). This observationsuggests that many apps’ popularity does haves significant correlation with the development ofpandemic, positively or negatively. , Vol. 1, No. 1, Article . Publication date: February 2020.
Liu Wang, Haoyu Wang, Yi Wang, and Gareth Tyson -
01 02 -
01 03 -
01 04 -
01 05 -
01 06 -
01 07 - China R a n k i n g -
01 02 -
01 03 -
01 04 -
01 05 -
01 06 - US C a s e s App New cases Active cases (a) Business -
01 02 -
01 03 -
01 04 -
01 05 -
01 06 -
01 07 - China R a n k i n g -
01 02 -
01 03 -
01 04 -
01 05 -
01 06 - US C a s e s App New cases Active cases (b) Education -
01 02 -
01 03 -
01 04 -
01 05 -
01 06 -
01 07 - China R a n k i n g -
01 02 -
01 03 -
01 04 -
01 05 -
01 06 - US C a s e s App New cases Active cases (c) Travel -
01 02 -
01 03 -
01 04 -
01 05 -
01 06 -
01 07 - China R a n k i n g -
01 02 -
01 03 -
01 04 -
01 05 -
01 06 - US C a s e s App New cases Active cases (d) Navigation
Fig. 2. Evolution of the daily ranking of apps and the number of diagnoses in four categories (Business,Education, Travel and Navigation) in two countries. -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1
Correlation coefficient C D F ChinaUS
Fig. 3. The overall distribution of correlation coefficients
We further study the distribution of correlations at the categorylevel. Figure 4 shows the boxplot of correlation coefficients for each category in
China . The red linein the box indicates the median, while the blue line represents the mean. First, we observe thatthe two most prominent categories are
Travel and
Navigation , with all correlation coefficientsbeing positive and a median greater than 0.75. This indicates that there is a significant positivecorrelation between pandemic situation and the ranking of apps in these two categories, i.e., theranking decreases as the pandemic situation worsens and increases when the situation improves(as measured by case rate).Moreover, most apps in the
Education category have negative correlation coefficients with amedian of about -0.6, suggesting that
Education apps are in general negatively correlated withthe pandemic situation. The findings are consistent with the evolution of the popularity of thesethree categories, as previously discussed. However, for the
Business category, there seems to beno significant negative correlation discernible. A possible explanation is the long term popularityfor most
Business apps after the outbreak, as we can observe in Figure 2(a). There are a number , Vol. 1, No. 1, Article . Publication date: February 2020.
Tale of Two Countries 9 B u s i n e ss E d u c a t i o n T r a v e l N a v i g a t i o n G a m e s U t ili t i e s E n t e r t a i n m e n t L i f e s t y l e P h o t o & V i d e o F i n a n c e H e a l t h & F i t n e ss P r o d u c t i v i t y M u s i c S h o pp i n gS o c i a l Sp o r t s F oo d & D r i n k R e f e r e n c e B oo k s N e w s M e d i c a l W e a t h e r C o rr e l a t i o n c o e ff i c i e n t Fig. 4. The distribution of correlation coefficients in each category in China of Business apps that jumped up the rankings during the outbreak and have been leading eversince, even though the number of active cases has dwindled to almost nothing. As a result, thereis no significant positive or negative correlation in the later phases, thereby undermining theoverall correlation coefficient. We posit that we can examine it in more detail by splitting it intotwo stages, which we will discuss later. In general, although most of the categories do not show asignificant positive or negative correlation, there are certain categories that do, such as
Travel and
Navigation showing a strong positive correlation and
Education presenting a strong negativecorrelation.The situation in the US is somewhat different. Figure 5 presents the boxplot of correlationcoefficients for each category in the US. Surprisingly, there seem to be no conspicuous categoriesshowing significant positive or negative correlations with the pandemic. Again focusing on theTravel category, we can see the correlation coefficients range between -0.5 and 0.5, with a meanclose to 0, failing to indicate a strong correlation.It is not hard to understand when we review Figure 2(c). Although the Travel apps have asignificant drop in ranking at the beginning, it only lasted for about two months and then graduallyrebounded back, during which time the number of diagnoses in general kept increasing, thus thereis no noticeable correlation between ranking and pandemic during the entire half year. In thesame vein, there is hardly any category in which app rankings are generally growing or decliningfor a extended period, which weakens the correlations. Therefore, in terms of category level, thecorrelation between app popularity and the pandemic seems to be less significant.
From the overall-level analysis, we can conclude that there are indeed a number ofapps that are strongly correlated with the outbreak situation over the six months of the pandemic,positively or negatively, in both countries. In terms of the category level, the vast majority ofcategories fail to demonstrate strong correlations. Besides, we can discover some distinctions inthe correlations between app rankings and pandemic conditions in China and the US. In China,there are certain categories (e.g., Travel and Navigation) showing significant correlations, whilein the US, the correlations are not very strong in any category. This confirms that the impact ofCOVID-19 on the mobile app ecosystem may differ from country to country. , Vol. 1, No. 1, Article . Publication date: February 2020. B u s i n e ss E d u c a t i o n T r a v e l N a v i g a t i o n G a m e s U t ili t i e s E n t e r t a i n m e n t L i f e s t y l e P h o t o & V i d e o F i n a n c e H e a l t h & F i t n e ss P r o d u c t i v i t y M u s i c S h o pp i n gS o c i a l Sp o r t s F oo d & D r i n k R e f e r e n c e B oo k s N e w s M e d i c a l W e a t h e r C o rr e l a t i o n c o e ff i c i e n t Fig. 5. The distribution of correlation coefficients in each category in US
We next introduce the findings related to the RQ . Up to now, we have identified that the case ratesof the outbreak have an immediate impact on app rankings, with a number of apps significantlycorrelated with the outbreak. We then delve into the different patterns that each app rankingconforms to, i.e., the different correlations with the pandemic trends. We make efforts to createa taxonomy and classify the apps based on their calculated correlation coefficients, where eachcategory corresponds to a pattern of change in rankings. Considering the different trends in theevolution of the pandemic situation in China and the United States over the first six months, i.e.,China has experienced a progression from deterioration to improvement, while the US has tended tobe more severe. We therefore examine the situation in China in two stages for a more fine-grainedcomparison.The first stage was when the pandemic was worsening, and the second stage was when it wasimproving. The cut-off date is February 17, when the number of new confirmed cases reached itshighest. Besides, since a larger correlation coefficient shows a more significant correlation, weconsider introducing a threshold n , and regard an app as strongly positively or negatively correlatedwith the outbreak (overall or in a stage) whose absolute correlation coefficient is greater than n .As such, the key challenge is to select an appropriate n . Nevertheless, the correlation coefficientrepresents an effect size and so we can verbally describe the strength of the correlation using theguide that Evans [27] suggests for the absolute value of r in Table 2. Thus, we set the threshold to 0 . 𝑟 > . 𝑟 < − . In China, we consider the situation from both an overall and a phased perspective.(1) Overall-level. We start with checking the correlation coefficient r between the app rankingand the overall period of the pandemic. If r >0.6 we classify the app as “SPC with the overall.”Similarly, we categorize the app as “SNC with the overall” if its r <-0.6.(2) Stage-level. For the apps without a strong correlation between its ranking and the overallpandemic period, we then examine their correlations with the two stages respectively, i.e., the , Vol. 1, No. 1, Article . Publication date: February 2020. Tale of Two Countries 11
Table 2. A commonly used interpretations of the r values Magnitude of Correlation Description of Strength0.01 - 0.19 Very Weak0.20 - 0.39 Weak0.40 - 0.59 Moderate0.60 - 0.79 Strong0.80 - 1.00 Very Stronggrowth stage of the pandemic and the decline stage of the pandemic. If an app’s rank is stronglycorrelated to only one of the stages, we label it as a one-stage correlation. Concretely, there arefour categories including “SPC with growth stage”, “SNC with growth stage”, “SPC with declinestage” and “SNC with decline stage.”If an app’s ranking is strongly correlated with both stages but in opposite directions, we deem itas two-stages correlation. Specifically, there are two categories including “SPC with growth stage,SNC with decline stage” and “SNC with growth stage, SPC with decline stage.”
In the US, the situation is relatively straightforward due to the single upward trend ofthe pandemic, and we can simply focus on two categories, i.e., “SPC with the overall” and “SNCwith the overall.”
Based on the above, we classify the apps in the two countries separately and theresults are shown in Table 3. For China, we have eight categories due to the shifts in the pandemictrend, with the number of apps ranging from 2 to 44. In the US, there are two categories with appquantities of 72 and 116, respectively.
Table 3. Eight categories in China and two categories in the US
Category Ranking/Popularity Change
Overall SPC with the overall The app ranking/popularity decreases significantly during the growth stage 41of the pandemic and increases significantly during the decline stage.SNC with the overall The app ranking/popularity increases significantly during the growth stage 44of the pandemic and decreases significantly during the decline stage.One-stage SPC with the growth stage The app ranking/popularity decreases significantly during the growth stage 20of the pandemic but no significant increase or decrease in the decline stage.SNC with the growth stage The app ranking/popularity increases significantly during the growth stage 50of the pandemic but no significant increase or decrease in the decline stage.SPC with the decline stage The app ranking/popularity increases significantly during the decline stage 22of the pandemic but no significant increase or decrease in the growth stage.SNC with the decline stage The app ranking/popularity decreases significantly during the decline stage 19of the pandemic but no significant increase or decrease in the growth stage.two-stages SPC with growth stage, The app ranking/popularity decreases significantly during the growth stage 7and SNC with decline stage of the pandemic and decreases significantly during the decline stage.SNC with growth stage, The app ranking/popularity increases significantly during the growth stage 2and SPC with decline stage of the pandemic and increases significantly during the decline stage.US SPC with the overall The app ranking/popularity decreases significantly during the period. 72SNC with the overall The app ranking/popularity increases significantly during the period. 116
As aforementioned, each of the four groups symbolize a pattern of how an app’s ranking changesin response to the outbreak. We next deep dive into each group.
There are 41 apps in China whose app rankings are stronglypositively related to the pandemic situation, i.e., they suffered a regression in ranking when the , Vol. 1, No. 1, Article . Publication date: February 2020. pandemic got worse (measured by case rate). Subsequently, their rankings rebounded as thesituation recovered. The app with the strongest correlation (0.911) is
Dianping , a leading locallifestyle information and trading platform in China, providing users with information services suchas merchant information, consumer reviews and offers, as well as transaction services includinggroup buying, restaurant reservations, takeaway and e-membership card. It is followed by
TikTakTravel (0.885),
Ctrip Travel (0.882) and
Tencent Maps (0.876), all of which cater to services liketravel and navigation. Unsurprisingly, most of the apps in this category are closely tied to outdooractivities. Given the strict quarantine policy during the outbreak in China, it is no wonder that therankings of this kind of apps undergo such a shift. The trend comparisons for two representativeapps are visualized in Figure 6. The red line shows the ranking of the app each day and the greenline presents the number of active confirmed cases per day. We can observe that their trends arealmost identical. - - - - - - - R a n k i n g C a s e s App rankingActive cases (a) Dianping - - - - - - - R a n k i n g C a s e s App rankingActive cases (b) TikTak Travel
Fig. 6. Two examples for SPC with the overall in China.
The situation in this category is opposite to the previousone. There is a strong negative correlation between the app ranking and the pandemic case ratetrends for the 44 apps in this category. This means that the outbreak has contributed to the growthof their rankings. The subsequent recovery has therefore resulted in a drop in the rankings. Forcontext, Figure 7 presents two example apps with these properties. We see opposite trends in theapp ranking vs. case rate. Note that since the Apple Store’s app ranking list offers up to 1,500 slots,for rankings that fall out of the top 1,500, we replace them with 1,500 in the chart.We highly four prominent examples that exhibit these behaviors: 1)
Happy Disappear , a triple-elimination casual game with the strongest Pearson correlation coefficient of -0.916, which may bea good choice for people to pass time during the lockdown. 2)
Diandu (-0.88), an educational appfor primary and secondary school language, mathematics, and English learning, which is helpfulfor tutoring children during the extended period of home study. 3)
Yong Hui Buy Food (-0.818),an online service platform that provides consumers with fresh food and other grocery productsthrough home delivery services, which alleviates the inconvenience of going out to buy food duringquarantine. 4)
Tinker Medicine (-0.776), a pharmaceutical and health product that helps pharmaciesprovide convenient services to the public and provides free delivery services for users who placeorders through the app. Overall, we observe that most apps in this category have a positive effecton people’s lives in some way during the pandemic.
In this category, the app ranking is strongly positivelycorrelated only with the growth stage of the pandemic. In other words, as the situation gets worse,the app ranking drops; yet it does not show a definitive upward or downward trend as the situationrecovers. For example, there are two typical apps shown in Figure 8. One is called
Eastern Airlines , Vol. 1, No. 1, Article . Publication date: February 2020.
Tale of Two Countries 13 - - - - - - - R a n k i n g C a s e s App rankingActive cases (a) Diandu - - - - - - - R a n k i n g C a s e s App rankingActive cases (b) Yong Hui Buy Food
Fig. 7. Two examples for SNC with the overall in China. that aims to provide safe and convenient ticketing and travel experiences. The other is
Damek , acomprehensive live entertainment ticket marketing platform in China, covering concerts, dramas,musicals, sporting events, etc. Both of them have a high correlation coefficient between app rankingand active cases in the pandemic growth stage (0.933 and 0.772, respectively). However, suchcorrelations are not that strong during the decline stage. This meas there are some apps that areadversely affected by the pandemic but do not recover in tandem with the pandemic. - - - - - - - R a n k i n g C a s e s App rankingActive cases (a) Eastern Airlines - - - - - - - R a n k i n g C a s e s App rankingActive cases (b) Damek
Fig. 8. Two examples for SPC with the growth stage in China.
Similarly, we investigate the situation where apprankings are only strongly negatively correlated with the growth stage of the pandemic, i.e., asthe pandemic worsened, the app ranking improved, but there was no clear trend of a growing ordeclining thereafter. We take a look at the following examples that have high correlation coefficients:1)
Tencent Classroom (-0.943) is a professional online education platform that provides teachers withonline teaching and students with interactive learning, which has been a great support for digitallearning due to the school closures. 2)
Anhui Things (-0.93) is a mobile app for Anhui ProvinceGovernment Services Network designed to provide government services, high-frequency publicservices and citizen-friendly service matters, which allows citizens to handle some business withoutleaving home. 3)
Tencent Meetings (-0.801) is a video conferencing product that features onlinemeetings, useful for the needs of users working from home. This indicates that there are a numberof apps that played an important role in the war against the pandemic, and continue to be used.This may indicate longer-term implications in how people live and work after the pandemic.
In this group, the correlation between app ranking and thepandemic case rate is not significant in the first stage of the pandemic, but as the situation turnsbetter, the app ranking rises. For example, the app with the highest correlation coefficient (0.924) , Vol. 1, No. 1, Article . Publication date: February 2020. - - - - - - - R a n k i n g C a s e s App rankingActive cases (a) Tencent Classroom - - - - - - - R a n k i n g C a s e s App rankingActive cases (b) Tencent Meetings
Fig. 9. Two examples for SNC with the growth stage in China. in the decline stage is
Shell Finder , a home searching platform featuring comprehensive and realproperty information as well as industry innovations such as VR viewings, home valuation, andintelligent recommendations. In addition,
Traffic Management 12123 (0.881) is the official clientof the Internet traffic safety management platform, providing a full range of traffic services andreservation of motor vehicle, driver’s license and illegal processing business, etc. As shown inFigure 10, we can observe that the rankings of these apps were only slightly affected in the early daysof the pandemic. This reflects the fact that as the situation improves and people’s lives graduallyreturn to normal, some apps are coming back to the forefront as soon as possible. - - - - - - - R a n k i n g C a s e s App rankingActive cases (a) Shell Finder - - - - - - - R a n k i n g C a s e s App rankingActive cases (b) Traffic Management 12123
Fig. 10. Two examples for SPC with the decline stage in China.
As expected, this category contains apps whose rankingshave a strong negative correlation with the active cases only in the decline stage, i.e., as thepandemic improves, the ranking declines. For this case, some apps may seem less important aspeople gradually return to their normal lives. For example, the app with the highest correlationcoefficient (-0.852) is
Fast View , an informational app that aggregates and provides users differentforms of content including graphics, videos, etc., as well as current affairs content and the latestreal-time information. Furthermore, there is an online gaming app called
Doudizhu (-0.762), whichis a very popular card game in China. This rose to the top of the rankings at the beginning of theoutbreak, but quickly declined for a long time as shown in Figure 11(b). These apps seem to becomepopular for a short period of the outbreak as a means for people to keep abreast of events or topass the time. However, they cannot last very long, especially when people return to their normallife routines.
In addition to the above apps, there are also a few apps that have strong correlations with , Vol. 1, No. 1, Article . Publication date: February 2020.
Tale of Two Countries 15 - - - - - - - R a n k i n g C a s e s App rankingActive cases (a) Fast View - - - - - - - R a n k i n g C a s e s App rankingActive cases (b) Doudizhu
Fig. 11. Two examples for SNC with the decline stage in China. both stages but in different directions. There are 7 apps that show strongly positive correlationsduring the growth stage of the pandemic while exhibiting negative correlations with the declinestage. Their rankings can be roughly treated as constantly declining over the outbreak. For example,there is a puzzle elimination game called
Cube Battle that has fallen steadily out of the top 1,500since the outbreak began, as shown in Figure 12(a). It has a high correlation coefficient of 0.803within the growth stage and -0.878 within the decline stage. In contrast, the 2 apps that are stronglynegatively correlated with the growth stage and positively related to the decline stage exhibit theopposite trend, i.e., the ranking is almost continuously rising for most of the time. For example,there is a beauty and cosmetic app called
SoYong that has a generally fluctuating upward trend inthe ranking from February to May, as displayed in Figure 12(b). It has the correlation coefficient of-0.86 within the growth stage and 0.717 within the decline stage. Note that both types are few innumber, indicating that the two evolving patterns may be very uncommon. - - - - - - - R a n k i n g C a s e s App rankingActive cases (a) Cube Battle - - - - - - - R a n k i n g C a s e s App rankingActive cases (b) SoYong
Fig. 12. Two examples for the two-stages correlations respectively.
Due to the single trend in the number of the active confirmed casesin the US, i.e., consistently increasing since mid-March, we construct two categories: SPC and SNCwith the overall, with 72 and 116 apps, respectively. Since the pandemic situation in the US hasbeen increasingly serious, the apps with an SPC to the pandemic case rate should show a largelydeclining ranking. For example, the app with the highest correlation coefficient (0.965) is a gamingapp called
Trivia Crack , which has its ranking change in parallel with the number of active cases,as shown in Figure 13(a). In contrast, the apps with an SNC with the pandemic display a generallyrising trend. An example is displayed in Figure 13(b), which is an app called
Videoleap Video Editor& Maker (-0.88), which is an app for producing videos. Its ranking has kept advancing during thesix months. , Vol. 1, No. 1, Article . Publication date: February 2020. - - - - - - - R a n k i n g C a s e s App rankingActive cases (a) Trivia Crack - - - - - - - R a n k i n g C a s e s App rankingActive cases (b) Videoleap Video Editor & Maker
Fig. 13. Two examples for the US.
In summary, the pandemic has diverse patterns in terms of its impact on app ranking. In China, 85out of 586 apps are strongly correlated with the half-year pandemic, either positively or negatively.Moreover, 111 apps have strong correlations (positive or negative) with only one stage of thepandemic (growth or decline). 9 apps even show strong correlations with the two stages in differentdirections. These correlation categories reflect different characteristics of the COVID-19 effecton app rankings. Most of the apps in the same category share some similarities in terms of theiradaptability to the pandemic situation. While for the US, since the vast majority of apps aregame apps, this somewhat limits our study of the adaptation of app functions and features to thepandemic.
We conclude by briefly inspecting side effects that may be caused by these rapid fluctuations in apppopularity. Particularly, an interesting phenomenon we observe is that the ratings of some appsare likely to decrease in varying degrees as their rankings go up. For example, Figure 14 showsthe trends in ranking and rating of the two app examples over this period. While the rankingand popularity of the app is rising rapidly, its rating is experiencing a decline. This prompts usto investigate whether the rise in app ranking and popularity may have some side effects on appmaintenance behavior. - - - - - - - R a n k i n g R a t i n g RankingRating (a) ViLin - - - - - - - R a n k i n g R a t i n g RankingRating (b) Tencent Meeting
Fig. 14. Two app examples where the rating goes down as the ranking goes up.
We briefly focus on those apps that gained popularity for a short or long time as COVID-19 hit,exploring the relationship between their ratings and rankings. Considering that there is a wide , Vol. 1, No. 1, Article . Publication date: February 2020.
Tale of Two Countries 17 diversity in the evolution of many app rankings throughout the time span, we shorten the timeframe to the period when COVID-19 was first spreading, and target only those apps whose rankingsrose rapidly during this period.Specifically, for
China , we choose the period when the ranking exhibits strong negative correlationwith the number of active confirmed cases during the worsening of the COVID-19, i.e., from Januaryto mid-February. For the US , we pay attention to the two months just after the outbreak of COVID-19,i.e., from March to May.We then calculate the correlation coefficients between the ratings and rankings of the focusedapps. The results are presented in Table 4. We see that there were 95 apps in China and 88 apps inthe US that improved their rankings significantly at the beginning of the COVID-19 outbreak. Ofthese, 63 apps in China and 24 apps in the US have a correlation coefficient r >0 between rating andranking. This means that their ratings decreased as their ranking rises, i.e., introducing side effects.Curiously, the results show that this particle side effect is more prevalent in China than the US. Table 4. Distribution of correlation coefficient between rankings and ratings for focused apps.
Country
We have shown that some apps that gains popularity while experiencing a decline in their ratings.We next dive into the reasons behind this phenomenon and investigate the challenges for appdevelopers. Previous studies [22, 43] showed that reviews represent a rich source of informationfor app developers, such as user requirements, bug reports, feature requests, and documentationof user experiences. Thus, we look at user reviews to analyze the possible reasons for the dropin app ratings. It is important to note that while the ratings of all of these apps have dropped,the magnitude of the drop varies greatly, with many showing only slight drops. For the sake ofrepresentativeness, we target apps where the correlation coefficient between app rankings andratings is greater than 0.6 and the absolute value of the range of rating decline is greater than 0.1.This leaves 37 apps for review analysis.We use AppBot to extract the app reviews. This tool provides a large number of data-mining andsentiment analysis features, and is used in many studies, e.g., [28]. For each app, we use AppBot forautomated text mining and sentiment analysis of reviews. To uncover new traits, we use reviewsfrom the second half of 2019 as the benchmark for comparision.Overall, we find a significant increase in the number of reviews for these apps in the first half of2020 compared to the second half of 2019. This is intuitive considering their increased demand. Atotal of 1,355,460 reviews occur in the first half of 2020, which is 4.3 times more than the 313,209reviews in the previous six months. More importantly, sentiment analysis reveals that the numberand percentage of negative reviews among them has also increased considerably, with a total of716,129 negative reviews in the first half of 2020 (53%) compared to 72,345 in the second half of2019 (23%), a visible number about 10 times higher. This is also the case for individual apps. Forexample,
Tencent Meeting had only 33 user reviews during the second half of 2019, of which only 5(15%) were negative, while the number of reviews during the first half of 2020 was astonishinglyhigh at 10,507, of which 4,583 (44%) were negative. This may mean that such apps have struggledto fulfil user expectations as their user base has grown. , Vol. 1, No. 1, Article . Publication date: February 2020.
Further, we focus on the negative reviews and seek to get a sense of the topics of user complaints.We thereofre rely on AppBot’s topic grouping feature and identify six core topics raised withinthese reviews:(1)
Bugs.
There are lots of negative reviews describing problems with the app which should becorrected, such as a crash, an erroneous behavior, or a performance issue. This is likely a productof more users experimenting with the features of the apps.(2)
Device.
There are many negative comments regarding the device, including device incom-patibility, interface mismatch, lack of matching function and device features, unfriendly to ipaddevices, triggering device heat and lag, etc. Again, such issues become more prevalent as your userbase diversifies.(3)
Connectivity.
This mainly refers to network connection problems, such as difficult WiFiconnections, slow loading, long waiting times, etc.(4)
Design & UX.
There are some negative reviews about the user interface, specifically that theuser interface is not attractive, and the UI design and interaction are not friendly enough, makingthe user experience poor.(5)
Video/Audio.
For some apps, there are some problems with opening video or voice, suchas delay, not smooth, etc. Usually such complaints arise in apps featuring multi-person onlinereal-time sessions, such as online meetings and online lecture apps.(6)
Privacy.
There are also some concerns about privacy and security of some apps, such asinvasion of privacy, sharing information with third-parties, stealing edits and tracking location, etc.We further measure the distribution of different kinds of topics for each app, as shown in Figure 15.As can be seen, the distribution of user concerns varies greatly across apps, where bugs and devicesissues are prevalent. We also notice that some of the issues are related to the increase in the numberof users. For example,
Play It , a multiplayer interactive trivia game app in China, suffered severalcrashes due to a surge in the number of users during the initial outbreak of COVID-19 in China aspeople were quarantined at home. Generally speaking, as more people use the app, the numberof complaints grows as well. Some issues that used to be obscure seem to be magnified, such asthe aesthetics of the user interface. This sudden popularity presents a great challenge to the appin many aspects including functionality, compatibility, stability, fault tolerance and UI design, etc.In a way, it serves as a wake-up call for developers to learn from the experience in dealing withunexpected situations in the future. Meanwhile, it offers unprecedented opportunities for appdevelopers to resolve the hidden problems.
The most significant result in our paper is that COVID-19 has had a significant impact on the apprankings in app store. This impact is reflected in both the popularity of several categories and therankings of individual apps. Moreover, the evolution of app rankings are diverse and have a lotto do with whether the features and functionalities of the apps are compatible with the currentenvironment. Besides, we notice that some apps that rapidly rise in popularity may also experiencethe side effect of rating drops. This reflects the challenges that rapid changes in popularity cancause app developers. We also note that findings are different between China and the United States.The relevant stakeholders should take note of these findings, as they play different roles atdifferent levels of the app market. Our findings have further implications for understanding thebehavior of the mobile app ecosystem during public health crises, and helps developers to makebetter decisions on app developmennt and management. As our first finding indicates, the popularityof apps may experience disturbances resulting from public events. Such disturbances demonstrate , Vol. 1, No. 1, Article . Publication date: February 2020.
Tale of Two Countries 19
Fig. 15. Distribution of the six negative review topics in each app. the importance of having a contingency plan. When some sudden public events such as the COVID-19 pandemic happens, a contingency plan could help app developers, operators, as well as managersbetter cope with them. For example, one may develop a quick server deployment plan whenexpecting high demand. App developers may diversify their portfolios to avoid potential losscaused by popularity decreases. The second finding suggests that the impact may be complicated,depending on how the event is going on. Thus, app developers may need to develop the abilityto develop the short-term and long-term insights, and plan accordingly. For example, for an appthat gains popularity at the early phase of the pandemic but gradually return to normal later,app developers need to decide if it is necessary to invest resources for the short-term spikes. Wealso notice that increases of app popularity often coincide with decreases in ratings, and growingnumber of reviews containing users’ insights. App developers thus need to be vigilant to users’feedback, and take this opportunities to improve the app quality. Moreover, our results and findingsare not restrict to a specific event; it could be helpful in other public emergencies at the global scale.. , Vol. 1, No. 1, Article . Publication date: February 2020.
We recognize that our study carries several limitations and threats to validity. First, since the mainimpact of the pandemic was reflected on only a small number of apps, we selected the top 100 appseach day, yet there was no accepted standard for selecting this number. Second, in determining thestrength of the correlation, we set a threshold value of 0.6. However, there is no standard definitionfor the selection of this threshold. Third, only a small number of apps have been used for ourreview analysis. The major reason is that many apps have insignificant side effects. Hence, we areconcerned that reviews of such apps are not representative and reliable enough.
This paper has presented the first longitudinal empirical study of the evolution of app rankingsduring COVID-19. Our analysis covers 586 apps in China and 590 apps in the US with records ofbeing ranked in the top 100 from January 1 to June 30, 2020, across 22 categories. We performedanalysis from the perspectives of category popularity. To further understand the evolution patternsof app rankings, we proposed a correlation-based method to classify them and then perform a percategory analysis. Besides, we characterized the side effect of declining ratings accompanied bythe rising popularity of some apps. Our observations reveal mobile users’ reactions in the mobileecosystem in the face of unexpected public crises, and provide insights for app developers to makebetter decisions on developing apps.
REFERENCES
IEEE Access , 8:134577–134601, 2020.[18] Nadeem Ahmed, Regio A Michelin, Wanli Xue, Sushmita Ruj, Robert Malaney, Salil S Kanhere, Aruna Seneviratne,Wen Hu, Helge Janicke, and Sanjay K Jha. A survey of covid-19 contact tracing apps.
IEEE Access , 8:134577–134601,, Vol. 1, No. 1, Article . Publication date: February 2020.
Tale of Two Countries 21
Dubai Medical Journal , 3(2):44–49, 2020.[20] Yan Bai, Lingsheng Yao, Tao Wei, Fei Tian, Dong-Yan Jin, Lijuan Chen, and Meiyun Wang. Presumed asymptomaticcarrier transmission of covid-19.
Jama , 323(14):1406–1407, 2020.[21] Arndt Büssing, Jutta Hübner, Stefanie Walter, Wolfgang Gießler, and Jens Büntzel. Tumor patients´ perceived changesof specific attitudes, perceptions, and behaviors due to the covid-19 pandemic and its relation to reduced wellbeing.
Frontiers in psychiatry , 11, 2020.[22] Laura V Galvis Carreno and Kristina Winbladh. Analysis of user comments: an approach for software requirementsevolution. In , pages 582–591. IEEE, 2013.[23] Hyunghoon Cho, Daphne Ippolito, and Yun William Yu. Contact tracing mobile apps for covid-19: Privacy considera-tions and related trade-offs. arXiv preprint arXiv:2003.11511 , 2020.[24] Feng Dong, Haoyu Wang, Li Li, Yao Guo, Tegawendé F Bissyandé, Tianming Liu, Guoai Xu, and Jacques Klein.Frauddroid: Automated ad fraud detection for android apps. In
Proceedings of the 2018 26th ACM Joint Meeting onEuropean Software Engineering Conference and Symposium on the Foundations of Software Engineering , pages 257–268.ACM, 2018.[25] Feng Dong, Haoyu Wang, Li Li, Yao Guo, Guoai Xu, and Shaodong Zhang. How do mobile apps violate the behavioralpolicy of advertisement libraries? In
Proceedings of the 19th International Workshop on Mobile Computing Systems &Applications , pages 75–80, 2018.[26] Yogesh K Dwivedi, D Laurie Hughes, Crispin Coombs, Ioanna Constantiou, Yanqing Duan, John S Edwards, BabitaGupta, Banita Lal, Santosh Misra, Prakhar Prashant, et al. Impact of covid-19 pandemic on information managementresearch and practice: Transforming education, work and life.
International Journal of Information Management ,55:102211, 2020.[27] James D Evans.
Straightforward statistics for the behavioral sciences.
Thomson Brooks/Cole Publishing Co, 1996.[28] Vahid Garousi, David Cutting, and Michael Felderer. Mining user reviews of covid contact-tracing apps: An exploratoryanalysis of nine european apps. arXiv preprint arXiv:2012.13589 , 2020.[29] Ren He, Haoyu Wang, Pengcheng Xia, Liu Wang, Yuanchun Li, Lei Wu, Yajin Zhou, Xiapu Luo, Yao Guo, and GuoaiXu. Beyond the virus: A first look at coronavirus-themed mobile malware. arXiv preprint arXiv:2005.14619 , 2020.[30] Michelle L. Holshue, Chas DeBolt, Scott Lindquist, Kathy H. Lofy, John Wiesman, Hollianne Bruce, Christopher Spitters,Keith Ericson, Sara Wilkerson, Ahmet Tural, George Diaz, Amanda Cohn, LeAnne Fox, Anita Patel, Susan I. Gerber,Lindsay Kim, Suxiang Tong, Xiaoyan Lu, Steve Lindstrom, Mark A. Pallansch, William C. Weldon, Holly M. Biggs,Timothy M. Uyeki, and Satish K. Pillai. First case of 2019 novel coronavirus in the united states.
New England Journalof Medicine , 382(10):929–936, 2020. PMID: 32004427.[31] Yangyu Hu, Haoyu Wang, Ren He, Li Li, Gareth Tyson, Ignacio Castro, Yao Guo, Lei Wu, and Guoai Xu. Mobile appsquatting. In
Proceedings of The Web Conference 2020 , pages 1727–1738, 2020.[32] Yangyu Hu, Haoyu Wang, Yajin Zhou, Yao Guo, Li Li, Bingxuan Luo, and Fangren Xu. Dating with scambots:Understanding the ecosystem of fraudulent dating applications. arXiv preprint arXiv:1807.04901 , 2018.[33] Amit Kramer and Karen Z Kramer. The potential impact of the covid-19 pandemic on occupational status, work fromhome, and occupational mobility, 2020.[34] Siddique Latif, Muhammad Usman, Sanaullah Manzoor, Waleed Iqbal, Junaid Qadir, Gareth Tyson, Ignacio Castro,Adeel Razi, Maged N Kamel Boulos, Adrian Weller, et al. Leveraging data science to combat covid-19: A comprehensivereview.
IEEE Transactions on Artificial Intelligence , 2020.[35] Li Li, Tegawendé F Bissyandé, and Jacques Klein. Rebooting research on detecting repackaged android apps: Literaturereview and benchmark.
IEEE Transactions on Software Engineering , 2019.[36] Li Li, Tegawendé F Bissyandé, Hao-Yu Wang, and Jacques Klein. On identifying and explaining similarities in androidapps.
Journal of Computer Science and Technology , 34(2):437–455, 2019.[37] Minxing Liu, Haoyu Wang, Yao Guo, and Jason Hong. Identifying and analyzing the privacy of apps for kids. In
Proceedings of the 17th International Workshop on Mobile Computing Systems and Applications , pages 105–110, 2016.[38] Tianming Liu, Haoyu Wang, Li Li, Guangdong Bai, Yao Guo, and Guoai Xu. Dapanda: Detecting aggressive pushnotifications in android apps. In , pages 66–78. IEEE, 2019.[39] Tianming Liu, Haoyu Wang, Li Li, Xiapu Luo, Feng Dong, Yao Guo, Liu Wang, Tegawende F. Bissyande, and JacquesKlein. Maddroid: Characterising and detecting devious ad content for android apps. In
Proceedings of the Web Conference2020 (WWW’20) , 2020.[40] Ziang Ma, Haoyu Wang, Yao Guo, and Xiangqun Chen. Libradar: fast and accurate detection of third-party libraries inandroid apps. In
Proceedings of the 38th International Conference on Software Engineering Companion , pages 653–656,2016. , Vol. 1, No. 1, Article . Publication date: February 2020. [41] Nuria Oliver, Emmanuel Letouzé, Harald Sterly, Sébastien Delataille, Marco De Nadai, Bruno Lepri, Renaud Lambiotte,Richard Benjamins, Ciro Cattuto, Vittoria Colizza, Nicolas de Cordes, Samuel P. Fraiberger, Till Koebe, Sune Lehmann,Juan Murillo, Alex Pentland, Phuong N Pham, Frédéric Pivetta, Albert Ali Salah, Jari Saramäki, Samuel V. Scarpino,Michele Tizzoni, Stefaan Verhulst, and Patrick Vinck. Mobile phone data and covid-19: Missing an opportunity?, 2020.[42] Graziano Onder, Giovanni Rezza, and Silvio Brusaferro. Case-fatality rate and characteristics of patients dying inrelation to covid-19 in italy.
Jama , 323(18):1775–1776, 2020.[43] Dennis Pagano and Walid Maalej. User feedback in the appstore: An empirical study. In , pages 125–134. IEEE, 2013.[44] Abbas Razaghpanah, Rishab Nithyanand, Narseo Vallina-Rodriguez, Srikanth Sundaresan, Mark Allman, ChristianKreibich, and Phillipa Gill. Apps, trackers, privacy, and regulators: A global study of the mobile tracking ecosystem.2018.[45] Joel Reardon, Álvaro Feal, Primal Wijesekera, Amit Elazari Bar On, Narseo Vallina-Rodriguez, and Serge Egelman. 50ways to leak your data: An exploration of apps’ circumvention of the android permissions system. In { USENIX } Security Symposium ( { USENIX } Security 19) , pages 603–620, 2019.[46] Tanusree Sharma, Tian Wang, and Masooda Bashir. Advocating for users’ privacy protections: A case study of covid-19apps. 2020.[47] Chenguang Shen, Zhaoqin Wang, Fang Zhao, Yang Yang, Jinxiu Li, Jing Yuan, Fuxiang Wang, Delin Li, Minghui Yang,Li Xing, et al. Treatment of 5 critically ill patients with covid-19 with convalescent plasma.
Jama , 323(16):1582–1589,2020.[48] R Sun, W Wang, M Xue, G Tyson, S Camtepe, and DC Ranasinghe. An empirical assessment of global covid-19 contacttracing applications. 2021.[49] Yutian Tang, Yulei Sui, Haoyu Wang, Xiapu Luo, Hao Zhou, and Zhou Xu. All your app links are belong to us:understanding the threats of instant apps based attacks. In
Proceedings of the 28th ACM Joint Meeting on EuropeanSoftware Engineering Conference and Symposium on the Foundations of Software Engineering , pages 914–926, 2020.[50] Haoyu Wang and Yao Guo. Understanding third-party libraries in mobile app analysis. In , pages 515–516. IEEE, 2017.[51] Haoyu Wang, Yao Guo, Zihao Tang, Guangdong Bai, and Xiangqun Chen. Reevaluating android permission gaps withstatic and dynamic analysis. In , pages 1–6. IEEE, 2015.[52] Haoyu Wang, Jason Hong, and Yao Guo. Using text mining to infer the purpose of permission use in mobile apps. In
Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing , pages 1107–1118,2015.[53] Haoyu Wang, Hao Li, and Yao Guo. Understanding the evolution of mobile app ecosystems: A longitudinal measurementstudy of google play. In
The World Wide Web Conference , pages 1988–1999. ACM, 2019.[54] Haoyu Wang, Hao Li, Li Li, Yao Guo, and Guoai Xu. Why are android apps removed from google play? a large-scaleempirical study. In
The 15th International Conference on Mining Software Repositories (MSR 2018) , 2018.[55] Haoyu Wang, Hongxuan Liu, Xusheng Xiao, Guozhu Meng, and Yao Guo. Characterizing android app signing issues.In , pages 280–292. IEEE, 2019.[56] Haoyu Wang, Zhe Liu, Yao Guo, Xiangqun Chen, Miao Zhang, Guoai Xu, and Jason Hong. An explorative study of themobile app ecosystem from app developers’ perspective. In
Proceedings of the 26th International Conference on WorldWide Web , pages 163–172, 2017.[57] Haoyu Wang, Zhe Liu, Jingyue Liang, Narseo Vallina-Rodriguez, Yao Guo, Li Li, Juan Tapiador, Jingcun Cao, and GuoaiXu. Beyond google play: A large-scale comparative study of chinese android app markets. In
Proceedings of the InternetMeasurement Conference 2018 , pages 293–307, 2018.[58] Haoyu Wang, Junjun Si, Hao Li, and Yao Guo. Rmvdroid: Towards a reliable android malware dataset with appmetadata. In
Proceedings of the 16th International Conference on Mining Software Repositories , page 404–408, 2019.[59] Haoyu Wang, Xupu Wang, and Yao Guo. Characterizing the global mobile app developers: a large-scale empiricalstudy. In , pages150–161. IEEE, 2019.[60] Huiyi Wang, Liu Wang, and Haoyu Wang. Market-level analysis of government-backed covid-19 contact tracing apps. arXiv preprint arXiv:2012.10866 , 2020.[61] Linda Wang and Alexander Wong. Covid-net: A tailored deep convolutional neural network design for detection ofcovid-19 cases from chest x-ray images. arXiv preprint arXiv:2003.09871 , 2020.[62] Wei Wang, Guozhu Meng, Haoyu Wang, Kai Chen, Weimin Ge, and Xiaohong Li. A3ident: A two-phased approachto identify the leading authors of android apps. In , pages 617–628. IEEE, 2020.[63] Shengqu Xi, Shao Yang, Xusheng Xiao, Yuan Yao, Yayuan Xiong, Fengyuan Xu, Haoyu Wang, Peng Gao, Zhuotao Liu,Feng Xu, et al. Deepintent: Deep icon-behavior learning for detecting intention-behavior discrepancy in mobile apps., Vol. 1, No. 1, Article . Publication date: February 2020.
Tale of Two Countries 23 In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security , pages 2421–2436, 2019.[64] Xu Xu, Yangyu Hu, Qian Guo, Ren He, Li Li, Guoai Xu, Zhihui Han, and Haoyu Wang. Dissecting mobile offerwalladvertisements: An explorative study. In , pages 518–526. IEEE, 2020.[65] Xian Zhan, Lingling Fan, Tianming Liu, Sen Chen, Li Li, Haoyu Wang, Yifei Xu, Xiapu Luo, and Yang Liu. Automatedthird-party library detection for android applications: Are we there yet? In , pages 919–930. IEEE, 2020.[66] Jianpeng Zhang, Yutong Xie, Yi Li, Chunhua Shen, and Yong Xia. Covid-19 screening on chest x-ray images usingdeep learning based anomaly detection. arXiv preprint arXiv:2003.12338 , 2020.[67] Ping Zhong, Songxue Guo, and Ting Chen. Correlation between travellers departing from Wuhan before the SpringFestival and subsequent spread of COVID-19 to all provinces in China.
Journal of Travel Medicine , 27(3), 03 2020.taaa036.[68] Hao Zhou, Haoyu Wang, Yajin Zhou, Xiapu Luo, Yutian Tang, Lei Xue, and Ting Wang. Demystifying diehard androidapps. In , pages 187–198. IEEE,2020.[69] Yajin Zhou and Xuxian Jiang. Dissecting android malware: Characterization and evolution. In , pages 95–109. IEEE, 2012.[70] Zi Yue Zu, Meng Di Jiang, Peng Peng Xu, Wen Chen, Qian Qian Ni, Guang Ming Lu, and Long Jiang Zhang. Coronavirusdisease 2019 (covid-19): a perspective from china.