Multi-Task Ordinal Regression for Jointly Predicting the Trustworthiness and the Leading Political Ideology of News Media
Ramy Baly, Georgi Karadzhov, Abdelrhman Saleh, James Glass, Preslav Nakov
MMulti-Task Ordinal Regression for Jointly Predictingthe Trustworthiness and the Leading Political Ideology of News Media
Ramy Baly , Georgi Karadzhov , Abdelrhman Saleh ,James Glass , Preslav Nakov MIT Computer Science and Artificial Intelligence Laboratory, MA, USA SiteGround Hosting EOOD, Bulgaria, Harvard University, MA, USA Qatar Computing Research Institute, HBKU, Qatar { baly, glass } @mit.edu , [email protected] [email protected] , [email protected] Abstract
In the context of fake news, bias, and propa-ganda, we study two important but relativelyunder-explored problems: ( i ) trustworthinessestimation (on a 3-point scale) and ( ii ) po-litical ideology detection (left/right bias on a7-point scale) of entire news outlets, as op-posed to evaluating individual articles. In par-ticular, we propose a multi-task ordinal re-gression framework that models the two prob-lems jointly. This is motivated by the obser-vation that hyper-partisanship is often linkedto low trustworthiness, e.g., appealing to emo-tions rather than sticking to the facts, whilecenter media tend to be generally more impar-tial and trustworthy. We further use severalauxiliary tasks, modeling centrality, hyper-partisanship, as well as left-vs.-right bias ona coarse-grained scale. The evaluation resultsshow sizable performance gains by the jointmodels over models that target the problemsin isolation. Recent years have seen the rise of social media,which has enabled people to virtually share in-formation with a large number of users withoutregulation or quality control. On the bright side,this has given an opportunity for anyone to be-come a content creator, and has also enabled amuch faster information dissemination. However,it has also opened the door for malicious users tospread disinformation and misinformation muchfaster, enabling them to easily reach audience ata scale that was never possible before. In somecases, this involved building sophisticated profilesfor individuals based on a combination of psycho-logical characteristics, meta-data, demographics,and location, and then micro-targeting them withpersonalized “fake news” with the aim of achiev-ing some political or financial gains (Lazer et al.,2018; Vosoughi et al., 2018). A number of fact-checking initiatives have beenlaunched so far, both manual and automatic, butthe whole enterprise remains in a state of cri-sis: by the time a claim is finally fact-checked, itcould have reached millions of users, and the harmcaused could hardly be undone. An arguably morepromising direction is to focus on fact-checkingentire news outlets, which can be done in advance.Then, we could fact-check the news before theywere even written: by checking how trustworthythe outlets that published them are. Knowing thereliability of a medium is important not only whenfact-checking a claim (Popat et al., 2017; Nguyenet al., 2018), but also when solving article-leveltasks such as “fake news” and click-bait detection(Brill, 2001; Finberg et al., 2002; Hardalov et al.,2016; Karadzhov et al., 2017; De Sarkar et al.,2018; Pan et al., 2018; P´erez-Rosas et al., 2018)Political ideology (or left/right bias) is a relatedcharacteristic, e.g., extreme left/right media tendto be propagandistic, while center media are morefactual, and thus generally more trustworthy. Thisconnection can be clearly seen in Figure 1.
Figure 1: Correlation between bias and factuality forthe news outlets in the Media Bias/Fact Check website. a r X i v : . [ c s . I R ] A p r espite the connection between factuality andbias, previous research has addressed them as in-dependent tasks, even when the underlying datasethad annotations for both (Baly et al., 2018). Incontrast, here we solve them jointly. Our contri-butions can be summarized as follows: • We study an under-explored but arguably im-portant problem: predicting the factuality ofreporting of news media. Moreover, unlikeprevious work, we do this jointly with thetask of predicting political bias. • As factuality and bias are naturally defined onan ordinal scale (factuality: from low to high ,and bias: from extreme-left to extreme-right ),we address them as ordinal regression. Us-ing multi-task ordinal regression is novel forthese tasks, and it is also an under-exploreddirection in machine learning in general. • We design a variety of auxiliary subtasksfrom the bias labels: modeling centrality,hyper-partisanship, as well as left-vs.-rightbias on a coarse-grained scale.
Factuality of Reporting
Previous work hasmodeled the factuality of reporting at the mediumlevel by checking the general stance of the tar-get medium with respect to known manually fact-checked claims, without access to gold labelsabout the overall medium-level factuality of re-porting (Mukherjee and Weikum, 2015; Popatet al., 2016, 2017, 2018).The trustworthiness of Web sources has alsobeen studied from a Data Analytics perspective,e.g., Dong et al. (2015) proposed that a trust-worthy source is one that contains very few falseclaims. In social media, there has been researchtargeting the user, e.g., finding malicious users(Mihaylov and Nakov, 2016; Mihaylova et al.,2018; Mihaylov et al., 2018), sockpuppets (Maityet al., 2017),
Internet water army (Chen et al.,2013), and seminar users (Darwish et al., 2017).Unlike the above work, here we study sourcereliability as a task in its own right, using man-ual gold annotations specific for the task and as-signed by independent fact-checking journalists.Moreover, we address the problem as one of ordi-nal regression on a three-point scale, and we solveit jointly with political ideology prediction in amulti-task learning setup, using several auxiliarytasks.
Predicting Political Ideology
In previous work,political ideology, also known as media bias, wasused as a feature for “fake news” detection (Horneet al., 2018a). It has also been the target ofclassification, e.g., Horne et al. (2018b) predictedwhether an article is biased ( political or bias ) vs.unbiased. Similarly, Potthast et al. (2018) classi-fied the bias in a target article as ( i ) left vs. rightvs. mainstream, or as ( ii ) hyper-partisan vs. main-stream. Left-vs-right bias classification at the ar-ticle level was also explored by Kulkarni et al.(2018), who modeled both the textual and the URLcontents of the target article. There has been alsowork targeting bias at the phrase or the sentencelevel (Iyyer et al., 2014), focusing on politicalspeeches (Sim et al., 2013) or legislative docu-ments (Gerrish and Blei, 2011), or targeting usersin Twitter (Preot¸iuc-Pietro et al., 2017). Anotherline of related work focuses on propaganda, whichcan be seen as a form of extreme bias (Rashkinet al., 2017; Barr´on-Cede˜no et al., 2019a,b). Seealso a recent position paper (Pitoura et al., 2018)and an overview paper on bias on the Web (Baeza-Yates, 2018). Unlike the above work, here we fo-cus on predicting the political ideology of newsmedia outlets.In our previous work (Baly et al., 2018), we didtarget the political bias of entire news outlets, asopposed to working at the article level (we alsomodeled factuality of reporting, but as a separatetask without trying multi-task learning). In addi-tion to the text of the articles published by the tar-get news medium, we used features extracted fromits corresponding Wikipedia page and Twitter pro-file, as well as analysis of its URL structure andtraffic information about it from Alexa rank. Inthe present work, we use a similar set of features,but we treat the problem as one of ordinal regres-sion. Moreover, we model the political ideologyand the factuality of reporting jointly in a multi-task learning setup, using several auxiliary tasks. Multitask Ordinal Regression
Ordinal regres-sion is well-studied and is commonly used for textclassification on an ordinal scale, e.g., for senti-ment analysis on a 5-point scale (He et al., 2016;Rosenthal et al., 2017a). However, multi-task or-dinal regression remains an understudied problem.Yu et al. (2006) proposed a Bayesian frameworkfor collaborative ordinal regression, and demon-strated that modeling multiple ordinal regressiontasks outperforms single-task models.alecki et al. (2016) were interested in jointlypredicting facial action units and their intensitylevel. They argued that, due to the high num-ber of classes, modeling these tasks independentlywould be inefficient. Thus, they proposed the cop-ula ordinal regression model for multi-task learn-ing and demonstrated that it can outperform vari-ous single-task setups. We use this model in ourexperiments below.Balikas et al. (2017) used multi-task ordinalregression for the task of fine-grained sentimentanalysis. In particular, they introduced an auxil-iary coarse-grained task on a 3-point scale, anddemonstrated that it can improve the results forsentiment analysis on the original 5-point scale.Inspired by this, below we experiment with dif-ferent granularity for political bias; however, weexplore a larger space of possible auxiliary tasks.
Copula Ordinal Regression
We use the
Cop-ula Ordinal Regression (COR) model, which wasoriginally proposed by Walecki et al. (2016) to es-timate the intensities of facial action units (AUs).The model uses copula functions and conditionalrandom fields (CRFs) to approximates the learningof the joint probability distribution function (PDF)of the facial AUs (random variables), using the bi-variate joint distributions capturing dependenciesbetween AU pairs. It was motivated by the factthat ( i ) many facial AUs co-exist with differentlevels of intensity, ( ii ) some AUs co-occur moreoften than others, and ( iii ) some AUs depend onthe intensity of other units.We can draw an analogy between modeling fa-cial AUs and modeling news media, where eachmedium expresses a particular bias (political ide-ology) and can also be associated with a particu-lar level of factuality. Therefore, bias and factual-ity can be analogous to the facial AUs in (Waleckiet al., 2016), and represent two aspects of news re-porting, each being modeled on a multi-point ordi-nal scale. In particular, we model bias on a 7-pointscale ( extreme-left , left , center-left , center , center-right , right , and extreme-right ), and factuality ona 3-point scale ( low , mixed , and high ).In our case, we train the COR model to predictthe joint PDF between political bias and factual-ity of reporting. This could potentially work wellgiven the inherent inter-dependency between thetwo tasks as we have seen on Figure 1. Auxiliary Tasks
We use a variety of auxiliarytasks, derived from the bias labels. This includesconverting the 7-point scale to ( i ) 5-point and 3-point scales, similarly to (Balikas et al., 2017), andto ( ii ) a 2-point scale in two ways to model ex-treme partisanship, and centrality. Here is the listof the auxiliary tasks we use with precise defini-tion of the label mappings: • Bias5-way:
Predict bias on a 5-pt scale;1: extreme-left , 2: left , 3: { center-left, center,center-right } , 4: right , and 5: extreme-right . • Bias3-way:
Predict bias on a 3-pt scale;1: { extreme-left, left } , 2: { center-left, center,center-right } , and 3: { right, extreme-right } . • Bias-extreme:
Predict extreme vs. non-extreme partisanship on a 2-pt scale;1: { extreme-left, extreme-right } , 2: { left,center-left, center, center-right, right } . • Bias-center:
Predict center vs. non-centerpolitical ideology on a 2-pt scale, ignoringpolarity: 1: { extreme-left, left, right, extreme-right } , 2: { center-left, center, center-right } . Features
We used the features from (Baly et al.,2018) . We gathered a sample of articles from thetarget medium, and we calculated features such asPOS tags, linguistic cues, sentiment scores, com-plexity, morality, as well as embeddings. We alsoused the Wikipedia page of the medium (if any)to generate document embedding. Then, we col-lected metadata from the medium’s Twitter ac-count (if any), e.g., whether is is verified, num-ber of followers, whether the URL in the Twitterpage matches the one of the medium. Finally, weadded Web-based features that ( i ) model the ortho-graphic structure of the medium’s URL address,and ( ii ) analyze the Web-traffic information aboutthe medium’s website, as found in Alexa rank. Data
We used the MBFC dataset (Baly et al.,2018) that has 1,066 news media manually anno-tated for factuality (3-pt scale: high , mixed , low )and political bias (7-pt scale: from extreme-left to extreme-right ). This dataset was annotated by vol-unteers using a detailed methodology that is de-signed to guarantee annotation objectivity. https://github.com/ramybaly/News-Media-Reliability For details, see https://mediabiasfactcheck.com/methodology/ ame URL Bias Factuality Twitter Handle Wikipedia page
London Web News londonwebnews.com
Extreme Left Low @londonwebnews N/ADaily Mirror
Left Mixed @DailyMirror ˜/Daily_Mirror
NBC News
Center-Left High @nbcnews ˜/NBC_News
Associated Press apnews.com
Center Very High @apnews ˜/Associated_Press
Gulf News gulfnews.com
Center-Right High @gulf news ˜/Gulf_News
Russia Insider russia-insider.com
Right Mixed @russiainsider ˜/Russia_Insider
Breitbart
Extreme Right Low @BreitbartNews ˜/Breitbart_News
Table 1: Examples of media and their labels for bias and factuality of reporting derived from MBFC.
Furthermore, readers can provide their own feed-back on existing annotations, and in case of a largediscrepancy, annotation is adjusted after a thor-ough review. Therefore, we believe the annotationquality is good enough to experiment with. Wenoticed that 117 media had low factuality becausethey publish satire and pseudo-science , neither ofwhich has a political perspective. Since we are in-terested in modeling the relation between factual-ity and bias, we excluded those websites, thus end-ing up with 949 news media. Some examples fromthis dataset are shown in Table 1 with both factual-ity and bias labels, in addition to their correspond-ing Twitter handles and Wikipedia pages. Overall,64% of the media in our dataset have Wikipediapages, and 65% have Twitter accounts. Table 2further provides detailed statistics about the labeldistribution in the MBFC dataset.
Factuality Bias
Low 198 Extreme-Left 23Mixed 282 Left 151High 469 Center-Left 200Center 139Center-Right 105Right 164Extreme-Right 167
Table 2: Labels counts in the MBFC dataset that weused in our experiments.
Experimental Setup
We used the implementa-tion of the Copula Ordinal Regression (COR)model as described in (Walecki et al., 2016). Inour experiments, we used 5-fold cross-validation,where for each fold we split the training datasetinto a training part and a validation part, and weused the latter to fine-tune the model’s hyper-parameters, optimizing for Mean Absolute Error(MAE). MAE is an appropriate evaluation mea-sure given the ordinal nature of the tasks. https://github.com/RWalecki/copula_ordinal_regression These hyper-parameters include the copula func-tion (
Gumbel vs.
Frank ), the marginal distribution( normal vs. sigmoid ), the number of training it-erations, the optimizer ( gradient descent , BFGS ),and the connection density of the CRFs. We reportboth MAE and MAE M , which is a variant of MAEthat is more robust to class imbalance. See (Bac-cianella et al., 2009; Rosenthal et al., 2017b) formore details about MAE M vs. MAE. We comparethe results to two baselines: ( i ) majority class, and( ii ) single-task ordinal regression. Results and Discussion
Table 3 shows the eval-uation results for the COR model when trainedto jointly model the main task ( shown in thecolumns ) using combinations of auxiliary tasks( shown in the rows ). We can see that the single-task ordinal regression model performs much bet-ter than the majority class baseline based on bothevaluation measures. We can further see thatthe performance on the main task improves whenjointly modeling several auxiliary tasks. This im-provement depends on the auxiliary tasks in use.For factuality prediction, it turns out that thecombination of bias-center + bias-extreme yieldsthe best overall MAE of 0.481. This makes senseand aligns well with the intuition that knowingwhether a medium is centric or hyper-partisan isimportant to predict the factuality of its reporting.For instance, a news medium without a politicalideology tends to be more trustworthy comparedto an extremely biased one, regardless of their po-larity (left or right), as we should expect based onthe data distribution shown in Figure 1 above.For bias prediction (at a 7-point left-to-rightscale), a joint model that uses political bias at dif-ferent levels of granularity (5-point and 3-point)as auxiliary tasks yields the best overall MAE of1.479. This means that jointly modeling bias withthe same information at coarser levels of granu-larity, i.e., adding 3-point and 5-point as auxiliarytasks, reduces the number of gross mistakes. actuality BiasAuxiliary Tasks MMAEM MMAE M M MAE MAE M (None) majority class . . . . . . . . . . . . . . . . . . . . . . . . . . . 0.714 1.000 1.798 1.857(None) single-task COR . . . . . . . . . . . . . . . . . . . . . . . . . 0.514 0.567 1.582 1.728 + bias . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 0.526 0.566 – – + factuality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . – – 1.584 1.695 + bias5-way . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 0.495 0.541 1.504 (1.485) (1.647) + bias3-way . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 0.497 0.548 1.528 (1.498) (1.654) + bias-center . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 0.509 0.561 1.594 (1.535) (1.695) + bias-extreme. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 0.498 0.550 1.584 (1.558) (1.726) + bias5-way + bias3-way . . . . . . . . . . . . . . . . . . . . . . . . . 0.493 0.541 1.479 ( ) ( ) + bias-center + bias-extreme . . . . . . . . . . . . . . . . . . . . . . (1.526) (1.672) + bias5-way + bias3-way + bias-center + bias-extreme 0.485 0.537 1.513 (1.504) (1.677) Table 3: Evaluating the copula ordinal regression model trained to jointly model the main task ( shown in thecolumns ) and different auxiliary tasks ( shown in the rows ). The results in parentheses correspond to the case whenfactuality is added as an additional auxiliary task (only applicable when the main task is bias prediction).
E.g., predicting extreme-left instead of extreme-right , since the model is encouraged by the aux-iliary tasks to learn the correct polarity, regard-less of its intensity. We can see that factuality is not very useful as an auxiliary task by itself(MAE=1.584 and MAE M =1.695). In other words,a medium with low factuality could be extremelybiased to either the right or to the left. Therefore,relying on factuality alone to predict bias might in-troduce severe errors, e.g., confusing extreme-leftwith extreme-right, thus leading to higher MAEscores. This can be remedied by adding factuality to the mix of other auxiliary tasks to model themain task (7-point bias prediction). The resultsof these experiments, shown in parentheses in Ta-ble 3, indicate that adding factuality to any combi-nation of auxiliary tasks consistently yields lowerMAE scores. In particular, modeling the combi-nation of factuality + bias5-way + bias3-way yieldsthe best results (MAE=1.475 and MAE M =1.623).This result indicates that factuality provides com-plementary information that can help predict bias.We ran a two-tailed t-test for statistical signif-icance, which is suitable for an evaluation mea-sure such as MAE, to confirm the improvementsthat were introduced by the multi-task setup. Wefound that the best models (shown in bold in Ta-ble 3) outperformed both the corresponding major-ity class baselines with a p-value ≤ ≤ We have presented a multi-task ordinal regres-sion framework for jointly predicting trustworthi-ness and political ideology of news media sources,using several auxiliary tasks, e.g., based on acoarser-grained scales or modeling extreme parti-sanship. Overall, we have observed sizable per-formance gains in terms of reduced MAE by themulti-task ordinal regression models over single-task models for each of the two individual tasks.In future work, we want to try more auxiliarytasks, and to experiment with other languages. Wefurther plan to go beyond left vs. right , which isnot universal and can exhibit regional specificity(Tavits and Letki, 2009), and to model other kindsof biases, e.g., eurosceptic vs. europhile , national-ist vs. globalist , islamist vs. secular , etc. Acknowledgments
This research is part of the Tanbih project, whichaims to limit the effect of “fake news”, propa-ganda and media bias by making users awareof what they are reading. The project is de-veloped in collaboration between the MIT Com-puter Science and Artificial Intelligence Labora-tory (CSAIL) and the Qatar Computing ResearchInstitute (QCRI), HBKU. http://tanbih.qcri.org/ eferences Stefano Baccianella, Andrea Esuli, and Fabrizio Sebas-tiani. 2009. Evaluation measures for ordinal regres-sion. In
Proceedings of the 9th IEEE InternationalConference on Intelligent Systems Design and Ap-plications , ISDA ’09, pages 283–287, Pisa, Italy.Ricardo Baeza-Yates. 2018. Bias on the web.
Com-mun. ACM , 61(6):54–61.Georgios Balikas, Simon Moura, and Massih-RezaAmini. 2017. Multitask learning for fine-grainedTwitter sentiment analysis. In
Proceedings of the40th International ACM SIGIR Conference on Re-search and Development in Information Retrieval ,SIGIR ’17, pages 1005–1008, Tokyo, Japan.Ramy Baly, Georgi Karadzhov, Dimitar Alexandrov,James Glass, and Preslav Nakov. 2018. Predict-ing factuality of reporting and bias of news mediasources. In
Proceedings of the Conference on Em-pirical Methods in Natural Language Processing ,EMNLP ’18, pages 3528–3539, Brussels, Belgium.Alberto Barr´on-Cede˜no, Giovanni Da San Martino, Is-raa Jaradat, and Preslav Nakov. 2019a. Proppy: Asystem to unmask propaganda in online news. In
Proceedings of the Thirty-Third AAAI Conferenceon Artificial Intelligence , AAAI’19, Honolulu, HI,USA.Alberto Barr´on-Cede˜no, Giovanni Da San Martino, Is-raa Jaradat, and Preslav Nakov. 2019b. Proppy: Or-ganizing news coverage on the basis of their propa-gandistic content.
Information Processing and Man-agement .Ann M Brill. 2001. Online journalists embrace newmarketing function.
Newspaper Research Journal ,22(2):28.Cheng Chen, Kui Wu, Venkatesh Srinivasan, andXudong Zhang. 2013. Battling the Internet WaterArmy: detection of hidden paid posters. In
Proceed-ings of the 2013 IEEE/ACM International Confer-ence on Advances in Social Networks Analysis andMining , ASONAM ’13, pages 116–120, Niagara,Canada.Kareem Darwish, Dimitar Alexandrov, Preslav Nakov,and Yelena Mejova. 2017. Seminar users in theArabic Twitter sphere. In
Proceedings of the9th International Conference on Social Informatics ,SocInfo ’17, pages 91–108, Oxford, UK.Sohan De Sarkar, Fan Yang, and Arjun Mukherjee.2018. Attending sentences to detect satirical fakenews. In
Proceedings of the 27th InternationalConference on Computational Linguistics , COL-ING ’18, pages 3371–3380, Santa Fe, NM, USA.Xin Luna Dong, Evgeniy Gabrilovich, Kevin Murphy,Van Dang, Wilko Horn, Camillo Lugaresi, Shao-hua Sun, and Wei Zhang. 2015. Knowledge-basedtrust: Estimating the trustworthiness of web sources.
Proc. VLDB Endow. , 8(9):938–949. Howard Finberg, Martha L Stone, and Diane Lynch.2002. Digital journalism credibility study.
OnlineNews Association. Retrieved November , 3:2003.Sean M. Gerrish and David M. Blei. 2011. Predict-ing legislative roll calls from text. In
Proceedings ofthe 28th International Conference on InternationalConference on Machine Learning , ICML ’11, pages489–496, Bellevue, Washington, USA.Momchil Hardalov, Ivan Koychev, and Preslav Nakov.2016. In search of credible news. In
Proceedingsof the 17th International Conference on Artificial In-telligence: Methodology, Systems, and Applications ,AIMSA ’16, pages 172–180, Varna, Bulgaria.Yunchao He, Liang-Chih Yu, Chin-Sheng Yang,K Robert Lai, and Weiyi Liu. 2016. YZU-NLP teamat semeval-2016 task 4: Ordinal sentiment classifi-cation using a recurrent convolutional network. In
Proceedings of the 10th International Workshop onSemantic Evaluation , SemEval ’16, pages 251–255,San Diego, CA, USA.Benjamin Horne, Sara Khedr, and Sibel Adali. 2018a.Sampling the news producers: A large news and fea-ture data set for the study of the complex media land-scape. In
Proceedings of the Twelfth InternationalConference on Web and Social Media , ICWSM ’18,pages 518–527, Stanford, CA, USA.Benjamin D. Horne, William Dron, Sara Khedr, andSibel Adali. 2018b. Assessing the news landscape:A multi-module toolkit for evaluating the credibilityof news. In
Proceedings of the The Web Conference ,WWW ’18, pages 235–238, Lyon, France.Mohit Iyyer, Peter Enns, Jordan Boyd-Graber, andPhilip Resnik. 2014. Political ideology detection us-ing recursive neural networks. In
Proceedings of the52nd Annual Meeting of the Association for Com-putational Linguistics , pages 1113–1122, Baltimore,MD, USA.Georgi Karadzhov, Pepa Gencheva, Preslav Nakov, andIvan Koychev. 2017. We built a fake news & click-bait filter: What happened next will blow your mind!In
Proceedings of the International Conference onRecent Advances in Natural Language Processing ,RANLP ’17, pages 334–343, Varna, Bulgaria.Vivek Kulkarni, Junting Ye, Steven Skiena, andWilliam Yang Wang. 2018. Multi-view models forpolitical ideology detection of news articles. In
Pro-ceedings of the Conference on Empirical Methods inNatural Language Processing , EMNLP ’18, pages3518–3527, Brussels, Belgium.David M.J. Lazer, Matthew A. Baum, Yochai Ben-kler, Adam J. Berinsky, Kelly M. Greenhill, FilippoMenczer, Miriam J. Metzger, Brendan Nyhan, Gor-don Pennycook, David Rothschild, Michael Schud-son, Steven A. Sloman, Cass R. Sunstein, Emily A.Thorson, Duncan J. Watts, and Jonathan L. Zit-train. 2018. The science of fake news.
Science ,359(6380):1094–1096.uman Kalyan Maity, Aishik Chakraborty, PawanGoyal, and Animesh Mukherjee. 2017. Detection ofsockpuppets in social media. In
Proceedings of theACM Conference on Computer Supported Coopera-tive Work and Social Computing , CSCW ’17, pages243–246, Portland, OR, USA.Todor Mihaylov, Tsvetomila Mihaylova, PreslavNakov, Llu´ıs M`arquez, Georgi Georgiev, and IvanKoychev. 2018. The dark side of news communityforums: Opinion manipulation trolls.
Internet Re-search , 28(5):1292–1312.Todor Mihaylov and Preslav Nakov. 2016. Hunting fortroll comments in news community forums. In
Pro-ceedings of the 54th Annual Meeting of the Associa-tion for Computational Linguistics , ACL ’16, pages399–405, Berlin, Germany.Tsvetomila Mihaylova, Preslav Nakov, Llu´ıs M`arquez,Alberto Barr´on-Cede˜no, Mitra Mohtarami, GeorgiKaradjov, and James Glass. 2018. Fact checking incommunity forums. In
Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence ,AAAI ’18, pages 879–886, New Orleans, LA, USA.Subhabrata Mukherjee and Gerhard Weikum. 2015.Leveraging joint interactions for credibility analy-sis in news communities. In
Proceedings of the24th ACM International on Conference on Informa-tion and Knowledge Management , CIKM ’15, pages353–362, Melbourne, Australia.An T. Nguyen, Aditya Kharosekar, Matthew Lease,and Byron C. Wallace. 2018. An interpretable jointgraphical model for fact-checking from crowds. In
Proceedings of the Thirty-Second AAAI Conferenceon Artificial Intelligence , AAAI ’18, New Orleans,LA, USA.Jeff Z. Pan, Siyana Pavlova, Chenxi Li, Ningxi Li,Yangmei Li, and Jinshuo Liu. 2018. Content basedfake news detection using knowledge graphs. In
Proceedings of the International Semantic Web Con-ference , ISWC ’18, Monterey, CA, USA.Ver´onica P´erez-Rosas, Bennett Kleinberg, AlexandraLefevre, and Rada Mihalcea. 2018. Automatic de-tection of fake news. In
Proceedings of the 27th In-ternational Conference on Computational Linguis-tics , COLING ’18, pages 3391–3401, Santa Fe, NM,USA.Evaggelia Pitoura, Panayiotis Tsaparas, GiorgosFlouris, Irini Fundulaki, Panagiotis Papadakos,Serge Abiteboul, and Gerhard Weikum. 2018. Onmeasuring bias in online information.
SIGMODRec. , 46(4):16–21.Kashyap Popat, Subhabrata Mukherjee, JannikStr¨otgen, and Gerhard Weikum. 2016. Credi-bility assessment of textual claims on the web.In
Proceedings of the 25th ACM Internationalon Conference on Information and KnowledgeManagement , CIKM ’16, pages 2173–2178,Indianapolis, IN, USA. Kashyap Popat, Subhabrata Mukherjee, JannikStr¨otgen, and Gerhard Weikum. 2017. Where thetruth lies: Explaining the credibility of emergingclaims on the Web and social media. In
Proceedingsof the 26th International Conference on World WideWeb Companion , WWW ’17, pages 1003–1012,Perth, Australia.Kashyap Popat, Subhabrata Mukherjee, JannikStr¨otgen, and Gerhard Weikum. 2018. CredEye: Acredibility lens for analyzing and explaining misin-formation. In
Proceedings of The Web Conference2018 , WWW ’18, pages 155–158, Lyon, France.Martin Potthast, Johannes Kiesel, Kevin Reinartz,Janek Bevendorff, and Benno Stein. 2018. A stylo-metric inquiry into hyperpartisan and fake news. In
Proceedings of the 56th Annual Meeting of the As-sociation for Computational Linguistics , ACL ’18,pages 231–240, Melbourne, Australia.Daniel Preot¸iuc-Pietro, Ye Liu, Daniel Hopkins, andLyle Ungar. 2017. Beyond binary labels: Politicalideology prediction of Twitter users. In
Proceed-ings of the 55th Annual Meeting of the Associationfor Computational Linguistics (Volume 1: Long Pa-pers) , ACL ’17, pages 729–740, Vancouver, Canada.Hannah Rashkin, Eunsol Choi, Jin Yea Jang, SvitlanaVolkova, and Yejin Choi. 2017. Truth of varyingshades: Analyzing language in fake news and polit-ical fact-checking. In
Proceedings of the 2017 Con-ference on Empirical Methods in Natural LanguageProcessing , EMNLP ’17, pages 2931–2937, Copen-hagen, Denmark.Sara Rosenthal, Noura Farra, and Preslav Nakov.2017a. SemEval-2017 task 4: Sentiment analysisin Twitter. In
Proceedings of the 11th InternationalWorkshop on Semantic Evaluation , SemEval ’17,pages 502–518, Vancouver, Canada.Sara Rosenthal, Noura Farra, and Preslav Nakov.2017b. SemEval-2017 task 4: Sentiment analysisin Twitter. In
Proceedings of the 11th InternationalWorkshop on Semantic Evaluation , SemEval ’17,pages 502–518, Vancouver, Canada.Yanchuan Sim, Brice D. L. Acree, Justin H. Gross, andNoah A. Smith. 2013. Measuring ideological pro-portions in political speeches. In
Proceedings of the2013 Conference on Empirical Methods in NaturalLanguage Processing , EMNLP ’13, pages 91–101,Seattle, WA, USA.Margit Tavits and Natalia Letki. 2009. When left isright: Party ideology and policy in Post-CommunistEurope.
The American Political Science Review ,103(4):555–569.Soroush Vosoughi, Deb Roy, and Sinan Aral. 2018.The spread of true and false news online.
Science ,359(6380):1146–1151.obert Walecki, Ognjen Rudovic, Vladimir Pavlovic,and Maja Pantic. 2016. Copula ordinal regressionfor joint estimation of facial action unit intensity. In
Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition , pages 4902–4910.Shipeng Yu, Kai Yu, Volker Tresp, and Hans-PeterKriegel. 2006. Collaborative ordinal regression. In