Defending Against Neural Fake News
Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, Yejin Choi
DDefending Against Neural Fake News
Rowan Zellers ♠ , Ari Holtzman ♠ , Hannah Rashkin ♠ , Yonatan Bisk ♠ Ali Farhadi ♠♥ , Franziska Roesner ♠ , Yejin Choi ♠♥♠ Paul G. Allen School of Computer Science & Engineering, University of Washington ♥ Allen Institute for Artificial Intelligence https://rowanzellers.com/grover
Abstract
Recent progress in natural language generation has raised dual-use concerns. Whileapplications like summarization and translation are positive, the underlying tech-nology also might enable adversaries to generate neural fake news : targeted propa-ganda that closely mimics the style of real news.Modern computer security relies on careful threat modeling : identifying potentialthreats and vulnerabilities from an adversary’s point of view, and exploring potentialmitigations to these threats. Likewise, developing robust defenses against neuralfake news requires us first to carefully investigate and characterize the risks of thesemodels. We thus present a model for controllable text generation called G rover .Given a headline like ‘Link Found Between Vaccines and Autism,’ G rover cangenerate the rest of the article; humans find these generations to be more trustworthythan human-written disinformation.Developing robust verification techniques against generators like G rover is critical.We find that best current discriminators can classify neural fake news from real,human-written, news with 73% accuracy, assuming access to a moderate level oftraining data. Counterintuitively, the best defense against G rover turns out to beG rover itself, with 92% accuracy, demonstrating the importance of public releaseof strong generators. We investigate these results further, showing that exposurebias – and sampling strategies that alleviate its e ff ects – both leave artifacts thatsimilar discriminators can pick up on. We conclude by discussing ethical issuesregarding the technology, and plan to release G rover publicly, helping pave theway for better detection of neural fake news. Online fake news – news designed to intentionally deceive – has recently emerged as a majorsocietal problem. Malicious actors spread fallacious viral stories in order to gain advertising revenue,influence opinions, and even tip elections (Faris et al., 2017; Wardle and Derakhshan, 2017). As such,countering the spread of disinformation online presents an urgent technical and political issue.To the best of our knowledge, most disinformation online today is manually written (Vargo et al., 2018).However, as progress continues in natural language generation, malicious actors will increasingly be
The Newest York Times
SCIENCE
SUBSCRIBE NOW LOG IN
Link Found Between Vaccines and Autism
By Paul Waldman
May 29, 2019
Those who have been vaccinated against measles have a more than 5-fold higher chance of developing autism, researchers at the University of California San Diego School of Medicine and the Centers for Disease Control and Prevention report today in the Journal of Epidemiology and Community Health. (continued) f a k e
Fake news!
News VerificationFake NewsGeneration
Figure 1: In this paper, we explore G rover , a model which can detect and generate neural fake news.Humans find the articles di ffi cult to distinguish from “real news” without high levels of scrutiny. a r X i v : . [ c s . C L ] O c t ble to controllably generate realistic-looking propaganda at scale. Thus, while we are excited aboutrecent progress in text generation (Józefowicz et al., 2016; Radford et al., 2018; 2019), we are alsoconcerned with the inevitability of AI-generated ‘neural’ fake news. With this paper, we seek to understand and respond to neural fake news before it manifests at scale.We draw on the field of computer security, which relies on threat modeling : analyzing the space ofpotential threats and vulnerabilities in a system to develop robust defenses. To scientifically study therisks of neural disinformation, we present a new generative model called G rover . Our model allowsfor controllable yet e ffi cient generation of an entire news article – not just the body, but also thetitle, news source, publication date, and author list. This lets us study an adversary with controllablegenerations (e.g. Figure 1, an example anti-vaccine article written in the style of the New YorkTimes).Humans rate the disinformation generated by G rover as trustworthy, even more so than human-written disinformation. Thus, developing robust verification techniques against generators such asG rover is an important research area. We consider a setting in which a discriminator has accessto 5000 G rover generations, but unlimited access to real news. In this setting, the best existingfake news discriminators are, themselves, deep pretrained language models (73% accuracy) (Peterset al., 2018; Radford et al., 2018; 2019; Devlin et al., 2018). However, we find that G rover , whenused in a discriminative setting, performs even better at 92% accuracy. This finding represents anexciting opportunity for defense against neural fake news: the best models for generating neuraldisinformation are also the best models at detecting it.Next, we investigate how deep pretrained language models distinguish between real and machine-generated text. We find that key artifacts are introduced during generation as a result of exposure bias:the generator is not perfect, so randomly sampling from its distribution results in generations that fallincreasingly out-of-distribution as length increases. However, sampling strategies that alleviate thesee ff ects also introduce artifacts that strong discriminators can pick up on.We conclude with a sketch of the ethical territory that must be mapped out in order to understand ourresponsibilities as researchers when studying fake news, and the potential negative implications ofreleasing models (Hecht et al., 2018; Zellers, 2019; Solaiman et al., 2019). Accordingly, we suggesta provisional policy of how such models should be released and why we believe it to be safe – andperhaps even imperative – to do so. We believe our proposed framework and accompanying modelsprovide a concrete initial proposal for an evolving conversation about ML-based disinformationthreats and how they can be countered. We present a framework – motivated by today’s dynamics of manually created fake news – forunderstanding what adversaries will attempt with deep models, and how verifiers should respond.
Scope of fake news.
There are many types of false news, ranging from satire to propaganda(Wardle, 2017). In this paper, we focus on text-only documents formatted as news articles: storiesand their corresponding metadata that contain purposefully false information. Existing fake news ispredominantly human-written, for two broad goals: monetization (ad revenue through clicks) andpropaganda (communicating targeted information) (Bradshaw and Howard, 2017; Melford and Fagan,2019). Achieving either goal requires the adversary to be selective about the news that they make,whether by producing only viral content, or content that advances a given agenda.
Fact checking and verification: related work.
There is considerable interest in fighting onlinedisinformation. Major platforms such as Facebook prioritize trustworthy sources and shut downaccounts linked to disinformation (Mosseri, 2018; Dwoskin and Romm, 2018). Some users ofthese platforms avoid fake news with tools such as NewsGuard and Hoaxy (Shao et al., 2016) andwebsites like Snopes and PolitiFact. These services rely on manual fact-checking e ff orts: verifyingthe accuracy of claims, articles, and entire websites. E ff orts to automate fake news detection generallypoint out stylistic biases that exist in the text (Rashkin et al., 2017; Wang, 2017; Pérez-Rosas et al., We thank past work, such as OpenAI’s Staged Release Policy for GPT2 for drawing attention to neuraldisinformation, alongside other dual-use implications. Short for G enerating a R ticles by O nly V iewing m E tadata R ecords. ff orts can help moderators on social media platforms shut down suspicious accounts.However, fact checking is not a panacea – cognitive biases such as the backfire e ff ect and confirmationbias make humans liable to believe fake news that fits their worldview (Swire et al., 2017). Framework.
We cast fake news generation and detection as an adversarial game, with two players: • Adversary . Their goal is to generate fake stories that match specified attributes: generally, beingviral or persuasive. The stories must read realistically to both human users as well as the verifier. • Verifier . Their goal is to classify news stories as real or fake. The verifier has access to unlimitedreal news stories, but few fake news stories from a specific adversary. This setup matches theexisting landscape: when a platform blocks an account or website, their disinformative storiesprovide training for the verifier; but it is di ffi cult to collect fake news from newly-created accounts.The dual objectives of these two players suggest an escalating “arms race” between attackers anddefenders. As verification systems get better, so too will adversaries. We must therefore be preparedto deal with ever-stronger adversarial attacks, which is the focus of the next section. rover : Modeling Conditional Generation of Neural Fake News Given existing online disinformation, we have reason to believe adversaries will try to generatetargeted content (e.g. clickbait and propaganda). Recently introduced large-scale generative modelsproduce realistic-looking text (Radford et al., 2019), but they do not lend themselves to producingcontrollable generations (Hu et al., 2017). Therefore, to probe the feasibility of realistic-lookingneural fake news, we introduce G rover , which produces both realistic and controlled generations.The current state-of-the-art in unconditional text generation views it as a language modeling problem(Bengio et al., 2003), in which the probability of a document x is the product of the conditionalprobability of generating each token x i given previous tokens: p p x q “ N ź i “ p p x i | x . . . x i ´ q . (1)The document is typically treated as a single unstructured text field , beginning with a
May 29, 2019 date authors headline body wired.com New Research Shows that Vaccines Cause Autism
New research from the University of California, Davis, finds that childhood … domain
May 29, 2019 date headline body
Justin Furillo authors wired.com Justin Furillo
New research from the University of California, Davis, finds that childhood … domain
May 29, 2019 date bodyauthors
Vaccines Might Be a Bigger Threat to Your Child's Future Than You Realized headline a)b)c)
Figure 2: A diagram of three G rover examples for article generation. In row a) , the body is generatedfrom partial context (the authors field is missing). In b) , the model generates the authors. In c) , themodel uses the new generations to regenerate the provided headline to one that is more realistic.Figure 2 shows an example of using G rover to generate an anti-vaccine article. Here, the adversaryspecifies a domain, date, and headline. After G rover generates the body, it can be used to generate afake author, before finally generating a new and more appropriate headline.During training, we simulate inference by randomly partitioning an article’s fields into two disjointsets F and F . We also randomly drop out individual fields with probability 10%, and drop out allbut the body with probability 35%. This allows the model to learn how to perform unconditionalgeneration. We sort the metadata fields in each set using our standard order, and concatenate theunderlying tokens. The model is then trained to minimize the cross-entropy of predicting the tokensin F followed by the tokens in F . Architecture.
We draw on recent progress in training large Transformers for language modeling(Vaswani et al., 2017), building G rover using the same architecture as for GPT2 (Radford et al.,2019). We consider three model sizes. Our smallest model, G rover -Base, has 12 layers and 124million parameters, on par with GPT and BERT-Base (Radford et al., 2018; Devlin et al., 2018). Ournext model, G rover -Large, has 24 layers and 355 million parameters, on par with BERT-Large. Ourlargest model, G rover -Mega, has 48 layers and 1.5 billion parameters, on par with GPT2.
Dataset.
We present R eal N ews , a large corpus of news articles from Common Crawl. TrainingG rover requires a large corpus of news articles with metadata, but none currently exists. Thus, weconstruct one by scraping dumps from Common Crawl, limiting ourselves to the 5000 news domainsindexed by Google News. We used the Newspaper Python library to extract the body and meta-data from each article. News from Common Crawl dumps from December 2016 through March 2019were used as training data; articles published in April 2019 from the April 2019 dump were used forevaluation. After deduplication, R eal N ews is 120 gigabytes without compression. Learning.
We trained each G rover model on randomly-sampled sequences from R eal N ews withlength 1024. Other optimization hyperparameters are in Appendix A. We trained G rover -Mega for800k iterations, using a batch size of 512 and 256 TPU v3 cores. Training time was two weeks. We validate G rover , versus standard unconditional language models, on the April 2019 test set. Weconsider two evaluation modes: unconditional , where no context is provided and the model mustgenerate the article body; and conditional , in which the full metadata is provided as context. In bothcases, we calculate the perplexity only over the article body.Our results, shown in Figure 3, show several conclusions. First, G rover noticeably improves (between.6 to .9 perplexity points) when conditioned on metadata. Second, perplexity decreases with size,with G rover -Mega obtaining 8.7 perplexity in the conditional setting. Third, the data distribution isstill important: though the GPT2 models with 124M parameters and 355M parameters respectivelymatch our G rover -Base and G rover -Large architectures, our model is over 5 perplexity points lowerin both cases, possibly because the OpenAI WebText corpus also contains non-news articles. All tokens use the same vocabulary. By using a standard order, but partitioning the fields into two sets, themodel can generate any field conditioned on others while only needing to learn 2 | F | orderings, versus | F | !. nconditional Conditional0510152025 P e r p l e x i t y GPT2 (124M)GPT2 (355M) Grover-Base (124M)Grover-Large (355M)Grover-Mega (1.5B)
Figure 3: Language Modeling results on thebody field of April 2019 articles. We evaluatein the
Unconditional setting (without providedmetadata) as well as in the
Conditional setting(with all metadata). G rover sees over a 0.6 pointdrop in perplexity when given metadata.
HumanNews MachineNews HumanPropaganda MachinePropaganda1.0(worst)1.52.02.53.0(best)
StyleContentOverall
Figure 4: Human evaluation. For each article,three annotators evaluated style, content, andthe overall trustworthiness; 100 articles of eachcategory were used. The results show that propa-ganda generated by G rover is rated more plausi-ble than the original human-written propaganda.
Sampling from G rover is straightforward as it behaves like a left-to-right language model duringdecoding. However, the choice of decoding algorithm is important. While likelihood-maximizationstrategies such as beam search work well for closed-ended generation tasks where the output containsthe same information as the context (like machine translation), these approaches have been shownto produce degenerate text during open-ended generation (Hashimoto et al., 2019; Holtzman et al.,2019). However, as we will show in Section 6, restricting the variance of generations is also crucial.In this paper, we primarily use Nucleus Sampling (top- p ): for a given threshold p , at each timestepwe sample from the most probable words whose cumulative probability comprises the top- p % of theentire vocabulary (Holtzman et al., 2019). rover -written Propaganda We evaluate the quality of disinformation generated by our largest model, G rover -Mega, using p “ . HumanNews ), G rover -written articles conditioned on the same metadata (
Machine News ), human-written arti-cles from known propaganda websites (
Human Propaganda ), and G rover -written articles conditionedon the propaganda metadata (
Machine Propaganda ). The domains used are in Appendix B; examplesare in Appendix F. We asked a pool of qualified workers on Amazon Mechanical Turk to rate eacharticle on three dimensions: stylistic consistency, content sensibility, and overall trustworthiness. Results (Figure 4) show a striking trend: though the quality of G rover -written news is not as highas human-written news, it is adept at rewriting propaganda. The overall trustworthiness score ofpropaganda increases from 2.19 to 2.42 (out of 3) when rewritten by G rover . In early experiments, we found Nucleus Sampling produced better and less-detectable generations thanalternatives like top- k sampling, wherein the most probable k tokens are used at each timestep (Fan et al., 2018). We use the technique described in Figure 2 to rewrite the propaganda: given the metadata, generate thearticle first, and then rewrite the headline. With these guidelines, we tried to separate style versus content. Overall trustworthiness asks ‘Does thearticle read like it comes from a trustworthy source?’ which emphasizes style, while content sensibility askswhether the content is believable on a semantic level. This di ff erence is statistically significant at p “ .
01. One possible hypothesis for this e ff ect is thatG rover ignores the provided context. To test this hypothesis, we did a human evaluation of the consistencyof the article body with the headline, date, and author. We found that human-written propaganda articles areconsistent with the headline with an average score of 2.85 of 3 on the same 1-3 scale, while machine-writtenpropaganda is consistent with 2.64 of 3. Neural Fake News Detection
The high quality of neural fake news written by G rover , as judged by humans, makes automatic neuralfake news detection an important research area. Using models (below) for the role of the
Verifier can mitigate the harm of neural fake news by classifying articles as
Human or Machine written. Thesedecisions can assist content moderators and end users in identifying likely (neural) disinformation. a . G rover . We consider a version of our model adapted for discrimination. Similar to GPT (Radfordet al., 2018), we place a special [CLS] token at the end of each article, and extract the final hiddenstate at that point. The hidden state is fed to a linear layer to predict the label Human or Machine .To simulate real conditions, and ensure minimal overlap between the generator and discriminatorparameters, we initialize G rover for discrimination using the checkpoint at iteration 700k, whereasthe generator uses the checkpoint at iteration 800k. b . GPT2, a 124M or 355M parameter pretrained Transformer language model. Similar to G rover ,we follow the GPT approach and extract the hidden state from a newly-added [CLS] token. c . BERT, a 110M parameter (BERT-Base) or 340M parameter (BERT-Large) bidirectional Trans-former encoder commonly used for discriminative tasks. We perform domain adaptation to adaptBERT to the news domain, as well as to account for long articles; details in Appendix C. d . FastText, an o ff -the-shelf library for bag-of-ngram text classification (Joulin et al., 2017). Thoughnot pretrained, similar models do well at detecting human-written fake news.All models are trained to minimize the cross-entropy loss of predicting the right label. Hyperparame-ters used during discrimination are in Appendix D. While there are many human-written articles online, most are from the distant past, whereas articles tobe detected will likely be set in the present. Likewise, there might be relatively few neural fake newsarticles from a given adversary. We thus frame neural fake news detection as a semi-supervisedproblem. A neural verifier (or discriminator ) has access to many human-written news articlesfrom March 2019 and before – the entire R eal N ews training set. However, it has limited access togenerations, and more recent news articles. Using 10k news articles from April 2019, we generatearticle body text; another 10k articles are used as a set of human-written news articles. We split thearticles in a balanced way, with 10k for training (5k per label), 2k for validation, and 8k for testing.We consider two evaluation modes. In the unpaired setting, a discriminator is provided singlenews articles, and must classify each independently as Human or Machine . In the paired setting,a model is given two news articles with the same metadata, one real and one machine-generated.The discriminator must assign the machine-written article a higher
Machine probability than thehuman-written article. We evaluate both modes in terms of accuracy. rover performs best at detecting G rover ’s fake news
We present experimental results in Table 1 for all generator and discriminator combinations. Foreach pair, we show the test results using the most adversarial generation hyperparameters (top- p ) asjudged on the validation set. The results show several trends. First, the paired setting appears mucheasier than the unpaired setting, suggesting that it is di ffi cult for the model to calibrate its predictions.Second, model size is highly important in the arms race between generators and discriminators. UsingG rover to discriminate G rover ’s generations results in roughly 90% accuracy across the range ofsizes. If a larger generator is used, accuracy slips below 81%; conversely, if the discriminator islarger, accuracy is above 98%. Third, other discriminators perform worse than G rover overall, evenwhen controlling for architecture size and (for both BERT models) the domain.That G rover is the best discriminator is possibly surprising: being unidirectional, it is less expressivethan deep bidirectional models such as BERT. That the more expressive model here is not the best at Moreover, since disinformation can be shared on a heterogeneous mix of platforms, it might be challengingto pin down a single generated model. For each discriminator / generator pair, we search over p P t . , . , . , . , . , . u . Indeed, bidirectional approaches perform best on leaderboards like GLUE (Wang et al., 2018). lowest validation accuracy. Compared with othermodels such as BERT, G rover is the best at de-tecting its own generations as neural fake news.
Unpaired Accuracy Paired AccuracyGenerator size Generator size1.5B 355M 124M 1.5B 355M 124MChance 50.0 50.0 D i s c r i m i n a t o r s i ze rover -Mega rover -Large BERT-Large 73.1 75.9 97.5 84.1 91.5 99.9GPT2 70.1 78.0 90.3 78.8 87.0 96.8124M G rover -Base
GPT2 66.2 71.9 83.5 72.5 79.6 89.611M FastText 63.8 65.6 69.7 65.9 69.0 74.4
16 64 256 1024 4096 G r o v e r- M ega U npa i r ed A cc u r a cy No weak supervision+Grover-Base Geneations+Grover-Large Generations
Figure 5: Exploring weak supervision for dis-criminating G rover -Mega generations. Withno weak supervision, the discriminator sees x machine-written articles (from G rover Mega).For ` G rover -Base and ` G rover -Mega, the dis-criminator sees 5000 ´ x machine-written articlesgiven by the weaker generator in question. See-ing weaker generations improves performancewhen few in-domain samples are given.discriminating between real and generated news articles suggests that neural fake news discriminationrequires having a similar inductive bias as the generator. rover -Mega? These results suggest that G rover is an e ff ective discriminator when we have a medium number offake news examples from the exact adversary that we will encounter at test time. What happens if werelax this assumption? Here, we consider the problem of detecting an adversary who is generatingnews with G rover -Mega and an unknown top- p threshold. In this setup, during training, we haveaccess to a weaker model (G rover -Base or G rover -Large). We consider the e ff ect of having only x examples from G rover -Mega, and sampling the missing 5000 ´ x articles from one of the weakermodels, where the top-p threshold is uniformly chosen for each article in the range of r . , . s .We show the results of this experiment in Figure 5. The results suggest that observing additionalgenerations greatly helps discrimination performance when few examples of G rover -Mega areavailable: weak supervision with between 16 and 256 examples from G rover -Large yields around78% accuracy, while accuracy remains around 50% without weak supervision. As the portion ofexamples that come from G rover -Mega increases, however, accuracy converges to 92%. In this section, we explore why G rover performs best at detecting fake news generated by otherG rover models. We find that there is a double-bind between exposure bias and variance-reduction algorithms that alleviate these biases while at the same time creating other artifacts.
Exposure Bias.
Models maximizing Equation 1 are trained only conditioned on human-writtentext, never on its own generations, creating a problem known as exposure bias (Ranzato et al., 2016).We investigate the importance of exposure bias towards creating artifacts. In Figure 6 we plot theperplexities given by G rover -Mega over each position for body text at top- p thresholds of 0 .
96, but we are not supposed to know this! In additional experiments we show that accuracy increases even more – up to 98% – when the number ofexamples is increased (Zellers et al., 2019c). We also find that G rover when trained to discriminate between realand fake G rover -generated news can detect GPT2-Mega generated news as fake with 96% accuracy.
50 100 150 200 250 300 350 400Position5101520 P o s i t i on w i s e pe r p l e x i t y p =1.00 p =0.96Human Figure 6: Perplexities of G rover -Mega, averaged overeach position in the body (after conditioning on meta-data). We consider human-written with G rover -Megagenerated text at p “ p “ . p U npa i r ed A cc u r a cy ( v a li da t i on ) DiscriminatorGrover-Mega (1.5B)Grover-Large (355M)BERT-Large (340M)
Figure 7: Unpaired validation accuracy,telling apart generated news articles (fromG rover
Mega) from real articles, at di ff er-ent variance reduction thresholds p (forNucleus Sampling). Results varying p show a sweet spot ( p “ .
92 – 0 . rover to fall increasingly out of the distribution of human language.However, limiting the variance ( p “ .
96) lowers the resulting perplexity and limits its growth.
Limiting the variance of a model also creates artifacts
On the other hand, clipping the model’svariance also leaves an artifact, as prior work has observed for top- k sampling (Strobelt and Gehrmann,2019). A similar phenomenon holds for Nucleus (top- p ) sampling. The probability of observing ahuman-written article where all tokens are drawn from the top- p % of the distribution is p n , where n is the document’s length. This probability goes to zero as n increases. However, for Nucleus Sampledtext – in which the final 1 ´ p is cut o ff – all tokens come from the top- p .The visibility of the artifacts depends on the choice of discriminator. The top- p at each timestepis calculated under the generator’s worldview, meaning that if the discriminator models text in adi ff erent way, it might have a harder time pinpointing the empty 1 ´ p tail. This could explain BERT’slower performance during discrimination. A sweet spot of careful variance reduction
Not reducing the variance, as well as significantlyreducing the variance, both cause problems. Might there be a sweet spot for how much to truncatethe variance, to make discrimination maximally hard? In Figure 7, we show results varying thetop- p threshold for the discrimination task applied to G rover -Mega’s generations. The results indeedshow a sweet spot, roughly between p “ .
92 and p “ .
98 depending on the discriminator, whereindiscrimination is hardest. Interestingly, we note that the most adversarial top- p threshold for BERT-Large is considerably lower than the corresponding top- p for G rover -Large of the same size. Thissupports our hypothesis that BERT’s view of language di ff ers markedly from G rover ; using a lowertop- p threshold does not seem to give it much more information about the missing tail. Overall , our analysis suggests that G rover might be the best at catching G rover because it is thebest at knowing where the tail is, and thus whether it was truncated. rover
This paper investigates the threats posed by adversaries seeking to spread disinformation. Our sketchof what these threats might look like – a controllable language model named G rover – suggests thatthese threats are real and dangerous. G rover can rewrite propaganda articles, with humans rating therewritten versions as more trustworthy. At the same time, there are defenses to these models – notably,in the form of G rover itself. We conclude with a discussion of next steps and ethical considerations.8 he Era of Neural Disinformation.
Though training G rover was challenging, it is easily achiev-able by real-world adversaries today. Obtaining the data required through Common Crawl cost$10k in AWS credits and can be massively parallelized over many CPUs. Training G rover -Mega isrelatively inexpensive: at a cost of $0.30 per TPU v3 core-hour and two weeks of training, the totalcost is $25k. Spending more money and engineering time could yield even more powerful generators.
Release of generators is critical.
At first, it would seem like keeping models like G rover privatewould make us safer. However, G rover serves as an e ff ective detector of neural fake news, evenwhen the generator is much larger (Section 5). If generators are kept private, then there will be littlerecourse against adversarial attacks. We thus released our models to researchers (Zellers, 2019). Future of progress in generation.
Models like BERT are strong discriminators for many NLPtasks, but they are not as good at detecting G rover ’s generations as left-to-right models like G rover ,even after domain adaptation. One hypothesis is that the artifacts shown in Section 6 are most visibleto a left-to-right discriminator. This also suggests that recent progress on generating text in any order(Gu et al., 2019; Stern et al., 2019; Ghazvininejad et al., 2019) may lead to models that evade aG rover discriminator. Likewise, models that are trained conditioned on their own predictions mightavoid exposure bias, however, these objectives often lead to low performance on language tasks(Caccia et al., 2018). One additional possibility is the use of Adversarial Filtering (Zellers et al., 2018;2019b) to oversample and then select a subset of generations. However, we found this didn’t workwell for very long sequences (up to 1024 BPE tokens), possibly as these are far from the ‘GoldilocksZone’ wherein discrimination is hard for machines.
Additional threat models.
In this paper, we studied the threat model whereby an adversary gener-ates an entire news article from scratch, given minimal context. Other threat models are possible: forinstance, an adversary might generate comments or have entire dialogue agents, they might start witha human-written news article and modify a few sentences, and they might fabricate images or video.These threat models ought to be studied by researchers also so that we can create better defenses.
Machine-generated real news?
Our study focused on detecting machine-written fake news,though the same G rover approach can be used for spotting human-written fake news as well (Zellerset al., 2019c). However, machines can also generate truthful news using templated systems. Domainswith templated news articles exist in our dataset, and are easy for G rover to spoof convincingly. Future of progress in discrimination.
Our discriminators are e ff ective, but they primarily leveragedistributional features rather than evidence. In contrast, humans assess whether an article is truthfulby relying on a model of the world, assessing whether the evidence in the article matches thatmodel. Future work should investigate integrating knowledge into the discriminator (e.g. for claimverification in FEVER; Thorne et al., 2018). An open question is to scale progress in this task towardsentire news articles, and without paired evidence (similar to open-domain QA; Chen et al., 2017). What should platforms do?
Video-sharing platforms like YouTube use deep neural networks toscan videos while they are uploaded, to filter out content like pornography (Hosseini et al., 2017).We suggest platforms do the same for news articles. An ensemble of deep generative models, such asG rover , can analyze the content of text – together with more shallow models that predict human-written disinformation. However, humans must still be in the loop due to dangers of flagging realnews as machine-generated, and possible unwanted social biases of these models.
Acknowledgments
We thank the anonymous reviewers, as well as Dan Weld, for their helpful feedback. Thanks also toZak Stone and the Google Cloud TPU team for help with the computing infrastructure. This workwas supported by the National Science Foundation through a Graduate Research Fellowship (DGE-1256082) and NSF grants (IIS-1524371, 1637479, 165205, 1703166), the DARPA CwC programthrough ARO (W911NF-15-1-0543), the Sloan Research Foundation through a Sloan Fellowship, theAllen Institute for Artificial Intelligence, the NVIDIA Artificial Intelligence Lab, Samsung through aSamsung AI research grant, and gifts by Google and Facebook. Computations on beaker.org weresupported in part by credits from Google Cloud. An example is https://americanbankingnews.com . eferences Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilisticlanguage model.
Journal of machine learning research , 3(Feb):1137–1155, 2003.Samantha Bradshaw and Philip Howard. Troops, trolls and troublemakers: A global inventory oforganized social media manipulation. Technical report, Oxford Internet Institute, 2017.Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Charlin.Language gans falling short. arXiv preprint arXiv:1811.02549 , 2018.Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading wikipedia to answer open-domain questions. In
Proceedings of the 55th Annual Meeting of the Association for ComputationalLinguistics (Volume 1: Long Papers) , pages 1870–1879, 2017.Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deepbidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.Rachel Dicker. Avoid These Fake News Sites at All Costs. , 2016. [Online; accessed 22-May-2019].Elizabeth Dwoskin and Tony Romm. Facebook says it has uncovered a coordinated disinformationoperation ahead of the 2018 midterm elections.
The Washington Post , 2018.Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In
Proceedingsof the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: LongPapers) , pages 889–898, 2018.Robert Faris, Hal Roberts, Bruce Etling, Nikki Bourassa, Ethan Zuckerman, and Yochai Benkler.Partisanship, propaganda, and disinformation: Online media and the 2016 us presidential election.
Berkman Klein Center Research Publication 2017-6. , 2017.Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Constant-time machinetranslation with conditional masked language models. arXiv preprint arXiv:1904.09324 , 2019.Jiatao Gu, Qi Liu, and Kyunghyun Cho. Insertion-based decoding with automatically inferredgeneration order. arXiv preprint arXiv:1902.01370 , 2019.Xiaochuang Han and Jacob Eisenstein. Unsupervised domain adaptation of contextualized embed-dings: A case study in early modern english. arXiv preprint arXiv:1904.02817 , 2019.Tatsunori B Hashimoto, Hugh Zhang, and Percy Liang. Unifying human and statistical evaluation fornatural language generation. arXiv preprint arXiv:1904.02792 , 2019.Brent Hecht, Lauren Wilcox, Je ff rey P. Bigham, Johannes Schöning, Ehsan Hoque, Jason Ernnst,Yonatan Bisk, Luigi De Russis, Lana Yarosh, Bushra Anjum, Danish Contractor, and Cathy Wu.It’s time to do something: Mitigating the negative impacts of computing through a change to thepeer review process. ACM Future of Computing Blog , 2018.Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. The curious case of neural text degenera-tion. arXiv preprint arXiv:1904.09751 , 2019.Hossein Hosseini, Baicen Xiao, Andrew Clark, and Radha Poovendran. Attacking automatic videoanalysis algorithms: A case study of google cloud video intelligence api. In
Proceedings of the2017 on Multimedia Privacy and Security , pages 21–32. ACM, 2017.Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. Toward controlledgeneration of text. In
Proceedings of the 34th International Conference on Machine Learning-Volume 70 , pages 1587–1596. JMLR. org, 2017.Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for e ffi cient textclassification. In Proceedings of the 15th Conference of the European Chapter of the Associationfor Computational Linguistics: Volume 2, Short Papers , volume 2, pages 427–431, 2017.10afal Józefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring thelimits of language modeling.
CoRR , abs / CoRR ,abs / FacebookNewsroom , 19, 2018.Myle Ott, Yejin Choi, Claire Cardie, and Je ff rey T Hancock. Finding deceptive opinion spam byany stretch of the imagination. In Proceedings of the 49th annual meeting of the association forcomputational linguistics: Human language technologies-volume 1 , pages 309–319. Associationfor Computational Linguistics, 2011.Verónica Pérez-Rosas, Bennett Kleinberg, Alexandra Lefevre, and Rada Mihalcea. Automaticdetection of fake news. In
Proceedings of the 27th International Conference on ComputationalLinguistics , pages 3391–3401, 2018.Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, andLuke Zettlemoyer. Deep contextualized word representations. In
Proceedings of the 2018 Confer-ence of the North American Chapter of the Association for Computational Linguistics: HumanLanguage Technologies, Volume 1 (Long Papers) , volume 1, pages 2227–2237, 2018.Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving languageunderstanding by generative pre-training. Technical report, OpenAI, 2018. URL https://blog.openai.com/language-unsupervised/ .Alec Radford, Je ff rey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Languagemodels are unsupervised multitask learners. Technical report, OpenAI, 2019.Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level trainingwith recurrent neural networks. In ICLR . ICLR, 2016.Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. Truth of varyingshades: Analyzing language in fake news and political fact-checking. In
Proceedings of the 2017Conference on Empirical Methods in Natural Language Processing , pages 2931–2937, 2017.Chengcheng Shao, Giovanni Luca Ciampaglia, Alessandro Flammini, and Filippo Menczer. Hoaxy:A platform for tracking online misinformation. In
Proceedings of the 25th international conferencecompanion on world wide web , pages 745–750. International World Wide Web ConferencesSteering Committee, 2016.Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost.In
International Conference on Machine Learning , pages 4603–4611, 2018.Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Je ff Wu, AlecRadford, and Jasmine Wang. Release strategies and the social impacts of language models. arXivpreprint arXiv:1908.09203 , 2019.Mitchell Stern, William Chan, Jamie Kiros, and Jakob Uszkoreit. Insertion transformer: Flexiblesequence generation via insertion operations. arXiv preprint arXiv:1902.03249 , 2019.Hendrik Strobelt and Sebastian Gehrmann. Catching a unicorn with gltr: A tool to detect automaticallygenerated text. Technical report, Harvard, 2019.Briony Swire, Ullrich KH Ecker, and Stephan Lewandowsky. The role of familiarity in correctinginaccurate information.
Journal of experimental psychology: learning, memory, and cognition , 43(12):1948, 2017. 11ames Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. Fever: a large-scale dataset for fact extraction and verification. In
Proceedings of the 2018 Conference of theNorth American Chapter of the Association for Computational Linguistics: Human LanguageTechnologies, Volume 1 (Long Papers) , pages 809–819, 2018.Chris J Vargo, Lei Guo, and Michelle A Amazeen. The agenda-setting power of fake news: A bigdata analysis of the online media landscape from 2014 to 2016.
New Media & Society , 20(5):2028–2049, 2018.Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ŁukaszKaiser, and Illia Polosukhin. Attention is all you need. In
Proceedings of the 31st InternationalConference on Neural Information Processing Systems , pages 6000–6010. Curran Associates Inc.,2017.Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue:A multi-task benchmark and analysis platform for natural language understanding. arXiv preprintarXiv:1804.07461 , 2018.William Yang Wang. “liar, liar pants on fire”: A new benchmark dataset for fake news detection. In
Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume2: Short Papers) , pages 422–426, 2017.Claire Wardle. Fake news. it’s complicated.
First Draft News , 16, 2017.Claire Wardle and Hossein Derakhshan. Information disorder: Toward an interdisciplinary frameworkfor research and policy making.
Council of Europe report, DGI (2017) , 9, 2017.Rowan Zellers. Why we released grover. Technical report, 2019. URL https://thegradient.pub/why-we-released-grover/ .Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. Swag: A large-scale adversarialdataset for grounded commonsense inference. In
Proceedings of the 2018 Conference on EmpiricalMethods in Natural Language Processing (EMNLP) , 2018.Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visualcommonsense reasoning. In
The IEEE Conference on Computer Vision and Pattern Recognition(CVPR) , 2019a.Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machinereally finish your sentence? In
Proceedings of the 57th Annual Meeting of the Association forComputational Linguistics , 2019b.Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner,Ali Farhadi Franziska Roesner Choi, Yejin Yonatan Bisk, and Yejin Choi. Counteracting neuraldisinformation with grover. Technical report, 2019c. URL https://medium.com/ai2-blog/counteracting-neural-disinformation-with-grover-6cf6690d463b .12 upplemental MaterialA Optimization Hyperparameters
For our input representation, we use the same BPE vocabulary as (Radford et al., 2019). We useAdafactor (Shazeer and Stern, 2018) as our optimizer. Common optimizers such as Adam (Kingmaand Ba, 2014) tend to work well, but the memory cost scales linearly with the number of parameters,which renders training G rover -Mega all but impossible. Adafactor alleviates this problem byfactoring the second-order momentum parameters into a tensor product of two vectors. We used amaximum learning rate of 1e-4 with linear warm-up over the first 10,000 iterations, and decay overthe remaining iterations. We set Adafactor’s β “ .
999 and clipped updates for each parameter toa root-mean-squared of at most 1. Last, we applied weight decay with coe ffi cient 0 .
01. We used abatch size of 512 on 256 TPU v3 cores. which corresponds to roughly 20 epochs through our newsdataset. The total training time required roughly two weeks.
B Real News and Propaganda Websites
In our generation experiments (Section 4), we consider a set of mainstream as well as propaganda web-sites. We used the following websites as ‘real news’: theguardian.com , reuters.com , nytimes.com , theatlantic.com , usatoday.com , huffingtonpost.com , and nbcnews.com . For propaganda sites, wechose sites that have notably spread misinformation (Dicker, 2016) or propaganda . These were breitbart.com , infowars.com , wnd.com , bigleaguepolitics.com , and naturalnews.com . C Domain Adaptation of BERT
BERT (Devlin et al., 2018) is a strong model for most classification tasks. However, care must betaken to format the input in the right way, particularly because BERT is pretrained in a setting where itis given two spans (separated by a special [SEP] token). We thus use the following input format. Thefirst span consists of the metadata, with each field prefixed by its name in brackets (e.g. ‘ [title] ’).The second span consists of the body. Because the generations are cased (with capital and lowercaseletters), we used the ‘cased’ version of BERT.Past work (e.g. Zellers et al. (2019a); Han and Eisenstein (2019)) has found that BERT, like otherlanguage models, benefits greatly from domain adaptation. We thus perform domain adaptation onBERT, adapting it to the news domain, by training it on R eal N ews for 50k iterations at a batch sizeof 256. Additionally, BERT was trained with a sequence length of at most 512 WordPiece tokens,but generations from G rover are much longer (1024 BPE tokens). Thus, we initialized new positionembeddings for positions 513-1024, and performed domain adaptation at a length of 1024 WordPiecetokens. D Hyperparameters for the Discriminators
For our discrimination experiments, we limited the lengths of generations (and human-written articles)to 1024 BPE tokens. This was needed because our discriminators only handle documents up to 1024words. However, we also found that the longer length empirically discrimination easier for models(see Section 6).For our discrimination experiments, we used di ff erent hyperparameters depending on the model,after an initial grid search. For BERT, we used the Adam (Kingma and Ba, 2014) optimizer with alearning rate of 2 e ´ rover , we used theAdam actor optimizer (Shazeer and Stern, 2018) optimizer with a learning rate of 2 e ´ ffi cient of 0 .
5. These models were trained for 10 epochs, with a linear warm-up over theinitial 20% iterations. For more information, see the Media Bias Chart at adfontesmedia.com/ . Human Evaluation Prompt
E.1 Evaluating Quality
For evaluating the quality of G rover -written versus human-written news articles, we asked workersthe following questions (shown exactly). The answer choices are shown next to the rating under our1-3 Likert scale (3 being the best, 1 being the worst for each attribute).(a) (Style) Is the style of this article consistent?3.
Yes , this sounds like an article I would find at an online news source.2.
Sort of , but there are certain sentences that are awkward or strange.1. No , it reads like it’s written by a madman.(b) (Content) Does the content of this article make sense?3. Yes , this article reads coherently.2.
Sort of , but I don’t understand what the author means in certain places.1. No , I have no (or almost no) idea what the author is trying to say.(c) (Overall) Does the article read like it comes from a trustworthy source?3. Yes , I feel that this article could come from a news source I would trust.2.
Sort of , but something seems a bit fishy.1. No , this seems like it comes from an unreliable source. E.2 Evaluating consistency
To measure consistency between the article and the metadata, we asked the following questions:(a) (Headline) How well does the article body match the following headline? [headline]3.
Yes , the article makes sense as something that I would see given the headline.2.
Sort of , the article is somewhat related to the headline, but seems slightly o ff .1. No , the article is completely o ff -topic.(b) (Authors) How well does the article body match the following author(s)? [authors]3. Yes , the article makes sense as something that could be written by the author(s).2.
Sort of , the article might have been written by the author(s) above, but it soundsunlikely.1. No , the article body contains information that says it was written by someone else.(c) (Date) How well does the article body match the following date? [date]3. Yes , the article makes sense as something that could have been written on [date].2.
Sort of , the article might have been written on [date], but it sounds unlikely.1. No , there’s information in the article that conflicts the proposed date. F Examples
In Figures 8 and 9, we include examples of articles with the average scores given by human raters,who were asked to evaluate the style, content, and overall trustworthiness. In Figure 8, we show a realarticle (
Human News ) posted by the Guardian along with an article from G rover ( Machine News ) madeusing the same metadata. Figure 9 shows a real propaganda article from the Natural News (
HumanPropaganda ) and an article made with G rover ( Machine Propaganda ) with the original headline andthe style of Hu ffi ngton Post (G rover was used to re-write the title to be more stylistically similar tothe Hu ffi ngton Post, as well).We also present several other generated examples, generated from G rover -Mega with a top- p threshold of p “ .
95. All of the examples are cut o ff to 1024 generated BPE tokens, since this is oursetup for discrimination. a . G rover can generate controlled propaganda. In Figure 10, we show the continuation from Figure 1,about a link found between autism and vaccines.14 iming of May’s ‘festival of Britain’ risks Irish anger April 13, 2019 theguardian.com
It was meant to be a glimmer of positivity to unite a divided nation – a festival to celebrate the best of British, bring communities together and strengthen “our precious union”. Yet Theresa May is being warned that her plan for a Festival of Great Britain and Northern Ireland risks doing the opposite. The planned 2022 event, announced at last year’s Conservative conference, was criticised as a headline-grabbing distraction. But May now faces concerns that the timing clashes with the centenary of Irish partition and the civil war. Arts industry figures in Northern Ireland and some of those involved in the peace process are also understood to have concerns. These worries are revealed in a report by the thinktank British Future, which examined the potential for arts and heritage to bring the nation together. The study calls on the festival to be delayed by at least three years. What is now the Irish republic became the Irish Free State in 1922, while Northern Ireland remained part of the UK. A civil war erupted among Irish nationalists over the remaining links with Britain and raged for a year. Sunder Katwala, the report’s author, said: “Holding a festival of Great Britain and Northern Ireland in 2022, on the centenary of Ireland’s partition and civil war, would be the worst possible timing. It is only likely to heighten tensions between communities – and that’s before we know Brexit’s implications for the border. Right across the UK, a festival so closely associated with Brexit may only reinforce divides when it could be bridging them.” Jonathan Powell, Tory Blair’s former chief of staff and one of the architects of the Good Friday agreement, also warned against anything that could inflame tensions, which have worsened given the prominent role of the Irish border in the Brexit debate. “There is going to be a lot of rebuilding to do in Anglo-Irish relations, which had been put on a completely new footing over the last 20 years and were then trashed by the Brexiters over the last couple of years,” he said. “An example is their constant attacks on [Irish prime minister Leo] Varadkar for hostility when all he is doing is protecting the interests of the Irish people. I fear it will take years to put this right.” The idea of a nationwide event was modelled on the Great Exhibition of 1851 and the Festival of Britain a century later, held to boost the nation’s spirits after the second world war. May was mocked on social media when she announced she was setting aside £120m to prepare for the event, with some suggesting the festival would only feature damp tents on village greens, an exhibition of the remaining avocados left in Britain and morris dancers. However, after holding focus groups, British Future found it could be more popular than critics think, should organisers get the tone and timing right. More than six in 10 thought the festival a good idea, with only one in 10 opposed, according to new ICM research. “The festival could be much more popular with the public than its social media critics think, and could help bridge social divides if done well,” Katwala said. “If not, it could be a second Millennium Dome. It risks being neglected, with the government stuck over Brexit and the prime minister who thought of it on her way out. The next prime minister will have to decide how to rescue this idea.”
Timing of May’s ‘festival of Britain’ risks Irish anger
April 13, 2019 theguardian.com
The prime minister’s decision to launch her much-hyped “great Britain” conference on Monday – the same day as a Lisbon treaty event paving the way for Brexit-free member states to leave the European Union – will be seen as provocative by some, according to senior Tories. Jo Johnson said May’s statement in Edinburgh was “instrumental” in chipping away at the strength of domestic opposition to Brexit. He added that the prime minister had also “churned” the membership of the ERG back into service. Tom Pursglove, another Tory MP involved in the campaign to prevent Brexit, said: “By lifting the gagging order on ERG members from the Liaison Committee and starting an intensification of the ethnic profiling of Remainers, the prime minister is doing herself and the ERG proud.” Announcing that the conference would launch her vision for the country, May will call for more global Britain to fight for global trade. Although still committed to leaving the single market and customs union, the Conservatives want to highlight the importance of these deals – as well as tackling climate change, tackling modern slavery and tackling poverty. The event will be on Monday 29 April, the day before the EU’s 2019 budget is agreed. May’s Treasury chief secretary, Liz Truss, is to try to convince European finance ministers that there is no alternative plan to Brexit. EU officials and political leaders are scheduled to decide the EU’s £1.2tn budget in mid-October. The Northern Ireland-based DUP, which failed to back May in the no confidence vote she suffered earlier this month, will be encouraged by the event. The DUP said it would be “easy to ignore” the motions at the conference, but would vote against any effort to transfer powers to Brussels. Labour MP Sir Keir Starmer, who now chairs the cross-party Brexit negotiations committee, said: “The timing of her conference announcement raises some worrying issues. We cannot allow the UK’s terms of exit to be dictated by no confidence votes. “These checks cannot be on the superficial level, where some make noises on the hill but are wholly unwilling to set out detailed proposals. Tighter controls at Heathrow are essential, and if May really wants to celebrate ‘all change’, then she should close Britain’s borders for a week and see how workable it is to stop EU nationals from flying in on the same visa system as Brits. “Brexit would be fantastic for the business world if you measure economic value only on the quality of the deal. But – and when we say ‘if’ the prime minister doesn’t care that she is still far short of securing that ‘good deal’ – she needs to work harder to deliver that for her negotiators.” Other critics, including party member James Ball, drew parallels with Brexit minister Dominic Raab’s similar focus on trade deals to stop other EU states leaving the bloc. They said Raab’s speech last week was “the latest Labour-held ploy to quietly delay Brexit, run out the clock or blame everyone except the UK for not being willing to walk away”. • Follow Guardian Opinion on Twitter at @gdnopinion
Human-written News Article Machine-written News Article
Original Headline: Timing of May’s ‘festival of Britain’ risks Irish anger
Ratings Style: 3.0 Content: 3.0 Overall: 2.3Ratings Style: 3.0 Content: 3.0 Overall: 3.0
Figure 8: Example of human-written news and machine-written news articles about the same headlinefrom The Guardian with the average ratings from human rating study. b . G rover can spoof the identity of writers. In Figure 11 we show a realistic-looking editorialseemingly from New York Times columnist Paul Krugman. c . G rover can generate fake political news. In Figure 12 we show an article generated about Trumpbeing impeached, written in the style of the Washington Post. d . G rover can generate fake movie reviews (opinion spam; Ott et al. (2011)). In Figure 13 we showa movie review, generated in the style of LA Times Movie Critic Kenneth Turan, for Sharknado 6,‘The Last Sharknado: It’s About Time’ e . G rover can generate fake business news. In Figure 14, we show an article generated about an‘Uber for Dogs’ startup. 15 on't drink the water: The dark side of water fluoridation March 13. 2019 naturalnews.com (Natural News) There are 7.7 billion people on this planet (as at March 2019). Only about 5 percent of them drink fluoridated water. Why? Because their governments recognize that fluoride in large amounts becomes a toxic chemical that is not fit for human consumption. The 328,000,000 citizens of the United States drink more fluoridated water than all other countries combined. Why? Because the U.S. government continues to doggedly insist that it is safe and improves dental health. But what do the facts say? As reported by Waking Times, dozens of peer-reviewed studies published in prestigious journals like The Lancet, have confirmed that fluoride is in fact toxic – especially to the developing brains of children. These chemicals are derived from unprocessed toxic waste which is not purified in any way before being pumped into the water supply. How could it possibly be anything but harmful? The history of water fluoridation in the United States So, what prompted the government to start adding something so obviously harmful to our precious water supply? Waking Times, quoting from an article by The Children’s Health Defense Team, explains a little about the history of this practice: During World War II, fluoride (a compound formed from the chemical element fluorine) came into large-scale production and use as part of the Manhattan Project. According to declassified government documents summarized by Project Censored, Manhattan Project scientists discovered early on that fluoride was a “leading health hazard to bomb program workers and surrounding communities.” In order to stave off lawsuits, government scientists “embarked on a campaign to calm the social panic about fluoride…by promoting its usefulness in preventing tooth decay.” The power of the elements: Discover Colloidal Silver Mouthwash with quality, natural ingredients like Sangre de Drago sap, black walnut hulls, menthol crystals and more. Zero artificial sweeteners, colors or alcohol. Learn more at the Health Ranger Store and help support this news site. To back up its decision, the government embarked on a series of flawed and poorly designed “scientific” studies, which an expert later lambasted as “especially rich in fallacies, improper design, invalid use of statistical methods, omissions of contrary data, and just plain muddleheadedness and hebetude.” They then used these sham studies to enforce a national policy of water fluoridation. Studies confirm fluoride lowers IQ and harms children in other ways Interestingly, even government-backed studies have confirmed the dangers of fluoride in drinking water. For example, a study published in 2017, which was largely funded by the government’s National Institutes of Health and National Institute of Environmental Health Sciences, uncovered a “strong relationship” between fluoride exposure in the womb and reduced cognitive function. In addition, Natural News previously reported: More than 50 peer-reviewed studies have linked the consumption of fluoridated water to lower IQ in children. A joint metanalysis by Harvard School of Public Health and China Medical University, which examined 27 studies on the subject, found “strong indications that fluoride may adversely affect cognitive development in children.” Fluoridation has also been linked to countless other devastating health effects in children, including premature birth, impaired neurological development, autism and preeclampsia. A recent study also confirmed a significant link between fluoridation and ADHD. Death rates up to 26 percent higher in the most fluoridated states And it gets worse. As reported by Waking Times, death rates among people who live in the 10 states with the highest fluoridation levels are between 5 and 26 percent higher than among people living in the 10 least fluoridated states. Furthermore, studies have linked fluoridated water to cancer, damaging effects on the brain, and dental fluorosis, which experts agree is the first visible sign of fluoride toxicity. The National Research Council of the National Academies also issued a warning in 2006 that fluoride exposure might be a risk factor in the development of Alzheimer’s disease. And to top it all off, there is literally no evidence that fluoride even prevents cavities! Even if it did, would that small benefit really be worth the massive potential for serious health problems that fluoridation has been scientifically linked to? Learn more at Fluoride.news or TapWater.news. Sources include: WakingTimes.com NaturalNews.com ChildrensHealthDefense.org
Fluoride in Your Water Isn’t Healthy, Even When It’s Recommended
March 13, 2019 huffingtonpost.com
Fluoride is regarded by researchers around the world as the “gold standard” in tooth care, and a safe, common ingredient that has almost been universally found to be safe in past studies of health effects. It’s absorbed without interference from the body’s natural minerals. Researchers at the U.S. Geological Survey (USGS) published the results of a multi-state environmental health study last month. It showed that during the first three decades of fluoridation of tap water systems, fluoride produced from the process alone increased rates of dental caries (the biggest contributor to tooth decay) by 16 percent in Mississippi and a whopping 45 percent in Arizona, which implemented fluoridation systems back in 1942. This increase was seen after a decade when fluoride levels didn’t change. USGS also found that fluoridation increased rates of other toxicants and petrochemicals, as well as deaths from brain, lung, kidney and bladder cancer. It bears noting that there is no clear proof that these specific contaminants were caused by fluoridation, but the USGS study at least hints that this was the case. The epidemic of brain cancers across the U.S. — especially in teenagers — has confounded researchers for decades. The USGS study points to links to numerous studies that have linked water fluoridation with increased risks of cancer. Even though the majority of studies on water fluoridation have not produced such alarming results, the mainstream medical community is, apparently, still skeptical. Two years ago, doctors from Harvard and Duke universities suggested that fluoride is associated with lower IQ scores and autoantibodies to water. The results of a recent study that followed more than 700 children over a period of four years demonstrated that the kids were more likely to have symptoms of illness, more likely to have higher blood pressure and sleep problems, had higher mean energy expenditure, more struggles with attention and poorer concentration and performance on educational tasks. The data also showed that the children were not more vulnerable to fluoride when it was administered by toothpaste. Even though the majority of studies on water fluoridation have not produced such alarming results, the mainstream medical community is, apparently, still skeptical. The link between bacteria and tooth decay is legendary. Experts agree that fluoride erases a cavity’s effect on surface-level bacteria and increases decay via dangerous pathways to deeper tissue. Fluoride actually quarantines surface-level bacteria to caries-causing toxins in the plaque and oral cavity. The longer the fluoride is in contact with those toxins, the more damage and damage can occur. Evaluations of the impacts of water fluoridation are well underway, but we’ve seen too many conflicts of interest and dirty laundry in the past few years. One of the largest of those is the Water Fluoridation Corporation (WFC), an industry network that carries on and even finances the work of paid “expert witnesses” in both domestic and international water fluoridation litigation. The WFC boasts 38 former and current members of the American Dental Association, including board members, former ADA commissioners, and representatives of the ADA-funded American Academy of Oral Medicine and American Dental Hygienists. It is arguably the second largest source of fluoride in the U.S. right behind the ADA. Despite denials, none of this new water research paints a rosy picture of water fluoridation. We can blame these new findings on ongoing conflicts of interest and blatant biases. Fluoride in your drinking water isn’t safe, even when it’s recommended. There are three risks that we as consumers need to ensure that are understood: 1) the hidden contaminants that may be causing harm; 2) the effects of over-medicating the fluoride vaccine; and 3) low-dose levels of fluoride in very old bones, causing unnecessary bone damage and irreparable nerve damage.
Human-written Propaganda Machine-written Propaganda
Original Headline: Don’t drink the water: The dark side of water fluoridation
Ratings Style: 2.3 Content: 2.7 Overall: 2.7Ratings Style: 2.3 Content: 2.3 Overall: 1.7
Figure 9: Example of human-written and machine-written articles arguing against fluoride with theaverage ratings from human rating study. 16rovided headline: ‘Link found between autism and vaccines’Generated body:
Those who have been vaccinated against measles have a more than 5-fold higher chance ofdeveloping autism, researchers at the University of California San Diego School of Medicine and the Centers forDisease Control and Prevention report today in the Journal of Epidemiology and Community Health.Those who have been vaccinated against measles have a more than 5-fold higher chance of developing autism.Earlier studies had linked past vaccines with the condition, but it was not clear whether those links were causal orindirect. Among research groups, the UCSD and CDC teams used new methods that enabled them to determinethe associations between the virus and the disease.While researchers in the previous studies had observed associations between those vaccinated against measlesand autism in children whose mothers had received the virus, the current UCSD / CDC team focused on childhoodvaccines given during the first two years of life because it is commonly believed those vaccines begin to clearthe body’s immune system before the onset of autism. The UCSD / CDC team compared children who had beenvaccinated against measles, mumps and rubella (MMR) and those who had not.“Because this is the first study to focus on MMR-vaccine syndrome, it was also the first time that this syndromehad been scientifically characterized,” said study lead author William Chan, MD, MPH, a researcher in theDepartment of Pediatrics in the UC San Diego School of Medicine. “We were able to identify certain subgroupsof those vaccinated against MMR and compare them to those not vaccinated.”Mumps, measles and rubella vaccines are administered in either three or five doses, depending on the age. Arecent study by the Public Health Agency of Canada found only 10 percent of children vaccinated againstmeasles had a developing autism disorder when compared to 73 percent of children vaccinated with one dose, 50percent of those vaccinated with two doses and 45 percent of those vaccinated with three doses. Those results donot surprise the UC San Diego researchers, who found those vaccinated with three or more doses were at anadditional 5- to 8-fold increased risk.The current UCSD / CDC analysis followed the same years of data as earlier studies, but used new methods toaccurately pinpoint the timing of those immunity-shifting epidemics. For example, previous studies showed thatautism was on the rise in 1995-96, but never topped 50 cases until 2001-02, when measles overtook mumps asthe leading cause of infant death due to disease. In 2003, several measles outbreaks were triggered by parentswho had ignored warnings about the vaccine.“Our research was critical in providing details of when outbreaks occurred in regions where there was significantvaccination e ff orts,” said study co-author Louis Cooley, MD, PhD, professor of Pediatrics and professor ofCommunicable Diseases and Epidemiology at UCSD School of Medicine. “We were able to show that if you getyour MMR vaccine in early childhood, you should reduce your risk of autism by 33 percent, with every passingyear bringing a slightly lower risk.”Researchers found that low vaccination rates — 42 percent in 2003-04 and 75 percent in 2007-08 — and morefrequent, and severe, cases of measles predicted higher rates of autism in later years. In 2004-05, the annualrate of autism and other disorders — including intellectual disability — increased by 20 percent, far more thanthe 5- to 7-fold increase in their model predicted, suggesting there may be a more subtle link between vaccineexposure and autism than previously understood. Preliminary findings from a separate analysis of the 2006-07data suggest that autism risk spikes occurred during times when children had fewer routine vaccinations andreceived older doses of vaccines.Autism spectrum disorders include autism spectrum disorders (ASD), a group of neurodevelopmental disordersthat can occur with or without symptoms of the immune system disorder, including Asperger’s syndrome,pervasive developmental disorder not otherwise specified (PDD-NOS) and pervasive developmental disorder-nototherwise specified (PDD-ENG).The study was supported by the National Institutes of Health’s National Institute of Child Health and HumanDevelopment (CDER), the NIH Program on Smoking and Health and the San Diego Veterans A ff airs (SDVA)Healthcare System.About the University of California, San Diego School of MedicineFounded in 1897, the School of Medicine has been recognized nationally for outstanding achievement ineducating clinical and research physicians. Nobel laureates and more than 170 physicians who practice in theUnited States and abroad are the graduates of UC San Diego. The school has been recognized as the first medicalschool in the world to o ff er its entire physician-practitioner curriculum online, currently serving more than 1,000medical students worldwide. The school also trains the next generation of scientists, physicians and publichealth leaders and ranks first in the nation for its graduates’ ability to train other healthcare professionals. UCSan Diego is associated with UC San Diego Health System, which has a faculty of more than 9,000 and an annual Figure 10: The rest of the example from the teaser figure, about a hypothetical link found betweenautism and vaccines. 17rovided domain: nytimes.com
Provided author: Paul KrugmanProvided date: May 23, 2019Generated headline: Why Trump could get impeachedGenerated body:
President Donald Trump could get impeached. Or he could get fired. Or he could be indictedfor special-counsel matters involving Stormy Daniels, or the Russians, or his own campaign. But, at the end ofthe day, what would matter is what House Speaker Nancy Pelosi decides. And here’s why:First, Pelosi seems open to the idea of impeachment, to the point where last week she endorsed the use of theword in a speech. And this is not to claim that “impeachment” is a possible solution to the current impasseon immigration. Nobody seriously thinks that. Rather, it’s to say that the dividing line between Trump’simpeachment and, on the other hand, the notion that he’s being treated adequately in his courtship of theRepublican base, is soft. If Trump runs a replay of his 2016 campaign, the focus on immigration will fade as anissue. Then his campaign’s fiery rhetoric on the economy and taxes will dominate, and get bigger. But as wepointed out yesterday, the “soul” of the Trump campaign was white nationalist sloganeering and authoritarianism.It seems unlikely that this will change.If it doesn’t change, impeachment seems like the only way to constrain him.Yet the power of the impeachment path is out of Trump’s hands. The recent passage of the Defense AuthorizationAct, to cite just one example, which codifies within the law the requirement that presidents get an honestbreak from investigations, seems to have guaranteed that Bob Mueller will be working hard into the fall. And“executive privilege” — the right of a president to withhold certain materials from the government — is clearlylimited by the extent to which federal judges have looked at and rejected it.So impeachment is less a decision than a historical fact: If you think the election of a president who promised toenrich himself through corruption has happened, a court-ordered investigation, or both, is probably necessary tocorrect the damage. But it can’t happen while he’s president, or until he’s gone. Even then, there are a few waysthat a president can claim executive privilege, or whatever other arguments he can throw out to minimize thepressure on him.So Congress must seize the moment. The first and most urgent must be an investigation of the Trump campaign’spossible ties to the Russians, and the subsequent question of whether the president knew what was going on.(For full coverage of the Russia story, see here.) The second, more abstract but no less important issue is whetherthe president’s own actions constitute obstruction of justice. And third, Trump needs to be impeached becausehis rhetoric has helped put members of his own party who could back his impeachment into near paralysis.If Republicans don’t take a bold stand, and appoint a special prosecutor to investigate all this — just as Democratsdid during Watergate — then their caucus will become dependent on impeachment proceedings to hold its seats.While that might be a good outcome in the short run, it could also signal a likely government shutdown — justas it signaled the risk of a government shutdown during the Clinton impeachment. So it’s a bad decision for thecountry, or at least for the country’s self-interest.That said, it could be an even worse one for Trump. True, other presidents have been impeached in part forbehavior far more extreme than Trump’s — for cases like Bill Clinton’s loveless marriage, and, more frequently,for their handling of women in private. But never before has the impeachment authority come at the point wherea man at the pinnacle of power and influence has cozied up to a hostile foreign power, and openly attacked notjust the judiciary, but the rule of law itself.For those who would recommend removal now for that alone, it’s worth noting that on a serious level, that kindof behavior isn’t akin to the brinkmanship that some U.S. presidents have brought to bear during a confrontationwith Cuba. It’s as close as Trump has come to laying out some of the first lines of battle for our uncertain anduneasy world.And to me, that is a form of obstruction of justice.
Figure 11: An example of spoofing someone’s identity using G rover -Mega. We generate a convincingNew York Times Op-Ed that is seemingly from columnist Paul Krugman.18rovided domain: washingtonpost.com
Provided date: May 29, 2019Provided headline: US House Votes to Impeach Donald Trump After New Evidence Released fromMueller ReportGenerated body:
WASHINGTON — The House voted to impeach President Donald Trump Wednesdayafter releasing hundreds of pages of text messages that point to clear evidence of obstruction of justice andcommunication with the head of the Trump Organization about a potential business deal in Russia.The 220-197 vote came after weeks of debate over whether new evidence released by special counsel RobertMueller’s o ffi ce signaled su ffi cient grounds for Trump’s removal from o ffi ce. The president personally denouncedthe move, announcing his intent to veto the resolution and accusing Democrats of plotting to remove him fromo ffi ce through a “con job.”“This is all a big con job,” he said during an appearance in Texas. “They got caught.”A number of Democratic lawmakers went further, calling for Trump’s impeachment on the grounds that heobstructed justice. Democrats hold the House majority, and if the Senate fails to convict Trump, the presidentcan be removed from o ffi ce by a two-thirds vote.But even with Trump’s political allies organizing from the White House, Republicans in the House stood unitedbehind their leader, noting the historical di ffi culty of making the case for impeachment.“This is not a decision for the president,” said House Speaker Paul Ryan, R-Wis. “It’s not one for this body, and Ithink it’s a dereliction of duty for the minority party to be making these calls.”Democrats on the House Judiciary Committee on Tuesday said they were not bound by Ryan’s defense. Theyreleased an 11-page joint report with Republicans on the House Oversight and Government Reform Committeethat took a di ff erent tack, emphasizing that Mueller had not decided whether the case warranted impeachment.Rep. Jerrold Nadler, D-N.Y., the committee’s chairman, told reporters before the vote that he hoped the tworeports would “send a clear message to the president that we in this body are not going to stand for obstructionof justice.”But when asked whether the report would build the case for impeachment, he said: “What’s far more likely tobuild a case for impeachment are the indictments and evidence produced through the special counsel’s work.”On Tuesday night, the White House issued a statement proclaiming that the book by a former FBI deputydirector, which has been cited by Democrats in support of impeachment, had been “made up and defamatory” tothe president.That statement drew swift and heavy rebukes from a number of Democrats, who said they had found thestatement “beyond astonishing.”The Democratic report, citing redacted text messages sent by former FBI counterintelligence agent Peter Strzok,revealed what the party called “a pattern of bias” and “an a ffi nity for the Russian regime.”The report did not include a full, uncensored text message Strzok sent to a friend after Trump was elected thatsaid: “I’ll stop it.” That led Democrats to bemoan the language used, questioning what is or is not in the specialcounsel’s investigative file.The report also quoted a review of emails and witness testimony to Mueller that said Trump “has obstructedjustice in the Russia investigation” by firing FBI Director James Comey, by urging Attorney General Je ff Sessions to recuse himself from the Russia investigation and by urging his son, Donald Trump Jr., to meet with aKremlin-connected Russian lawyer at Trump Tower in 2016.“This has never happened in the history of our country,” Nadler said of the president. “No president has evertried to interfere with an investigation into his own conduct.”The vote was also led by Democrats who have been outspoken against Trump, including Rep. Al Green, D-Texas,who took out newspaper ads calling for the president’s impeachment. Rep. Maxine Waters, D-Calif., anotheroutspoken Trump critic, said Trump’s ties to Russia were “becoming more and more ominous,” and said “as wego on, we are going to see if we can’t get the articles of impeachment out.”Rep. Tony Cárdenas, D-Calif., a member of the Judiciary Committee, said many Democrats would be eager tosee whether the newly discovered evidence could be used to support impeachment.“I think it will be the nail in the co ffi n of the ‘non-con,’” he said. Generated headline: Congress voted to impeach President Trump Wednesday; the President vowed toveto the resolution.Figure 12: An example of using G rover -Mega to generate fake political news, written in the styleof the Washington Post. Here, it generates a mostly-realistic-looking article about Trump beingimpeached. 19rovided domain: latimes.com
Provided date: August 1, 2018Provided authors: Kenneth TuranProvided headline: Review: "The Last Sharknado: It’s About Time" is a gem of an action movieGenerated body:
Some heroes take a vacation every now and then, even temporarily, a couple of beachfrontcondos in a movie world they hope, a la Rick James in “The Beautiful Daze,” won’t have to live through.Unfortunately, that has never happened to one of the most exceptional directors in the business, Anthony & JoeRusso, so the moment is altogether too good to spoil, at least for us Southern Californians.In “The Last Sharknado: It’s About Time,” which had its Los Angeles premiere at the Tower Theatre on Tuesday,we can expect the San Diego-born Anthony Russo to somehow make it through seven years without being insidea sharknado.Oh, he has a couple of other movies during that time, mostly that stupid’d-up New York “Avengers” picture, butthey have nothing to do with sharks or serious action-film plotting.Don’t worry, he lives to fight another day.Written by Jason Friedberg and Aaron Seltzer (“Vampires Suck”) and directed by John Hlavin (“Code NameVerity”), “The Last Sharknado” is exactly what you would expect: very silly and thrilling.Once again the Ryan Newman- and Ian Ziering-costumed stars of the YouTube smash “Sharknado” have a lot offun poking fun at themselves and the silly film business (“I don’t know if being caged for a few days after we getout is too much to ask”) along with helping a once-beleaguered government figure out what to do.We start on a lake in North Carolina as Newman, now an exotic dancer, does a good-natured impersonation ofSeltzer’s Agent Slutz. He’s not a bad actor, though, and by the end of the first reel “it’s clear he’s never a lunatic.”So Slutz has just been rehired by the Trump Administration, even though the agency has been e ff ectively takenover by the tag team of the Governator and Alex Jones (the viewer is never told why they are being hired).As Slutz, Newman is working with his deputy, Ziering, the impetuous marine o ffi cer no longer needled by hisloyalty to his best friend, Tara Reid, who showed up in the third “Sharknado” and is absent this time.The three are doing their best to fix things, with Newman looking in particular for more information on how todeal with the five remaining sharks in New York who seem to think they should go with the Trump administration.Svelt’s plotting goes way beyond the call of duty, of course, for despite the presence of Anthony Mackie andJuliette Lewis, for example, there are many disappointments here, especially in the casting of a grouchy Je ff Goldblum as the villain of the piece.Still, like the original “Sharknado,” all the elements are exactly right, including some fancy e ff ects (stuntcoordinator Zak Penn, “The Avengers”) and some deeply, bizarrely funny dialogue (try spending a few days in asharknado and after a while questioning your sanity).Knowing that everything is going to be fine, Ziering tells Newman: “Whether we go sharknado-free is on you.”Come again?Might we welcome it? If by a miracle we’re not sated by our popcorn, that would be the best way to look at [email protected] me on Twitter @KennethTuran Generated headline: ’The Last Sharknado’ returns with the dramatic thriller you expected — andsome unexpected laughsFigure 13: An example of using G rover -Mega to generate a fake movie review. Here, we spoof LATimes Film Critic Kenneth Turan and generate a positive movie review for ‘The Last Sharknado: It’sAbout Time,’ the sixth installment in the Sharknado series of movies.20rovided domain: techcrunch.com
Provided date: May 29, 2019Provided headline: Rowan Zellers’ new startup raises 17M in Series A round for "Uber for Dogs"Generated body:
In today’s competitive eco-system of startups trying to convince consumers to eschew popularapps in favor of their own, you would think that the dog-walking market would be the easiest to crack. Well, it’snot. But at least one startup believes it has a new and clever solution: apply its core o ff ering of mobile-basedrecommendation engine algorithms to locate dog owners near them with the end goal of making dog walkingmore e ffi cient.The company, owned by Australian startup Yfrog, is called DogSpotter. It is co-founded by entrepreneur RowanZellers, along with King’s College London-trained dogsman Craig Osbaldoon.Launched in Melbourne in November, DogSpotter has now launched in Sydney, Canberra, Adelaide, Perth, andis set to be introduced to Brisbane and Hobart in June, with a platform the company expects to be available in allcapital cities in Australia this year. In addition, it is now prepared to accept payment via credit cards in Australiaand New Zealand.In exchange for its name recognition and the unique proposition that it is aiming to turn into a software-as-a-service, DogSpotter has raised $17 million in a Series A round of funding led by Plug and Play Tech Center,with participation from Open Ocean Capital and previous investors, including the Cahill Foundation.So what does DogSpotter o ff er, beyond a fairly basic service that could theoretically be used by anyone, whichis at the base of a human-side algorithm that determines the best potential location for a dog owner’s pet — aproblem that dogs are notoriously impatient for in cities where dog walking is often unprofitable or a lost art formany?Well, DogSpotter leverages its core recommendation engine to process the thousands of now readily availablereviews of dog care providers via its app that users can find by searching for their own city or a curated selectionof recommendations. That way, it’s much quicker than traditional alternatives, where you have to wade througha fair number of reviews to find the best-rated providers in your city, and does it with far less hassle.From there, the DogSpotter app makes use of the data it provides to recommend relevant walking services,among them a variety of preferred types of walks for your dog, based on everything from their preferences andsorts of paths to the weather, geographic di ff erences, and degree of safety.DogSpotter’s founders also contend that their software can — and in some cases already has been — used tomake sure that the actual human responsible for dog walking is nearby, by using the location-based data tofacilitate long-distance walkings or picking out a dog for a veterinarian who can be contacted.Speaking of which, they’re not just operating on the basis that it’s an e ff ective way to walk dogs withoutsu ff ering losses. Although I won’t deny that I’ve seen some big cities su ff er from too many small businessesadvertising purely online and with little regard for actual human location. As it happens, DogSpotter tells me ithas its own network of paying businesses in places like U.S. cities, but it claims to be seeing a company likeDogBuddy.com.au as a market that it is disrupting.My hope is that the company ends up developing a number of other application-specific verticals such asveterinary procedures — again, I have not seen this yet but that would be a great and easy way to monetize.The business model is to o ff er DogSpotter services for a set fee per user, per location per month. It includescomplimentary services for canine community or adoption groups, dog run assessment and mapping, dogboarding facilities, dog walking packages, and a myriad of other services.“DogSpotter is di ff erent from other dog walking platforms because we have the leading mobile app which is themost e ff ective tool for location-based visual targeting for relevant content and experiences,” said Rowan Zellers,founder and CEO of DogSpotter, in a statement. “This means that DogSpotter is able to provide more relevantproducts and services and, for early users, we can allow companies to target only the users they want to engagewith.”The problem, of course, is that not all dog walkers have the best apps. DogSpotter says it will compete onperformance of its o ff ering, not with other dog walking apps but rather with making it easier for dog ownersto find the people they are looking for and, importantly, for those individual dog walkers to save a small bit ofmoney on their operating costs, by saving time and e ff ort with a website. And so on. Generated headline: DogSpotter raises $17M to improve your search for walking servicesFigure 14: An example of using G roverrover