Heteroglossia: In-Situ Story Ideation with the Crowd
HHeteroglossia: In-Situ Story Ideation with the Crowd
Chieh-Yang Huang, Shih-Hong Huang, and Ting-Hao (Kenneth) Huang
College of Information Sciences and TechnologyPennsylvania State University, University Park, PA, USA{chiehyang, szh277, txh710}@psu.edu
ABSTRACT
Ideation is essential for creative writing. Many authors strug-gle to come up with ideas throughout the writing process, yetmodern writing tools fail to provide on-the-spot assistancefor writers when they get stuck. This paper introduces Het-eroglossia, an add-on for Google Docs that allows writers toelicit story ideas from the online crowd using their text editors.Writers can share snippets of their working drafts and ask thecrowd to provide follow-up story ideas based on it. Heteroglos-sia employs a strategy called “ role play ”, where each worker isassigned a fictional character in a story and asked to brainstormplot ideas from that character’s perspective. Our deploymentwith two experienced story writers shows that Heteroglossia iseasy to use and can generate interesting ideas. Heteroglossiaallows us to gain insight into how future technologies can bedeveloped to support ideation in creative writing.
Author Keywords
Crowdsourcing; Creative Writing; Ideation; Role Play; Story
CCS Concepts • Information systems → Crowdsourcing; • Human-centered computing → Human computer interaction(HCI);
User studies;
INTRODUCTION
Storytelling is one of the oldest known human activities [39].People engage in storytelling to communicate, teach, entertain,establish identity, or simply relate to each other in meaningfulways [33]. Storytelling is important, but writing a good story isa challenging and complicated task, and many creative writersstruggle to come up with ideas throughout the process. RolandBarthes said: “A creative writer is one for whom writing isa problem.” Despite this common experience, research intotechnological writing support does not have much to say tohelp story writers. Writing support systems have long beenfocused on business and technical writing. Researchers havecreated systems that can automatically generate follow-uptext in an auto-complete manner [8]; decompose and recom-pose complicated writing tasks [21]; outsource writing jobs
Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACMmust be honored. Abstracting with credit is permitted. To copy otherwise, or republish,to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from [email protected].
CHI ’20, April 25–30, 2020, Honolulu, HI, USA. © 2020 Association of Computing Machinery.ACM ISBN 978-1-4503-6708-0/20/04 ...$15.00.http://dx.doi.org/10.1145/3313831.3376715
Figure 1. The overview of the role play ideation strategy. In order toobtain follow-up story ideas for a working draft, we recruit a group ofcrowd workers and ask them to imagine they were a character in thegiven story. Each worker is instructed to assume the role of a characterin the story and generate plot ideas from this character’s perspective. to online crowds [3, 14], collaborators [36], or writers them-selves [37]; or even allow the user to write a paper solely usinga smartwatch [29]. However, these prior works were largelydeveloped and tested for producing technical reports [36, 37,29], Wikipedia-like essays [21], or business documents [13],rather than short stories or novels. One of the few exceptionsis the work done by Kim et al. , who created Ensemble [18]and Mechanical Novel [19]. These two systems pushed theboundaries of collaborative story writing, but did not focus onhelping creative writers, who mostly write alone [26, 12, 32].This paper introduces
Heteroglossia , a crowd-powered sys-tem that allows writers to elicit story ideas simply using theirtext editors. Figure 1 overviews the Heteroglossia system,which we built as an add-on for Google Docs. A writer canselect a part of a story draft as the prompt and ask the onlinecrowd to provide follow-up story ideas based on it. Heteroglos-sia employs an ideation strategy called “ role play ,” where eachworker is assigned a fictional character in the story and askedto brainstorm plot ideas from the character’s perspective. Thiswork is motivated by the fact that role-playing and acting tra-ditionally have had a role in the creative writing process [9].Some professional novelists also use role play to help writing.Also relevant is the well-known “six hats” method, which askspeople to wear metaphorical hats representing different think-ing perspectives [10]. Teevan et al. proposed to use the sixhats schema to assign different thinking roles to the authorsthemselves in order to promote self-reflection from differentangles [38]. Chou et al. showed that perspective-taking can Heteroglossia: a diversity of voices, styles of discourse, or points ofview in a literary work and especially a novel [27]. a r X i v : . [ c s . H C ] J a n elease the fixation and thus help generate new ideas by ask-ing people to imagine themselves in different roles involvedin different activities [7]. We developed our work based onthese inspiring prior works. In this paper, we first use a set ofcontrolled experiments to quantitatively illustrate the propertyof the role play ideation strategy, and then overview our sys-tem deployment with two experienced creative writers. Webelieve Heteroglossia allows us to gain insight into how futuretechnologies can be developed to support creative writing. RELATED WORK
This work is related to (i) crowd ideation, (ii) crowd writing, (iii) supporting creative writing, and (iv) crowd feedback.
Crowd Ideation
Prior work has used the online crowd as a source of new ideas,primarily for problem-solving and product design. Chan etal. introduced IdeaGens, an ideation system where a group ofworkers proposes ideas in real-time and the expert monitors theincoming ideas and provides instant feedback to the crowd [4].Yu et al. explored using a schematic representation for thetarget design problem to guide the crowd to “think outsideof the box” [42]. Online crowds were also used to providereal-time creative input during early-stage design activities [2].
Crowd Writing
Heteroglossia also builds upon the work in crowd writing,which aims to allow a group of people, including experts andnon-experts, to work together to write an article. Many crowdwriting projects focused on decompose and recompose compli-cated writing tasks. For example, the Knowledge Acceleratorused a complex workflow where each worker contributes smallamounts of effort to synthesize online information, generatinga Wikipedia-like article for open-ended questions [14]. Soy-lent used a Find-Fix-Verify workflow to allow crowd workersto identify problems in a draft, propose solutions, and selectthe best solution for each identified problem [3]. MicroWriterdecomposed the task of writing into three subtasks: idea gener-ation, labeling, and writing [36]. Meanwhile, some other workhas pushed the boundaries of the classic workflow approachfor crowd writing. For example, WearWrite explored usingwearable devices, such as smart watches, to guide a group ofcrowd workers to write articles [29]. Agapie et al. exploredusing local crowds to generate event reports [1]. However,these projects all focused on business or technical writing,rather than creative writing.One of the few exceptions is the work done by Kim et al. , whocreated Ensemble [18] and Mechanical Novel [19]. Ensembleis a volunteer-based collaborative story competition platformwhere Leaders set high-level creative goals and constraintsfor a story and Contributors participate in low-level tasks,such as drafting, commenting, and voting. Mechanical Novelembodies a more organic workflow, the “Reflect-and-Reviseloop,” that allows crowd workers to revisit and revise theirwriting goal. These works pushed the boundaries of creativewriting and helped to answer why collaborative novels at ascale similar to that of Wikipedia do not exist.While collaborative writing has opened new possibilities, mostwriters still write alone. Professional novelists write alone [26], freelance writers write alone [12], and, even within an industrywith a collaborative culture, many TV screenwriters still writealone [32]. Our goal is to assist creative writers, who oftenwrite alone, without drastically changing the way they work.
Supporting Creative Writing
A few researchers have developed technologies to supportcreative writing. Most of them focused on lower-level textgeneration or proofreading. For example, the Creative Helpsystem used a recurrent neural network model to generatesuggestions for creative writing [34]. The Scheherazade sys-tem was developed for interactive narrative generation [24].InkWell produced stylistic variations on texts to assist creativewriters [11]. More recently, Clark et al. studied machine-in-the-loop story writing and suggested that machine interventionshould balance between generating coherent and surprisingsuggestions [8].
Crowd Feedback Systems
Researchers have also attempted to use online crowds to gener-ate critiques and feedback. Xu et al. created Voyant, a systemthat used non-expert crowd workers to generate structuredfeedback on visual designs [40]. Their classroom study furtherdemonstrated the effectiveness of using crowd feedback in thedesign process [41]. As for visual designs, Luther et al. alsocreated CrowdCrit, a system that aggregated multiple critiquesfrom non-expert crowd workers [25]. Luther showed in exper-iments that the critiques generated by CrowdCrit could helpdesigners improve their design processes. On the other hand,some other researchers focused on generating writing feed-back. For example, Huang et al. used workers from AmazonMechanical Turk (Mturk), who are often fluent in English, toprovide structural feedback for ESL writing [15].
HETEROGLOSSIA SYSTEM
Heteroglossia incorporates a web site to manage informationand a Google Docs add-on for writing. Figure 2 shows thescreenshot of each page of Heteroglossia website. Users startby creating characters (Figure 2A) and forming teams of char-acters (Figure 2B and 2C) on the Heteroglossia website. Aftersetting up the characters and teams, users start writing thestory in Google Docs. When they get stuck, users can select astory snippet to initiate an ideation task through Heteroglossia(Figure 2D) and acquire follow-up story ideas (Figure 2E).
Creating Characters
Figure 2A shows that to create a new character, users specifyan image, name, and description. Notice that only the nameand description will be shown to workers. Users can provide adetailed setting for a character in the description, such as innergoal, outer goal, and personality, to help workers understandthe story background and come up with new story ideas. Edit-ing and deleting an existing character can be done through thesetting button in the upper right corner.
Forming a Team of Characters
A team represents a group of characters used in role playideation. Figure 2B shows the interface of editing a team.Available characters are listed in the “Team Members” block igure 2. The system overview of Heteroglossia. Heteroglossia incorporates a web site to manage information and a Google Docs add-on for writing.Users start by creating characters (A) and forming teams of characters (B and C) on the Heteroglossia website. After setting up the characters andteams, users start writing their own story in Google Docs. When they get stuck, users can select a story snippet to initiate an ideation task throughHeteroglossia (D) and acquire follow-up story ideas (E). with selected characters highlighted in gray. Existing teamsare listed row by row, as shown in Figure 2C, and are availableto edit and delete.
In-Situ Story Ideation with the Crowd
Heteroglossia adopts Google Docs as its main platform, takingadvantage of its convenient and well-maintained comment functionality. The Google Docs add-on is implemented inGoogle Apps Script. As shown in Figure 2D, users can easilyinitiate a new ideation task in Heteroglossia by (i) selectinga snippet of text on the Google Docs, which Heteroglossiauses as the story prompt (the “Content” field), (ii) picking upa suitable team for role-play (using the “Team” button), and (iii) writing down guiding information to workers in the “Note”field (optional).After submitting a new ideation task, Heteroglossia will au-tomatically generate corresponding pages and create HumanIntelligence Tasks (HITs) on MTurk to recruit workers. Atthe same time, Heteroglossia will create a new comment that overviews this ideation task ( e.g., which team is used, whichcharacters are in this team) and associate it with the selectedtext in Google Docs. The first comment in Figure 2E is an ex-ample. Upon receiving a worker’s assignment, Heteroglossiawill present the story idea to the user as a reply to the initialoverview comment. Figure 2E shows four different story ideasfrom both characters. Since Google doesn’t provide enoughsupport to manipulate comments, Heteroglossia’s commentfunction is implemented by Google Doc API, Google DriveAPI, and a bot built with Selenium.
Worker Interface
The worker interface contains an instruction pane, a storypane, and an idea pane, as shown in Figure 3. User-definedinformation, a character description, and a task note are givenin the instruction pane. Workers are required to read the storyprompt in the story pane and enter an idea in the idea pane.To emphasize the role-play strategy, we display the charactername in all three panes highlighted in red. To prevent workersfrom behaving differently from our expectations, three rules igure 3. Heteroglossia worker interface. The interface contains an in-struction pane, a story pane, and an idea pane. are implemented on the worker interface: a 30-second timelock for HIT submission, a reach-to-the-bottom check for thestory prompt, and a prohibition of copy-paste functionality inthe idea pane.
Dynamic Payment for Workers
We proposed a formula to dynamically estimate working timeand set up corresponding payment for workers. The estimationis based on two factors: reading comprehension and writing.The average reading speed of English native speakers is 200-300 words per minute with reasonable comprehension whenusing LCD monitors [35]. We empirically estimated thatwriting a fifty-word-long story idea in Heteroglossia takesapproximately 5-6 minutes. Aiming at providing a $10 hourlywage, we implemented the formula as follows:
Cost ( HIT ) =
Cost ( Reading ) +
Cost ( W riting )= $ ( words / ) + $1 . words refers to the word count of the story prompt.Heteroglossia then creates HITs with the reward dynamicallycomputed according to the designed formula. STUDY 1: THE EFFECTS OF ROLE PLAY STRATEGY
The goal of Heteroglossia is to provide inspiring ideas tocreative writers, especially when they get stuck during writing.Heteroglossia particularly uses an ideation strategy called “roleplay.” To understand the effects of the role-play strategy andinform the design of Heteroglossia, we conducted two sets ofexperiments. We would like to answer these two questionsthat are motivated by literature: (i) can the role play strategyproduce more useful story ideas? and (ii) what are some trade-offs of using this strategy?
Role Play Produces Semantically-Far Story Ideas
Per Chan et al. [6, 5], when a creator reaches an impasse,ideas that are semantically far from current working ideasare more helpful than those that are nearer. Chan’s workis powered by the Search for Ideas in Associative Memory
Figure 4. The overview of the Study 1. Two settings of ideas were col-lected: (A) role-play and (B) no-role. Notice that the same number ofideas was collected for a fair comparison. (SIAM) theory [30] and verified with crowd ideation experi-ments. SIAM [30] assumes that idea generation is proceed intwo stages, knowledge activation and idea production. In thefirst stage, an image will be retrieved according to the problem.The given image is assumed to have several features that isthen used to generate ideas. Chan et al. [6, 5] showed thatin the idea production stage, relevant stimulations help gener-ate more ideas. However, after exhausting the related ideas,semantically far stimulations would help people to changethe category of images and thus generate more ideas. Apply-ing Chan’s conclusion to our system says that the theoreticalprerequisites for resolving writer’s block are to come up withstory ideas that with greater semantic distance from the currentworking draft. In this subsection, we conducted a set of exper-iments to examine if role-play strategy results in semanticallydistant ideas.
Pilot Study:
The overview procedure of the pilot study isshown in Figure 4. We first conducted a pilot study usingfive Taiwanese folk stories . These stories are unfamiliar tomany crowd workers in order to simulate the workers’ sense offreshness when reading the stories. We then took the first 30%and first 60% of each story (based on word count) to simulatea writer’s working story drafts. For each of these ten (5 × (story draft, character) tuple, we recruited five workersfrom MTurk to read the story draft and provide a follow-upstory idea in free text from the character’s perspective. Forcomparison, we also recruited ( character ×
5) workers foreach draft without assigning any characters, and asked workersto write story ideas. We paid $0.5 for 30% story drafts and$0.8 for 60% story drafts. Each worker was allowed to workon each story once, i.e. , five was the maximum. In total, 105workers participated in our pilot study.The pilot study leads to four main findings: . Paragraph-level semantic distance measurement isneeded.
Chan et. al ’s work either focused on short text [6]or large-scale ideation data [5]. However, in Heteroglossia,the prompts and ideas will be in paragraphs, necessitat-ing a paragraph-level semantic distance measurement. Weintroduce using doc2vec to measure semantic distances au-tomatically, which we will describe in later subsections.2.
The role play strategy resulted in longer semantic dis-tance.
Using doc2vec (Wiki), we estimated the semanticdistance between the story draft and ideas. The ideas thatcame from workers with a role measured 0.490 and withouta role measured 0.468.3.
There’s a need to know where the writer actually gotstuck.
The story drafts used in the pilot study were notactually interrupted where the writer got stuck. Some of thedrafts were even segmented at where the follow-up plot isstraightforward.4.
Story prompts could miss critical context.
Workerssometimes provided ideas that conflicted with the core char-acter setting because they simply did not know it. Priorworks have demonstrated that maintaining context is criticalfor designing efficient crowdsourcing workflows. We willallow users to supplement the background information ofeach character in Heteroglossia.
Data Preparation:
In response to the need of knowing wherethe writer actually got stuck, we acquired the data collectedby the Creative Help system [34] for further study. CreativeHelp is an online writing application where users can freelywrite stories. When the writer explicitly requests for help,the system automatically generates suggestions for the nextsentence in a story. Users can modify, delete, or adopt thesuggestions. We considered a user request in Creative Helpas a strong signal indicating the writer gets stuck. CreativeHelp collected 1,078 stories during its deployment between2015 to 2018. A story contains one or more “instances”, eachrepresents a help request sent from the writer to the CreativeHelp system. We removed stories with fewer than 20 sentencesor fewer than three requests, resulting in a total of 107 stories.Each story on average contains 37.8 (SD=17.4) sentences and510.9 (SD=252.8) words. We further removed stories thatare obviously copied from the Internet or generated entirelyby Creative Help without any human-written parts. One co-author and one collaborator then labeled the characters thatappeared in each story, respectively. Only 14 stories, whosecharacters were totally agreed by two annotators, were usedin the following experiment. We segmented each story at the second last request and used it as the story prompt.
Story Ideation Using Role Play Strategy:
We used the sameinterface as Heteroglossia (Figure 3) to collect story ideas.Identical to the pilot study, two conditions, [role] and [no-role], were subject to experiment. In the [role] condition,workers were instructed to imagine that they were one of themain characters in the story and provide a follow-up plot ideain free text. In the [no-role] condition, we asked workers toprovide ideas without any constraints. A total of 330 storieswere collected contributed by 101 workers.
Human Evaluation:
For each received story idea, we re-cruited another five workers from MTurk to rate the semanticdistance to the story prompt. We collected the rating scoresusing a 5-point Likert scale of agreement with the statement,“This story idea is a relevant follow-up of the original storyprompt.” (1 = Strongly Disagree, 5 = Strongly Agree.) Table 2shows the results. The story ideas collected in the [role] condi-tion had an average relevance score of 3.869, while the ideasin the [no-role] condition had an average relevance score of3.998. The difference is statistically significant (paired t-test, p < . N = therole play strategy generated semantically further ideas. Researchers used MTurk to evaluate creative works and re-sulted in high inter-annotator agreements [6, 25]. However,some prior works also raised concerns about using non-expertsto assess creative works such as graphic designs [16] and po-ems [17]. To further examine our findings, in the followingsubsection, we measured the semantic distance between textsnippets using vector representations.
Automatic Evaluation:
Most of the automated distance mea-sures require first representing text snippets as numeric vectors,called “document representation.” Previous studies [6], whereideas were short pieces of text, simply summed up the corre-sponding GloVe vectors [31] and used cosine similarity fordistance measurement. However, in our study, the collectedideas on average contains 78.0 words (SD=27.8) and thusrequire a paragraph-based representation. To this end, we ex-perimented using the following six document representationsto measure semantic distance:1.
GloVe:
We used pretrained GloVe vectors(glove.6B.300d) [31]. The document vector was ob-tained by summing up the corresponding word vectors and1 − cosine similarity was applied for distance measurement.2. Doc2Vec (Wikipedia):
We used the Doc2Vecmodel [23, 22], which was trained on the Wikipediadataset (github.com/jhlau/doc2vec), and applied1 − cosine similarity as a function for distance mea-surement.3. Doc2Vec (News):
Same as
Doc2Vec (Story):
Same as
Skip-thought Vector:
We used the pretrained skip-thoughtmodel [20] to encode the document and applied 1 − cosine similarity as a function for distance measurement.6. Sentence-level Skip-thought Vector (Mean):
We seg-mented a document into sentences first and encoded eachsentence using the pretrained skip-thought model. Thus,the document can be represented as a set of vectors V sst = { v , v , ..., v n }. When computing the distance be-tween two sentence-level skip-thought vectors, we com-puted 1 − cosine similarity among pairs. The mean overthe distances of each pair was used as the distance measure. etric GloVe D2V-Wiki D2V-News D2V-Story ST S-ST Mean S-ST Min S-ST Median ρ -0.132 -0.153 -0.021 0.202 0.129 0.178 τ -0.053 -0.128 Table 1. Correlation between human-rated relevance and automatic evaluation scores. Note that a good semantic distance indicator should negativelycorrelate with relevance scores. Only GloVe and Doc2Vec-Story generate scores that are negatively correlated to relevance in both Pearson ( ρ ) andKendall ( τ ) correlation coefficients. Doc2Vec-Story is a stronger indicator than GloVe because it yields higher correlation scores.No-Role Role d Mean 95% CI Mean 95% CIRelevance *3.998 [3.925, 4.070] 3.869 [3.789, 3.948] 0.89
Table 2. Relevance of ideas, rated by human judges. The role play strat-egy (Role) generated semantically further ( i.e., less relevant) ideas. (*: p < . . Paired t-test. N = . Cohen’s d reported as [no-role] - [role].Large effect size: | d | > . .)No-Role Role d Mean 95% CI Mean 95% CID2V-Story
GloVe
Table 3. Automatic evaluation metrics of semantic distance. Both D2V-Story and GloVe methods suggested that the story ideas collected in the[Role] condition had an longer semantic distance to the story prompt.(Cohen’s d reported as [no-role] - [role]. Small effect size: | d | > . .) Sentence-level Skip-thought Vector (Min):
Same as min over the distances of every pair as thedistance measure.8.
Sentence-level Skip-thought Vector (Median):
Same as median over the distances of every pair asthe distance measure.In order to evaluate how well these methods reflect humanjudgements, we calculated the correlation coefficients betweenthe automatic score and human scores collected above. Notethat a good semantic distance indicator should negatively corre-late with relevance scores. Table 1 shows that only
GloVe andDoc2Vec-Story generate scores that are negatively corre-lated to human judgements of relevance in both Pearson( ρ ) and Kendall ( τ ) correlation coefficients, where Doc2Vec-Story is a stronger indicator than GloVe because it yieldshigher correlation scores.Finally, we used GloVe and Doc2Vec-Story to measure thesemantic distance automatically. Table 3 shows that both meth-ods suggest the story ideas collected in the [role] conditionhad an longer semantic distance to the story prompt. Theseautomatic measurement can be used in Heteroglossia to au-tomatically rank the usefulness of received story ideas, or tofilter out ideas that are abnormally similar to each other. Trade-offs Between Task Structures and Creativity
Per Kim et al. [18], in collaborative story writing, task struc-tures and creativity have some trade-offs. Too little structureleads to “unfocused, sprawling narratives”, and too muchstructure “stifles creativity.” The role play strategy enforces aschema of characters for ideation and could possibly sacrificethe quality or creativity of story ideas submitted by workers.
Aspects No-Role Role d Mean 95% CI Mean 95% CILegitimate **3.97 [3.882, 4.052] 3.81 [3.735, 3.885] 1.02
Creative
Interesting
Willing-to-Read *3.60 [3.473, 3.733] 3.49 [3.396, 3.587] 0.51
Surprising *3.37 [3.242, 3.503] 3.23 [3.105, 3.348] 0.61
Table 4. Trade-offs between task structures and creativity. Five humanjudges on MTurk rate each story ideas on the following five aspects, us-ing a 5-point Likert scale: Legitimate, Creative, Interesting, Willing-to-Read, and Surprising. (*: p < . ; **: p < . . Paired t-test. N = .Cohen’s d reported as [no-role] - [role]. Medium effect size: | d | > . .Large effect size: | d | > . .) To understand the effect of role play strategy thoroughly, weconducted experiments to examine this possible trade-off.For each story idea received in Study 1, we recruited fiveworkers from MTurk to rate the quality of the idea in vari-ous aspects , using a 5-point Likert scale of agreement (1 =Strongly Disagree, 5 = Strongly Agree.) Table 4 shows theresults, which echo Kim et al. ’s observation that enforcingtask structure could stifle creativity. As for design implication,this trade-off should be made explicit to users and allow themto freely decide which strategy to use. STUDY 2: DEPLOYMENT WITH CREATIVE WRITERS
To understand how writers would use Heteroglossia, we con-ducted a three-day deployment study with two experiencedcreative writers and held pre-study and post-study interviews.
Participants
Two experienced creative writers, P1 and P2, participated inour deployment study. Both participants are women and nativespeakers of American English. They were recruited throughour personal networks. P1 has been writing since she couldpick up a pencil and has always written stories. She wrote a lotof fan fiction in middle school and high school. P1 receivedan English minor in undergraduate school. She started writingher own book in 2018 and is currently at the stage of pitchingagents for publication. P1 has also translated video gamesfrom Japanese into English. P1 has never done technicalwriting, and her main genre focus is fantasy. P2 Legitimate (“This story idea makes sense given the story prompt.”),Creative (“This story idea is creative.”), Interesting (“This story ideais interesting.”), Willing-to-Read (“I’m willing to read the final storythat is written based on this story idea.”), and Surprising (“This is asurprising story idea.”) igure 5. User activity logs of P1 shown in the cumulative word count with respect to time. When different numbers of characters were used, theresulting number of ideas varied. Therefore, the total number of ideas is shown in the legends. As we can see, participants usually requested ideas andpaused writing. After hours, when most of ideas had appeared, they came back and resumed writing.Figure 6. User activity logs of P2 shown in the cumulative word countwith respect to time. P2 requested multiple tasks at the same time. great deal. She has done technical writing before and mainlyfocuses on science fiction and science fantasy. P2 uses GoogleDocs as her primary text editor.
Study Protocol
Before the study, we had a semi-structured pre-study inter-view via Skype with the participants to understand their back-grounds, needs, and creative writing processes. At the endof the pre-study interview, one of the authors gave a brieftutorial of Heteroglossia and demonstrated how to use thesystem. Note that we explicitly informed the participants thatHeteroglossia is a crowd-powered system and that their storieswould be viewed by online crowd workers. In the study, theparticipants were asked to use Heteroglossia to write a storyof approximately 1,000 words. We asked the participants tofinish the story in a time span of three days, during whichthey needed to use Heteroglossia’s ideation function at leastthree times when writing their stories. After the study, wehad a semi-structured post-study interview via Skype withthe participants to understand their experience and feedback.The pre- and post-study interviews were both around thirtyminutes long. The audio was recorded and transcribed by theauthors. Each participant was compensated with $50 after thepost-study interview. Table 5 shows one example of crowdideation created by P2.
How Did the Participants Use Heteroglossia?
To capture how P1 and P2 used the system, we plotted the evo-lution of the cumulative word count to visualize their writingprogress, aligned with the time they requested ideas. Figure 5 and 6 show the logs from P1 and P2, respectively. Both par-ticipants usually requested ideas and paused writing, whichmight signal getting stuck. After a few hours, the participantscame back, read all the returned story ideas and continued writ-ing. We also asked participants about how they interacted withHeteroglossia in the post-study interview. Both participantswrote sequentially without any outlines.“... I would write until I didn’t know what to do next andthen I would use the tool. The next day, I would readover everyone’s responses and then write until I got stuckand then use the tool.” (P1)P2 finished all the writing within a day, so she tried variouslengths of story prompts and launched several requests at thesame time, as shown in Figure 6. Note that we allowed theparticipants to write freely (see the “Study Protocol” Section)and did not enforce any writing processes.
Findings
We summarize our findings of the study below, supported byquotes from the participants.
The output of Heteroglossia is interesting.
Both P1 and P2expressed that the ideas are interesting and fun:“Yeah, there are some very, very creative answers in there.Some people would just be like ... “well, if I was thischaracter, I would do this, this, and this.” ... Some peoplewould write a whole paragraph continuing the story. AndI thought that was really interesting.” (P1)“I really like it; it’s pretty fun... that it came up withinteresting stuff. There’s one... “Oh my gosh, that weirdo.I don’t like her, booo.” And it’s just so funny... One ofmy favorite comments was like, [P2 read one idea] I waslike, Oh, that’s really interesting.... I thought that wasreally fun.” (P2)
Heteroglossia is useful in generating inspiration.
Both P1and P2 think Heteroglossia can be useful for getting inspiringideas from the crowd.“Yeah, it’s helpful, even if I don’t use their ideas.” (P1)“It was nice when I got stuck on what to do next to be ableto ask people. That gave me more inspiration to continueand also more insight into what people’s expectationswere for the story.” (P1) tory Prompt
Detective Opal considered her seargant. Like all werewolves, Seargant Subwoofer looked like a normal person most of the time–he onlywent furry during the full moon–and so he could be contained in a Full Metal concrete barrier, and no harm would happen. Because of this,Werewolves were considered a class-A Supernatural. Other Supernaturals–who were a bigger threat to humanity–had classes B, C, D, and F.Even though Seargant Subwoofer was one of the least dangerous, I could see how people looked at him–it grated on him, I could tell.
Detective Opal - Detective Opal has a murder to solve in a fantasy world: and it’s not as obvious who the killer is that she thought
1. I would have to clear the Sergeant’s name even though I had a my doubts about his guilt. Finding his alibi should be easy enough as Iclearly remember seeing a camera that eyed the only way in and out of his concrete self imposed prison. I can’t deny that I how odd itwas though when he asked to see the warrant for the tape I requested.2. It bothered him so much that one day he hired a witch to remove the curse that was placed on him. The curse that turned him into awerewolf. After the curse was removed he aged significantly. Being a werewolf kept him young and now he was his real age. He didn’tcare though, because he was finally free from the curse.3. Detective Opal is one of the unique supernaturals. She doesn’t change every full moon, but rather has the powers to change betweenhuman and werewolf whenever she pleases. She generally uses the powers during investigation, smelling for scents, and chasing downperpetrators. Her supernatural abilities come in handy as a detective.
Siren Eris - Eris is a siren. She is being accused of murder, but she is innocent.
1. I was aware of how they could view me as a potential suspect. We were known to have our fights and screaming matches, but it neverwent any further. I was determined to set the record straight. I knew that the best way to do that was to provide them with my alibi.2. I will approach them telling them the true story of how the crime happened. if there is a negative response, i will now tell them i wish tosee sergeant Subwoofer and explain myself to him and beg him to show empathy on me. if it does not bring a positive result i will ask totalk to their boss.3. Eris sat in the interrogation room wringing her hands. "I was just in the wrong place at the wrong time. There can’t be any conclusiveevidence against me." She though to herself. Detective Opal entered the room and shut the door behind her. She sat across the table fromEris and looked deep into her eyes. "Things aren’t looking good for you." She started solemnly.
Dead Doctor - Doctor is dead. She is the murder that the detective is trying to solve.
1. Lying on the floor, not much I can do. They think I’m dead, but I’m not. They should know this, they checked my pulse, I can’t be dead.Am I dead? As I wonder, I see my whole life replay back before my eyes, it felt like years, but in reality it was only 20 minutes, and Iregained consciousness, but couldn’t move.2. Having fought and clawed at her attacker until the bitter end she had played her role in helping to solve her own murder. The evidencewas their hiding in plain sight beneath her darkly painted fingernails. Microscopic clumps of fur were embedded into her cuticles justwaiting to plucked and analyzed.3. I lie quietly in the morgue, not moving or seeing or breathing. The slab is quite cold and it is terribly quiet. The detective and the sergeanthad stopped by earlier, but even though the Sergeant was a werewolf, he hadn’t noticed and just thought I was another dead doctor. I am aClass F supernatural, and there are few who can recognize me – for I am the Queen of the Vampires. I wait for night to fall when I willmake my next move and begin hunting them down one by one.
Table 5. An ideation example created by P2. Three characters are involved and their descriptions are listed accordingly.
P2 said that, as an idea generator, Heteroglossia producesrelevant story ideas:“It’s like an idea generator. I was really surprised by howmuch the ideas were actually related to the story.” (P2)P2 also mentioned that she would like to use the system forher next NaNoWriMo:“NaNoWriMo is coming up ... where you have to write awhole novel in a month ... sometimes it can be tricky tocome up with ideas.” (P2)
Writers benefit from Heteroglossia in different ways.
Wenoticed that, although both participants think the system isuseful, the way they used it was slightly different. WhenP1 did not have ideas, people helped her figure out how toproceed, even inspiring the next part of the story.“Very helpful for when I don’t have any ideas ... then Ican ask a lot of other people and they’ll help me figureit out. Even if I don’t take any of their ideas, ... it mightinspire something else that I will think of for the nextpart of the story.” (P1)P1 also agreed that Heteroglossia can capture the personalityof the role that is assigned to it.“Again, I think it depends on how much informationthe writer ... gives the people taking on the roles. And I think it can also (help) even if people don’t get the correctpersonality, it still helps you learn ... “well, they wouldn’tdo that, but they would do this instead.” So it’s still it’shelpful to rule things out in that way too.” (P1)For P2, the ideas can be relevant to the characters: either theymatched the character’s personality or figured out a personalitythat had not been provided by P2.“...There’s also... some things that are actually relevantto the characters. One of the characters ... was verydramatic. And then it (Heteroglossia) came up with thisidea that she would go make a lot of money and go toVegas and like, sip martinis on an island somewhere...That’s exactly like that!” (P2)“There was another one... the surgeon says “I’m goingto do nothing. Except taking aspirin for my headache.”And I was like, wait, Heteroglossia remembered that hehad a headache!... I didn’t give them that much to go on.And it still had some personality figured out.” (P2)Not all the ideas was used in the stories, but they were stillconsidered useful by participants because they inspired newthought. For example, P2 specified in the role profile that thecharacter “Dead Doctor” is a human, but a worker wrote anidea saying this character is a “supernatural” (the last row inTable 4). igure 7. The histogram of the latency for getting responses from Het-eroglossia. A total of 81 responses were collected. “(the worker) decided that one of my characters was asupernatural when I had said they were human. But this(the idea of setting “Dead Doctor” as a “supernatural”)sounds like that’s an interesting way of doing this.” (P2)Although P2 did not adopt this idea, this idea gave her story anew interesting direction to go. Such ideas stimulated partici-pants ideas and thus were still considered useful.
System latency did not greatly affect writing.
We investi-gated the system latency statistic and the resulting impact onusers. System latency is defined as the duration between thetime the overview comment was automatically created andthe time the story idea was received. A histogram of systemlatency is shown in Figure 7. The latency for getting the firstidea was around 15 minutes (median=14.42; mean=23.64;SD=25.14). For each character to get at least one idea, thelatency was about 50 minutes (median=53.32; mean=56.67;SD=30.47). The latency for getting the last idea was about160 minutes (median=167.12; mean=167.68; SD=52.72).We asked the participants if the system latency disrupted theirwriting processes. Both P1 and P2 said that the fact thatHeteroglossia needs a certain amount of time to respond didnot affect their writing.“... because I had other things that I was doing. So, Iwould write, and then I would â ˘A˛e do the other thingsduring the day that I needed to do... I kind of had aschedule so it didn’t really affect my time.” (P1)“Probably, but it wasn’t too bad... when you’re comingup with stuff, it’s always ... a long process anyway...it’s not really like, “okay, I got the whole idea downnow” ... some of the ideas, it’s kind of like “that’s reallyinteresting; maybe I’ll go back and change it.” But it’snot that big of a deal to go back and change it becauseyou have to ... make like four or five drafts anyway.” (P2)
The role play strategy does not fit some use cases.
Both par-ticipants pointed out some problems they encountered, someof which were caused by the nature of the role play strategy. P1would like to use the baseline strategy (no-role) and dynamicteam management in some cases:“I’m conflicted between wanting to be able to assign acertain task to just one character versus the whole teambecause sometimes characters don’t fall in... I wouldimagine if you have a larger cast of characters, the teamswould overlap quite a bit, so it might be easier to be ableto... assign tasks to single characters.” (P1) P1 also pointed out that some scenarios might be hard to usethe role play strategy, since the structure of the story will betoo complex:“... it depends on how detailed you are when you writethe character, because I only wrote a couple sentencesfor my characters, but if you wrote ... a whole biography,then maybe. I think it depends on the complexity of thestory and the complexity of the characters.” (P1)
Working with stranger workers have trade-offs.
We askedthe participants to compare working with strangers versusworking with friends, families, or colleagues. P1 explainedthe trade-offs between them.“(Using Heteroglossia) It’s less pressure because youdon’t know the people, but it’s also a little more nerve-wracking because you don’t know the people. So there’sgood and bad... It’s better to have someone that youknow and have a good relationship with. It’s hard to truststrangers with a story, especially a story as complex as abook.” (P1)Copyright issues were also raised if users were to use Het-eroglossia for their own professional work.“I also think that if a professional writer was going touse the tool in their professional work, it might raisecopyright issues. ... If you are getting ideas from otherpeople, and you implement them in a book that you’regoing to sell, who gets the credit for them?” (P1)
Handling overwhelming number of story ideas.
Both par-ticipants thought the number of ideas provided by Heteroglos-sia was overwhelming.“Depending on how big it gets, it might be overwhelmingto have to read through all those responses.” (P1)“It’s not that there are too many it’s just that I didn’trealize how many.” (P2)
DISCUSSION
In this section we discuss topics that are broader than the scopeof the Heteroglossia system.
Differences Between Technical and Creative Writing
P2 in Experiment 2 had experience in both technical writingand creative writing. In order to better inform our futuresystem design, we asked P2 in pre-study interview what aredifferences between these two. P2 said she thinks technicalwriting is much easier because the goal and style is more clear.“Often (in technical writing) you have a style guide, andyou have the a goal. The goal with technical writingis to make something that’s confusing understandable.Whereas the goal with creative writing is usually to givesome kind of feeling to the reader.” (P2)She further explained that, in creative writing, the writerssometimes need to intentionally avoid clear explanations,whereas technical writing is all about making things clearand understandable.So, with technical writing, you’re just explaining some-thing, you’re trying to make something very clear. Butsometimes with creative writing, you might not necessar-ily want to be making something clear. You might wantto introduce moral quandaries to your reader and makethem think about all of the gray areas and like ... how it’snot as clear as you thought, I guess.
That’s a big, bigdifference. ” (P2)
What Do Writers Do to Resolve Writer’s Block
P1 struggles with plot the most. She said that knowing thecharacters more can help with the situation since charactersand plots are intertwined.“Plot is my weakest skill when it comes to writing... So Ifind that if I really get to know my characters really welland understand the choices that they would make in anygiven situation, then the plot can kind of unravel itselffrom there.” (P1)P2 usually let the characters talk when getting stuck. Even ifthe conversations are deleted afterwards, it helps her under-stand the character more.“When I get stuck, what I usually do is just have char-acters talk about dumb stuff... just have two characterstalk to each other... And then sometimes you figure outmore about what you want to do by that conversation...It helps you understand the characters more, if you havethem talk to each other. And then knowing “now I knowthat this person wants to do this.”” (P2)
How Do Writers Write
P1 stated that her writing process was to come up with an idea,create characters, and finally, design a plot. P1 also said thatshe would write one draft first and later revise for plot andcharacter.“The idea always comes first. So I always have “what ifthis happened; that would be an interesting story.” Andthen I create the characters for that idea. The plot comeslast. I will write one draft and then revise for plot andcharacter. But it’s always idea, characters, plot.” (P1)P2 usually thinks about the message she wants to send, picksoverall “concepts of things,” thinks about characters and setsup the inner/outer goals, and figures out what she needs in theplot to satisfy the characters’ needs.“So first, I think about what message I want to send.What ... is something that I want to talk about or discuss?And I’ll pick ... the setting and the time period. Youknow, overall concepts of things like social change thatI want to talk about. Then I’ll think about the character.Because for me, the character drives the plot. ... Usuallycharacters have two goals. The first is the outer goal andthe second is an inner goal. ... You figure out what youwant in the plot based on how it’ll satisfy the characters’needs.” (P2)
Limitations
The need of non-role strategy.
Heteroglossia currently onlysupports a role-playing strategy, so we do not observe any cases where the user prefers to request ideas without roleschema. We will add new features to Heteroglossia to allowusers to request ideas using different strategies.
Scalability.
Our experiment focused on short stories (under1,000 words) with a few characters who have relatively simplebackstories. When working on long stories, the structures andcharacters may become complicated, raising two issues: han-dling a large number of characters and conveying complicatedbackstories. Heteroglossia currently requires users to definecharacters and teams before writing stories. However, whena story has more characters, it can become difficult to handleall the characters and teams. Features such as automaticallysuggesting teams based on context might help. We will alsoexplore automatic summarization technologies to produce orupdate character backstories automatically.
Insufficient amount of participants.
Only two participantswere recruited in this study, as it is hard to recruit creativewriters who are willing to participate in a multiple-day study.A one-day study might be too short for creative writers to comeup with good ideas for stories. In the future, a deploymentseveral months long with more users would allow us to betterunderstand how people interact with Heteroglossia.
Evidence for relieving writer’s block.
In this paper, weshowed that the role-playing strategy produces semanticallyfar story plot ideas (Study 1), and participants were satisfiedwith the ideas provided by Heteroglossia (Study 2). How-ever, we did not directly examine whether the system helpedrelieve writer’s block. Evaluating the usefulness of an ideais challenging because an idea can still be considered usefulor inspiring even if it is not directly adopted. A large-scaledeployment will allow us to observe whether writers use theideation feature frequently, which could better validate theusefulness of Heteroglossia.
CONCLUSION AND FUTURE WORK
This paper introduces Heteroglossia, a crowd-powered systemthat applied crowd ideation to help creative writers. We builtHeteroglossia as a Google Docs add-on, and writers can sim-ply use the editor to elicit story ideas with the online crowd.Heteroglossia adopts the role play strategy for story ideation.In controlled experiments, this strategy produced story ideasthat are semantically more distant to the working story draft,which is known to be more useful to creator who reaches animpasse. We also conducted a deployment study with twoexperienced creative writers. In the deployment study, wefound that the outputs of Heteroglossia is generally interestingand useful, while two participants benefit from the system indifferent ways. In the future, we will relax the definition of“characters” in our system, allowing writers to use these “roles”in Heteroglossia in a more general way. For example, each rolecould be a “thinking hat” that represents one perspective [10].
ACKNOWLEDGEMENTS
We thank Sooyeon Lee, Tiffany Knearem, Chi-Yang (Ethan)Hsu, Lisa Yu, and Frank Ritter for their valuable feedback andhelp. We also thank the workers on Mechanical Turk whoparticipated in our studies.
EFERENCES [1] Elena Agapie, Jaime Teevan, and AndrésMonroy-Hernández. 2015. Crowdsourcing in the field:A case study using local crowds for event reporting. In
Third AAAI Conference on Human Computation andCrowdsourcing .[2] Salvatore Andolina, Hendrik Schneider, Joel Chan,Khalil Klouche, Giulio Jacucci, and Steven Dow. 2017.Crowdboard: augmenting in-person idea generation withreal-time crowds. In
Proceedings of the 2017 ACMSIGCHI Conference on Creativity and Cognition . ACM,106–118.[3] Michael S Bernstein, Greg Little, Robert C Miller, BjörnHartmann, Mark S Ackerman, David R Karger, DavidCrowell, and Katrina Panovich. 2010. Soylent: a wordprocessor with a crowd inside. In
Proceedings of the23nd annual ACM symposium on User interfacesoftware and technology . ACM, 313–322.[4] Joel Chan, Steven Dang, and Steven P Dow. 2016.Improving crowd innovation with expert facilitation. In
Proceedings of the 19th ACM Conference onComputer-Supported Cooperative Work & SocialComputing . ACM, 1223–1235.[5] Joel Chan, Steven P Dow, and Christian D Schunn.2018. Do the best design ideas (really) come fromconceptually distant sources of inspiration? In
Engineering a Better Future . Springer, 111–139.[6] Joel Chan, Pao Siangliulue, Denisa Qori McDonald,Ruixue Liu, Reza Moradinezhad, Safa Aman, Erin TSolovey, Krzysztof Z Gajos, and Steven P Dow. 2017.Semantically Far Inspirations Considered Harmful?:Accounting for Cognitive States in CollaborativeIdeation. In
Proceedings of the 2017 ACM SIGCHIConference on Creativity and Cognition . ACM, 93–105.[7] Yung-Yi Juliet Chou and Barbara Tversky. 2017.Finding Creative New Ideas: Human-Centric MindsetOvershadows Mind-Wandering.. In
CogSci .[8] Elizabeth Clark, Anne Spencer Ross, Chenhao Tan,Yangfeng Ji, and Noah A. Smith. 2018. Creative Writingwith a Machine in the Loop: Case Studies on Slogansand Stories. In . ACM, New York,NY, USA, 329–340.
DOI: http://dx.doi.org/10.1145/3172944.3172983 [9] Colette Daiute. 1989. Play as thought: Thinkingstrategies of young writers.
Harvard educational review
59, 1 (1989), 1–24.[10] Edward De Bono. 2017.
Six thinking hats . Penguin UK.[11] Richard P Gabriel, Jilin Chen, and Jeffrey Nichols. 2015.InkWell: A Creative Writer’s Creative Assistant. In
Proceedings of the 2015 ACM SIGCHI Conference onCreativity and Cognition . ACM, 93–102.[12] John L Geiger and Howard Suber. 2019.
Creativity andCopyright: Legal Essentials for Screenwriters andCreative Artists . University of California Press. [13] Nick Greer, Jaime Teevan, and Shamsi T Iqbal. 2016.
An introduction to technological support for writing .Technical Report. Technical Report. Microsoft ResearchTech Report MSR-TR-2016-001.[14] Nathan Hahn, Joseph Chang, Ji Eun Kim, and AniketKittur. 2016. The Knowledge Accelerator: Big picturethinking in small pieces. In
Proceedings of the 2016CHI Conference on Human Factors in ComputingSystems . ACM, 2258–2270.[15] Yi-Ching Huang, Jiunn-Chia Huang, Hao-Chuan Wang,and Jane Yung-jen Hsu. 2017. Supporting ESL Writingby Prompting Crowdsourced Structural Feedback. In
Fifth AAAI Conference on Human Computation andCrowdsourcing .[16] Karl K Jeffries. 2017. A CAT with caveats: is theConsensual Assessment Technique a reliable measure ofgraphic design creativity?
International Journal ofDesign Creativity and Innovation
5, 1-2 (2017), 16–28.[17] James C Kaufman, John Baer, Jason C Cole, and Janel DSexton*. 2008. A comparison of expert and nonexpertraters using the consensual assessment technique.
Creativity Research Journal
20, 2 (2008), 171–178.[18] Joy Kim, Justin Cheng, and Michael S Bernstein. 2014.Ensemble: exploring complementary strengths ofleaders and crowds in creative collaboration. In
Proceedings of the 17th ACM conference on Computersupported cooperative work & social computing . ACM,745–755.[19] Joy Kim, Sarah Sterman, Allegra Argent Beal Cohen,and Michael S Bernstein. 2017. Mechanical novel:Crowdsourcing complex work through reflection andrevision. In
Proceedings of the 2017 ACM Conferenceon Computer Supported Cooperative Work and SocialComputing . ACM, 233–245.[20] Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov,Richard Zemel, Raquel Urtasun, Antonio Torralba, andSanja Fidler. 2015. Skip-thought vectors. In
Advances inneural information processing systems . 3294–3302.[21] Aniket Kittur, Boris Smus, Susheel Khamkar, andRobert E Kraut. 2011. Crowdforge: Crowdsourcingcomplex work. In
Proceedings of the 24th annual ACMsymposium on User interface software and technology .ACM, 43–52.[22] Jey Han Lau and Timothy Baldwin. 2016. An empiricalevaluation of doc2vec with practical insights intodocument embedding generation. arXiv preprintarXiv:1607.05368 (2016).[23] Quoc Le and Tomas Mikolov. 2014. Distributedrepresentations of sentences and documents. In
International conference on machine learning .1188–1196.[24] Boyang Li and Mark Riedl. 2015. Scheherazade:Crowd-powered interactive narrative generation. In
Twenty-Ninth AAAI Conference on Artificial Intelligence .25] Kurt Luther, Jari-Lee Tolentino, Wei Wu, Amy Pavel,Brian P Bailey, Maneesh Agrawala, Björn Hartmann,and Steven P Dow. 2015. Structuring, aggregating, andevaluating crowdsourced design critique. In
Proceedingsof the 18th ACM Conference on Computer SupportedCooperative Work & Social Computing . ACM, 473–485.[26] James Maxey. 2007. You Never Write Alone. (Nov2007). http://jamesmaxey.blogspot.com/2007/11/you-never-write-alone.html [27] Merriam-Webster. 2014. Heteroglossia. (2014). [28] Nasrin Mostafazadeh, Nathanael Chambers, XiaodongHe, Devi Parikh, Dhruv Batra, Lucy Vanderwende,Pushmeet Kohli, and James Allen. 2016. A corpus andevaluation framework for deeper understanding ofcommonsense stories. arXiv preprint arXiv:1604.01696 (2016).[29] Michael Nebeling, Alexandra To, Anhong Guo,Adrian A de Freitas, Jaime Teevan, Steven P Dow, andJeffrey P Bigham. 2016. WearWrite: Crowd-assistedwriting from smartwatches. In
Proceedings of the 2016CHI Conference on Human Factors in ComputingSystems . ACM, 3834–3846.[30] Bernard A Nijstad and Wolfgang Stroebe. 2006. Howthe group affects the mind: A cognitive model of ideageneration in groups.
Personality and social psychologyreview
10, 3 (2006), 186–213.[31] Jeffrey Pennington, Richard Socher, and ChristopherManning. 2014. Glove: Global vectors for wordrepresentation. In
Proceedings of the 2014 conferenceon empirical methods in natural language processing(EMNLP) . 1532–1543.[32] Eva Novrup Redvall. 2012. European TV Drama SeriesLab. Summary of Module 1. (2012).[33] Melissa Roemmele. 2018.
Neural Networks forNarrative Continuation . Ph.D. Dissertation. Universityof Southern California.[34] Melissa Roemmele and Andrew S Gordon. 2015.Creative help: a story writing assistant. In
InternationalConference on Interactive Digital Storytelling . Springer,81–92. [35] Eva Siegenthaler, Yves Bochud, Per Bergamin, andPascal Wurtz. 2012. Reading on LCD vs e-Ink displays:effects on fatigue and visual strain.
Ophthalmic andPhysiological Optics
32, 5 (2012), 367–374.[36] Jaime Teevan, Shamsi T Iqbal, and Curtis Von Veh.2016. Supporting collaborative writing with microtasks.In
Proceedings of the 2016 CHI conference on humanfactors in computing systems . ACM, 2657–2668.[37] Jaime Teevan, Daniel J Liebling, and Walter S Lasecki.2014. Selfsourcing personal tasks. In
CHI’14 ExtendedAbstracts on Human Factors in Computing Systems .ACM, 2527–2532.[38] Jaime Teevan and Lisa Yu. 2017. Bringing the Wisdomof the Crowd to an Individual by Having the IndividualAssume Different Roles. In
Proceedings of the 2017ACM SIGCHI Conference on Creativity and Cognition .ACM, 131–135.[39] Polly W Wiessner. 2014. Embers of society: Firelighttalk among the Ju/’hoansi Bushmen.
Proceedings of theNational Academy of Sciences
Proceedings of the 17thACM conference on Computer supported cooperativework & social computing . ACM, 1433–1444.[41] Anbang Xu, Huaming Rao, Steven P Dow, and Brian PBailey. 2015. A classroom study of using crowdfeedback in the iterative design process. In
Proceedingsof the 18th ACM conference on computer supportedcooperative work & social computing . ACM,1637–1648.[42] Lixiu Yu, Aniket Kittur, and Robert E. Kraut. 2016.Encouraging “Outside- The- Box” Thinking in CrowdInnovation Through Identifying Domains of Expertise.In
Proceedings of the 19th ACM Conference onComputer-Supported Cooperative Work & SocialComputing (CSCW ’16) . ACM, New York, NY, USA,1214–1222.