Seeker or Avoider? User Modeling for Inspiration Deployment in Large-Scale Ideation
Maximilian Mackeprang, Kim Kern, Thomas Hadler, Claudia Müller-Birn
SSeeker or Avoider? User Modeling for Inspiration Deploymentin Large-Scale Ideation
Maximilian Mackeprang [email protected] Computing, Freie Universität Berlin
Kim Kern [email protected] Computing, Freie Universität Berlin
Thomas Hadler [email protected] Computing, Freie Universität Berlin
Claudia Müller-Birn [email protected] Computing, Freie Universität Berlin
ABSTRACT
People react differently to inspirations shown to them during brain-storming. Existing research on large-scale ideation systems hasinvestigated this phenomenon through aspects of timing, inspi-ration similarity and inspiration integration. However, these ap-proaches do not address people’s individual preferences. In theresearch presented, we aim to address this lack with regards toinspirations. In a first step, we conducted a co-located brainstorm-ing study with 15 participants, which allowed us to differentiatetwo types of ideators: Inspiration seekers and inspiration avoiders.These insights informed the study design of the second step, wherewe propose a user model for classifying people depending on theirideator types, which was translated into a rule-based and a randomforest-based classifier. We evaluated the validity of our user modelby conducting an online experiment with 380 participants. Theresults confirmed our proposed ideator types, showing that, whileseekers benefit from the availability of inspiration, avoiders wereinfluenced negatively. The random forest classifier enabled us todifferentiate people with a 73 % accuracy after only three minutesof ideation. These insights show that the proposed ideator typesare a promising user model for large-scale ideation. In future work,this distinction may help to design more personalized large-scaleideation systems that recommend inspirations adaptively.
CCS CONCEPTS • Information systems → Crowdsourcing ; •
Human-centeredcomputing → User models ; User studies . KEYWORDS
Creativity; brainstorming; large-scale ideation; adaptive systems
ACM Reference Format:
Maximilian Mackeprang, Kim Kern, Thomas Hadler, and Claudia Müller-Birn. 2020. Seeker or Avoider? User Modeling for Inspiration Deploymentin Large-Scale Ideation. In
Proceedings of Arxiv Preprint (not submitted)(Preprint).
ACM, New York, NY, USA, 11 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn
Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.For all other uses, contact the owner/author(s).
Preprint, January 2020, © 2020 Copyright held by the owner/author(s).ACM ISBN 978-x-xxxx-xxxx-x/YY/MM.https://doi.org/10.1145/nnnnnnn.nnnnnnn
Generating ideas is a complex, multi-faceted process. People ap-proach this process of idea generation, i.e. ideation, differently;they use distinct strategies or employ various methods [6]. Oneestablished technique for improving creativity during ideation isproviding people with others’ ideas (subsequently called inspira-tions). The effect of others’ ideas as an inspiration technique hasled to the original proposal of brainstorming [14]. The rise of digitalmedia allowed the moving of co-located, small-scale brainstorm-ing into large-scale online settings. In the latter, inspirations havebecome a critical aspect in the algorithmic support of brainstorm-ing [20]. The ideation process itself is defined as follows: A problemor topic is provided as a challenge on an online ideation system. In acrowdsourced fashion, participants provide their ideas in individualtime-bounded ideation sessions after reading the challenge. Relatedwork has found that the effectiveness of inspirations depends on aperson’s cognitive state [22], the semantic similarity of the inspi-rations provided [3] and the level of attention a person devotes toan inspiration [7]. However, when summarizing the existing ap-proaches of providing inspirations in large-scale ideation systems,we can surmise that these approaches disregard individual prefer-ences people might have when using inspirations during ideation.This gap in research led to our overarching goal: We envision thatlarge-scale ideation systems consider individual user preferences(e.g. their cognitive state, the available type of inspirations and theparticular context of ideation) in order to provide the most effectiveinspirations during ideation.In the context of this work, we focus on one aspect in the con-text of user modeling and tackle the research question: Can wedistinguish individual preferences towards inspirations in ideationsession and how do these preferences influence the ideation out-come? To approach this question, we conducted an exploratorypre-study, which allowed us to differentiate people who either seekor avoid inspirations during ideation. We translated this observa-tion into an actionable user model and implemented the model in aclassifier. We conducted an online experiment to validate our usermodel, consisting of two sessions. In the first session, participa-tions were classified into one of the two ideator types (seeker oravoider) identified in the pre-study. Based on the type identified,we assign ideators to two conditions during the subsequent second,actual, ideation session. We then evaluated the impact of inspirationavailability on the two types. Finally, we propose and evaluate aheuristic to dynamically detect inspiration seekers and avoiders inthe context of a large-scale ideation session. a r X i v : . [ c s . H C ] F e b reprint, January 2020, Maximilian Mackeprang, Kim Kern, Thomas Hadler, and Claudia Müller-Birn Existing research in the research field of ideation can be dividedinto two main research streams. On the one hand, there is researchfrom psychology, which analyzes mainly individual differences ingroup-brainstorming contexts. Garfield et al., for example, showthat personality types influence the outcome of ideation [6]. Eventhough these personality types might be a promising approach formodeling user preferences, the effect of inspiration has not yetbeen evaluated. Furthermore, Garfield’s research is based on dataobtained in a co-located group setting. Gamper et al. also conducteda study in the context of small scale brainstorming where they gaveparticipants the chance to have their ideas critiqued and reviewedby other participants during a session. They show that in terms ofthis type of feedback, people can be classified into participants want-ing feedback early, and others who deferred it to later [5]. Theseinsights can potentially be transferred to inspirations: Both feed-back and inspiration are instances of external input during ideation.However, while inspiration is inspiring (divergent) input, feedbackis normally viewed as restricting (convergent) input[23]. On theother hand, there is an extensive body of work on inspirations inthe context of large-scale ideation [2, 8, 12, 21, 24, 25]. Siangliu-lue et al., for example, compared different timing mechanisms forproviding inspirations in large-scale ideation systems [22]. Theauthors analyzed the impact of the inspiration mechanism in threeformats: Inspirations are shown to the user (1) on demand, (2) onidle time, i.e. when the user is pausing, or (3) in fixed time intervals.However no user model was deployed in this research–individualpreferences were thereby neglected. In their research, Girotto etal. [8] proposed a user model. When users submit an idea, in theirsystem, they have to choose one or two topical categories for it. Thisinformation is used to build a matrix model of categories to whicheach user has submitted ideas. This model can then be used to infercategories a user is likely to be fluent in (based on collaborativefiltering techniques) and recommend inspirations according to thisinformation. Although this approach introduced a user model, themodel is based only on the categorization of ideas. There was noinvestigation of individual differences regarding how participantsreacted to, or used the inspirations. In summary, research in psy-chology highlights individual differences in ideation; however, thisinsight is evaluated only in co-located group settings. Research inthe area of large-scale ideation focuses on either state modeling(timing) or categories during ideation; individual preferences havenot been considered so far. This situation motivated our research onusers’ preferences when providing inspirations, which is presentedin the following.
While related work investigated the impact of inspiration timing orindividual preferences towards feedback, preferences towards inspi-rations have not yet been analyzed. We conducted an exploratorypre-study to better understand the role of individual preferencestowards inspiration, i.e. how people react to and use inspirations.
The goals of the exploratory study were to identify potential individ-ual preferences of participants and understand how the participants
Figure 1: Study setup for the exploratory study. themselves assess their experiences. As these qualitative insightsare hard to obtain in a large-scale ideation context [10], we decidedto conduct three brainstorming sessions in a co-located group set-ting to get a more profound understanding about people’s behaviorduring ideation. We attain a more detailed perspective on users’preferences by studying the situated practice of people during suchco-located settings. During the session, we collected a variety ofdata. Two of the authors directly observed the brainstorming ses-sions. At the end of each session, we asked participants to fill out aquestionnaire that consisted of seven questions focusing on partici-pants’ individual experience. We asked participants, for example,when they used the provided inspirations, how they used the pro-vided inspirations, and if the inspirations distracted or inspiredthem . We used the brainwriting pool method for the brainstormingsessions [19]. Participants in a brainwriting pool write down ideason sheets of paper. These sheets are then placed in the middle ofthe table (the pool), where other participants can take and readthem. We chose the brainwriting pool as it resembles many of thecharacteristics of large-scale ideation approaches. We recruited 15participants via a mailing list, posters and personal networks. Weconducted three sessions each with five people. The participantsfirst received a short introduction to the method and brainstormingchallenge (cp. Smart Coating in Table 2) and then brainstormedideas for 30 minutes. Afterwards, we provided the survey and theparticipants had 15 minutes to fill it out.
Overall, the 15 participants generated a total of 225ideas and read 220 ideas for inspiration. Furthermore, all of theparticipants filled out the questionnaire. The questionnaires wereanalyzed using thematic analysis [15] by one of the authors with afocus on how the participants perceived the study setup, its influ-ence on their performance and differences in their behavior duringthe ideation session. This way, similar statements (e.g. P4: “[ oth-ers’ ideas ] mostly inspired me because they showed very diverseuse cases I was not thinking about”, P15:“Most of [ others’ ideas ]inspired me. I’ve noticed that my ideas were more diverse afterreading others’.”) occurring multiple times were condensed intothemes. By considering these themes, the completed questionnaireswere re-read to ensure that all answers matching the themes areidentified. We derived three main themes from this qualitative anal-ysis. These themes were social pressure (P9: “made me nervous howproductive they seemed to be”), inspiration integration (e.g. annotat-ing how participants’ ideas re-used inspiration aspects) and attitudetowards inspirations (P13:“some of the ideas helped me think in The complete survey can be found at
Inspiration Seekers are actively looking for inspirationsduring an ideation process and derive their ideation strate-gies from them.
Inspiration Avoiders feel distracted by inspirations. Theyfollow their own ideation strategies to come up with moreor better ideas.However, these two ideator types are derived from qualitativeinsights and obtained in the context of co-located group settings. Inorder to further substantiate the definition, we analyzed data fromlarge-scale ideation sessions collected in previous research. In thenext section, we describe the results of this data analysis.
We conducted an exploratory data analysis to transfer our prelim-inary model into the context of large-scale ideation. The goal ofthis data analysis was to compare the insights about the ideatortypes found in the co-located setting with existing data obtainedin a large-scale ideation context. We wanted to define the differentideator types in terms of data available in large-scale ideation (e.g.tracking data). We used tracking data of three studies . The datawas obtained by using a web-based prototype similar to the oneoutlined in Section 4.1.1. In these ideation sessions, participantswere presented with a challenge and got a fixed amount of time towrite ideas and submit them via the user interface. Furthermore,they were shown a button that allowed them to request inspira-tions . Overall, the data comprised 193 individual ideation sessionsfrom three challenges (cp. Table 1). Each participant completed onesession for one of the data sets. The data sets (DS1, DS2, DS3) con-sisted of timestamped events for all sessions, namely session start,session end, whether the tab with the application was focused, ideasubmissions and inspiration requests. We chose inspiration requestsas the most relevant metric for a potential operationalization of theuser model described previously. We decided to segment the par-ticipants into how often they requested inspiration to find the twoapproaches within the session data. Furthermore, we disregarded The data is published anonymously at https://osf.io/7wjya/?view_only=0ba9e138d22e414abd8b868ed594e93e The inspirations are based on others’ ideas.
Table 1: Data sets (DS) used for validating the seeker /avoider model. Data sets varied in their number of partici-pants (each participant completed one session for one dataset), length of the sessions and input challenge (see Table 2for a full description of the challenges).
Name Sessions Length (min) ChallengeDS1 89 15 Smart CoatingDS2 30 25 Smart CoatingDS3 74 20 Bionic Radarall participants with fewer than three idea submitted (similar tostudies in related work [22]) which resulted into 20 filtered sessions(10 for DS1, 1 for DS2 and 9 for DS3).
We show the distribution of inspiration requestsin each data set in Figure 2. We summarized users that requestedinspirations for more than 8 times in the last bin to remove outliers.The participants requested inspiration up to 40 times in DS1, 20times in DS2 and 15 times in DS3. The median number of inspirationrequests were six for DS1, three for DS2, two for DS3.
DS 1
Inspiration requests P a r t i c i p a n t s DS 2
Inspiration requests P a r t i c i p a n t s DS 3
Inspiration requests P a r t i c i p a n t s Figure 2: Distribution of inspiration requests over all par-ticipants for all data sets. The vertical line represents themedian of inspiration requests. Participants are shown inabsolute numbers.
One goal for the data analysis was to develop a model of seekersand avoiders based on the patterns in the data sets. When visualiz-ing the inspiration request distributions, we found that inspirationrequests formed an almost bimodal distribution. However, therewere some outliers in inspiration requests, where people requestedinspiration heavily towards the end of the session. We hypothe-size that these requests happened out of idleness and not with theintention of getting inspiration. As a first classification rule we,therefore, chose the median number of requests over all partici-pants as the cut-off point between seekers and avoiders. We optedfor the median because it is not as susceptible to outliers as themean. When using the median as the cut-off point, we found thatparticipants having five inspiration requests would still be classifiedas avoiders for DS1. Based on the insights from the co-located study,we decided to change our definition so that only people with oneinspiration request at most were classified as avoiders. It turnedout that the request was often conducted right at the beginningof the session in sessions with only one inspiration request. Weassume that the participants in these sessions represent inspirationavoiders, because participants supposedly tested the inspiration reprint, January 2020, Maximilian Mackeprang, Kim Kern, Thomas Hadler, and Claudia Müller-Birn
200 400 600 800 . . . DS 1
Seconds F a il u r e r a t e SeekerAvoider
200 600 1000 1400 . . . DS 2
Seconds F a il u r e r a t e SeekerAvoider
200 600 1000 . . . DS 3
Seconds F a il u r e r a t e SeekerAvoider
Figure 3: Ratio of incorrectly classified seekers/avoiders(failure rate). For each point in time, the classified ideatortypes are compared to the results after the whole study du-ration. button once and discarded it completely later during ideation. We,therefore, chose to expand our avoider definition to include partici-pants with one request. Based on these insights, we defined seekersand avoiders more precisely:If participants request more than the median number of in-spiration requests, we assign them to the group of
InspirationSeeker .If participants request inspiration at most once during theideation session, we assign them to the group of
InspirationAvoider .Based on these definitions, we classified 80 participants as seek-ers and 51 as avoiders. We assigned all participants that fall in be-tween these categories to the group
Undetermined . The rule-baseddefinition of seekers and avoiders allowed us to classify existingsessions based on the complete information about the sessions.However, to understand how the ideator type impacts users weneed typed participants to test with. We, therefore, analyzed partialdata from the sessions to develop an understanding on when toclassify participants.
The results of the visualization of the distribution of inspirationrequests (see Figure 2) confirmed our hypothesis of the existenceof ideator types and provided us with a rule-based definition basedon finished sessions and ideation challenges. We chose to conductan online experiment to test the effects of ideator types in a large-scale ideation context. We decided to classify the participants of theexperiment by conducting a classification session. However, oneopen question when choosing this approach is: How long should aclassification session be? We conducted an analysis of partial datafrom the data sets to answer this question, which we present in thefollowing. The analysis was based on the ideator types we assignedto participants using the rule-based classification. We divided thesessions into inspiration sequences to develop a feeling for howlong a classification session should be. An inspiration sequencedescribes the number of inspiration requests in bins of 60 secondseach. We applied the rule-based classification on partial sequencesand then compared the results with the label at the end of thesession to find out a good length for a classification session.
We show the failure rates of applying this model inFigure 3. The number of incorrectly classified ideators decreasesover time compared to the classifications at the very end of each session (e.g., in DS1 60% of the avoiders were classified correctlyafter 300 s).
In the qualitative analysis of the co-located brainstorming setting,we found that some participants relied heavily on other participants’inspiring ideas, while others were either confused by inspirationsor actively rejected them. The qualitative feedback from the studyparticipants allowed us to get more insights into the characteristicsof the participants. It turned out that avoiders are more confidentabout the task of ideation and their idea generation strategies, whichwe assume, increases their fluency, i.e. number of ideas generatedduring a session. We conducted a data analysis of existing ideationdata to validate these insights in the context of a large-scale onlinesettings. We found in our analysis that the classification errors foravoiders decreased over time. Seekers, on average, could be classi-fied faster (e.g. in DS1, 70% of seekers are classified correctly after300 s). The failure rate analysis in Figure 3 shows that at 10 min, theclassification for both seekers and avoiders is better than by chancefor all three data sets analyzed. Furthermore, the failure rate seemsto decrease non linearly for DS1 and DS2. We decided to investigatethis insight further, since an early classification of seekers/avoiderscan enable an adaptive recommendation of inspirations in large-scale online ideation. The results of the data analysis allowed thedefinition of a user model for inspiration seekers and avoiders. Weoperationalized this distinction based on the number of inspirationrequests they submit during an ideation session. We defined thefollowing research questions to evaluate this user model: RQ . To what extend does ideator type (based on the rule-based classification) depend on the challenge? RQ . How do ideation metrics, such as fluency, differ betweenideator types? RQ . To what extent are ideator types impacted by the avail-ability of inspiration? RQ . Are we able to predict the ideator type without fullsession data?We carried out an online experiment using Amazon MechanicalTurk (MTurk) to investigate these questions. The study setup andresults are described next. The qualitative feedback from the co-located ideation sessions andthe preliminary data analysis from previous studies led us to the hy-pothesis that a classification of ideators exists relying on inspiration(seekers) and ideators distracted or annoyed by it (avoiders). How-ever, this hypothesis is based only on qualitative findings. Further-more, the existence of these ideator types says nothing about theireffect on brainstorming or inspiration deployment. We conductedan online experiment to systematically evaluate the differences ofideator types in large-scale ideation. The goals of this experimentwere, firstly, to validate the user model proposed, secondly, to testwhether the model has relevant impact on user interface configu-ration and, thirdly, to see if we can classify ideators heuristically,even without having data on a full ideation session. eeker or Avoider? User Modeling for Inspiration Deployment in Large-Scale Ideation Preprint, January 2020,
Table 2: Ideation challenges used in the studies.
Name Used In DescriptionSmart Coating Pre-Study
Imagine you could have acoating that could turn ev-ery surface into a touch dis-play. Brainstorm cool prod-ucts, systems, gadgets orservices that could be builtwith it.
Bionic Radar Session A
A technology can perceivethe movement of an ob-ject, such as humans, liv-ing beings or objects, likea bat. Once remembered,the technology can sub-sequently recognize it (...)The technology is approxi-mately hand-sized and canbe used anywhere.
Fabric Display Session B
Imagine there was a touch-sensitive “fabric display”that could render high res-olution images and videoson any fabric througha penny-sized connector[22].
We first needed to distinguish participants within our experimentto understand the impact of the model systematically. We, therefore,conducted a first ideation session as a classification-session (sessionA). Based on the inspiration requests in this session, we classifiedparticipants into the types described above. We then asked themto do another ideation session (session B). We collected data aboutideation metrics in the second session, based on the classification.We, furthermore, split participants into two conditions (see below)to understand the impact of the ideator types as a user modelin large-scale ideation. We decided to conduct both sessions inone Human Intelligence Task (HIT), because we did not have torecruit the people again but could allow participants to immediatelycontinue working (to avoid low retention rates).
The study consisted of two sessions. The firstsession used the same software for all participants. The software(as shown in Figure 4) consisted of a challenge description panel(A), an idea input panel (B), where participants could enter ideas,an idea history (C) and an inspiration button (D).The data obtained in the classification session was used to assigna type to the participant. If a participant issued zero or one inspira-tion request, we classified them as an avoider. As we did not havethe median number of inspirations available, we chose a cut-off offour to identify a participant as a seeker. This cut-off was chosenqualitatively after the exploratory data analysis. It differed fromthe median calculated after the experiment (5) by one. This meansthat participants that requested between two and four inspirations
Figure 4: The brainstorming user interface, showing thechallenge panel (A) showing the current ideation challenge,the idea input panel (B) where users type their ideas, an ideahistory list (C) showing all of the users’ previous ideas andan inspiration button (D). In the baseline condition, the in-spiration button was not shown. were classified as undetermined . We classified participants submit-ting three or fewer ideas as unmotivated . We used a 2x2 factorialdesign for the second session to combine ideator types with twodifferent conditions. Participants in the on-demand conditions wereshown an [inspire me] button. The button was not shown forparticipants in the baseline condition.
Participants started the study with a short textintroduction describing the brainstorming challenge and the userinterface. Participants were then shown the main brainstormingapp (as shown in Figure 4) and were asked to brainstorm ideas for10 min (session A). The timing of session A was chosen based on theresults from Section 3.3. The challenge for this session was bionicradar , with the prompt listed in Table 2. People with a classificationof seeker or avoider moved on to the second session (session B).The other participants were directed to the survey. In session B,participants had to brainstorm for 15 min on the challenge fabricdisplay (see Table 2). We chose to change the challenge betweensession A and B to evaluate whether seeker / avoider is a feature ofthe participant or the challenge at hand. All participants filled outa user experience survey at the end.
The inspiration mechanism imple-mented was an [inspire me] button that participants could clickon when they were stuck. This pull -approach to inspiration isbased on related work comparing different timings of inspirationdeployment [3, 22]. When the inspiration button was clicked on,the participant received one inspiring idea. In order to obtain theinspirations, 200 ideas for both challenges from previous studieswere manually rated for quality (in terms of novelty and value)by two of the authors. The inspirational ideas were sorted by thesum of novelty and value and then sorted (each inspiration buttonclick returned the next idea, in descending order). This approachensured that only high-quality ideas were shown as inspirations(as suggested by related work [22]). Furthermore, this approachensured that inspirations were equal for all participants.
We recruited 380 participants using AmazonMechanical Turk. We limited participation to U.S. workers with atleast 1,000 HITS and an approval rate > reprint, January 2020, Maximilian Mackeprang, Kim Kern, Thomas Hadler, and Claudia Müller-Birn $ 4 if they completed one session and $ 7 if they completed bothsessions (~$12/h). Our focus in the first part of the study, was on the ideation process:By conducting the classification session, our goal was to analyze thebehavior of seekers and avoiders separately, and understand howthe proposed model translates into ideation metrics. We were ableto analyze whether the availability of inspiration impacts avoidersdifferently than seekers by having the availability of inspiration asa condition in the second study. Lastly, we wanted to find out if theavoider/seeker classification is dependent on the challenge or theparticipant by having two sessions with different challenges. Wedefined the following hypotheses to test these goals: H : The seeker / avoider distinction is a characteristic of theuser and not the challenge. Therefore, participants classifiedas avoiders in the first session will not request inspiration inthe second session. H : The availability of inspiration distracts avoiders, leadingto a lower quality of their ideas in the on-demand condition. We chose to employ standard ideationmetrics, as used in related work [17], to analyse the effects of ideatortypes and the availability of inspiration. Firstly, we measured
Flu-ency : the number of ideas generated in a session. Furthermore, wetracked the number of times, a participant clicked on the inspirationbutton (in session A and in the on-demand condition of session B).We employed novelty and value ratings from external crowdwork-ers to analyse the effect on ideation outcome. We collected ratingsfor all ideas generated in session B by asking workers to rate nov-elty and value on a five-point Likert scale for a batch of fifteen ideasper HIT. Ratings were obtained redundantly, with at least threeratings per idea (some of the ideas were rated more often becauseassigned batches were returned unfinished by workers, leading toa non-uniform distribution of batches). We normalized the ratingsfor each batch to account for systemic bias in the ratings (e.g. aworker rating all ideas in their batch at 5 ( very good )). We decidedto test the maximum novelty and value per participant. This deci-sion was motivated by the overarching goal of brainstorming: Thestatistical chance of high quality ideas is increased by contributinglarge quantities of ideas. Therefore, we compared only the ideaswith the highest novelty and highest value within their session tomeasure the effect of inspiration availability on ideators.
A total of 380 MTurk crowd workers participated inthe experiment. Of these, 134 participants continued with sessionB, from whom 81 sessions were considered usable. We filtered 41sessions that were placed in a faulty condition, due to a softwareerror. The remaining 12 participants were sorted out because theyhad either submitted only non-sense ideas (e.g. “this a machine toknow underwater scenery that makes the more deeps goes that dueto make the changes”) or completely misunderstood the challenge.The other 246 participants only completed session A and the survey,because they were either not classified as a seeker or avoider, orthe target number of participants per condition had already beenreached (e.g. a participant who was classified as a seeker did notcomplete session B if we already had obtained 40 observations classified as seekers). Out of the 380 participants, based on the in-spiration requests, we classified 201 as seekers, 59 as avoiders, 83 asundetermined and 37 as unmotivated. In session A, the participantsrequested inspiration 3,018 times overall, with a median number ofinspiration requests of five.
Ideator Types P a r t i c i p a n t s Figure 5: Number of participants classified into types aftersession A.
Seeker Avoider
Ideator Types I n s p i r a t i o n s Figure 6: Number of inspirations requested by participantsin the on-demand condition of Session B, segmented by label.The error bars represent the standard error.
Ideator Types.
To understand whether the ideator type dependson the challenge, we compared the behavior of the participants insession A to the participants in the on-demand condition of sessionB. Intuitively, participants that were classified as avoiders in the firstsession should not use the inspiration button in session B. Figure 6shows the inspiration requests of seekers and avoiders in sessionB. To test H we deployed a linear model predicting the number ofinspiration requests for ideators in the on-demand condition. Themodel shows a highly significant difference between seekers andavoiders ( p < . , r = . , r = . , estimate = . ) . Wefurthermore used the rule-based classification based on the trackingdata of session B in the on-demand condition. Table 3 shows theparticipants’ labels after session A and then, if the label changed,how many participants were re-classified in session B. We found eeker or Avoider? User Modeling for Inspiration Deployment in Large-Scale Ideation Preprint, January 2020, out that all the participants classified as seekers in session A stayedin that classification. On the avoider side, 11 of the 20 avoiderswould be classified as avoiders again. The rest was classified asundetermined (5), seekers (3) or unmotivated (1). Table 3: Classifications after Session A and Session B
Session A Session B
Impact of Ideator Type and Condition on Ideation Metrics.
We usedlinear regression models to test H . All linear models were testedagainst their assumptions according to Field et al. [4]. The data werelog-transformed for idea submits to ensure a normal distribution ofmodel residuals as suggested by the Box-Cox test [1]. A probablecause for the positively skewed distribution is that ideators witha low number of submits were sorted out as unmotivated. Thecontrasts were coded as described by Schad et al. [18]: The ideatortype was coded as a sliding difference contrast, and the conditionwas coded as a custom contrast with on-demand vs. baseline sothat the estimates can be interpreted. Seeker Avoider Overall baseline on-demand
Condition baseline on-demand
Condition baseline on-demand
Condition I d e a s ub m i t s (a) Fluency Figure 7: Fluency for session B per ideator type and condi-tion. Overall shows seekers and avoiders together. The errorbars represent the standard error.
We collected ideation metrics (fluency, novelty, value) for ses-sion B and segmented them by condition. Figure 7a shows thefluency results for types and conditions in session B. There wereno main or interaction effects in the multiple linear regression onfluency between ideator types and conditions ( p = . , r = . , estimate = − . ) . Fluency was slightly lower for avoidersthan seekers, albeit non-significant. In the multiple linear regres-sion on maximum novelty between ideator types and conditions,there was a significant main effect for ideator types ( p < . p = . on-demand to baseline , whereas it is the otherway around for seekers, see Figure 9a. Table 4: Multiple linear regression model predicting maxi-mum novelty based on ideator type and condition.
Predictors Estimates SE Statistic pGrand Mean 0.83 0.03 24.61 <0.001Seeker vs. Avoider -0.11 0.07 -1.68 0.096on-demand vs. baseline -0.02 0.08 -0.26 0.793Seeker vs. Avoider :on-demand vs. baseline 0.34 0.16 2.04 0.043Observations 122 R / adjustedR Seeker Avoider Overall baseline on-demand
Condition baseline on-demand
Condition baseline on-demand
Condition N o v e l t y (a) Max Novelty Seeker Avoider Overall baseline on-demand
Condition baseline on-demand
Condition baseline on-demand
Condition V a l u e (b) Max Value Figure 8: Maximum novelty/value ratings per ideator typeand condition. Maximum novelty/value is defined as theidea with the highest novelty/value rating within a session.The error bars represent the standard error. baseline on-demand
Condition M a x i m u m N o v e l t y Ideator Type
AvoiderSeeker (a) Max Novelty baseline on-demand
Condition M a x i m u m V a l u e Ideator Type
AvoiderSeeker (b) Max Value
Figure 9: Interaction Effects for conditions and ideator types
There were no main or interaction effects in the multiple linearregression on maximum value between ideator types and condi-tions, see Figure 9b. Differences in maximum value were marginal,see Figure 8b.
The second goal of the study was to find out whether we can predictideator types based on incomplete sessions. This objective was reprint, January 2020, Maximilian Mackeprang, Kim Kern, Thomas Hadler, and Claudia Müller-Birn motivated by the results of the failure rate analysis in the pre-studies(cf Section 3.3) and the vision of an personalized adaptive inspirationrecommender system. The input data are the number of inspirationrequests of the sessions summed in bins over equal amounts of time.We used both sessions as the input data. Session A contains 343sessions (we filtered 37 unmotivated participants from the original380 samples) and took 10 min. Session B in the on-demand conditioncontains 41 samples and took 15 min. Since we want to determinea user’s ideator type before the session ends, we restrict the inputdata’s end point in time, looking at only the first x minutes of thesession. The output is a classification of the participant as seeker oravoider (based on the entire session's data). Table 5 shows a samplesession, where the user requested 0 inspirations in the first bin,then 1 inspiration in the second bin. This session was labelled asan avoider by the procedure detailed in Section 3.2.1.
Table 5: Example 10-minute data set with a 15 seconds bin-size which gives us 40 bins.
Session id Bin 1 Bin 2 ... Bin 40 Label0001 2 0 ... 1 seeker... ... ... ... ... seeker0318 0 1 ... 0 avoider
We used decision tree regressorsas well as random forest regressors as models. Decision trees arepredictive models with a tree-like structure [16]. The regressorsmodel the relationship of input data within the branches to the classlabels in the leaves. During training the data is inserted into the rootnode. From there, it travels to the leaves though the branches. Thebranches contain conjunctions of data feature restrictions whichredirect the instances into child nodes. The leaves contain the meanvalues of the training data’s class labels. We chose decision treesmainly because they can be visualized and give a good intuitionon which features are most relevant. We used regression insteadof straight forward classification in order to better incorporate thecontinuous transition of user types. In addition regressors can beevaluated by receiver operator characteristic (ROC) curves whichallow for a more detailed evaluation. Random forests are ensemblesof decision trees [11]. In order to train a random forest severaldecision trees are trained on randomized subsets of the data. Duringtraining time the data features they are allowed to put restrictionson are also randomly limited. This randomization decorrelates thetrees from each other while preserving their ability to generalizeacross large parts of the data set. During prediction, all trees withinthe forest vote on the instance's value. However, regressors modelcontinuous relationships. In order to extract classes from regressors,the continuous space must be divided into sections representingclasses via thresholds. We determine these thresholds by evaluatingthe regressors on a ROC curve. A ROC curve considers differentthresholds on which a regressor’s predictions could be split intotwo classes and plots the amount of true positives (correctly labeledseekers) against the amount of false positives (sessions that werepredicted to be seekers but are actually avoiders) for each of thesethresholds. We then choose the threshold in order to maximize theratio of true positives to false positives. We set the bin size to 15 s for the analysis. We trained the randomforest classifier with 200 decision trees. The maximal tree depth wasnot limited. When looking at the labels assigned to the participants,we found that we had classified 201 participants as seekers andonly 59 participants as avoiders. To adapt the model, we weighedideator type classes by their frequency in the dataset . The dataconsists of input values of the bins which are the sum of inspirationrequests within the bin’s time period. For our experiments wecreated sub-data sets which consist of the first x minutes of the binsbut exclude the rest. These sub-data sets are called x-minute datasets. With these parameters we conducted two experiments: Thefirst experiment trains and predicts scores on session A. We trainthe random forest on x-minute data sets of on a subset of users,the train data set. Then we score the seekers and avoiders with theregressors on x-minute data sets of the rest of users, the test datasets. We create ROC-curves to determine optimal thresholds withwhich we classify the test data sets. The metrics we use to evaluatethe configurations are defined as follows:Accuracy: T P + T NT P + FP + F N + T N
Precision:
T PT P + FP Recall:
T PT P + F N
Then we visualize one of the trees in order to understand qualita-tively how the predictions were made. In the second experimentthe classifiers are trained on session A and predict on session B,thus, generalizing across challenges. The random forest is trainedon x-minute data sets of all instances in session A and predicts theclasses of all instances on session B. Random forests take the samelength inputs that they were trained on, which means that the last5 min of the second session’s bins are not considered as input atany point in the experiment. Otherwise the evaluation is analogousto the first experiment.
For the first experiment, we created one ROC curveper minute (cf Figure 10a for an example ROC curve at 3 min). Wethen calculated the area under the curve (AUC) for each ROC curve(Figure 10b shows the result). The AUC increases steadily but moststrongly for the first 3-4 minutes then the increase ebbs off quickly.Having determined the thresholds, we calculate accuracy, precisionand recall for the prediction with an x-minute input data set. Resultsfor accuracy, precision and recall for the prediction results using thesession A data are shown in Figure 11. The increase in predictivepower is largest in the first minutes and declines subsequently. Wevisualize an individual decision tree for a more qualitative analysis(see Figure 12). Every bin in the data concerns 15 s, for example,X1-X4 contains minute 1. The decisions made close to the tree’s rootconcern more relevant bins than the decisions made closer to theleaves. The computed tree shows that more fundamental decisionsfocus on the latest available minutes, if the tree uses the first 5 min,then the bins of the fifth minute are most relevant. The same istrue for all 10 min. For the second experiment, we also calculatedthe ROC curves, AUC, thresholds, and metrics for each one to tenminutes. Figure 13 shows accuracy, precision and recall plotted for For the code and data, see https://osf.io/7wjya/?view_only=0ba9e138d22e414abd8b868ed594e93e eeker or Avoider? User Modeling for Inspiration Deployment in Large-Scale Ideation Preprint, January 2020,
FPR T P R ROC Curve for Random Forest on 3 Minute Data Sets (a) ROC curve on a 3 minute dataset for experiment 1.
Minutes A U C AUC for Random Forest on X Minute Data Sets (b) AUC plot for experiment 1summarizing the ROC curves.
Figure 10: receiver operator characteristic (ROC) and areaunder the curve (AUC) examples for experiment 1
Sub Data Set for X Minutes V a l u e s accuracy precision recall Figure 11: Accuracy, precision and recall for session Aideator type predictions based on a decision tree trained ondata sets from session A.Figure 12: Visualization of a 5-minute decision tree gener-ated by the random forest classifier for experiment 1. x-minute data sets. As the prediction is made on session B, whilebeing trained on session A data, the last 5 minutes of session B arenot accounted for by the classifier. Similar to the results of Figure11, the predictive power is greatest at the start. Due to the small number of samples in session B, we see no increase in predictionquality for 2 to 6 min.
Sub Data Set for X Minutes V a l u e s accuracy precision recall Figure 13: Accuracy, precision and recall for session Bideator type predictions based on a decision tree trained ondata sets from session A.
In the following, we discuss the results of our crowdsourced ideationsessions in the context of our research questions which were statedabove (cf Section 3.4).
When discussing the insights from the co-located brainstormingsessions, we speculated whether the ideator type either is a char-acter trait or depends on the provided challenge. The latter makessense insofar that some challenges might be too difficult and othersuninteresting for some participants, which might influence theirideation behavior. We elaborated this question by three analyses.First, we analyzed inspiration requests in the on-demand condi-tion. As expected, participants assigned to the group of seekersrequested significantly more inspirations. This aligns with our intu-ition, derived from the pre-study in a co-located setting, that thereare individual preferences regarding inspirations. However, therelationship between a person and the provided challenge seems tobe more complex than expected: Three participants changed theirgroup membership from avoiders to seekers in the second session.In that study, we presented the participants with two similar, rathertechnical challenges. Our insights from this condition are inconclu-sive. In future work, we plan to investigate whether the assignedideator type remains stable even when two or more fundamen-tally different challenges (e.g. technical, social, ludic) are compared.Additionally, we analyzed the accuracy of the between-session pre-dictions of our random forest classifiers and found that, while theaccuracy initially increased in the predictive power (similar to thewithin-challenge accuracy), the cross-domain setting shows no rele-vant improvement after 3 to 4 min of inspiration requests. In futureresearch, we plan to elaborate on the more complex relationshipbetween ideator type and the challenge provided, for example byexplicit elicitation of confidence or challenge comprehension. reprint, January 2020, Maximilian Mackeprang, Kim Kern, Thomas Hadler, and Claudia Müller-Birn
Inspired by the insights from the co-located setting, we supposedthat avoiders have a better understanding of their idea generationstrategies, thus, might generate more ideas. However, when dif-ferentiating study participants based on our heuristic, we werenot able to find significant differences in fluency between ideatortypes. Even though both participant groups use inspirations dif-ferently, no differences can be seen by only looking at the fluency.We, therefore, assume that inspirations have a positive effect onideation outcome but do not have the same impact on all users.More research is needed to better understand the effects of inspira-tions on different user groups. Furthermore, in future research weplan to refine the user model, to include other factors that couldpotentially outweigh the seeker / avoider classification such as taskmotivation, task comprehension, confidence and prior experiencein brainstorming. We plan to elicit these data based on explicit userpreferences provided by the study participants as has already beenused in recommendation and information filtering applications(e.g. [13]).
Based on the crowdsourced ideation session, we learnt that, whenproviding avoiders with the ability to request inspirations, it im-pacted the novelty of their ideas negatively. This effect confirmsour initial intuition that avoiders have their own set of strategieswhen generating ideas. This observation contrasts seekers, who useinspirations as a kind of guideline or scaffold for both understand-ing the challenge better and becoming inspired by topics, entitiesor activities in the inspiration provided. This effect, however, showsthat there is a distinction between seekers and avoiders that is rele-vant for providing inspirations. We plan to further investigate thisdistinction by conducting research in which we use the heuristicapproach to adapt the inspirations available in a timely manner(e.g. by disabling or fading out the inspiration button). We assumethat this could further help avoiders in leveraging their own ideageneration strategies, while at the same time supporting seekerswho need inspirations during ideation.
We employed random forest classifiers to predict the user’s type.The classifier worked especially well in the first few minutes ofthe ideation session. The accuracy plots show a quick increase inpredictive power for both avoiders and seekers and the ROC-curvesprove that the increase is due to structures within the user-behaviorand not random chance. The increase in predictive power is largestin the first few minutes and declines later, although it never ebbs offentirely. The increase is in contrast to the decision tree visualized,that assigns later minutes a greater predictive power. This effect canbe explained because an avoider has a higher probability of tryingthe inspiration button in the early minutes without using it laterwhereas it is unlikely in later minutes. The random forests wereused because they are reliable in dealing with noisy data, achieving comparatively accurate results and finding difficult patterns [9].In future research, other classifiers more suited to the recursivelist-like nature of the data, such as recurrent neural networks orbayesian networks might be more suitable when finding avoidersand seekers on cross-domain settings and different time scales.
One potential limitation of the study is the participants’ under-standing of the procedure and the task of the study. Although thestudy included an introduction and a short tutorial of the interface,it is not clear that all avoiders deliberately chose not to use inspira-tions or whether they might not have understood the inspirationmechanism. Future work should include an interactive tutorial toensure that participants understand the mechanism. At the currentstate, the classification into seekers and avoiders is only done bythe analysis of inspiration requests. Future work could comparethis inspiration request-based model with self-assessment (bothprior and after the session) of the participants preferences regardinginspiration.
Research has shown that the creative outcome of large-scale ideationcan be improved by showing inspirations to ideators. However, ex-isting approaches neglect individual preferences. In the researchpresented here, we sought to close this gap, and investigated howindividual preferences can inform a user model for personalizedadaptive inspirations. We conducted our research in two stages.Firstly, we conducted co-located brainstorming sessions which in-formed an exploratory data analysis. The insights collected in thesestudies finally informed a user model. Using this model, we differ-entiate ideation participants into seekers, i.e. users who appreciateinspirations, and avoiders, i.e. users which are distracted by inspi-rations and dislike them. We validated this user model in onlineexperiments. In the latter, we investigated how stable the assign-ment of people to one ideator type is. We, furthermore, evaluatedthe impact of the type and the availability of inspiration on ideationmetrics (e.g. fluency). Lastly, we trained and deployed a randomforest approach to test how accurately we can predict ideator typeswith incomplete session information. Our results show that, interms of numbers of ideas generated, there is no significant differ-ences between ideator types. However, when looking at the ideasrated most novel, we found that while seekers were influenced pos-itively by the availability of inspiration, avoiders’ highest noveltydecreased with inspiration availability. When using the randomforest-based approach for classifying participants, we found thatthe first three minutes of a session provided non-linear informationgain for the predictor. After a time of three minutes, we were ableto classify 73% of the ideator types correctly. These findings showthat ideator types provide relevant information when designingadaptive inspirations within large-scale ideation systems. Further-more, random forests provide a promising heuristic to determineideator types based on incomplete session information. However,our research has a number of limitations. The relationship betweenideator type and challenge proved more complex than anticipated.In future work, we will approach this by evaluating the impact of eeker or Avoider? User Modeling for Inspiration Deployment in Large-Scale Ideation Preprint, January 2020, thematically diverse challenges. Moreover, we focused on the prin-cipal differences between ideator types by using static conditions.In future work we will therefore evaluate the impact of dynamicallyadding or removing possibilities to request inspirations.
ACKNOWLEDGMENTS
This work is supported by the German Federal Ministry of Educa-tion and Research, grant 01IO1617 (âĂIJIdeas to MarketâĂİ).
REFERENCES [1] George EP Box and David R Cox. 1964. An analysis of transformations.
Journalof the Royal Statistical Society: Series B (Methodological)
26, 2 (1964), 211–243.[2] Joel Chan, Steven Dang, and Steven P Dow. 2016. IdeaGens: enabling expertfacilitation of crowd brainstorming. In
Proceedings of the 19th ACM Conference onComputer Supported Cooperative Work and Social Computing Companion . ACM,13–16.[3] Joel Chan, Pao Siangliulue, Denisa Qori McDonald, Ruixue Liu, RezaMoradinezhad, Safa Aman, Erin T Solovey, Krzysztof Z Gajos, and Steven PDow. 2017. Semantically Far Inspirations Considered Harmful?: Accounting forCognitive States in Collaborative Ideation. In
Proceedings of the 2017 ACM SIGCHIConference on Creativity and Cognition . ACM, 93–105.[4] Andy P Field, Jeremy Miles, and Zoë Field. 2012. Discovering statistics usingR/Andy Field, Jeremy Miles, Zoë Field.[5] Veronika Gamper, Andreas Butz, and Klaus Diepold. 2017. Sooner or later?:immediate feedback as a source of inspiration in electronic brainstorming. In
Proceedings of the 29th Australian Conference on Computer-Human Interaction .ACM, 182–190.[6] Monica J Garfield, Nolan J Taylor, Alan R Dennis, and John W Satzinger. 2001.Modifying paradigmsâĂŤIndividual differences, creativity techniques, and expo-sure to ideas in group idea generation.
Information Systems Research
12, 3 (2001),322–333.[7] Victor Girotto, Erin Walker, and Winslow Burleson. 2017. The effect of peripheralmicro-tasks on crowd ideation. In
Proceedings of the 2017 CHI Conference onHuman Factors in Computing Systems . ACM, 1843–1854.[8] Victor Girotto, Erin Walker, and Winslow Burleson. 2019. CrowdMuse: Support-ing Crowd Idea Generation through User Modeling and Adaptation. In
Proceedingsof the 2019 on Creativity and Cognition . ACM, 95–106.[9] Trevor Hastie, Robert Tibshirani, and Jerome Friedman. 2009. Random forests.In
The elements of statistical learning . Springer, 587–604.[10] Dirk Heerwegh. 2009. Mode differences between face-to-face and web surveys:an experimental investigation of data quality and social desirability effects.
In-ternational Journal of Public Opinion Research
21, 1 (2009), 111–121.[11] Tin Kam Ho. 1995. Random decision forests. In
Proceedings of 3rd internationalconference on document analysis and recognition , Vol. 1. IEEE, 278–282.[12] Elahe Javadi and Judith Gebauer. 2019. An Experimental Study of the Attention-based View of Idea Integration: The Need for a Multi-level Dependent Variable.
Journal of Information Technology Theory and Application (JITTA)
20, 1 (2019), 1.[13] Bart P Knijnenburg, Niels JM Reijmer, and Martijn C Willemsen. 2011. Each to hisown: how different users call for different interaction methods in recommendersystems. In
Proceedings of the fifth ACM conference on Recommender systems .ACM, 141–148.[14] Alex F Osborn. 1953. Applied imagination. (1953).[15] Brittany L Peterson. 2017. Thematic analysis/interpretive thematic analysis.
Theinternational encyclopedia of communication research methods (2017), 1–9.[16] J. Ross Quinlan. 1986. Induction of decision trees.
Machine learning
1, 1 (1986),81–106.[17] Mark A Runco and Garrett J Jaeger. 2012. The standard definition of creativity.
Creativity research journal
24, 1 (2012), 92–96.[18] Daniel J Schad, Shravan Vasishth, Sven Hohenstein, and Reinhold Kliegl. 2018.How to capitalize on a priori contrasts in linear (mixed) models: A tutorial. arXivpreprint arXiv:1807.10451 (2018).[19] Helmut Schlicksupp. 1975.
Grundlagen der Ideenfindung und Problemlösung .[20] Pao Siangliulue, Kenneth C Arnold, Krzysztof Z Gajos, and Steven P Dow. 2015.Toward collaborative ideation at scale: Leveraging ideas from others to generatemore creative and diverse ideas. In
Proceedings of the 18th ACM Conference onComputer Supported Cooperative Work & Social Computing . ACM, 937–945.[21] Pao Siangliulue, Joel Chan, Steven P Dow, and Krzysztof Z Gajos. 2016. Idea-Hound: improving large-scale collaborative ideation with crowd-powered real-time semantic modeling. In
Proceedings of the 29th Annual Symposium on UserInterface Software and Technology . ACM, 609–624.[22] Pao Siangliulue, Joel Chan, Krzysztof Z Gajos, and Steven P Dow. 2015. Providingtimely examples improves the quantity and quality of generated ideas. In
Pro-ceedings of the 2015 ACM SIGCHI Conference on Creativity and Cognition . ACM,83–92. [23] Kai Wang and Jeffrey V Nickerson. 2017. A literature review on individualcreativity support systems.
Computers in Human Behavior
74 (2017), 139–151.[24] Lixiu Yu, Aniket Kittur, and Robert E Kraut. 2016. Encouraging âĂIJoutside-the-boxâĂİ thinking in crowd innovation through identifying domains of expertise.In
Proceedings of the 19th ACM Conference on Computer-Supported CooperativeWork & Social Computing . ACM, 1214–1222.[25] Lixiu Yu, Robert E Kraut, and Aniket Kittur. 2016. Distributed analogical ideageneration with multiple constraints. In