Mordechai Z. Juni

Flexible human collective wisdom

Group decisions typically outperform individual decisions. But how do groups combine their individual decisions to reach their collective decisions? Previous studies conceptualize collective decision making using static combination rules, be it a majority-voting rule or a weighted-averaging rule. Unknown is whether groups adapt their combination rules to changing information environments. We implemented a novel paradigm for which information obeyed a mixture of distributions, such that the optimal Bayesian rule is nonlinear and often follows minority opinions, while the majority rule leads to suboptimal but above chance performance. Using perceptual (Experiment 1) and cognitive (Experiment 2) signal-detection tasks, we switched the information environment halfway through the experiments to a mixture of distributions without informing participants. Groups gradually abandoned the majority rule to follow any minority opinion advocating signal presence with high confidence. Furthermore, groups with greater ability to abandon the majority rule achieved higher collective-decision accuracies. It is important to note that this abandonment was not triggered by performance loss for the majority rule relative to the first half of the experiment. Our results propose a new theory of human collective decision making: Humans make inferences about how information is distributed across individuals and time, and dynamically alter their joint decision algorithms to enhance the benefits of collective wisdom.

The wisdom of crowds for visual search

Significance Simple majority voting is a widespread, effective mechanism to exploit the wisdom of crowds. We explored scenarios where, from decision to decision, a varying minority of group members often has increased information relative to the majority of the group. We show how this happens for visual search with large image data and how the resulting pooling benefits are greater than previously thought based on simpler perceptual tasks. Furthermore, we show how simple majority voting obtains inferior benefits for such scenarios relative to averaging people’s confidences. These findings could apply to life-critical medical and geospatial imaging decisions that require searching large data volumes and, more generally, to any decision-making task for which the minority of group members with high expertise varies across decisions. Decision-making accuracy typically increases through collective integration of people’s judgments into group decisions, a phenomenon known as the wisdom of crowds. For simple perceptual laboratory tasks, classic signal detection theory specifies the upper limit for collective integration benefits obtained by weighted averaging of people’s confidences, and simple majority voting can often approximate that limit. Life-critical perceptual decisions often involve searching large image data (e.g., medical, security, and aerial imagery), but the expected benefits and merits of using different pooling algorithms are unknown for such tasks. Here, we show that expected pooling benefits are significantly greater for visual search than for single-location perceptual tasks and the prediction given by classic signal detection theory. In addition, we show that simple majority voting obtains inferior accuracy benefits for visual search relative to averaging and weighted averaging of observers’ confidences. Analysis of gaze behavior across observers suggests that the greater collective integration benefits for visual search arise from an interaction between the foveated properties of the human visual system (high foveal acuity and low peripheral acuity) and observers’ nonexhaustive search patterns, and can be predicted by an extended signal detection theory framework with trial to trial sampling from a varying mixture of high and low target detectabilities across observers (SDT-MIX). These findings advance our theoretical understanding of how to predict and enhance the wisdom of crowds for real world search tasks and could apply more generally to any decision-making task for which the minority of group members with high expertise varies from decision to decision.

Integration of visual information across time

Methods. On each trial, we sampled nine values from a spatial univariate Gaussian (SD = 3.32 cm). The mean of the Gaussian varied from trial to trial. We drew small vertical ticks whose x-coordinate was the sample value and whose y-coordinate was the center of the display marked by a horizontal reference line. Each tick was visible for 150 msec followed by a 150 msec delay between successive ticks. Observers estimated the center of the Gaussian by clicking on the horizontal reference line. Observers completed 200 trials without feedback, followed by 300 trials with corrective feedback indicating the true center of the Gaussian.

Benefits of Independent Double Reading in Digital Mammography: A Theoretical Evaluation of All Possible Pairing Methodologies

RATIONALE AND OBJECTIVES To establish the efficacy of pairing readers randomly and evaluate the merits of developing optimal pairing methodologies. MATERIALS AND METHODS Sensitivity, specificity, and proportion correct were computed for three different case sets that were independently read by 16 radiologists. Performance of radiologists as single readers was compared to expected double reading performance. We theoretically evaluated all possible pairing methodologies. Bootstrap resampling methods were used for statistical analyses. RESULTS Significant improvements in expected performance for double versus single reading (ie, delta performance) were shown for all performance measures and case-sets (p ≤ .003), with overall delta performance across all theoretically possible pairing schemes (n = 10,395) ranging between .05 and .08. Delta performance for the 20 best pairing schemes was significant (p < .001) and ranged between .07 and .10. Delta performance for 20 random pairing schemes was also significant (p ≤ .003) and ranged between .05 and .08. Delta performance for the 20 worst pairing schemes ranged between .03 and .06, reaching significance in delta proportion correct (p ≤ .021) for all three case-sets and in delta specificity for two case-sets (p ≤ .033) but not for a third case-set (p = .131), and not reaching significance in delta sensitivity for any of the three case-sets (.098 ≥ p ≥ .067). CONCLUSION Significant benefits accrue from double reading, and while random reader pairing achieves most double reading benefits, a strategic pairing approach may maximize the benefits of double reading.

Learning efficient perceptual sampling

We tested adults and children aged 7-9 and 10-12 years in a stochastic judgment task. Adult observers compensate in part for perceptual uncertainty. However, the manner in which perceptual systems represent and compute with probabilistic estimates remains largely unknown. Developmental studies provide insight into the nature and origins of these capabilities. In our task, subjects could earn a reward by touching an invisible target circle marked by dots (cues) drawn from a Gaussian distribution centred on the target. Subjects could sample up to 20 cues but each cue reduced the possible reward by a fixed amount. Each additional cue improved the reliability of the location estimate by reducing the standard error of the mean. Subjects therefore had to trade off localization accuracy against the cost of additional cues. There were two conditions that differed in the variance of the Gaussian. We computed the optimal sample size that maximized expected reward in each condition: 4 cues (low variance) and 8 cues (high). We assumed that observers aimed for the mean location of each dot cloud; control conditions showed that deviations from this strategy were small across all age groups. Strikingly, across both variance conditions, in both child and adult groups, numbers of cues sampled were indistinguishable from optimal. However, sampling in child groups was more variable trial-to-trial, with a cost to their final rewards as compared with adults. Childrens relatively mature abilities to compute with probabilistic estimates here contrast with their much poorer abilities to take uncertainty into account in difficult perceptual and motor tasks (e.g. Nardini et al, PNAS 2010; Dekker et al, VSS 2012). This apparent dissociation suggests that probabilities dependent mainly on external factors (samples of dots, in this task) are computed separately to those dependent mainly on internal noise (sensory uncertainty, in previous tasks). Meeting abstract presented at VSS 2015.