Chad Dubé

Binary ROCs in Perception and Recognition Memory Are Curved

In recognition memory, a classic finding is that receiver operating characteristics (ROCs) are curvilinear. This has been taken to support the fundamental assumptions of signal detection theory (SDT) over discrete-state models such as the double high-threshold model (2HTM), which predicts linear ROCs. Recently, however, Bröder and Schütz (2009) challenged this argument by noting that most of the data on which support for SDT is based have involved confidence ratings. The authors argued that certain types of rating scale usage may result in curved ROCs even if the generating process is thresholded in nature. From this point of view, only ROCs constructed via experimental bias manipulations are useful for discriminating between the models. Bröder and Schütz conducted a meta-analysis and new experiments that compared SDT and the 2HTM using binary (yes-no) ROCs and found that many of these functions were linear, supporting 2HTM over SDT. We examine all the data reported by Bröder and Schütz, noting important limitations in their methodology, analyses, and conclusions. We report a new meta-analysis and 2 new experiments to examine the issue more closely while avoiding the limitations of Bröder and Schützs study. These new data indicate that binary ROCs are curved in recognition, consistent with previous findings in perception and reasoning. Our results support classic arguments in favor of SDT and indicate that curvature in ratings ROCs is not task specific. We recommend the ratings procedure and suggest that analyses based on threshold models be treated with caution.

A critical comparison of discrete-state and continuous models of recognition memory: implications for recognition and beyond.

Multinomial processing tree (MPT) models such as the single high-threshold, double high-threshold, and low-threshold models are discrete-state decision models that map internal cognitive events onto overt responses. The apparent benefit of these models is that they provide independent measures of accuracy and response bias, a claim that has motivated their frequent application in many areas of psychological science including perception, item and source memory, social cognition, reasoning, educational testing, eyewitness testimony, and psychopathology. Before appropriate conclusions about a given analysis can be drawn, however, one must first confirm that the models assumptions about the underlying structure of the data are valid. The current review outlines the assumptions of several popular MPT models and assesses their validity using multiple sources of evidence, including receiver operating characteristics, direct model fits, and experimental tests of qualitative predictions. We argue that the majority of the evidence is inconsistent with these models and that, instead, the evidence supports continuous models such as those based on signal detection theory (SDT). Hybrid models that incorporate both SDT and MPT processes are also explored, and we conclude that these models retain the limitations associated with their threshold model predecessors. The potentially severe consequences associated with using an invalid model to interpret data are discussed, and a simple tutorial and model-fitting tool is provided to allow implementation of the empirically supported SDT model.

When more data steer us wrong: replications with the wrong dependent measure perpetuate erroneous conclusions.

There is a replication crisis in science, to which psychological research has not been immune: Many effects have proven uncomfortably difficult to reproduce. Although the reliability of data is a serious concern, we argue that there is a deeper and more insidious problem in the field: the persistent and dramatic misinterpretation of empirical results that replicate easily and consistently. Using a series of four highly studied “textbook” examples from different research domains (eyewitness memory, deductive reasoning, social psychology, and child welfare), we show how simple unrecognized incompatibilities among dependent measures, analysis tools, and the properties of data can lead to fundamental interpretive errors. These errors, which are not reduced by additional data collection, may lead to misguided research efforts and policy recommendations. We conclude with a set of recommended strategies and research tools to reduce the probability of these persistent and largely unrecognized errors. The use of receiver operating characteristic (ROC) curves is highlighted as one such recommendation.

The nomological network of self-reported distress tolerance

Distress tolerance (DT), or the ability to withstand psychological distress, is a popular construct in the psychological literature. However, research has not specified the nomological network of DT across self-report measures. The purpose of the current investigation was to understand what personality features, environmental stressors, current affective states, and behaviors contribute to DT in two different samples: college students and those in residential substance use treatment. Correlations revealed that self-reported DT was most strongly associated with trait negative emotionality, state negative affect, impulsivity, and perceived stress. In comparisons across samples, self-harm exhibited a stronger relationship with self-reported DT in the drug treatment than in the student sample, whereas perceived stress had a stronger association in the student sample. Correlations between self-report and behavioral measures of DT were nonsignificant. To understand this lack of associations, associations of outcomes with behavioral measures were assessed. In contrast to self-reported DT, behavioral DT was more closely related to achievement orientation, state negative affect, and state positive affect, but was not significantly related to psychopathology and maladaptive behaviors. It is necessary to continue investigating the construct validity of behavioral DT measures via the use of incremental utility analyses and experimental approaches.

Paying Attention to Attention in Recognition Memory Insights From Models and Electrophysiology

Reliance on remembered facts or events requires memory for their sources, that is, the contexts in which those facts or events were embedded. Understanding of source retrieval has been stymied by the fact that uncontrolled fluctuations of attention during encoding can cloud results of key importance to theoretical development. To address this issue, we combined electrophysiology (high-density electroencephalogram, EEG, recordings) with computational modeling of behavioral results. We manipulated subjects’ attention to an auditory attribute, whether the source of individual study words was a male or female speaker. Posterior alpha-band (8–14 Hz) power in subjects’ EEG increased after a cue to ignore the voice of the person who was about to speak. Receiver-operating-characteristic analysis validated our interpretation of oscillatory dynamics as a marker of attention to source information. With attention under experimental control, computational modeling showed unequivocally that memory for source (male or female speaker) reflected a continuous signal detection process rather than a threshold recollection process.

Obligatory and adaptive averaging in visual short-term memory.

Visual memory can draw upon averaged perceptual representations, a dependence that could be both adaptive and obligatory. In support of this idea, we review a wide range of evidence, including findings from our own lab. This evidence shows that time- and space-averaged memory representations influence detection and recognition responses, and do so without instruction to compute or report an average. Some of the work reviewed exploits fine-grained measures of retrieval from visual short-term memory to closely track the influence of stored averages on recall and recognition of briefly presented visual textures. Results show that reliance on perceptual averages is greatest when memory resources are taxed or when subjects are uncertain about the fidelity of their memory representation. We relate these findings to models of how summary statistics impact visual short-term memory, and discuss a neural signature for contexts in which perceptual averaging exerts maximal influence.

Social influences on adaptive criterion learning

People adaptively shift decision criteria when given biased feedback encouraging specific types of errors. Given that work on this topic has been conducted in nonsocial contexts, we extended the literature by examining adaptive criterion learning in both social and nonsocial contexts. Specifically, we compared potential differences in criterion shifting given performance feedback from social sources varying in reliability and from a nonsocial source. Participants became lax when given false positive feedback for false alarms, and became conservative when given false positive feedback for misses, replicating prior work. In terms of a social influence on adaptive criterion learning, people became more lax in response style over time if feedback was provided by a nonsocial source or by a social source meant to be perceived as unreliable and low-achieving. In contrast, people adopted a more conservative response style over time if performance feedback came from a high-achieving and reliable source. Awareness that a reliable and high-achieving person had not provided their feedback reduced the tendency to become more conservative, relative to those unaware of the source manipulation. Because teaching and learning often occur in a social context, these findings may have important implications for many scenarios in which people fine-tune their behaviors, given cues from others.

Characterizing belief bias in syllogistic reasoning: A hierarchical Bayesian meta-analysis of ROC data

The belief-bias effect is one of the most-studied biases in reasoning. A recent study of the phenomenon using the signal detection theory (SDT) model called into question all theoretical accounts of belief bias by demonstrating that belief-based differences in the ability to discriminate between valid and invalid syllogisms may be an artifact stemming from the use of inappropriate linear measurement models such as analysis of variance (Dube et al., Psychological Review, 117(3), 831–863, 2010). The discrepancy between Dube et al.’s, Psychological Review, 117(3), 831–863 (2010) results and the previous three decades of work, together with former’s methodological criticisms suggests the need to revisit earlier results, this time collecting confidence-rating responses. Using a hierarchical Bayesian meta-analysis, we reanalyzed a corpus of 22 confidence-rating studies (N = 993). The results indicated that extensive replications using confidence-rating data are unnecessary as the observed receiver operating characteristic functions are not systematically asymmetric. These results were subsequently corroborated by a novel experimental design based on SDT’s generalized area theorem. Although the meta-analysis confirms that believability does not influence discriminability unconditionally, it also confirmed previous results that factors such as individual differences mediate the effect. The main point is that data from previous and future studies can be safely analyzed using appropriate hierarchical methods that do not require confidence ratings. More generally, our results set a new standard for analyzing data and evaluating theories in reasoning. Important methodological and theoretical considerations for future work on belief bias and related domains are discussed.

Signal Detection Theory (SDT) Is Effective for Modeling User Behavior Toward Phishing and Spear-Phishing Attacks

Objective: To examine the utility of equal-variance signal detection theory (EVSDT) for evaluating and understanding human detection of phishing and spear-phishing e-mail scams. Background: Although the majority of cybersecurity breaches are due to erroneous responses to deceptive phishing e-mails, it is unclear how best to quantify performance in this context. In particular, it is unclear whether equal variances can safely be assumed in the SDT model, or, relatedly, whether degree of targeting, or threat level, primarily affects mean separation or evidence variability. Method: Through an online inbox simulation, the present research found that differences in susceptibility to phishing and spear-phishing e-mails could be carefully quantified with respect to detection accuracy and response bias through the use of an EVSDT framework. Results: The results indicated that EVSDT-based point metrics are effective for modeling and measuring phishing susceptibility in the inbox task, without the need for parameter estimation or model comparison involving unequal-variance SDT (UVSDT). Threat level modulated mean separation, with no effects on signal variances. Conclusion: These findings support the viability of using EVSDT to initially assess and subsequently monitor training effectiveness for phishing susceptibility, thereby providing measures that are superior to more intuitive metrics, which typically confound an individual’s bias and accuracy. Effects of threat level mapped clearly onto distribution means with no effect on variances, suggesting phishing susceptibility primarily reflects temporally stable discriminative characteristics of observers. Notably, results indicated that people are particularly poor at identifying spear-phishing e-mail threats (demonstrating only 40% accuracy).

A Bayesian Approach to Discriminating Between Biased Responding and Sequential Dependencies in Binary Choice Data.

Sequential dependencies occur when prior decisions affect subsequent decisions, and they have been observed in both memory and perception tasks. For binary response tasks, an inherent problem in measuring sequential dependencies is the ability to distinguish between sequential dependencies and response bias. The problem arises because the sequential dependency estimate within the frequentist architecture does not contain information regarding the number of observations upon which it is based. One solution to the problem is to use a Bayesian approach that takes the uncertainty of the sequential dependency estimate into account. We describe 2 Bayesian measurement models of sequential dependencies in binary response tasks and test them using simulated data with known degrees of response bias and sequential dependencies. Both models were able to distinguish between fluctuations in sequential dependencies and response bias. We then use the model to measure the contributions of sequential dependencies and response bias to the decisions made in recognition memory and perceptual categorization.