David A. Lieberman

Learning when reward is delayed: a marking hypothesis.

Rats were trained on spatial discriminations in which reward was delayed for 1 min. Experiment 1 tested Letts hypothesis that responses made in the home cage during the delay interval are less likely to interfere with learning than responses made in the maze. Experimental subjects were transferred to their home cages during the delay interval, and control subjects were picked up but then immediately replaced in the maze. Contrary to Letts hypothesis, both groups learned. Further experiments suggested that handling following a choice response was the crucial variable in producing learning: No learning occurred when handling was delayed (Experiment 2) or omitted (Experiment 3). One possible explanation for the fact that handling facilitated learning is that it served to mark the preceding choice response in memory so that subjects were then more likely to recall it when subsequently reinforced. In accordance with this interpretation, learning was found to be just as strong when the choice response was followed by an intense light or noise as by handling (Experiment 4). The implication of marking for other phenomena such as avoidance, quasi-reinforcement, and the paradoxical effects of punishment is also discussed.

The role of marking when reward is delayed.

Two-choice spatial discrimination by rats is enhanced if a salient stimulus marker occurs immediately after every choice response and again after a delay interval (Lieberman, McIntosh & Thomas, 1979). Three experiments further explore this effect. Experiment 1 found that the second marker is unnecessary. Experiment 2 found that a marker presented before a response is as effective as one presented after. Both effects could be explained in terms of markers focusing attention on subsequent cues. Experiment 3, however, found that markers after choice enhance learning even when no discriminative cues are present following the marker. Markers thus appear to initiate both a backward search through memory and attention to subsequent events; both processes help to identify events that might be related to the unexpected marking stimulus.

Reinforcement Without Awareness: I. Voice Level

Although some studies have reported that reinforcement can strengthen peoples behaviour without their awareness, most studies that have incorporated adequate procedures for assessing awareness have been unsuccessful. Using rigorous procedures, we report two studies designed to provide more scope for learning without awareness by making the reinforcement contingency as unobtrusive as possible. Subjects were told that the experiment was on ESP, and that they were to say which of two words the experimenter was thinking about. In fact, reinforcement was contingent on the loudness of the subjects voice when responding. Even though subjects reported no awareness of a relationship between trial outcomes and their behaviour, the probability of the reinforced responses increased significantly over trials in both experiments. These results support the view that reinforcement can occur without awareness.

Reinforcement without awareness: II. Word class.

In a previous study of reinforcement without awareness by Lieberman, Sunnucks, & Kirk (this issue), subjects were told that the experiment was on ESP; two words were presented on every trial, and their task was to choose the word the experimenter was thinking of. In fact, reinforcement was contingent on the loudness of the subjects’ voices when responding. They found stronger evidence for reinforcement without awareness than in many previous studies, and they attributed their success in part to the reinforcement of a response to which subjects were unlikely to attend. To explore this factor further, we again used an ESP cover story but reinforced subjects for choosing the word in each pair that contained a double letter. We found evidence of reinforcement without awareness in three experiments and also identified two factors that influence these effects: Learning did not occur if subjects (a) were encouraged to test irrelevant hypotheses, or (b) sat in an uncomfortable chair. We speculate that learning without awareness may be more likely when subjects are relaxed and that hypothesis testing and uncomfortable chairs impair such learning because they prevent relaxation.

Why do the effects of delaying reinforcement in animals and delaying feedback in humans differ? A working-memory analysis.

Animal research has shown that reinforcement is substantially less effective when it is delayed, but in studies of human motor learning delays in providing feedback typically have much less effect. One possible explanation is that in human research participants know the response to be learned and can thus focus on it during the delay; that is not the case in experiments on animals. We tested this hypothesis using a task in which participants had minimal information on what movement was correct and found that, as in animal experiments, participants learned only when feedback was immediate. A second experiment confirmed that the effects of the delay depended on how many responses had to be held in working memory: the greater the memory load, the poorer the learning. The results point to the importance of activity during a delay on learning; implications for the teaching of motor skills are discussed.

Commentary: Determinants of success and failure in experiments on marking

Abstract Lieberman, McIntosh, & Thomas (1979) found that presenting a brief salient stimulus can facilitate long delay learning in the T-maze by marking choice responses in memory, but Urcuioli and Kasprow (1988) have recently reported a partial failure to replicate our original finding. We identify procedural differences which might explain the discrepancies and then conduct a wider ranging review of the evidence which has convinced us that marking is a reliable and robust phenomenon. We also recount some of the problems we have encountered in developing well-controlled, fully automated procedures for studying marking. For operant learning, the major design problems centered on controlling for the confounding effects of secondary reinforcement and of adventitious reinforcement of choice responses repeated during the delay. For Pavlovian conditioning, ultimate success depended on the identification of adequately salient stimuli that could be presented as markers following key light-conditioned stimuli in a trace autoshaping procedure. Experiments in which these technical problems have been resolved have yielded data which cannot be explained with previously established principles of learning and consequently require the postulation of additional mechanisms, such as marking.

The Feeling of Freedom: Unconscious Associations

You may have been persuaded by the material in Chapter 4 that our brains play a major role in generating our thoughts and feelings, perhaps more than you had realized, but you may still be reluctant to accept the claim that all behaviour is determined. If so, one reason could be that this conclusion is so profoundly counterintuitive. If you decided to have pancakes for breakfast one morning rather than cereal, you wouldn’t have any sense of compulsion; quite the contrary, you would feel free to choose whichever one you wanted. How can our behaviour be determined if every decision we make feels so free?

Determinism’s Implications

We’ve suggested that one reason why people resist the idea that behaviour is determined is the feeling of freedom—if we constantly feel free to choose between alternative plans of action, it is difficult to believe that our choices are determined. There is, however, another reason, namely determinism’s implications. If our behaviour really were determined, that would seem to imply a view of the world that most of us find unpleasant, even abhorrent. If behaviour is determined, would that mean that we are helpless, like billiard balls hurtling blindly through space, propelled by forces we cannot resist?

Behaviour Must Be Lawful

We have looked at evidence that our genes and experiences can powerfully influence our behaviour, sometimes without our realizing it. In this chapter we are going to consider a more radical claim, not that our behaviour might be determined but that it must be. This claim is based on two assumptions: Our brains control our behaviour. As with any biological system, the brain’s operations obey the laws of physics and are thus entirely lawful.

The Feeling of Freedom: Making Decisions and Solving Problems

To recapitulate, we’ve been examining the claim that the reason our choices feel free is simply that we aren’t aware of the unconscious processes that have shaped them. The examples we’ve looked at so far, however, have involved relatively simple forms of behaviour, such as associating a light with a puff of air in the eye. Conditioning like this is very far from the kinds of sophisticated thought required to solve problems or make complex decisions. So in this chapter we’ll turn to these more complex forms of thought. Do unconscious processes also play an important role here?