José E. Burgos

Theoretical note: simulating latent inhibition with selection neural networks.

The selection neural-network model proposed by Donahoe et al. [J. Exp. Anal. Behav. 60 (1993) 17] was used to simulate latent inhibition (LI). The model can simulate increases of LI by the number, intensity, and duration of preexposed conditioned stimulus (CS). It can also simulate dependence on total CS preexposure time, CS specificity, and attenuation by preexposure to a compound that includes the to-be-trained CS. It also predicts a potentially new phenomenon: acquisition facilitation by preexposure to a stimulus that is orthogonal to and synaptically competitive with the to-be-trained CS. The basic mechanism is the same through which the model simulates extinction, namely, weight decrement. The realization of this mechanism in the present simulations required two conditions. First, networks had to come to the experimental situation with substantial initial connection weights in the sensory-association subnetwork (0.15, compared to the 0.01 value we have used in all previous simulations). Second, the discrepancy threshold for deciding whether to increase or decrease weights had to be larger than zero (the value we have used in all published simulations). A value of 0.001 was sufficient to produce all the effects.

Theoretical note: the C/T ratio in artificial neural networks

This paper describes computer simulations of the effect of the C/T ratio on acquisition rate in artificial neural networks. The networks consisted of neural processing elements that functioned according to a neurocomputational model whose learning rule is consistent with information on dopaminergic mechanisms of reinforcement. In Simulation 1, three comparisons were made: constant C and variable T, variable C and constant T, and a constant C/T with variable C and T. In the last two comparisons, C was manipulated by changing the probability of reinforcement within the intertrial interval (ITI), in the absence of the conditioned stimulus (CS). Acquisition rate tended to increase with C/T, and the invariant ratio had no effect. In Simulation 2, C was manipulated by changing the ITI, with continuous reinforcement in the presence of the CS and no reinforcements in its absence. Results were comparable to those obtained in Simulation 1. Simulation 3 further explored the effect of the invariant ratio, but with larger absolute values of C and T, which slowed acquisition significantly. The results parallel some experimental findings and theoretical implications of the Gibbon-Balsam model, showing that they can emerge from the moment-to-moment dynamics of a neural-network model. In contrast to that model, however, Simulation 3 suggests that the effect of invariant C/T ratios may be bounded.

Neural-network simulations of two context-dependence phenomena

This paper describes simulations of two context-dependence phenomena in Pavlovian conditioning, using a neural-network model that draws on knowledge from neuroscience and makes no distinction between operant and respondent learning mechanisms. One phenomenon is context specificity or the context-shift effect, the decrease of conditioned responding (CR) when the conditioned stimulus (CS) is tested in a context different from the one in which it had been paired with the unconditioned stimulus (US). The other effect is renewal, the recovery of CR in the training context after extinction in another context. For specificity (simulation 1), two neural networks were first given 200 CS-US pairings in a context. Then, the CS was tested either in the training context or a new context. Output activations in the new context were substantially lower. For renewal (simulation 2), two networks were first given 200 CS-US pairings in a context, then 100 extinction trials in either the same context or a new one, and then tested back in the training context. Output activations during the test phase were substantially higher after extinction in a new context. The results are interpreted in terms of the dynamics of activations and weights.

Pavlovian conditioning: Pigeon nictitating membrane

A new Pavlovian conditioning preparation was developed using the nictitating membrane of the restrained pigeon. Either visual or auditory stimuli served as conditioned stimuli (CSs) with an unconditioned stimulus (US) of a puff of air to the cornea. Movement of the nictitating membrane constituted the conditioned and unconditioned responses (CR and UR). Conditioning was studied with the Kamin blocking procedure. In agreement with findings from other conditioning preparations, responding to the redundant stimulus was attenuated relative to a stimulus that received the same number of CS-US pairings in a compound-conditioning procedure. Although response attenuation occurred, substantial individual variation was observed within the blocking procedure, a finding with some precedent in the experimental literature. Theoretical analysis and neural-network simulations indicate that inter-subject variation in response attenuation may result from differences in the extent to which contextual stimuli contribute to the functional CS.

A simultaneous procedure facilitates acquisition under an optimal interstimulus interval in artificial neural networks and rats

In a computer simulation, a neural network first received a simultaneous procedure, where the interstimulus interval (ISI) was 0 time-steps (ts). Output activations were near zero under this procedure. The network then received a forward-delay procedure where the ISI was 8 ts. Output activations increased to the near-maximum level faster than those of a control network that first received an explicitly unpaired procedure. Comparable results were obtained with rats that first received trials where a retractable lever was presented for 3s concurrently with access to water. Low-lever pressing was observed under this procedure. The rats then received trials where the lever was followed 15s after by water. Lever pressing appeared faster than a control group that received the 15-s ISI after an explicitly unpaired procedure. The model used in the simulation explains these results as connection-weight increments that promote little output activations in a simultaneous procedure, but facilitate acquisition in an optimal ISI.

Unified principle of reinforcement in a neural-network model: Reply to N. T. Calvin and J. J. McDowell.

An article published in Behavioural Processes (Calvin and McDowell, 2015) contemplated that the approach to neural networks developed by the present authors cannot simulate certain behavioral findings, notably the Kamin blocking effect and successive conditioning. Here we demonstrate that these concerns are unwarranted as an overall characterization of the approach. In addition, several other more general issues identified in the target article are addressed as well. These include the determination of network architectures, the assignment-of-credit problem, the potential for catastrophic interference, and the falsifiability of the model.

Autoshaped choice in artificial neural networks: implications for behavioral economics and neuroeconomics.

An existing neural network model of conditioning was used to simulate autoshaped choice. In this phenomenon, pigeons first receive an autoshaping procedure with two keylight stimuli X and Y separately paired with food in a forward-delay manner, intermittently for X and continuously for Y. Then pigeons receive unreinforced choice test trials of X and Y concurrently present. Most pigeons choose Y. This preference for a more valuable response alternative is a form of economic behavior that makes the phenomenon relevant to behavioral economics. The phenomenon also suggests a role for Pavlovian contingencies in economic behavior. The model used, in contrast to others, predicts autoshaping and automaintenance, so it is uniquely positioned to predict autoshaped choice. The model also contemplates neural substrates of economic behavior in neuroeconomics, such as dopaminergic and hippocampal systems. A feedforward neural network architecture was designed to simulate a neuroanatomical differentiation between two environment-behavior relations X-R1 and Y-R2, [corrected] where R1 and R2 denote two different emitted responses (not unconditionally elicited by the reward). Networks with this architecture received a training protocol that simulated an autoshaped-choice procedure. Most networks simulated the phenomenon. Implications for behavioral economics and neuroeconomics, limitations, and the issue of model appraisal are discussed.

Transitive Inference Remains Despite Overtraining on Premise Pair C+D-

Transitive inference (TI) has been studied in humans and several animals such as rats, pigeons and fishes. Using different methods for training premises it has been shown that a non-trained relation between stimuli can be stablished, so that if A > B > C > D > E, then B > D. Despite the widely reported cases of TI, the specific mechanisms underlying this phenomenon remain under discussion. In the present experiment pigeons were trained in a TI procedure with four premises. After being exposed to all premises, the pigeons showed a consistent preference for B over D during the test. After overtraining C+D- alone, B was still preferred over D. However, the expected pattern of training performance (referred to as serial position effect) was distorted, whereas TI remained unaltered. The results are discussed regarding value transfer and reinforcement contingencies as possible mechanisms. We conclude that reinforcement contingencies can affect training performance without altering TI.

Selection by reinforcement: A critical reappraisal

This essay is a critical reappraisal of the idea of ontogenetic selection by reinforcement, according to which learning, specifically conditioning, in the individual animal is deeply analogous to phylogenetic evolution by natural selection. I focus on two general versions of this idea. The traditional Skinnerian version restricts the idea to operant conditioning and excludes Pavlovian conditioning, based on a sharp dichotomy between the two types of conditioning. The other version extends the idea to Pavlovian conditioning, based on a unified principle of reinforcement that applies to both types of conditioning, and linked to a neural-network model. I criticize both versions on the same grounds, for being: 1) unable to capture Pavlovian conditioning; 2) unnecessary to formulate said model and use it for explanation and prediction (its combination with a genetic algorithm allows for a substantive contact with the theory of evolution by selection, without the idea of selection by reinforcement), and 3) metaphysically unsound. Non-selectionist accounts of conditioning are not only possible but also more intelligible, explanatory, and heuristic.