Symbolic Behaviour in Artificial Intelligence
Adam Santoro, Andrew Lampinen, Kory Mathewson, Timothy Lillicrap, David Raposo
SSymbolic Behaviour in Artificial Intelligence
Adam Santoro *,1 , Andrew Lampinen *,1 , Kory Mathewson , Timothy Lillicrap and David Raposo * Equal contributions, DeepMind
The ability to use symbols is the pinnacle of human intelligence, but has yet to be fully replicated inmachines. Here we argue that the path towards symbolically fluent artificial intelligence (AI) beginswith a reinterpretation of what symbols are, how they come to exist, and how a system behaves when ituses them. We begin by offering an interpretation of symbols as entities whose meaning is establishedby convention. But crucially, something is a symbol only for those who demonstrably and actively par-ticipate in this convention. We then outline how this interpretation thematically unifies the behaviouraltraits humans exhibit when they use symbols. This motivates our proposal that the field place a greateremphasis on symbolic behaviour rather than particular computational mechanisms inspired by morerestrictive interpretations of symbols. Finally, we suggest that AI research explore social and culturalengagement as a tool to develop the cognitive machinery necessary for symbolic behaviour to emerge.This approach will allow for AI to interpret something as symbolic on its own rather than simply manip-ulate things that are only symbols to human onlookers, and thus will ultimately lead to AI with morehuman-like symbolic fluency.
Keywords: Symbolic AI, Connectionism
1. Introduction
The ability to use symbols makes it possible to reason about the laws of nature, fly to the moon, writeand recite poetry, and evoke thoughts and feelings in the minds of others. Newell and Simon extolledthe virtues of symbols in their Turing award winning lecture in 1976, claiming that “[s]ymbols lie atthe root of intelligent action” and should therefore be a central component in the design of artificialintelligence (Newell and Simon, 1976). With this in mind they formulated the physical symbol systemhypothesis (PSSH), which states that a physical symbol system has the necessary and sufficient meansfor general intelligent action (Newell, 1980; Newell and Simon, 1976). Thus began a decades-longresearch program in AI (often referred to as Good Old-Fashioned AI, or GOFAI) to create intelligenceby applying the syntactic mechanisms developed in computer science, logic, mathematics, physics,and psychology.Newell and Simon argued that their hypothesis is empirical, meaning that experimental andobservational evidence can prove it to be false (Newell and Simon, 1976). In the time since itsproclamation, there have been demonstrations that human thought is not best explained with formal,syntax-centric models (Johnson-Laird, 1999; Kosslyn and Hatfield, 1984; McClelland et al., 2010;Rumelhart and McClelland, 1986), and accumulating feats of intelligence that heavily depend onnon-GOFAI methods (Akkaya et al., 2019; Brown et al., 2020; Mnih et al., 2015; Silver et al., 2016,2017). There have also been pointed rational arguments, stating that GOFAI methods alone areinsufficient because they do not connect to the real world (Harnad, 1990), and philosophical attackson the fundamental assumptions underlying GOFAI methods (Dreyfus, 1972, 1965; Smith, 2019).Contemporary AI research strays from GOFAI methods, and instead leverages learning-basedartificial neural networks (LeCun et al., 2015). Nevertheless, the apparent weaknesses of currentconnectionist-based approaches are leading some to call for a return of symbolic methods, a de-
Corresponding author(s): [email protected] © 2021 DeepMind. All rights reserved a r X i v : . [ c s . A I] F e b ymbolic Behaviour in Artificial Intelligence emphasis of connectionist methods, or some reconciliation of the two (Garcez and Lamb, 2020;Garnelo and Shanahan, 2019; Lake et al., 2017; Marcus, 2018, 2020). As the argument goes, perhapsit is not true that these methods alone lead to human-like intelligent action, but rather, the pathtowards intelligent action lies in a chimeric assembly of old methods with the new.However, here we argue that building a system that exhibits human-like engagement with symbolsis not a necessary consequence of deploying “symbolic methods” (Boden, 2014) (which we termGOFAI-like), whether in conjunction with neural networks or not. This is because these approachesmiss much of the full spectrum of symbolic capabilities that humans exhibit. To that end, we focuson describing the key components of symbolic behaviour , which is the observable consequence ofsymbol-use that we are ultimately concerned with, and which can be directly measured and targetedwhen building AI. Indeed, it was behavioral evidence from humans—including their introspectionabout problem solving—that Newell and Simon (1976) cited in support of the PSSH. Our argumentwill center on the question: how does a symbolic thinker behave if they interpret something as asymbol, and can these behaviours be replicated in machines? An intriguing aspect of our analysis isthat the traits associated with symbolic behaviour can be made thematically coherent if we reinterpretwhat symbols are and how they come to exist. To that end, our argument will proceed as follows:We will start by presenting this interpretation of symbols, which underpins the broader character-istics of symbol-use in humans, and contrasts with narrower interpretations that focus on the syntacticmanipulation of discrete tokens. In particular, our interpretation emphasizes how symbol meaningis established by convention, how symbols only exist with respect to an interpreter that exhibitscertain behaviours, how symbol meaning is independent of the properties of their substrate, and howsymbols are situated in broader symbolic frameworks. We will then articulate some behavioral traitsassociated with symbol-use in humans and link them back to our interpretation of symbols. We thendraw inspiration from developmental psychology to argue for social and cultural engagement as apositive pressure for creating AI that can exhibit symbolic behaviour. Finally, we discuss how ourperspective relates to other views on symbols and AI, and conclude by summarizing our argumentand the research directions we prescribe.
2. Properties of Symbols and Symbolic Systems
Although symbols are conceptually and historically important in the study of AI, they do not have acanonical definition in the literature. Working definitions are often implicit, or only vaguely referenced,presumably because we can leverage intuitions gained from the study of logic, computer science,philosophy, mathematics, or semiotics. But the lack of definition has led to tension and disagreement.For example, in a series of articles about the first self-driving car, ALVINN (Pomerleau, 1989), Touretskyand Pomerleau expressed frustration that seemingly anything that “designates” or “denotes” is raisedto the status of “symbol” by some researchers, rendering the concept operationally vacuous (Touretzkyand Pomerleau, 1994; Vera and Simon, 1993).Newell and Simon define symbols as a set of interrelated “physical patterns” that could “designateany expression whatsoever” (Newell and Simon, 1976). Indeed, the property of arbitrary designationis a common theme throughout the literature. But, crucially, Newell and Simon do not explicitly statewho creates or is aware of the designation. What we hope to show is that considering the answer to“who creates or is aware of the designation” is fundamental for creating AI with symbolic competence.First, we offer a disclaimer. Our goal is not to proclaim that any particular definition is objectivelycorrect. Rather, different uses of the word “symbol” offer unique perspectives. We hope to offerguidelines for thinking about symbols that offer promise for building AI that can achieve what wedescribe as “symbolic behavior”, and further, to argue that achieving symbolic behavior is a useful goal. For ease of writing, we will speak declaratively in the remainder of the paper (e.g., “symbolsare...”), but we do not wish to reject the validity of other perspectives. Similarly, we will use termslike “understanding,” and “meaning” that have acquired a conventional meaning but are difficult todefine precisely. While this vagueness is unavoidable in some of the issues we must discuss, we willattempt to describe our proposed path forward concretely in terms of operationalizable behavioralcriteria and research directions that stem from our interpretation of a symbol.A symbol consists of some substrate—a scratch on a page, an air vibration, or an electrical pulse—that is imbued with meaning by matter of convention. However, the substrate is only a symbol to thosewho demonstrably: (i) understand the established conventional meaning, and (ii) understand thatthe meaning is what it is because of convention. That symbols’ meanings are a matter of conventionaligns with Newell and Simon’s criterion of arbitrary designation. But we extend this criterion byemphasizing the role of the interpreter who understands the conventional relationship between asubstrate and what it represents. A symbol’s meaning is therefore not intrinsic to its substrate, noris it objective. Symbols are subjective entities; to those who do not participate in the establishedconvention, a scratch on a page is simply a scratch on a page (or at best is recognizable as the substrateof a symbol, but is devoid of meaning, as is a character from an alphabet we do not understand).Our definition of a symbol draws on the work of the philosopher Charles Sanders Peirce. Peirceoutlined three categories of relation—icons, indices, and symbols—whose definitions illuminate therole of convention in establishing meaning (Deacon, 1998; Hartshorne et al., 1931). According toPeirce, icons make reference by way of similarity. A sculpture of an elephant bears literal physicalresemblance to a real elephant, and hence is iconic. Indices, on the other hand, leverage sometemporal or physical connection to the things to which they refer. The mercury in a thermometer, forexample, changes its height in accordance with the temperature. But to Peirce, symbols depend on an“agreed upon link”, regardless of the actual physical or temporal characteristics of the medium. It isonly by agreement that a flag comes to symbolize one country rather than another. There is nothinginherent in the properties of cotton or ink that makes a flag symbolize Canada. The delineationbetween icons, indices, and symbols need not be sharp, and in reality Peirce’s proposed categorizationis probably best understood as a guideline. Nevertheless, for the purposes of our argument we canentertain an idealized notion of a symbol wherein meaning is established purely by convention.There is much to unpack in this interpretation of symbols. We’ll first describe the relationshipbetween representation and meaning as determined by convention, and in the later section titled“Symbolic Behaviour” we’ll discuss the role of the symbolic interpreter, what it means to understandmeaning established by convention, and what it means to understand that meaning is what it isbecause of convention.
A first consequence to construing symbol meaning as a matter of convention (an “agreed upon link”) isthat a symbol’s meaning is rendered independent of the properties of the symbol’s substrate. When arepresentation’s properties do not convey meaning in and of themselves, representations can come tobe used to symbolize anything . And similarly, the same meaning can be ascribed to representationswith wildly different characteristics. For example, consider the differences between written text andoral speech. The underlying meaning can be similar, even though the mediums are distinct. Theconverse is also possible: similar representations can symbolize very different things to differentinterpreters (such as a cross-lingual homophone). This connects with the “arbitrariness” emphasized To emphasize this point by taking it to the extreme, as computer scientists are well aware even a single bit (i.e. 0 or 1)can come to symbolize an elephant, or democracy, or your state of mind when you visited a grocer and they were out oforanges. by (Newell and Simon, 1976), but with a different focus—it is important to speak about symbols withreference to a particular interpreter who generates a particular interpretation, and to not speak aboutsymbols as if they transmit objective meanings to any possible interpreter.Shannon’s communication theory exploits the fact that material substrates do not inherentlysymbolize anything (Shannon, 1948). Once one establishes a coding scheme, then and only thendo sequences of electrical pulses, for example, come to be about something (i.e., they symbolizesomething). But the pulses are only symbolic to those privy to the coding scheme. Part of the insightbehind Shannon’s theory is that it measures quantities (e.g. entropy) based on properties of a symbol’ssubstrate (e.g. an electrical pulse), and not based on what the arbitrary meaning of a message happensto be. In other words, Shannon realized that we can communicate symbols efficiently because we canexploit the fact that symbol meaning is imposed, and hence because we can indiscriminately factormeaning out of the equation when we shuttle signals around. Crucially, to a telephone, or an electricalcable, or a drum, electrical pulses do not mean, nor symbolize, anything. And there is nothing theycan glean from the physical properties of these pulses to make them mean what they do for humans.A second consequence of construing symbol meaning as a matter of convention, therefore, isthat something can be a symbol from the perspective of one system, but not another. Turning tocommunication theory again, suppose Bob establishes a coding scheme with Alice but not with Sally.The sequences of electrical pulses that Bob sends Alice are symbolic to Alice (that is, they have anassociated meaning based on an agreed convention). But to Sally, they are just electrical pulses. Theonly meaning Sally can hope to glean from the message must be extracted from the literal, physicalproperties of electricity (such as its charge), or the temporal electric signal (such as its entropy), orfrom some complicated inference process wherein she predicts the coding scheme using privileged,outside knowledge. Indeed, this fact is the base upon which open, encrypted communication is built.A third consequence of construing symbol meaning as convention is that we can situate symbolsin a broader symbolic system, wherein symbols refer to other symbols in the system in addition toreferring to entities in the world. This property is also central to GOFAI-like approaches to symbols.Indeed, the consequences of taking this observation to the extreme were popularly explored in Searle’sChinese Room Argument (Searle et al., 1980). Just as in these classical approaches, convention candictate that a symbol’s meaning is tied to the meaning of another symbol. We emphasize that there isnot simply a unidirectional definition of one symbol by another, but rather that symbol meaning is aholistic property of the entire symbol system. Deacon similarly argues that the referential power ofsymbols is derived from their “positions in an organized system of other symbols” (Deacon, 1998).So, in characterizing symbols it is often not enough to make reference to a fixed system into whichthey happen to be placed, or to treat symbols as isolated, independent entities. Instead we mustconsider the bi-directional influences caused by a symbol’s placement in a broader symbolic system.As an example, consider the concept of a vector in mathematics. One can use the symbol “v” todenote an arbitrary example of a vector, such as (0.2, -0.5, 0.7) for 3-dimensional Euclidean space.But a vector is more than a symbolic pointer to a collection of numbers. What vectors are, generally,is determined by the notion of a vector space, which is an agreed upon set of axioms and rules thatdescribe a grander symbolic framework within which vectors live. So, describing a vector as simplyan ordered collection of elements is in many ways impoverished. For example, ordered collections donot tell us anything about the rules used to combine or shape vectors. Notably, the axioms and rulesthat describe the grander symbolic system are also symbolic, creating a webbed relationship whereinone type of convention impacts another.
3. Symbolic Behaviour
The previous section offered a view of symbols that emphasized the role of an interpreter. It claimedthat an entity is not a symbol in any objective sense, but rather, is a symbol for an interpreter who treatsit as such . We will now ask: how does an interpreter behave if it interprets something as a symbol? Ifwe can identify the particular behavioural traits that are consequences of engaging with symbols,then we can use them as tangible goals for creating AI that is as symbolically fluent as humans, andfurther, use their presence as evidence that a system interprets something as a symbol. What wehope to show is that the behavioural traits exhibited by symbol-users indicate an active participationin an infrastructure of meaning by convention, with the accompanying understanding that meaning isconventional . In other words, the behavioural traits exhibited by symbol users directly motivate theinterpretation of symbols provided in the previous section, which paid particular attention to the roleof convention in establishing symbol meaning.We will begin with a few examples. Suppose a symbolic thinker creates a symbol. For example, aperson can use the word “dax” as a command for their dog to sit. Consequently, their dog might learnto sit when it hears the word “dax”. Is the word “dax” a symbol to the dog? Similarly, consider apesthat learn some simple sign language. Apes can enact the motor program for certain gestures, andthey can form associations between these gestures and some outcomes. For example, suppose theylearn that a particular hand movement results in them receiving a treat. There is nothing inherentto the motor movement itself that entails that the ape should get a treat. So, the gesture and itsoutcome are more-or-less arbitrarily linked, which aligns with the properties we previously outlinedabout symbols, as well as with the “arbitrariness” highlighted by Newell and Simon (1976). Is thegesture a symbol for the ape?Now consider an adult human performing the gesture. Humans understand, to a degree notparalleled in other animals, that they are participating in a cooperative interaction involving a sharedunderstanding of a situation. A human gestures with the knowledge that other humans understandthe gesture’s meaning, and can use the gesture themselves for their own ends. Moreover, they knowthat the gesture is useful because other humans agree on its meaning. They can even participate withothers to alter the gesture’s purpose to be “I’d like the sweet treat, not the salty one.” To humans, thegesture is not just a tool, or a means to an end that might happen to involve others. It is a movementthat others could similarly use, or modify, because of its shared, conventional meaning. Therefore,it is the uniquely human recognition of symbols as conventional that allows us to construct newsymbols, to change the meaning of existing symbols, and to situate symbols in the context of othersymbols. These abilities allow us to flexibly communicate with others, to aid our own thinking, andto construct new ideas or revise old ones. Apes, on the other hand, have a comparatively reducedtendency to participate in established convention of what gestural movements come to mean, orought to mean (Tomasello, 2019). Apes instead gesture because they want something, and they’veassociated the gesture with getting that thing; that is, they merely exploit, via learned association,what has already been established as convention by humans.So at least in humans, symbolic behaviour involves a cognitive capacity that pushes beyond mereassociation of action with outcome. It pushes beyond the association of some gesture, marking, orutterance with meaning. Symbolic behaviour instead exemplifies an active engagement with symbolsas entities whose meaning is established by convention. The distinctions we draw here might seemunimportant, or merely a game of semantics. But the added burden of actively engaging with symbolsin this way is a critical piece of the puzzle for designing AI that is as symbolically fluent as humans.Thus is true whether we imagine symbol-use in a social setting, or their use purely within an individual(e.g., to aid one’s reasoning, as Feynmann recalls doing when he invented new notation to studytrigonometry; Feynman and Sackett, 1985). For example, do we want AI that can only form brute associations between scratches on a page and things in the world, or do we want AI that can conjureup a new symbol to aid its reasoning, as a mathematician would when exploring the frontiers of ourknowledge? Do we want AI that instinctively performs some action when given a particular signal, ordo we want AI that cooperates with us to devise new signs and signals so that our future interactionsare more efficient and fruitful? And how would AI do the latter if it has no conceptual understandingof meaning being constructed through convention? How would it do the latter if it doesn’t know howto actively participate in an infrastructure of shared meaning? So, while we do not claim that theway we construe symbol use is “correct”, we nevertheless argue that it is better aligned with what wewant symbolic fluency to look like (and what it looks like in humans), and hence, it is an importantdirection for AI research.To this end, we propose some interrelated behavioral traits that reveal an active participation inan infrastructure of meaning by convention. In particular, symbolic behaviour is receptive, constructive,embedded, malleable, separable, meaningful, and graded . We will unpack each of these in turn, andspeak briefly to how they are being addressed (or how they might be addressed if they are not) in thecurrent AI literature. Receptive . Symbolic behavior includes the ability to appreciate existing conventions, and toreceive new ones. For example, humans can use a new word when its meaning is conveyed, whileacknowledging that others also have this meaning in mind when they use the same word. This criterionmight seem odd given the examples at the beginning of the section regarding dogs and apes, which de-emphasized the understanding of a symbol’s meaning for establishing evidence of symbolic thought.However, being receptive to an entity and its associated meaning is still a basic ability requiredfor symbolic understanding, even if it might not constitute sufficient proof that one understandssomething as symbolic. As we’ve seen, many animals can associate sounds, gestures, words, or imageswith certain meanings through brute association, without an accompanying understanding thatthese meanings are a matter of convention. So, while being receptive to meaning is a basic mode ofparticipation in a symbolic framework, it does not on its own entail fluent symbolic thought.Receptiveness to meaning is arguably the predominant behavioural trait explored in current AIresearch. For example, researchers assess language models by their ability to engage with existingconvention (as measured by the quality of the text they produce, and whether it is meaningful to ahuman onlooker). Work on multi-modal perception measures engagement with existing conventionby asking a model to convert between words and some of the things humans have in mind when theyuse those words (e.g., by producing an image of a scene given some textual description) (Radfordet al., 2021, for a recent example). Instruction-following agents map between symbolic inputs andgoals (Hermann et al., 2017). New research also shows that deep learning models can be receptive toestablished convention with only a single exposure (or a few exposures) to a new symbol (Brownet al., 2020; Hill et al., 2020). Evidently, the ability to engage with established convention manifestsin a number of ways, and research in this space is currently thriving.
Constructive . A second trait of symbolic behaviour is the ability to form new conventions byimposing new meaning on effectively arbitrary substrates. Whereas the previous behavioural trait(receptiveness) speaks to a system’s ability to appreciate the meaning of a symbol that is imposed bysomeone else, this property refers to the dual ability to create a new symbol when doing so would beuseful. This ability is critical for communicating efficiently (e.g. creating a new term for a recurringsituation) as well as for creating new systems of knowledge (much scientific progress begins with thecreation of a new symbol, e.g. a “gene”). Indeed, one substantial benefit of engaging with symbolsis that one can reduce the mental burden of thinking about a complex concept by denoting it witha symbol, even if the symbol is used purely internally. Constructive use of symbols demonstratesboth the understanding that creating symbols is useful and understanding of the fact that, because meaning is conventional, it can be imposed arbitrarily on top of any substrate.Evidence for this behavioural trait is scarce in current AI research. While much of what we docomprises creating models that engage with the conventions humans have already established, lesswork probes a model’s capacity to construct new conventions by imposing meaning on arbitrarysubstrates. Few-shot learning explores a model’s ability to rapidly assign meaning to a new substrate,but the meaning is usually still human-constructed (e.g., categories of images that are relevant tohumans). Evidence for this behavioural trait might instead look like the following: (i) A model isasked a question, and when explaining its answer it invents a new symbol to streamline or clarify itsreasoning process or communication. (ii) Agents cooperating on a team invent terminology so thatthey can communicate without exposing their strategy to their opponents. (iii) A model constructsan algebra and uses it to prove something meaningful, or similarly, we withhold knowledge of aparticular mathematical concept and its associated symbol and see whether a model proposes a wayto solve a problem that invokes that concept and symbol.
Embedded . Symbols conform and contribute to the principles of organization that describe thebroader symbolic system in which they’re associated, and symbolic behaviour reflects this understand-ing. Deacon again:“The learning problem associated with symbolic reference is a consequence of the factthat what determines the pairing between a symbol (like a word) and some object orevent is not their probability of co-occurrence, but rather some complex function of therelationship that the symbol has to other symbols. . . symbols cannot be understood as anunstructured collection of tokens that map to a collection of referents because symbolsdon’t just represent things in the world, they also represent each other.” (Deacon, 1998)Because the meaning of a symbol is determined in part by how that symbol interacts with the restof the symbol system, introducing new symbols can radically alter the interpretation of other symbols.For example, the discovery of category theory as a unifying perspective on many mathematical researchareas fundamentally changed the questions researchers ask within many of those areas (Mac Lane,1988).There are a few ways that current AI models demonstrate behaviour that appreciates the embed-ded nature of symbols. Neural networks in particular have strong representational and functionalpriors that bias them toward such behaviour: since their internal vector representations can becompared in magnitude and angle, as any one representation changes so too does its relation toother representations. One way this manifests is with distributed word embeddings trained usinglarge corpuses of human generated text, which capture semantic content like analogies (Mikolovet al., 2013). However, this way of hard-coding the embedded nature of symbols can sometimeshave undesirable consequences, such as when symbols indiscriminately absorb the biases of theinput data (Bolukbasi et al., 2016; Caliskan et al., 2017). While Lake and Baroni (2018) show thatlearning relationships among symbols in a restricted context can lead to poor generalization fromdeep learning models, these models may generalize better when language is embedded within a richer,more embodied context (Hill et al., 2019; see also McClelland et al., 2020). Indeed, humans canuse embodied understanding, such as physical gestures, to help them grasp abstract mathematical orscientific concepts (Goldin-Meadow, 1999). Similarly, knowledge of underlying social constructs aidslogical reasoning in analogues of the Wason Selection Task (JOHNSON-LAIRD et al., 1972; Sperberet al., 1995). The whole structure of knowledge in which symbols are embedded can, and usuallydoes, affect symbol-use.
Malleable . A symbolic thinker demonstrates an understanding that the conventionality of meaning allows for change, and thus that the commitment to the current meaning of a symbol is not ironclad.This allows for meaning to be altered by the creation of other symbols or concepts, as well as theability to integrate a redefinition of a word, or to be able to redefine a symbol oneself.Indeed, human language use exemplifies the malleability of a symbolic system, because ourlanguage use is highly collaborative and pragmatic (Clark, 1996; Clark and Wilkes-Gibbs, 1986; Frankand Goodman, 2012). The interpretation of language depends heavily on the situation in which it isused and the mutual understanding between the speaker and listener, and so understanding humancommunication requires an ability to form pragmatic conventions on the fly (Clark and Wilkes-Gibbs,1986; Hawkins et al., 2018). That is, using and responding to words and utterances in conventionaland contextual ways, rather than fixed and unchanging ones, is necessary for achieving the flexiblelanguage use that humans exhibit. But the malleability of symbols is not only important in everyday communication. Another benefitof understanding that symbol meaning is conventional is the consequent understanding that meaningcould be otherwise, or even should be otherwise. This epistemic humility can be critical to insight,and many important human ideas could not have been obtained without expanding or altering themeaning of a symbol. For example, extending the concept of numbers to include complex numbersallowed humans to mathematically describe the wave nature of fundamental particles, and revisingthe geocentric concept of the universe allowed us to develop a heliocentric model of the solar system.To reason flexibly and productively, it is often necessary to reshape the conventions of meaning,when the extant meaning is misaligned with the physical world or with other aspects of the symbolsystem. But to do this, one must first know that meaning can be reshaped because it is established byconvention.It’s not obvious whether any extant AI system exhibits behaviour demonstrative of a malleableunderstanding of symbols, at least compared to how this behaviour manifests in humans. While wepreviously provided evidence that current models can rapidly appreciate new convention, it’s less clearwhether they exhibit behaviour that demonstrates that they can, or should, shift their understandingof an already known symbol. Returning to the case of language, consider a person from North Americavisiting Britain, who will quickly change their internal definitions of the words “quite”, “boot”, and“lift”. Any learning-based models will surely alter their understanding of a symbol as they experiencenew data; in other words, malleability is entailed by learning. However, this type of malleability ispassive, and depends on researchers to drive it. Humans more actively expand or alter the meaningof symbols. That is, they are malleable for a reason , whether that reason is to permit more fluidcommunication, or to come to a deeper understanding of some phenomenon.
Separable . A symbol user should be able to demonstrate partitioning of their understanding of asubstrate and symbol, or multiple symbols that reuse the same substrate. For example, a maple leafcan be both an entity with inherent properties (e.g., leaves are light and carbon-based), and a symbolfor something else (e.g. Canada). Crucially, the understanding of the leaf as a symbol is not affectedby superficial changes to the leaf (e.g. if the leaf were protected with a fungus-resistant chemical,unbeknownst to the person using it as a symbol). Furthermore, a single symbol may have multiplemeanings which are separable from each other. For example, the word “bank” in English can refer toa border of a river or to a financial institution. But the separability of meaning goes far beyond aword having multiple fixed meanings. We can readily receive or construct a new convention withan existing symbol, without changing our understanding of the extant denotations of that symbol.For example, if someone is introduced to the programming language Python, they can adopt the We note that some might dispute at what stage context plays a role in human language understanding — for examplewhether context only constrains the referents of indexicals or the implicatures drawn (Lasersohn, 2012) — but this debateis not relevant to our behavioral claims. convention that Python = programming language in future conversations with programmers, withoutchanging the way they interpret the word python when talking to a veterinarian.Although contemporary language models show the ability to partition symbol meaning amongknown senses, for example disambiguating the word “bank” in context, their ability to fully separatemeaning remains to be seen. In particular, can these models apply new meaning to a known word?For example, could these models handle challenges like “When I say ‘walk’ I mean ‘swim’ and whenI say ‘park’ I mean lake. If I walk in the park, will I get wet?” Or can they handle a new use of aword created after they were trained? This will be important for models to adapt appropriately toshifting language across contexts and time. The ability to partition substrate properties from symbolproperties will also need further investigation. Meaningful . Symbolic behavior should demonstrate use and communication of the meaningbehind the symbols and reasoning processes that employ them. Consider what happens when weinterpret symbol-use as consisting purely of syntactic manipulation. Harkening back to the symbolgrounding problem (Harnad, 1990), in purely syntactic systems meaning must somehow propagatethrough meaning-agnostic syntactic reasoning processes. But meaning itself is essential to effectiveuse of symbols. We don’t employ algebras because we get an arbitrary physical token at the otherend, but rather because we can interpret the reasoning process and the resulting arbitrary physicaltokens as meaningful. Fields medalist Paul Cohen highlighted this aspect of symbolic thinking inmathematics: “I never was able to successfully analyze proofs as a combinatorial ‘game’ played withsymbols on paper” instead, to reason productively “one must essentially forget that all proofs areeventually transcribed in this formal language” (Cohen, 2002). Mac Lane (2012) explains why:“strict formalism can’t explain which of many formulas matter [...] the choice of form is determinedby ideas and experience.” The meaning of symbols and their relationships gives mathematiciansintuitions for why a theorem might be true, and thereby ideas of how to prove it (as do examplesillustrating the theorem, see Polya, 1990). McClelland et al. (2016) make similar arguments aboutthe primacy of meaning in mathematics. Similarly, Marder (2020) describes theory in biology as“disciplined dreaming” wherein we face the “challenge of creatively marrying the rules of mathematicsand physics with what is known of fundamental biological principles.” It is meaning that solves theframe problem (Dennett, 2006). It is meaning that drives effective symbolic thinking.Meaning is not just useful for constraining reasoning, however. A major goal of theories or proofsis to convey understanding. Thus, mathematicians are often unsatisfied with a formal proof of atheorem that does not clearly communicate understanding of the underlying ideas (Lange, 2015;Mac Lane, 1988):“Mathematicians must search for the most illuminating proof. What matters is proofswhich are “deep”— but not in a logistic sense of depth. The logical rules of inferenceprovide a careful syntax for each proof, but they do not serve to indicate the steps thatare crucial — those steps where one understands why the proof works. For this reason,Mathematics is not just logical deduction.” Mac Lane (1988)Models must therefore understand their own reasoning processes as meaningful to exhibit fullymeaningful symbolic behaviour. Harnad alluded to this fact — despite focusing on groundingtokens, he noted that “the syntactic manipulations both actual and possible and the rules [must be]‘semantically interpretable”’ (Harnad, 1990). That is, the principles by which the model manipulatesits symbols must themselves be understood symbolically by the system. Symbolic behaviour shouldreveal the construction of “deep” arguments, not merely chains of valid logical inferences that somehowarrive at the correct conclusion.
Evidence for meaningful understanding of symbols and reasoning will therefore come in the formof both being able to use semantics to reason more effectively (for example, finding a proof of atheorem efficiently rather than by blindly searching) as well as to communicate how the interactionbetween the semantics and the reasoning process licenses those conclusions (i.e. communicatingwhy the semantics motivates particular steps of the proof, and what that implies about the conceptsin question). A number of recent advances in AI have used deep learning to partially address thefirst point. For example, AlphaZero (Silver et al., 2017) effectively uses the learned semantics of itsestimates of move value as strong heuristics for reducing the search space of possible actions. However,because AlphaZero’s reasoning process (Monte-Carlo Tree Search) is hand engineered and relieson strictly rule-based processes, it is incapable of achieving the second goal. This motivates futureresearch toward systems that can understand their reasoning processes as meaningful.
Graded . Finally, symbol use is not a binary capacity (c.f. McClelland et al., 2010, 2016). We’vepresented behavioural criteria that collectively demonstrate symbolic fluency, and each encapsulatesa spectrum of competencies. As a simple example, young children might be receptive to existinglanguage conventions, but only simple words and phrases. The reasons why a child is not receptive tosome words or phrases might have to do with seemingly tangential cognitive faculties. For example,perhaps the child is inexperienced with parsing certain sounds, and hence has difficulty perceiving aword’s aural boundary. Or perhaps they cannot reproduce the word vocally, and by virtue of havinglittle experience using the word, they have difficulty appreciating its meaning. One can imagine anumber of other scenarios involving the limitations of memory, attention, and nearly every othercognitive faculty. What this example further illustrates is that symbolic thinking is not an independent,siloed function, but is rather an interdependent component of our broader cognition. And since otheraspects of our cognition—attention, memory, perception—can be graded, so too can our ability tothink symbolically.For example, although we highlighted the importance of malleability for allowing the revision ofsystems of knowledge, this does not mean that every human will be able to come up with the rightalterations. Only a few may have access to the particular combination of information and experiencethat allow insight in any particular context, for example to realize that the heliocentric model of thesolar system was more parsimonious than the geocentric one. Similarly, one can imagine inventingnew conventions, but not with unbounded freedom. For example, consider the Pirahã people of theAmazon who do not have words for exact numbers, and instead refer to imprecise quantities such as“small amount” or “larger amount” (Everett et al., 2005). While we might not expect them to createa symbol that meant “precisely five trees”, they clearly are symbolic thinkers, as evidenced e.g. bytheir use of language. Symbol use will always be constrained in one way or another (by cognitive,conceptual, or cultural forces) and hence will inevitably present as a graded capacity.As it regards AI, acknowledging that the ability to use symbols is graded entails acknowledgingthat symbol use is not synonymous with a particular computation (e.g. syntactic control over entities),or the use of a particular type of representation (e.g. discrete tokens). We must instead ask howcertain computations or representations alter the behaviour of a system, and use analyses of the othercriteria presented above to characterize how a system expresses engagement with symbols. Symbolicbehavior manifests in a variety of ways in humans, and it’s reasonable to expect that it will manifestin a number of ways in AI as well.
When motivating the physical symbol system hypothesis, Newell and Simon made reference to thedevelopment of formal logic as a “symbol game”, stating that: “Logic, and by incorporation all of mathematics, was a game played with meaninglesstokens according to certain purely syntactic rules. All meaning had been purged. Onehad a mechanical, though permissive (we would now say nondeterministic), system aboutwhich various things could be proved. Thus progress was first made by walking awayfrom all that seemed relevant to meaning and human symbols. We could call this thestage of formal symbol manipulation.” (Newell and Simon, 1976)This view dominated our conception of symbols in AI. Symbols became synonymous with discreteentities, symbolic processing was what we did when pushing symbols around with formal algebras,and the properties that came to be associated with symbolic systems (e.g., compositionality) wereinherited from these algebras.But it is because we are already symbolically fluent that we find value in purely mechanical, syntac-tic operations. As we noted above, mathematicians generally rely on meaning in their reasoning aboutwhich manipulations to make. Similarly, as we mentioned in regards to Shannon’s communicationtheory, purely mechanical computations are motivated by a deep, extant understanding of symbols.They’re motivated by the knowledge that human interpreters can bring meaning back into the foldonce the syntactic manipulation is complete. But crucially, purely syntactic operations per se are notsynonymous with symbol-use because they do not consider the steps where meaning is extractedfrom, and later imbued in arbitrary physical tokens, which we argue is where a large part of theintelligence behind symbolic thinking lies. Claiming that AI is symbolically fluent because we equip itwith syntactic-control of discrete entities therefore puts the cart before the horse; it assumes thatsuch manipulations would be useful to a system that has yet to demonstrate an understanding ofsymbols as symbols, or worse, it assumes that symbolic fluency consists only of an ability to exhibitsyntactic control of discrete entities.Notably, the importance of establishing symbol meaning was known by Newell and Simon, whowrote that “it is a hypothesis that these symbols are in fact the same symbols that we humans haveand use everyday of our lives” (Newell, 1980). This has become an underappreciated, but criticalassumption underlying GOFAI-like methods—namely, that it is possible to assume away the difficultpiece of the symbolic puzzle: what meaning is, where it comes from, and who it is known to. To befair, at the time this wasn’t an outlandish hypothesis, because if GOFAI-like systems indeed provedcapable of feats of intelligence rivalling humans, and if human intelligence were proved to be bestmodeled by these kinds of symbolic methods, then the hypothesis would be corroborated. And at thetime, no other class of model was as promising. But our discussion here highlights a key reason tosuspect that this hypothesis does not hold: it’s not obvious how GOFAI-like models could interpret theentities over which they reason to be symbols, in order to exhibit the behavioral abilities that we haveoutlined. There is little evidence of these capabilities in GOFAI models, nor is there evidence that themeanings of the entities they manipulate are shared with humans (Harnad, 1990; Smith, 2019).Many contemporary approaches to symbolic computation—including hybrid architectures com-bining GOFAI-like methods with deep learning—take a syntax-centric view of symbolic processing,and hence do not advocate for symbolic behaviour as we have delineated it. Instead, they advocate aparticular set of computational mechanisms that operate over entities that are symbols to the humansthat create the systems. Crucially missing is a solution to how these things actually come to be symbolsto the systems themselves. The only known proof-of-principle systems capable of symbolic thoughtare biological neural networks, and we have highlighted how many aspects of symbolic behavior canalready be observed to some degree in contemporary neural network models. This suggests that thekey to bringing symbols into the fold in hybrid systems might be their neural network “front-ends”and that, ironically, their GOFAI-like “back-ends” might be fundamentally preventing them fromachieving human-like symbolic behavior.
Symbolic behaviour does not have to depend on rules. Rule-based reasoning is a useful tool, but likepurely syntactic reasoning it is most useful when it is guided by a deeper capacity for symbolic thought,including the meaningful understanding of the reasoning process. That is, it is necessary to understandthe meaning of the manipulations, and the relationship of the manipulations to the ultimate goal.Even in the most formal domains humanity has created, such as mathematics, rules are subsidiaryto understanding. We noted above that meaning is often needed to guide mathematicians toward aproof, and to identify “deep” proofs that communicate the reason why the theorem is true. Symbolicbehavior should likewise enable meaningfully constructed threads of reasoning that communicateunderstanding, which is notably not observed in syntax-centric GOFAI-like methods (Harnad, 1990).These matters come to the fore when thinking about important issues of safety and ethics. Whileit might be tempting to think that rule based systems could guarantee that AI will be safe or ethical,we suggest an alternative perspective, echoing others who have highlighted the challenges of creatingrule-based machine ethics (e.g. Allen et al., 2005; Brundage, 2014). Applying strict rules will beproblematic in domains of ethics and safety for reasons similar to those we have outlined in our otherarguments: rule-based systems are incapable of understanding the meaning behind the rules, so theywould only follow the letter of the law, not the spirit (the consequences of which were popularlyfictionalized by Asimov in his Robots series of stories; Asimov, 1950). For example, we could imagineimposing a rule like “never hurt a human,” but in fact humans hurt other humans frequently in waysthat we would consider ethical. What if you need to break a bone to set it cleanly, or amputate agangrenous limb before it sickens the rest of the body? Or what if we have to tell someone about thedeath of a parent? Should we tell them right away? What if we need to withhold the news until theyare in a safe situation to process it, not driving? Our ethical principles are profoundly contextual.Similarly, we might have a rule like “don’t discriminate on the basis of race,” but as practices suchas redlining (in the US) have illustrated, it’s easy to find a proxy variable for race, like neighborhood,and have essentially the same discriminatory effect without directly using the protected category(Woods, 2012). We need a system that behaves in accordance with the meaning behind its principles,not one that just superficially follows the letter of the law. We need systems with judgement (Smith,2019). We suggest that this goal will be more easily achieved by pursuing AI that can exhibit symbolicbehavior within a holistic, meaningful framework of ethics.
4. How Can Symbolic Behaviour Come About?
Symbolic behaviour can be characterized as an engagement with an infrastructure of meaningestablished by convention. In the last section we described the characteristics of this behaviour. Herewe hope to answer: what are the cognitive steps a system must take to form or appreciate an agreedupon link between a substrate and what it (arbitrarily) means?
Our contention is that a necessary(but not necessarily sufficient) cognitive trait is the ability to reconcile perspectives between oneselfand others, and that this trait may emerge as a consequence of effective and efficient social andcultural engagement. Further, we’ll argue that the emergence of symbolic behavior (McClellandet al., 2010) will be most effectively induced through external social and cultural forces, even insystems that primarily engage in introspective symbol use. This perspective is certainly not unique,and has appeared in a number of guises in the literature. For example, specifically in the context oflanguage, many researchers have considered the role of successful communication as a bottleneck forlearning the use of symbols (Kirby, 2002; Lazaridou and Baroni, 2020). Nevertheless, our perspectiveconsiders the more general case of symbol use (language or otherwise). And as we will describe, ithighlights some cognitive prerequisites for symbol-use, rather than suggest algorithmic features (suchas “bottlenecking”, compression, efficiency, etc.) that might necessarily entail symbol-use.
One strategy to elicit an engagement with an infrastructure of meaning by convention in AI isto first identify the types of inferences that humans make when engaged in symbol-use, and thentarget these same inferences when building AI. These inferences have a clear developmental ontogeny.When children are around two years old, they certainly use words as tools. They also come to learnthat seemingly arbitrary behaviours, such as nodding their head, have meaning. But at around threeyears old, children show some evidence of inferring—and then obeying—social norms based onothers behaviour (Schmidt et al., 2016). At around three years old, children also begin to exhibitan appreciation of language as a matter of convention. They assume that words will be known byspeakers of the same language ( but not others ), they expect speakers of their language to use words inaccordance with what they conventionally mean, they reinterpret or ignore noun-object bindings if aspeaker is unreliable later, and (if they are bilingual) they mix languages less (Dautriche et al., 2020;Tomasello, 2019; Wei, 2020). At a similar age, they begin to show signs of considering what could bein the mind of others—what they might know or how they might intend to use a word (Diesendruck,2005). As they age further, their ability to understand meaning as a matter of convention solidifies.Tomasello argues that effective social engagements, like those listed above, require the abilityto recognize distinct perspectives of different social members, and to take an objective perspectivewhich subsumes them. He states:“A chimpanzee sees a monkey escaping, and he knows that his conspecific sitting nextto him sees the monkey escaping also...They are both attending to the monkey escaping,and each knows that the other is too. But they are not jointly attending to it; they are notattending to it as a “we”. Two humans in that same situation could, if so motivated, attendto the monkey escaping together in joint attention. This creates between the two of them akind of shared world, within which they each distinguish their two perspectives. They eachalso understand that both of their perspectives—that is, their beliefs—on the situationcould potentially contrast with an objective (perspectiveless) view of it.” (Tomasello,2019)Humans construct cognitive representations that do not merely reflect a passive, subjectiveexperience. Rather, these representations strive towards objectivity; they reflect coordination acrosssubjective perspectives and what exists in the world, and hence come to be about things that aretrue from multiple perspectives, and perhaps even regardless of perspective. For example, suppose Icome in from outside and start acting as though it’s raining outside. Perhaps I collect my umbrellaand put on a rain coat so that I do not get wet when I go out again. But you recently looked out thewindow, and saw that it was not raining. You must then reconcile and coordinate our perspectives.But crucially, you coordinate under the pretense that there is an objective truth to the matter; it iseither raining or it is not regardless of what one of us might think. One of our subjective perspectivesis mistaken, and perhaps it’s your own because in the time since you looked out the window it couldhave started raining. You therefore come to believe it is raining, despite your subjective experience.The capacity to achieve these coordinated, relatively objective perspectives can emerge as aresult of optimizing social “tasks” (cf. Graziano and Kastner, 2011). Consider the classic false-beliefparadigm, wherein your colleague observes an object being moved from one container to anotherwithout your awareness. If you then search for the object in the wrong container and ask for yourcolleague’s help, they will not help by continuing to search the incorrect container. Rather, they willimmediately go to the new container, because they understand that your perspective conflicts with theobjective state of the world. There is no way you could know the object moved, whereas your colleaguecan because they perceived it, and you must reconcile their behaviour (and hence, their perspective onthe state of the world) with your perspective on the state of the world. Thus, a successful interaction,wherein you retrieve the object from the proper location, can fuel the cognitive machinery necessary for making the types of inferences involved in reconciling various subjective beliefs. And since wehave a notion of “success”, we can imagine a situation wherein it is optimized for across experiences.One can imagine a glut of other situations that can drive the cognitive development required forobjective understanding when optimized, such as jointly attending to a situation or sharing intentionswhen performing a task. For example, children draw upon reconciling perspectives in many situationsto understand what to learn from a social interaction (Gweon, 2020), and even what to teach toothers — for example, they will choose to teach things that would be more difficult to discover ormore valuable to know (Bridgers et al., 2020). Teaching and learning could be both intrinsically andextrinsically rewarding for children, which could encourage them to improve their ability to reconcileperspectives in order to achieve these rewards.Understanding the objective “truth” of a situation by reconciling subjective perspectives mightseem unrelated to understanding convention. However, the skills required for reconciling perspectivesare similar to — and may well be prerequisite for — those required for understanding conventions.Consider again what the understanding of objective perspectives requires: one must reconcile subjec-tive perspectives in the context of an external world that provides the “ground truth”. Understandingand creating convention, on the other hand, requires reconciling subjective perspectives withoutnecessarily being able to appeal to, or depend on any “ground truth”. In particular regards to symbols,understanding a symbol’s meaning cannot come from understanding the objective properties of thesymbol’s substrate, which do not convey meaning in and of themselves by definition. Symbol meaningis instead inherently subjective, and often heavily depends on a reconciliation and coordination ofsubjective perspectives alone.While we’re calling attention to an extreme case here (i.e., reconciliation of subjective perspectiveswithout any hope of appealing to some source of objective truth), how things play out in the realworld is naturally more graded. For example, consider children’s first experiences with symbols, suchas the word “dog”. The people with whom they communicate are predictable users of the word “dog”,and can be counted on as dependable sources to ground the word (and hence, give it meaning).This renders the meaning of the word “dog” nearly as “objective” as any visual, tactile, or auditoryexperiences of actual dogs. But it’s important to note that in these cases meaning is grounded inthings in the world that are not the symbol per se . The symbol substrate’s properties are not doingany work to convey meaning, and initially, it’s only via brute association that children come to learnthat an essentially arbitrary substrate (air vibrations) is being used to refer to something else entirely(a dog). In particular, noun learning in children comprises a gentle learning path to the importantmeta-lessons that: (1) seemingly arbitrary things can be connected (e.g., certain air vibrations anddogs), (2) there are established connections that other people in their environment adhere to.Nevertheless, children eventually cope with situations where they can’t depend on “objective”things in the world to understand meaning. For example, they encounter metaphoric uses of language,they participate in analogical reasoning without immediate access to relevant aspects of the physicalworld (e.g., when told that “a zebra is like a horse with stripes” without having ever seen a zebra;(Harnad, 1990)), they learn words that do not have clear connections to the physical world (such as“wish” or “no”), and they might even participate in the invention of a new word when playing a game.These experiences require coordination of subjective perspectives between themselves and the peoplewith whom they’re communicating, without being able to depend on a source of objective truth ofthe matter (which does not yet exist). The shift in cognitive abilities is indeed reflected in humandevelopment: children first absorb established convention from their social and cultural context beforethey develop the fluency to be active participants, and creators of new convention (Tomasello, 2019).In many ways, reconciling subjective perspectives is harder when we cannot appeal to an objectivestate of the world as an arbiter, which could explain why it appears later in cognitive development,why early child language contains more words for concrete entities than adult language (Brown,
While we’ve emphasized the ability to reconcile subjective experiences between individuals in somesocial setting as a path towards symbolic fluency, there is an alternative possibility. Perhaps this abilityoriginates internally, and just happens to be exploited in social settings. If this were true, then thecapacity for understanding convention in social settings would be a cognitive spandrel (Gould andLewontin, 1979). That is, the ability to engage with social conventions would be a consequence ofinternal machinery that originated from other demands. We will discuss this possibility in this section.Note, however, that the following discussion is not about how a system that is already symbolicallyfluent can and does use symbols of their own creation, outside of any social context. As we noted above,symbolically fluent systems undoubtedly use these abilities for internal reasoning as well as externalbehaviours. What we ask here is how this ability to use symbols—introspective or otherwise—can comeabout in the first place . Can the ability to use symbols emerge entirely within an individual?Many researchers have answered this question in the positive. Some have gone as far as to positthat “symbol manipulation” is an innate cognitive capacity that is a prerequisite for acquiring languageand other human abilities. For example, Fodor argued that humans must have an innate (not learned)internal language of thought that possesses the characteristics of classical symbolic methods (Fodor,1975). Similar arguments were historically advanced by Chomsky about the need for a UniversalGrammar in order to learn language (Chomsky, 1995), although this view has been extensivelycritiqued (e.g. Dąbrowska, 2015) and Chomsky and colleagues have more recently suggested thatrecursion alone might suffice (Hauser et al., 2002). It is unclear what AI can learn from disputednotions of innateness—what should we build in and why? The onus is on us researchers to describeand demonstrate how any mechanisms we propose lead to the behavioral characteristics highlightedhere.We can, however, entertain an “internal” origin story of symbolic fluency without succumbing to the problems of innate machinery. Consider the case where a system still must reconcile perspectives,but not between itself and another. Rather, it must reconcile the perspectives of its past and currentself. Therefore, reconciliation and coordination of internal perspectives might provide similar sorts ofcognitive pressures to those that exist in social situations. Indeed, Karmiloff-Smith (1994) positeda process of representational redescription in cognitive development that could involve a kind ofinternal reconciliation in order to obtain a more objective representation. Thus, it is conceivable thatan isolated agent could eventually discover self-symbolic thinking, without social pressures to drive it.Of course, this origin story of symbolic thought comes with its own difficulties. For example, a systemmust have an impeccable memory in order to store the perspectives it must eventually coordinate,and it must have reasons for reconciling them that induce the same kinds of pressures as those thatexist in socio-cultural settings.If symbolic fluency can be a consequence of purely internal processes or events—whether mech-anisms per se , or forces like those described in the previous paragraph—then we’d expect to seeevidence of this in nature. We might expect, for example, that humans exhibit symbolic fluency froma very young age. And if this ability were evolved, they might also exhibit symbolic fluency (in theabsence of social pressures) with great ease. We believe the evidence for this is scarce, however.While humans do show fascinating behavioural characteristics from a relatively early age, such asthe ability to learn and generalize abstract patterns from only positive examples (Marcus et al.,1999), these abilities do not require a classical symbol manipulation mechanism. Instead, they can bedemonstrated in simple neural network models given appropriate learning experience (Geiger et al.,2020). Furthermore, learning formal domains like mathematics is rather difficult for humans: evenby the end of a proof-based high school geometry class most students do not achieve formal deductiveunderstanding of the concepts taught unless (or until) they become undergraduate mathematicsmajors (Burger and Shaughnessy, 1986). Similar patterns have been observed in linguistics. Gleitmanand Gleitman (1970) showed that only graduate students (compared to undergraduates or those witha high-school degree) exhibited a strong and consistent ability to understand three word compoundnouns, despite compound nouns being a relatively basic construction that should be easily parsableaccording to the syntactic rules of language. If social engagement is a secondary consideration forthe development of symbolic fluency, why would our ability to understand formal symbolic domainsemerge so much later than our ability to engage socially? We suggest that this provides some evidencefor social pressures as primary, although it is by no means conclusive.The above evidence is inconclusive in part because it is challenging to accurately measure “internal”symbolic fluency, especially in an experiment that is not confounded with social engagement. It ischallenging to design a test that can only be passed if one were to have the capacity to use symbols,because alternative solutions are always possible. For example, although symbolic thinking was surelyessential to humanity achieving artificial flight, evolution was able to discover flight as well, and wewould not want to attribute symbolic thinking to evolution. So while we cannot rule out the possibilityof self-symbolic thinking arising without a social context, it would be very difficult to ascertain.Indeed, the difficulty of measuring internally-developed use of symbols is itself an argumentfor exploring social forces as a tool for teaching AI to exhibit symbolic behaviour. The social forceshighlighted by Tomasello and others provide directions for training systems to engage with structuresof conventional understanding (Tomasello, 2019). It seems necessary for a system to develop symbolicbehaviour in order to fully engage in these social situations. By contrast, for the same reasons that itis difficult to assess an internally-originating use of symbols, it is correspondingly difficult to imaginesituations that would ensure the emergence of an internally-originating use of symbols rather thansome alternative solution. Thus, we suggest that AI research should focus more on interactive socialsituations, especially interacting with humans, because this will provide more direct pressure towarddeveloping symbolic behaviour.
5. Reconciling Our View With The Current Debate
Much of the current and past debate regarding symbolic AI and connectionism revolves around theparticularities and properties of certain ways of computing. For example, strengths and weaknessesof neural networks and GOFAI-like methods are contrasted and compared, and a reconciliation of thetwo is presented as a remedy to AI’s current woes (Garcez and Lamb, 2020; Garnelo and Shanahan,2019; Lake et al., 2017; Marcus, 2018, 2020). Our argument, on the other hand, takes a decidedlybehavioural perspective, with no explicit mention or prescription of the functions or mechanismsrequired for symbolic thought to emerge. This approach has a number of advantages (c.f. McClellandet al., 2010). First, it allows us to explicitly characterize what we expect from a system that usessymbols. Notably, we go beyond base mechanistic capacities (such as “variable binding”), which donot engage with what symbols are in the most general sense, but rather focus on what we humanstend to do with symbols in the particular setting of digital computers or formal algebras. Second, itleads us to the important conclusion that symbolic behaviour is a graded capacity that is not suddenlyenabled if a system comes equipped with a certain type of computational mechanism. Third, it betteraligns with symbolic behaviour as observed in nature, where it has a developmental ontogeny and isgraded based on experience, maturation, and formal education.The evidence cited in arguments for GOFAI-like mechanisms in humans is often behavioral. Forexample, humans show a capacity for relatively systematic or compositional generalization. Thiscapacity is often assumed to rely on particular GOFAI-like compositional mechanisms (e.g. Fodor andPylyshyn, 1988), which have these properties by definition. Our views differ from that perspective inseveral crucial ways (see also Hill et al., 2019; Potts, 2019). First, under our interpretation symbol-usedoes not entail systematicity since it emphasizes the graded, flawed nature of behaviour, rather thansomething lost in the translation of some “purer”, internal formal algebra that has these properties bydefinition. Indeed, even adult humans are often far from perfectly systematic, achieving compositionalgeneralization performance of only 70-90% even within relatively simple domains (Lake et al., 2019). Moreover, symbolic frameworks need not be strictly compositional, even if some, like formal logic,are compositional by design. Natural language, for example, is a symbolic system that isn’t strictlycompositional (Fodor, 2001; Szabó, 2020), and it’s trivial to invent a symbolic system that is notcompositional. The view we’ve presented embraces this lesson, and does not conflate symbols orsymbolic frameworks with possible properties those frameworks can have. This view has an importantconsequence: if symbolic behaviour is not inherently systematic, then mechanisms that guaranteesystematicity (through, for example, built-in compositional computations) may be neither necessarynor sufficient for achieving the type of symbol-use we advocate for here. Rather, systematic behaviormay be a graded competency afforded by environmental and educational factors (Hill et al., 2019;McClelland et al., 2010, 2020).Many perspectives on symbolic AI implicitly define symbols differently than we do in this work,often synonymizing them with discrete tokens or even localist representations (Garcez and Lamb,2020, e.g.). That view meshes well with the types of entities we operate with using digital computers,and has allowed us to leverage the algorithms and mechanisms developed for digital computerswhen designing AI. But what is discrete about air vibrations, or the fluctuating intensities of photonsreflected off a cloth waving in the wind? Humans undoubtedly have an ability to interpret what we’dtraditionally not call discrete entities as symbolic, even if they also interpret them as discrete units with a broader symbolic framework (the important distinction here is that a particular entity does nothave to itself be discrete for it to be a member of a discrete set, much like how a continuous vector This imperfection is often elided in claims about differences between humans and models, as noted by Lampinen(2020): “When humans make a compositionally-valid interpretation 80% or 95% of the time, it is presented as evidence oftheir compositional symbolic skill, yet when a deep model achieves 98.4% accuracy on much more difficult [mathematicalreasoning] problems, it is depicted as a failure of the model class to exhibit compositional symbolic reasoning.” can be a member of a discrete set of basis vectors). Why would we not expect the same for the AI wecreate? If we are to appreciate symbolic thought in its most potent, general form, then we need toembrace the idea that symbols come in many shapes and forms. Indeed, according to the definitionpresented here, any substrate can be symbolic, and a fluent symbolic thinker should appreciate andengage with this fact.Some recent work has attempted to explain how symbols can come to be within more classicalframeworks, for example by representing symbols through mechanisms like combinatoric logic(Piantadosi, 2020), or deriving symbolic abstractions through probabilistic program induction (Elliset al., 2020). While we find these directions intriguing, we are not optimistic that they will ultimatelyprovide a path to the symbolic behavior we have outlined. These systems are sometimes capable ofconstructing new abstractions, though not in a manner that demonstrates a command over convention.It is less clear whether they can achieve other attributes of symbolic behaviour that we highlightedabove. In particular, it is not easy for these systems to have malleable symbols in the way that we havedescribed, as the definition of a symbol is still generally expressed as a single program or formula.That is, it may be difficult for these systems to accommodate the many subtle constraints that weargue influence human interpretation of e.g. a word in context. Furthermore, it is still unclear howthese systems could demonstrate their understanding of their reasoning as meaningful, and as wesuggested above, it is the meaning behind reasoning that solves the frame problem. Thus, thesesystems, like the symbolic mechanisms of GOFAI, are prone to combinatorial explosions of possibleinferences. For example, even though Ellis et al. (2020) use deep learning to guide program search,their approach is still restricted to relatively simple domains with tightly constrained underlyingstructure. It is thus still unclear whether these approaches are a pathway toward human-like symbolicfluency, especially in real-world settings. However, as we noted above, our arguments are relativelyagnostic to the particular mechanisms underlying behavior. Our arguments about the power of socialinteractions for encouraging symbolic behavior should apply to these mechanisms as well. Thus,we suggest that those interested in such models should consider how they could accommodate therichness of human social interaction.
6. Summary & Conclusion
A symbol consists of some substrate that is imbued with meaning by matter of convention. Crucially,a substrate is only a symbol to those who demonstrably: (i) understand the established conventionalmeaning, and (ii) understand that meaning is what it is because of convention.To determine whether a system understands something as a symbol, we require evidence thatit understands that it is actively participating in an infrastructure of meaning by convention. Inthe biological world, this evidence presents as knowledge that some meaning is a matter of socialnorm, and as a capacity to establish new conventions and update old ones. To this end, we definesome behavioural characteristics that demonstrate symbolic fluency: symbolic behaviour is receptive,constructive, embedded, malleable, separable, meaningful, and graded.We can imagine two possible origins of symbolic behaviour that are not mutually exclusive:symbolic behaviour as an emergent consequence of social engagement, and symbolic behaviour as aconsequence of internal forces. Although we do not hope to fully adjudicate which course led to thehuman development of symbolic behaviour (and some evidence points to a co-evolution; Deacon,1998), we have argued that considering the social setting might be the most promising pathwaytoward developing AI capable using symbols in a human-like way.The views we’ve presented here seem to run counter to the past and current AI zeitgeist surroundingsymbols, which emphasizes rigid mechanisms and particular representational forms. Nevertheless, it would be difficult to create a symbolically fluent AI without carefully characterizing the behavioursthat result from the use of symbols. We need AI that is driven by a deep understanding of symbols asmembers of an infrastructure of meaning by convention, which manifests as behaviour that is nuanced,flexible, and subject to various shaping forces. We therefore prescribe that AI practitioners focuson instilling symbolic behavior in AI if they hope to build something that engages as profoundly ashumans do with issues of ethics, reasoning, and even beauty.
References
J. Abramson, A. Ahuja, A. Brussee, F. Carnevale, M. Cassin, S. Clark, A. Dudzik, P. Georgiev, A. Guy,T. Harley, et al. Imitating interactive intelligence. arXiv preprint arXiv:2012.05672 , 2020.I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert,G. Powell, R. Ribas, et al. Solving rubik’s cube with a robot hand. arXiv preprint arXiv:1910.07113 ,2019.C. Allen, I. Smit, and W. Wallach. Artificial morality: Top-down, bottom-up, and hybrid approaches.
Ethics and information technology , 7(3):149–155, 2005.I. Asimov.
I, robot . 1950.M. A. Boden. Gofai. In
The Cambridge handbook of artificial intelligence . Cambridge University Press,2014.T. Bolukbasi, K.-W. Chang, J. Y. Zou, V. Saligrama, and A. T. Kalai. Man is to computer programmer aswoman is to homemaker? debiasing word embeddings.
Advances in neural information processingsystems , 29:4349–4357, 2016.S. Bridgers, J. Jara-Ettinger, and H. Gweon. Young children consider the expected utility of others’learning to decide what to teach.
Nature Human Behaviour , 4(2):144–152, 2020.R. Brown.
A first language: The early stages.
Harvard U. Press, 1973.T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam,G. Sastry, A. Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 ,2020.M. Brundage. Limitations and risks of machine ethics.
Journal of Experimental & Theoretical ArtificialIntelligence , 26(3):355–372, 2014.W. F. Burger and J. M. Shaughnessy. Characterizing the van hiele levels of development in geometry.
Journal for research in mathematics education , pages 31–48, 1986.A. Caliskan, J. J. Bryson, and A. Narayanan. Semantics derived automatically from language corporacontain human-like biases.
Science , 356(6334):183–186, 2017.N. Chomsky.
The minimalist program . MIT press, 1995.H. H. Clark.
Using language . Cambridge university press, 1996.H. H. Clark and D. Wilkes-Gibbs. Referring as a collaborative process.
Cognition , 22(1):1–39, 1986.P. Cohen. The discovery of forcing.
The Rocky Mountain journal of mathematics , 32(4):1071–1100,2002.
E. Dąbrowska. What exactly is universal grammar, and has anyone seen it?
Frontiers in psychology , 6:852, 2015.I. Dautriche, L. Goupil, K. Smith, and H. Rabagliati. Knowing how you know: Toddlers re-evaluatewords learnt from an unreliable speaker, Nov 2020. URL psyarxiv.com/ruwdk .T. W. Deacon.
The symbolic species: The co-evolution of language and the brain . Number 202. WWNorton & Company, 1998.D. C. Dennett. Cognitive wheels: The frame problem of ai. 2006.G. Diesendruck. The principles of conventionality and contrast in word learning: an empiricalexamination.
Developmental Psychology , 41(3):451, 2005.H. Dreyfus. What computers can’t do: The limits of artificial intelligence. 1972.H. L. Dreyfus. Alchemy and artificial intelligence. Technical report, 1965.K. Ellis, C. Wong, M. Nye, M. Sable-Meyer, L. Cary, L. Morales, L. Hewitt, A. Solar-Lezama, andJ. B. Tenenbaum. Dreamcoder: Growing generalizable, interpretable knowledge with wake-sleepbayesian program learning. arXiv preprint arXiv:2006.08381 , 2020.D. Everett, B. Berlin, M. Gonalves, P. Kay, S. Levinson, A. Pawley, A. Surralls, M. Tomasello,A. Wierzbicka, and D. Everett. Cultural constraints on grammar and cognition in pirahã: An-other look at the design features of human language.
Current anthropology , 46(4):621–646,2005.R. P. Feynman and P. D. Sackett. “surely you’re joking mr. feynman!”adventures of a curious character.
American Journal of Physics , 53(12):1214–1216, 1985.J. Fodor. Language, thought and compositionality.
Royal Institute of Philosophy Supplements , 48:227–242, 2001.J. A. Fodor.
The language of thought , volume 5. Harvard university press, 1975.J. A. Fodor and Z. W. Pylyshyn. Connectionism and cognitive architecture: A critical analysis.
Cognition ,28(1-2):3–71, 1988.M. C. Frank and N. D. Goodman. Predicting pragmatic reasoning in language games.
Science , 336(6084):998–998, 2012.A. d. Garcez and L. C. Lamb. Neurosymbolic ai: The 3rd wave. arXiv preprint arXiv:2012.05876 ,2020.M. Garnelo and M. Shanahan. Reconciling deep learning with symbolic artificial intelligence: repre-senting objects and relations.
Current Opinion in Behavioral Sciences , 29:17–23, 2019.A. Geiger, A. C. M. C. Frank, and C. Potts. Relational reasoning and generalization using non-symbolicneural networks. arXiv preprint arXiv:2006.07968 , 2020.L. R. Gleitman and H. Gleitman. Phrase and paraphrase: Some innovative uses of language. 1970.S. Goldin-Meadow. The role of gesture in communication and thinking.
Trends in cognitive sciences , 3(11):419–429, 1999.
S. J. Gould and R. C. Lewontin. The spandrels of san marco and the panglossian paradigm: a critiqueof the adaptationist programme.
Proceedings of the royal society of London. Series B. BiologicalSciences , 205(1161):581–598, 1979.M. S. Graziano and S. Kastner. Human consciousness and its relationship to social neuroscience: anovel hypothesis.
Cognitive neuroscience , 2(2):98–113, 2011.H. Gweon. The role of communication in acquisition, curation, and transmission of culture.
Behavioraland Brain Sciences , 43, 2020.S. Harnad. The symbol grounding problem.
Physica D: Nonlinear Phenomena , 42(1-3):335–346, 1990.C. Hartshorne, P. Weiss, A. W. Burks, et al.
Collected Papers of Charles Sanders Peirce , volume 1.Harvard University Press Cambridge, 1931.M. D. Hauser, N. Chomsky, and W. T. Fitch. The faculty of language: what is it, who has it, and howdid it evolve? science , 298(5598):1569–1579, 2002.R. X. Hawkins, M. Franke, K. Smith, and N. D. Goodman. Emerging abstractions: Lexical conventionsare shaped by communicative context. In
CogSci , 2018.K. M. Hermann, F. Hill, S. Green, F. Wang, R. Faulkner, H. Soyer, D. Szepesvari, W. M. Czarnecki,M. Jaderberg, D. Teplyashin, et al. Grounded language learning in a simulated 3d world. arXivpreprint arXiv:1706.06551 , 2017.F. Hill, A. Lampinen, R. Schneider, S. Clark, M. Botvinick, J. L. McClelland, and A. Santoro. Environ-mental drivers of systematicity and generalization in a situated agent. In
International Conferenceon Learning Representations , 2019.F. Hill, O. Tieleman, T. von Glehn, N. Wong, H. Merzic, and S. Clark. Grounded language learningfast and slow. arXiv preprint arXiv:2009.01719 , 2020.P. N. Johnson-Laird. Deductive reasoning.
Annual review of psychology , 50(1):109–135, 1999.P. N. JOHNSON-LAIRD, P. Legrenzi, and M. S. Legrenzi. Reasoning and a sense of reality.
Britishjournal of Psychology , 63(3):395–400, 1972.A. Karmiloff-Smith. Précis of beyond modularity: A developmental perspective on cognitive science.
Behavioral and brain sciences , 17(4):693–707, 1994.S. Kirby. Learning, bottlenecks and the evolution of recursive syntax. 2002.S. M. Kosslyn and G. Hatfield. Representation without symbol systems.
Social Research , pages1019–1045, 1984.B. Lake and M. Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In
International Conference on Machine Learning , pages 2873–2882.PMLR, 2018.B. M. Lake, T. D. Ullman, J. B. Tenenbaum, and S. J. Gershman. Building machines that learn andthink like people.
Behavioral and brain sciences , 40, 2017.B. M. Lake, T. Linzen, and M. Baroni. Human few-shot learning of compositional instructions. In
Proceedings of the 41st Annual Conference of the Cognitive Science Society , 2019.
A. K. Lampinen.
A Computational Framework for Learning and Transforming Task Representations .PhD thesis, Stanford University, 2020.M. Lange. Depth and explanation in mathematics.
Philosophia Mathematica , 23(2):196–214, 2015.P. Lasersohn. Contextualism and compositionality.
Linguistics and Philosophy , 35(2):171–189, 2012.A. Lazaridou and M. Baroni. Emergent multi-agent communication in the deep learning era. arXivpreprint arXiv:2006.02419 , 2020.Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. nature , 521(7553):436–444, 2015.S. Mac Lane. Concepts and categories in perspective.
Duren P, A century of mathematics in AmericaPart , 3:353–365, 1988.S. Mac Lane.
Mathematics form and function . Springer Science & Business Media, 2012.G. Marcus. Deep learning: A critical appraisal. arXiv preprint arXiv:1801.00631 , 2018.G. Marcus. The next decade in ai: four steps towards robust artificial intelligence. arXiv preprintarXiv:2002.06177 , 2020.G. F. Marcus, S. Vijayan, S. B. Rao, and P. M. Vishton. Rule learning by seven-month-old infants.
Science , 283(5398):77–80, 1999.E. Marder. Living science: Theoretical musings.
Elife , 9:e60703, 2020.J. L. McClelland, M. M. Botvinick, D. C. Noelle, D. C. Plaut, T. T. Rogers, M. S. Seidenberg, and L. B.Smith. Letting structure emerge: connectionist and dynamical systems approaches to cognition.
Trends in cognitive sciences , 14(8):348–356, 2010.J. L. McClelland, K. Mickey, S. Hansen, A. Yuan, and Q. Lu. A parallel-distributed processing approachto mathematical cognition.
Manuscript, Stanford University , 2016.J. L. McClelland, F. Hill, M. Rudolph, J. Baldridge, and H. Schütze. Placing language in an integratedunderstanding system: Next steps toward human-level performance in neural language models.
Proceedings of the National Academy of Sciences , 117(42):25966–25974, 2020.T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of word representations in vectorspace. arXiv preprint arXiv:1301.3781 , 2013.V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller,A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning. nature , 518(7540):529–533, 2015.A. Newell. Physical symbol systems.
Cognitive science , 4(2):135–183, 1980.A. Newell and H. A. Simon. Computer science as empirical inquiry: Symbols and search. In
ACMTuring award lectures , page 1975. 1976.S. T. Piantadosi. The computational origin of representation.
Minds and Machines , pages 1–58, 2020.G. Polya.
Mathematics and Plausible Reasoning: Patterns of plausible inference , volume 2. PrincetonUniversity Press, 1990.D. A. Pomerleau. Alvinn: An autonomous land vehicle in a neural network. In
Advances in neuralinformation processing systems , pages 305–313, 1989.
C. Potts. A case for deep learning in semantics: Response to pater.
Language , 95(1):e115–e124, 2019.A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin,J. Clark, et al. Learning transferable visual models from natural language supervision. preprint ,2021.D. E. Rumelhart and J. L. McClelland. On learning the past tenses of english verbs. In J. L. McClelland,D. E. Rumelhart, and the PDP Research Group, editors,
Parallel Distributed Processing: Explorationsin the Microstructure of Cognition , pages 216–271. MIT Press Cambridge, MA, 1986.M. F. Schmidt, L. P. Butler, J. Heinz, and M. Tomasello. Young children see a single action and infer asocial norm: Promiscuous normativity in 3-year-olds.
Psychological Science , 27(10):1360–1370,2016.J. R. Searle et al. Minds, brains, and programs.
Behavioural & Brain Sciences , pages 417–457, 1980.C. E. Shannon. A mathematical theory of communication.
The Bell system technical journal , 27(3):379–423, 1948.D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser,I. Antonoglou, V. Panneershelvam, M. Lanctot, et al. Mastering the game of go with deep neuralnetworks and tree search. nature , 529(7587):484–489, 2016.D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai,A. Bolton, et al. Mastering the game of go without human knowledge. nature , 550(7676):354–359,2017.B. C. Smith.
The promise of artificial intelligence: reckoning and judgment . Mit Press, 2019.D. Sperber, F. Cara, and V. Girotto. Relevance theory explains the selection task.
Cognition , 57(1):31–95, 1995.Z. G. Szabó. Compositionality. In E. N. Zalta, editor,
The Stanford Encyclopedia of Philosophy .Metaphysics Research Lab, Stanford University, fall 2020 edition, 2020.C. Tennie, J. Call, and M. Tomasello. Ratcheting up the ratchet: on the evolution of cumulativeculture.
Philosophical Transactions of the Royal Society B: Biological Sciences , 364(1528):2405–2415,2009.M. Tomasello.
Becoming human: A theory of ontogeny . Belknap Press, 2019.D. S. Touretzky and D. Pomerleau. Reconstructing physical symbol systems.
Cognitive Science , 18(2):345–353, 1994.N. Vélez and H. Gweon. Learning from other minds: An optimistic critique of reinforcement learningmodels of social learning. 2020.A. H. Vera and H. A. Simon. Situated action: A symbolic interpretation.
Cognitive science , 17(1):7–48,1993.L. Wei.
The bilingualism reader . Routledge, 2020.L. L. Woods. The federal home loan bank board, redlining, and the national proliferation of raciallending discrimination, 1921–1950.
Journal of Urban History , 38(6):1036–1059, 2012.
Acknowledgements