Bayesian mechanics of perceptual inference and motor control in the brain
aa r X i v : . [ q - b i o . N C ] A ug Bayesian mechanics of perceptual inference andmotor control in the brain
Chang Sub Kim
Department of Physics, Chonnam National University, Gwangju 61186, Republic ofKoreaE-mail: [email protected]
Abstract.
The free energy principle in the neurosciences stipulates that all viableagents embody and minimize informational free energy to fit their environmental niche.We implement free energy minimization in a more physically principled manner byeffectively casting it to Bayesian control dynamics. Specifically, we build a continuous-state formulation that prescribes the brain’s computation for actively inferring thecauses of sensory inputs, based on the principle of least action. For this, we considerthat the motor signal arises from the residual errors in the cognitive expectationabout the nonstationary sensory inputs at the proprioceptive level. Consequently, wederive the effective Hamilton equations that determine the optimal trajectories in theperceptual phase space, which minimize the sensory uncertainty. We also demonstratetheir utility using a simple model with biological relevance to sensorimotor regulationprocesses such as motor reflex arcs or saccadic eye movement.
Keywords free energy principle, active inference, recognition dynamics, continuousstate-space models, limit cycles
25 August 2020 ayesian mechanics of perceptual inference and motor control in the brain
1. Introduction
The free energy principle (FEP) in the field of neurosciences rationalizes that all viableorganisms cognize and behave in the natural world by calling forth the probabilisticmodels embodied in their neural system — the brain — in a manner that ensurestheir adaptive fitness (Friston 2010a). The neurobiological mechanism that endows anorganism’s brain — the neural observer — with this ability is theoretically framed intoan inequality that weighs two information-theoretical measures, namely, the surprisaland the informational free energy (IFE) (Buckley and Kim et al. 2017). The surprisalprovides a measure of the atypicality of an environmental niche, and the IFE is theupper bound of the surprisal. The inequality enables a cognitive agent to indirectlyminimize the IFE as a variational objective function instead of the intractable surprisal.The minimization corresponds to inferring the external causes of afferent sensory data,encoded as a probability density at the sensory interface, e.g., sensory organs. Anorganism’s brain neurophysically performs the Bayesian computation of minimizing theinduced IFE. This is termed as recognition dynamics (RD), which emulates — under theLaplace approximation (Buckley and Kim et al. 2017) — the predictive coding schemeof message processing or recognition (Rao and Ballard 1999).The neurobiological mechanisms of the abductive inference of the physical brain arenot yet understood; therefore, researchers mostly rely on information-theoretic concepts(Elfwing 2016; Ramstead et al. 2019; Kuzma 2019; Shimazaki 2019; Kiefer 2020, Sanderset al. 2020). The FEP facilitates dynamic causal models in the brain’s generalized-statespace (Friston 2008b; Friston et al. 2010), which pose a mixed discrete-continuousBayesian filtering (Jazwinski 1970); Balaji and Friston 2011). In the present work,we consider that the brain confronts the continuous influx of stochastic sensations andconducts the Bayesian inversion of inferring external causes in the continuous staterepresentations. Biological phenomena are naturally continuous spatiotemporal events;accordingly, we suggest that the continuous-state approaches that are used to describecognition and behavior are better suited than discrete-state descriptions for studyingthe perceptual computation in the brain. Despite its explanatory power as a unifiedbiological principle (Friston 2013; Isomura et al. 2015; Bogacz2017; Colombo andWright 2018), the conventional FEP assumes several extra-theoretical facets that requireevaluation. The specific issues that have drawn our attention are i) the non-Newtonianextension of dynamical states in the generalized state space and ii) the heuristic gradient-descent implementation of minimization on the instantaneous IFE landscape.Recently, we carefully evaluated the FEP while clarifying the technical assumptionsthat underlie the continuous state-space formulation of the FEP Buckley and Kim etal. 2017). A full account of the discrete-state formulation, which is complementary toour formulation, can be found in (Da Costa 2020). In a subsequent study (Kim 2018),we clarified some technical facets in its conventional formalism, which necessitated areformulation of the working hypotheses described above. In doing so, we postulatedthat “surprisal” plays the role of a Lagrangian in theoretical mechanics (Landau and ayesian mechanics of perceptual inference and motor control in the brain ż dt t´ ln p p ϕ qu ď ż dt F r q p ϑ q , p p ϕ, ϑ qs , (1)where ϕ and ϑ collectively denote the sensory inputs and their environmental causes,respectively. The integrand on the left-hand side (LHS) of the preceding equation ´ ln p p ϕ q is the aforementioned surprisal , which measures the “self-information”contained in the sensory density p p ϕ q (Cover 2006), and F on the right-hand side (RHS)is the variational IFE defined as F r q p ϑ q , p p ϕ, ϑ qs ” ż dϑq p ϑ q ln q p ϑ q p p ϕ, ϑ q , (2)which encapsulates the recognition (R-) density q p ϑ q and the generative (G-) density p p ϑ, ϕ q (see Buckley and Kim et al. 2017). While the G-density is the brain’s priorbelief about the environmental dynamics and sensory generation, the R-density is thebrain’s current guess of the environmental cause of the sensory perturbation. Theyinduce variational IFE when receptors at the brain-environment interface are excited bysensory perturbations, and the induced variational IFE is encoded by the brain variablessuch as the action potential, synaptic plasticity, or brain waves.According to Eq. (1), the FEP articulates that the brain minimizes the upperbound of the sensory uncertainty, which is a long-term averaged surprisal and notan instant one. We identify this bound as an informational action (IA) within thescope of the mechanical principle of least action (Landau and Lifshitz 1976). Then,by complying with the revised FEP, we derive the RD without invoking gradient-descent schemes. The RD neurophysically performs the computation for minimizingthe IA when the neural observer encounters continuous streams of sensory data. Theadvantage of our formulation is that the brain and the environmental states are specifiedby using only bare continuous variables and their first-order derivatives (velocities orequivalent momenta), thereby absolving the need for the aforementioned non-Newtonianassumptions. The momentum variables represent the prediction errors, which quantifythe discrepancy between an observed input and its top-down belief of a cognitive agent inthe predictive coding language (Huang and Rao 2011; de Gardelle et al. 2013; Kozunovet al. 2020). In the present study, we continue our effort to make the FEP a morephysically principled formalism.The goal of this work is to extend our previous study to include the agent’s motorcontrol, which acts on the environment to alter sensory inputs. ; Previously, by utilizing ; In this work, we use the term control , instead of the frequently used term “action” to mean themotion of a living agent’s effectors (muscles) acting on the environment. This is done to avoid anyconfusion with the term action appearing in the nomenclature of “the principle of least action”. ayesian mechanics of perceptual inference and motor control in the brain active inference derived from the FEP (Fristonet al. 2009; Friston et al. 2010c; Friston et al. 2011a), which proposes that organismscan minimize the IFE further by altering sensory observations when the outcome ofperceptual inference alone is not in accordance with an internal representation of theenvironment (Buckley and Kim et al. 2017). It is suggested that living systems areendowed with the ability to adjust their sensations via proprioceptive feedback, whichis attributed to an inherited trait of all motile animals embodied in the reflex pathways(Tuthill and Azim 2018). In this respect, motor control is considered as an inferenceof the causes of motor signals that are encoded as prediction errors at proprioceptors,and motor inference is realized at the spinal level by classical reflex arcs (Friston 2011b;Adams et al. 2013). Our formulation evinces that the ensuing RD encompasses motorsignals as a time-dependent source, thereby describing active inference as an extensionof the optimal control theory embedding Bayesian logic (Friston 2011b; Baltieri andBuckley 2019). Another notable feature of our work is that it considers the correlationbetween the causal states and the sensory-data measurement, which results in significantmodulation in the perceptual and motor-control outcomes.Technically, a functional variation of the IA yields the RD that computes theBayesian inversion, which is given as a set of coupled differential equations for thebrain variables and their conjugate momenta. The brain variables are ascribed to thebrain’s representation of the environmental states, and their conjugate momenta are thecombined prediction errors of the sensory data and the rate of the state representations.The neural computation of active inference corresponds to integrating the RD and issubject to nonautonomous motor signals. Its solution results in optimal trajectoriesin the perceptual state space, which minimizes the accumulation of the IFE overcontinuous time, i.e., the IA. In our formulation, both descending predictions andascending prediction errors constitute the dynamical states governed by the closed-loopRD in the functional hierarchy of the brain’s sensorimotor system. This feature is incontrast to the conventional implementation of the FEP, which considers the backwardprediction — belief propagation — as neural dynamics and the forward prediction erroras an instant message passing without any causal dynamics (Friston 2010a; Buckley andKim et al. 2017).The remainder of this paper is organized as follows. In Sect. 2, we provide a compactoverview of our recent efforts in finessing the technical subtleties of the standard FEPtheory. In Sect. 3, we formulate the Bayesian mechanics of the sensorimotor cycle byutilizing the principle of least action. Then, in Sect. 4, we present a parsimonious modelfor a concrete manifestation of our formulation. Finally, in Sect. 5, we provide theconcluding remarks. ayesian mechanics of perceptual inference and motor control in the brain
2. Outline of technical developments
Below, we recapitulate the significant developments from our continuous-stateformulation, while discussing the technical features that distinguish our theory fromthe conventional state-space implementation of the FEP.
The Bayesian filtering formalism of the FEP adopts the concept of generalized motionof a physical object by defining its mechanical state beyond position and velocity(momentum). The introduction of this theoretical construct is argued to provide a moredetailed specification of the dynamical states (Friston 2008a; Friston 2008b; Friston etal. 2010b). The mechanical states are generalized by recursively taking higher-orderderivatives of the bare states. A point in hyperspace defined by the generalized states isinterpreted as an instantaneous trajectory. This notion provides an essential theoreticalbasis for ensuring an equilibrium solution of the RD in the conventional formulation ofthe FEP (Kim 2018), and it is commonly employed by researchers (Parr and Friston2018; Baltieri and Buckley 2019). However, the non-Newtonian construct has subtletiesin assigning physical observables to higher-order motions. Standard physics does notconsider jerk (third order) as a cause of the changing acceleration (second order), snap(fourth order) as a cause of the changing jerk, or the continued generalization. Instead,only the position (zeroth order) and velocity (first order) are adopted as the two requireddynamical observables (Landau and Lifshitz 1976). Our formulation avoids invoking thegeneralized states and follows the normative rules (Kim 2018); however, it still providesa natural way to determine the equilibrium solutions for the RD. Besides, it can alsohandle temporal correlation, which is another justification for the conventional use ofthe generalized states (see Sect. 2.3).
The conventional FEP employs the gradient descent method in the optimal theory tominimize the variational IFE, i.e., the upper bound on the surprisal. To incorporatethe time-varying feature of the sensory inputs, it further adopts an adroit procedure oftechnically distinguishing the path of a mode and the mode of a path in the generalizedstate space (Friston 2008b; Friston et al. 2010b). This extra-theoretical constructconsiders the nonequilibrium dynamics of the generalized brain states as drift-diffusionflows that locally conserve the probability density in the hyperspace of the generalizedstates. In the revised formulation, we replace the gradient descent scheme with thestandard mechanical formulation of the least action principle (Kim 2018). The resultingnovel RD entails optimal trajectories but no single fixed points in the canonical state(phase) space, which provides an estimate of the minimum sensory uncertainty, i.e.,the average surprisal over a finite temporal horizon. The phase space comprises thepositions (predictions) and momenta (prediction errors) of the brain’s representations ayesian mechanics of perceptual inference and motor control in the brain
The FEP requires the brain’s internal model of the G-density p p ϕ, ϑ q with the likelihood p p ϕ | ϑ q and prior p p ϑ q in a top-down manner. The likelihood density is determined bythe random fluctuation in the expected sensory-data generation, and the prior densityis determined by that in the believed environmental dynamics. The brain encounterscontinuous sensory signals on a fast timescale, often shorter than the correlationtime of the random processes (Friston 2008a); accordingly, in general, the noisesembrace a non-Markovian stochastic process with an intrinsic temporal correlation thatsurmounts the ideal white-noise stochasticity. It is argued that the noises are analytic(i.e., differentiable) to allow correlation between the distinct dynamical orders of thecontinuous states (Friston 2008b; Friston et al. 2010b). In practice, to furnish a closeddynamics for a finite number of variables, the recursive equations of motion for thecontinued generalized states need to be truncated at an arbitrary order by hand. TheWiener processes with a delta-correlated (white) noise are mathematically singular;they need to be smoothed to describe fast biophysical processes. However, withoutresorting to the generalized states, the non-Markovian processes with colored noisescan be considered within conventional stochastic theories (van Kampen:1981; Fox 1987;Risken 1989; Moon and Wettlaufer 2014). Besides, the rules of ordinary physics for acontinuous motion presume the position and its first-order derivative (velocity) to bestatistically uncorrelated observables at a microscopic level. The conventional FEP facilitates gradient descent minimization for the mechanisticimplementation of active inference, which makes the motor-control dynamics availablein the brain’s RD (Friston et al. 2009; Friston et al. 2010c; Friston et al. 2011a). Thegradient-descent scheme is mathematically expressed as a “ ´ ∇ a F Ñ ´ B F B ϕ dϕda , (3)where a denotes an agent’s motor variable, and F is the Laplace-encoded IFE by thebiophysical brain variables (see Buckley and Kim et al. 2017). An agent’s capabilityof subjecting sensory inputs to motor control is considered as a functional dependence ϕ “ ϕ p a q in the environmental generative processes (Friston et al. 2009). According toEq. (3), an agent performs the minimization by effectuating the sensory data dϕ { da and obtains the best result for motor inference when a “
0, where the condition B F {B ϕ “ B F {B ϕ produces terms proportional to the sensoryprediction errors, the fulfillment of motor inference is equivalent to suppressing theproprioceptive errors. Thus, the motor control tries to minimize the prediction errors, ayesian mechanics of perceptual inference and motor control in the brain a : Equation (3) evidently handles a as a dynamical state;however, the corresponding equation of motion governing its dynamics is not given inthe environmental processes. Instead, the mechanism of motor control that vicariouslyalters the sensory-data generation is presumed (Friston et al. 2009). In addition,motor variables are represented as the active states of the brain, e.g., motor-neuronactivities in the ventral horn of the spinal cord (Friston et al. 2010c); however, theyare treated differently from other hidden-state representations. Recall that the internalstate representations are expressed as generalized states, whereas the active states arenot. In the following, we formulate a semi-active inference problem, which does notexplicitly address motor planning in the RD but encompasses motor control as a time-dependent driving term. Our theory does not employ the theoretical construct of thegeneralized states and evades the gradient descent implementation of the active statedynamics. We also consider the correlation between the environmental dynamics andthe generative process of sensory inputs. We regard the environmental states and thesensory variables as fluctuating, coarse-grained hydrodynamic fields, not dynamicallyindependent microscopic states.
3. Closed-loop dynamics of perception and motion
The brain is not divided into sensory and motor systems. Instead, it is one inferencemachine that performs the closed-loop dynamics of perception and motor control. Here,we develop a framework of active inference within the scope of the least action principleby employing the Laplace-encoded IFE as an informational Lagrangian.The environmental states ϑ undergo deterministic or stochastic dynamics byobeying physical laws and principles. Here, we do not explicitly consider their equationsof motion because they are hidden from the brain’s perspective, i.e., the brain as aneural observer does not possess direct epistemic access. Similarly, the sensory data ϕ are physically generated by an externally hidden process at a sensory receptor, whichconstitutes the brain-environment interface. However, to emphasize the effect of anagent’s motor control a on sensory generation, we facilitate the generative process ofsensory data, as usual, by an instantaneous mapping ϕ “ h p ϑ, a q ` z gp , (4)where h p ϑ, a q is the linear or nonlinear map of the input generation, and z gp is thenoise involved. Note that an agent’s motor-control a is explicitly included in thegenerative map. However, the neural observer does not know how the sensory streamsare effectuated by the agent’s motion in the environment (Friston et al. 2010c).The FEP circumvents this epistemic difficulty by hypothesizing a formal homologybetween the external physical processes and the corresponding internal models foreseen ayesian mechanics of perceptual inference and motor control in the brain q p ϑ q to infer the external causes via variational Bayes.The R-density is the probabilistic representation of the environment, whose sufficientstatistics are assumed to be encoded by neurophysical brain variables, e.g., neuronalactivity or synaptic efficacy. When a fixed-form Gaussian density is taken for theR-density, which is called the Laplace approximation, only the first-order sufficientstatistic, i.e., the mean µ is needed to specify the IFE effectively (Buckley and Kim etal. 2017). The brain continually updates the R-density by using its internal dynamics,which is described here as a Langevin-type equation dµdt “ f p µ q ` w, (5)where f p µ q represents the brain’s belief about the external dynamics encoded by aneurophysical driving mechanism of the brain variables µ , and w is random noise. Thesensory perturbations at the receptors are predicted by the neural observer via theinstantaneous mapping ϕ p a q “ g p µ q ` z, (6)where the belief g p µ q is encoded by the internal variables, and z is the associated noise.Our sensory generative model provides a mechanism for sampling the sensory data ϕ byusing the brain’s active states a , which represent an external motor control embedded inEq. (4). Note that Eq. (4) describes the environmental processes that generate sensoryinputs ϕ , while its homolog ’Eq. (6)’ prescribes the brains’ prior belief about ϕ thatcan be altered by the active states a . The instantaneous state of the brain µ , whichis specified by Eq. (5), picks out a particular R-density q p ϑ q when the brain seeks thetrue posterior (the goal of perceptual inference). The motor control fulfills the priorexpectations by modifying the sensory generation via active-state effectuation at theproprioceptors.Through Laplace approximation (Buckley and Kim et al. 2017), the G-density p p ϕ, ϑ q is encoded in the brain as p “ p p ϕ, µ q , where the sensory stimuli ϕ are predictedby the neural observer µ via Eq. (6). Here, we argue that the physical sensory-recordingprocess is conditionally independent of the brain’s internal dynamics; however, the brainstates must be neurophysically involved in computing the sensory prediction. In otherwords, the sensory perturbation ϕ at the interface is a source for exciting the neuronalactivity µ . This observation renders the set of Eqs. (5) and (6) to be dynamicallycoupled, not conditionally independent. We incorporate this conditional dependenceinto our formulation by introducing a correlation between the noises w and z .For simplicity, we consider the stationary Gaussian processes for the bivariatevariable Z as a column vector Z ” ˜ wz ¸ , ayesian mechanics of perceptual inference and motor control in the brain w “ µ ´ f p µ q and z “ ϕ ´ g p µ q , and specify the Laplace-encoded G-density p p ϕ, µ q as p p ϕ, µ q “ a p π q | Σ | exp ˆ ´ Z T Σ ´ Z ˙ , (7)where Z T is the transpose of Z . The covariance matrix Σ for the above is given asΣ “ ˜ σ w φ p t q φ p t q σ z ¸ , where the stationary variances σ i ( i “ w, z ) and the transient covariance φ are defined,respectively, as σ w p q “ x w y , σ z p q “ x z y , and φ p t q “ x w p q z p t qy . With the prescribed internal model of the brain for the G-density, the Laplace-encodedIFE can be specified as F p ϕ, µ q “ ´ ln p p ϕ, µ q (for details, see Buckley and Kim et al.2017). Then, it follows that F p ϕ, µ ; t q “ m w p µ ´ f p µ qq ` m z p ϕ ´ g p µ qq (8) ´ ? m w m z ρ p µ ´ f p µ qq p ϕ ´ g p µ qq `
12 ln ` π p ´ ρ q σ w σ z ˘ , where ρ is the correlation function defined as a normalized covariance ρ ” φ ? σ w σ z . (9)We also introduce the notations m i p i “ w, z q as m i ” σ i p ´ ρ q , (10)which are precisions , scaled by the correlation, in the conventional FEP.Next, as we proposed in (Kim 2018), we identify F as an informational Lagrangian L in the scope of the principle of least action, and define L ” m w p µ ´ f qq ` m z p ϕ ´ g qq ´ ? m w m z ρ p µ ´ f q p ϕ ´ g q , (11)which is viewed as a function of µ and µ for the given sensory inputs ϕ p t q , i.e., L “ L p µ, µ ; ϕ q . Note that we dropped the last term in Eq. (8) when translating F into L because it can be expressed as a total time-derivative term that does not affectthe resulting equations of motion (Landau and Lifshitz 1976). Then, the theoreticalaction S that effectuates the variational objective functional under the revised FEP isset up as S r µ p t qs “ ż F p µ p t q , µ p t q ; ϕ q dt. (12)The Lagrange equation of motion, which determine the trajectory µ “ µ p t q for a giveninitial condition µ p q , is derived by minimizing the action δS ” µ and its conjugate momentum p , instead of the position µ and velocity µ . We have used ayesian mechanics of perceptual inference and motor control in the brain µ and µ , respectively. For this, we need to convert the Lagrangian L into the Hamiltonian H by performing a Legendre transformation H p µ, p q “ p µ ´ L p µ, µ q , where p is the canonical momentum conjugate to µ , which is calculated from L as p “ B L B µ “ m w p µ ´ f q ´ ? m w m z p ϕ ´ g q ρ. (13)After some manipulation, the functional form of H can be obtained explicitly as follows H p µ, p ; ϕ q “ T p µ, p ; ϕ q ` V p µ ; ϕ q , (14)where we have indicated its dependence on the sensory influx ϕ . In addition, the terms T and V on the RHS are defined as T p µ, p ; ϕ p t qq ” p m w ` ˆ ρ c m z m w ´ ϕ ´ g p µ q ¯ ` f p µ q ˙ p, (15) V p µ ; ϕ p t qq ” m z ` ´ ρ ˘ ´ ϕ ´ g p µ q ¯ . (16)Here, T and V represent the kinetic and potential energies , respectively, which definethe informational Hamiltonian of the brain. Similarly, m w and m z represent the neuralinertial masses , again as a metaphor. Unlike that in standard mechanics, the secondterm in the expression for kinetic energy is dependent on the linear momentum andposition.Next, we generate the Hamilton equations of motion, which are equivalent to theLagrange equation, by carrying out µ “ B H B p and p “ ´ B H B µ . As described below, the Hamilton equations are better suited for our purposes, sincethey specify the RD as coupled first-order differential equations of the brain state µ and its conjugate momentum p . In contrast, the Lagrange equation is a second-orderdifferential equation of the state variable (Landau and Lifshitz 1976). The results areas follows: µ “ m w p ` f p µ q ` α ∆ ϕ , (17) p “ ´ ˆ B f B µ ´ β B g B µ ˙ p ´ ` ´ γ ˘ B g B µ ∆ ϕ , (18)where the parameters α , β , and γ have been respectively defined for notationalconvenience as α ” ρ {? m z m w , β ” ρ a m z { m w , and γ ” ρ a m w { m z . (19)In Eqs. (17) and (18), we have also defined the notation ∆ ϕ as∆ ϕ p µ ; t q ” m z p ϕ p a q ´ g p µ qq . (20) ayesian mechanics of perceptual inference and motor control in the brain ϕ by an agent’s motorcontrol a and the top-down neural prediction g p µ q , weighted by the neural inertial mass m z . Below, we appraise the Bayesian mechanics prescribed by Eqs. (17) and (18) andnote some significant aspects:(i) The derived RD suggests that both the brain activities µ and their conjugatemomenta p are dynamic variables. The instantaneous values of µ and p correspondto a point in the brain’s perceptual state space, and the continuous solution over atemporal horizon forms an optimal trajectory that minimizes the theoretical action,which represents the sensory uncertainty.(ii) The canonical momentum p defined in Eq. (13) can be rewritten as p “ m w p µ ´ f q´ ρ a m w { m z ∆ ϕ . Accordingly, when the normalized correlation ρ is nonvanishing, themomentum quantifies the combined errors in predicting the changing states and thesensory stimuli. The prediction errors propagate through the brain by obeying thecoupled dynamics according to Eqs. (17) and (18).(iii) The terms involving the time-dependent ∆ ϕ in Eqs. (17) and (18) are identified asthe driving forces C i , i “ µ, p , C µ ” α ∆ ϕ , (21) C p ” ´ B V B µ “ ´ ` ´ γ ˘ B g B µ ∆ ϕ . (22)The sensory prediction error ∆ ϕ , defined in Eq. (20), quantifies the motor signalsengaging with the brain’s nervous control in integrating the RD.Equations (17) and (18) are the highlights of our formulation, which prescribes thebrain’s Bayesian mechanics of actively inferring the external causes of sensory inputsunder the revised FEP. Note that the motor variable a is not explicitly included in ourderived RD; instead, it implicitly induces nonautonomous sensory inputs ϕ p t q in themotor signal ∆ ϕ . The motor signal appears as a time-dependent driving term, whichmakes the RD more akin to the motor-control dynamics. The driving forces are primarilyinduced by the discrepancy between the sensory streams ϕ p t q and those predicted by thebrain. The neural observer continuously integrates the RD subject to a motor signalto perform the sensory-uncertainty minimization, thereby closing the perception andmotion control within a reflex arc. When we neglect the correlation ρ between thesensory perturbation and the neuronal response to it, we can recover the RD that wasreported in the previous publication (Kim 2018), which demonstrates the consistencyof our formulation. The full implication of our formulation in the hierarchical brain canbe straightforwardly made as done in (Kim 2018), which admits the bidirectional facetin information flow of descending predictions and ascending prediction errors (Markovand Kennedy 2013; Michalareas et al. 2016). ayesian mechanics of perceptual inference and motor control in the brain
4. Simple Bayesian-agent model: an implicit motor control
In this section, we numerically demonstrate the utility of our formulation using anagent-based model, which is based on an early publication (Buckley and Kim et al.2017). Unlike that in the previous study, the current model does not employ generalizedstates and their motions; instead, the RD is specified using only the position µ and itsconjugate momentum p for a given sensory input ϕ p t q . The environmental objectsinvoking an agent’s sensations can be either static or time-dependent, and in turn, thetime dependence can also be either stationary (not moving on average) or nonstationary.According to the framework of active inference, the inference of static propertiescorresponds to passive perception without motor control a . Meanwhile, the inference oftime-varying properties renders an agent’s active perception of proprioceptive sensationsby discharging motor signals ∆ ϕ via classic reflex arcs.In the present simulation, the external hidden state ϑ is a point property, e.g.,temperature or a salient visual feature, which varies with the field point x unless theenvironment is in global thermal equilibrium. As the simplest environmental map, wetake h p ϑ, a q “ ϑ p x p a qq and assume that the sensory influx at the corresponding receptoris given by ϕ “ ϑ ` z gp , (23)where z gp is the random fluctuation. The external property, e.g., temperature, isassumed to display a spatial profile as follows: ϑ p x q “ ϑ {p x ` q , where ϑ is the value at the field origin, and the desired environmental niche is situatedat x “ x d , where ϑ p x d q “ ϑ d . The biological agent that senses temperature is allowed tonavigate through a one-dimensional environment by exploiting the hidden property. Theagent initiates its motion from x p q , where the temperature does not accord with thedesired value. In this case, the agent must fulfill its allostasis at the cost of biochemicalenergy by exploiting the environment according to x p t q “ x p q ` ż t a p t q dt , (24)where a p t q is a motor variable, e.g., agent’s velocity. Accordingly, the nonstationarysensory data ϕ p t q are afferent at the receptor with respect to the noise z gp . Thetime dependence is caused by the agent’s own motion, i.e., ϕ p t q “ ϑ p x p a p t qqq , whichis assumed to be latent to the agent’s brain in the current model. With the prescribedsensorimotor control, the rate of averaged sensory data with respect to the noise isrelated to the control variable as follows: ϕ “ B ϕ p x qB x a. As discussed in Sect. 3, the neural observer does not know how the sensory inputsat the proprioceptor are affected by the agent’s motor reflex control. In the case ofsaccadic motor control (Friston et al. 2012), an agent may as well stand at a field point ayesian mechanics of perceptual inference and motor control in the brain t φ ( t ) Figure 1.
Influx of stochastic sensory data ϕ p t q generated by the environmentalprocess Eq. (23), where we have set ϑ “
20. The dashed curve represents the agent’sposition x p t q as a function of time, with its movement starting from x p q “
10. Thedotted curve represents the magnitude of the latent motor variable a p t q that controlsthe agent’s location. [All curves are in arbitrary units.] without changing its position but sampling the salient visual features of the environmentthrough a fast eye movement a p t q , which makes the visual input nonstationary, i.e., ϕ p t q “ ϑ p a p t qq .In Fig. 1, we depict the streams of sensory data at the agent’s receptor as a functionof time. For this simulation, the latent motor variable in Eq. (24) is taken as § a p t q “ a p q e t p ` e t q , which renders the agent’s position in the environment as x p t q “ x p q{p ` e t q with x p q “ ´ a p q . The figure shows that the agent, initially located at x p q “
10, sensesan undesirable stimulus ϑ p q . “ .
2; accordingly, it reacts by using motor control to findan acceptable ambient niche. After a period of ∆ t “
5, the agent finds itself at theorigin x “
0, where the environmental state is marked by the value ϑ . “ g p µ q “ µ, (25) f p µ q “ ´p µ ´ ϑ d q . (26)Note that the motor control a is not included in the generative model, and the desiredsensory data ϑ d , e.g., temperature, appears as the brain’s prior belief about the hidden § For simplicity, we assume that this is hardwired in the agent’s reflex pathway over evolutionary anddevelopmental time scales. ayesian mechanics of perceptual inference and motor control in the brain µ and its conjugate momentum p : µ “ ´ p µ ´ ϑ d q ` m w p ` α ∆ ϕ , (27) p “ p ` β q p ´ ` ´ γ ˘ ∆ ϕ . (28)The parameters α , β , and γ are proportional to the correlation ρ ; see Eq. (19). Hence,they become zero when the neural response to the sensory inputs is uncorrelated withthe neural dynamics, which is not the case in general. The time-dependent drivingterms appearing on the RHS of both equations, namely Eqs. (27) and (28), include thesensorimotor signal ∆ ϕ p µ ; ϕ p t qq given in Eq. (20). The motor variable a , which drivesthe nonstationary inputs ϕ p t q , is unknown to the neural observer in our implementation.In the following, for a compact mathematical description, we denote the brain’sperceptual state as a column vector:Ψ ” ˜ µp ¸ . The vector Ψ represents the brain’s current expectation µ and the associated predictionerror p with respect to the sensory causes, encoded by the neuronal activities performedwhen encountering a sensory influx. Therefore, in terms of the perceptual vector Ψ,Eqs. (27) and (27) are expressed as d Ψ dt ` R Ψ “ S , (29)where the relaxation matrix R is defined as R “ ˜ ` φm w ´ m w ´ m z ` φ m w ´p ` φm w q ¸ , (30)and the source vector S encompassing the sensory influx ϕ p t q is defined as S “ ˜ ϑ d ` φm w ϕ ´p m z ´ φ m z q ϕ ¸ . (31)Unless it is a pathological case, the steady-state (or equilibrium) solution ψ eq of Eq. (29)is uniquely obtained asΨ eq “ R ´ S ” ˜ µ eq p eq ¸ . (32)We find it informative to consider the general solution Ψ p t q of Eq. (29) with respect tothe fixed point ψ eq by setting ψ p t q ” Ψ p t q ´ Ψ eq . To this end, we seek time-dependent solutions for the shifted measure ψ p t q as follows dψdt ` R ψ “ δ S , ayesian mechanics of perceptual inference and motor control in the brain δ S “ S p t q ´ S p8q . It is straightforward to integrate the above inhomogeneousdifferential equation to obtain a formal solution, which is given by ψ p t q “ e ´ R t ψ p q ` ż t dt e ´ R p t ´ t q δ S p t q . (33)Note that δ S becomes zero identically for static sensory inputs; therefore, the relaxationadmits simple homogeneous dynamics. In contrast, for time-varying sensory inputs, theinhomogeneous dynamics driven by the source term is expected to be predominant.However, on time scales longer than the sensory-influx saturation time τ , it can beshown that δ S Ñ
0; for instance, τ . “ λ l and eigenvectors ξ p l q of the relaxation matrix R as follows: ψ p t q “ ÿ l “ c l e ´ λ l t ξ p l q , (34)where the expansion coefficients c l are fixed by the initial conditions ψ p q . The initialconditions ψ p q represent a spontaneous or resting cognitive state. In Eq. (34), theeigenvalues and eigenvectors are determined by the secular equation R ξ p l q “ λ l ξ p l q . (35)Then, the solution for the linear RD Eq. (29) is given byΨ p t q “ Ψ eq ` ÿ l “ c l e ´ λ l t ξ p l q , (36)which is exact for perceptual inference, and legitimate for active inference on timescales t ą τ .Before presenting the numerical outcome, we first inspect the nature of the fixedpoints by analyzing the eigenvalues of the relaxation matrix R given in Eq. (30). First,it can be seen that the trace of R is zero, which indicates that the two eigenvalues haveopposite signs, i.e., λ “ ´ λ . Second, the determinant of R can be calculated as followsDet p R q “ ´ p ` φ { m w q ` m w ` ´ m z ` φ { m z ˘ . Therefore, if the correlation φ Ñ
0, it can be conjectured that both eigenvalues are real.This is because Det p R q “ λ λ Ñ ´ ´ m z { m w ă
0, which gives rise to λ “ λ ą φ “
0, the solution will be unstable. In contrast,when the correlation is retained, Det p R q can be positive for a suitable choice of statisticalparameters, namely m w , m z , and φ . In the latter case, the condition λ λ ą λ l ă l “ ,
2. Accordingly, λ and λ that have opposite signs are purelyimaginary, which makes the fixed point Ψ eq a center (Strogatz 2015). If we define λ , ” ˘ iω , the long-time solution of RD with respect to Ψ eq is expressed as ψ p t q “ c e iωt ξ p q ` c e ´ iωt ξ p q , ayesian mechanics of perceptual inference and motor control in the brain
10 20 30 40 t - ( t ) - - - - - - - - - p Figure 2.
Perceptual inference of static sensory data: (Left) Oscillatory brainvariables µ “ µ p t q developed from a common spontaneous state p µ p q , p p qq “ p , q by responding to sensory inputs ϕ “
10 (gray), 15 (red), and 20 (blue). Horizontaldotted line indicates the agent’s prior belief about the sensory input. (Right) Limitcycles in the perceptual state space from an input ϕ “ . p µ p q , p p qq “ p , q , p´ , ´ q , and p´ , ´ q . The common fixed point isindicated by an orange bullet at the center of the orbits, which predicts the sensorycause incorrectly. [All curves are in arbitrary units.] which specifies a limit cycle with an angular frequency ω . Thus, according to ourformulation, the effect of correlation on the brain’s RD is not a subsidiary but a crucialcomponent. Here, we consider numerical illustrations with finite correlation.We exploited a wide range of parameters for numerically solving Eqs. (27) and(28) and found that there exists a narrow window in the statistical parameters σ w , σ z , and φ , within which a stable trajectory is allowed for a successful inference. Thisfinding implies that the agent’s brain must learn and hardwire this narrow parameterrange over evolutionary and developmental timescales; namely, the generative modelsare conditioned on an individual biological agent. We denote the instantaneous cognitivestate as p µ p t q , p p t qq for notational convenience.In Fig. 2, we present the numerical outcome from the perceptual inference of thestatic sensory inputs. To obtain the results, we have chosen a particular set of statisticalparameters as follows: σ w “ . , σ z “ , and φ “ ´ . , which specify the neural inertial masses m w . “ . m z “ . ˆ m w as well as the coefficients that enter the RD, namely α . “ ´ . , β “ ´ . , and γ “ ´ . . In Fig. 2 (Left), we depict the brain variable µ as a function of time, which representsthe cognitive expectation of a registered sensory input, under the generative model ayesian mechanics of perceptual inference and motor control in the brain
10 20 30 40 t - - Figure 3.
Active perception: Time-development of the perceptual state inferringthe external causes of sensory inputs that are altered by the agent’s motor control.The blue and magenta curves depict the brain activity µ p t q and the correspondingmomentum p p t q , respectively, and the noisy curve indicates the nonstationary sensoryinputs ϕ p t q at the receptor. For numerical illustration, we have used σ w “ . , σ z “ , and φ “ ´ .
8. [All curves are in arbitrary units.] [Eq. (25)] for three values, namely ϕ “ , , and 20. For all illustrations, the agent’sprior belief with regard to the sensory input is set to be ϑ d “ , which is indicated by the horizontal dotted line. The blue curve represents the case inwhich the sensory data are in line with the belief. The RD of the perceptual inferencedelivers an exact output p µ eq , p eq q “ p , q . Note that µ eq and p eq are the temporalaverages of µ p t q and p p t q , respectively, in the stationary limit. The other two inferencesunderscore the correct answer. Figure 2 (Right) corresponds to the case of a singlesensory data ϕ “ .
0, which the standing agent senses at the field point x “
2. Theensuing trajectories from all three initial spontaneous states have their limit cycles inthe state space defined by µ and p . We numerically determined the fixed point to be p µ eq , p eq q . “ p´ . , ´ q and the two eigenvalues of the relaxation matrix R to be p λ , λ q . “ p . i, ´ . i q , which are purely imaginary and have opposite signs. Again,the perceptual outcome does not accord with the sensory input; it deviates significantly.Next, in Fig. 3, we present the results for active inference, which were calculatedusing the same generative parameters used in Fig. 2. The agent is initially situated at x p q “
2, where it senses the sensory influx ϑ p q “
4, which does not match the desiredvalue ϑ d “
20. Therefore, the agent reacts to find a comfortable environmental nichematching its prior belief, which generates nonstationary sensory inputs at the receptors(see Fig. 1). It can be observed that the brain variable µ initially undergos a transientperiod at t ď
5. The RD commences from the resting condition p µ p q , p p qq “ p , q and then develops a stationary evolution. We have also numerically confirmed that thebrain’s stationary prediction µ eq , which is the brain’s perceptual outcome of the sensorycause, is close to but not in line with the prior belief ϑ d . The stationary value p eq isestimated to be approximately 8 .
0, which is the average of the stationary oscillation ofthe prediction error p p t q . ayesian mechanics of perceptual inference and motor control in the brain -
50 50 100 μ - - - p Figure 4.
Active inference: Temporal development of trajectories rendering stationarylimit cycles in the perceptual phase space, obtained from the same statisticalparameters used in Fig. 2. The blue, red, and gray curves correspond to the threeinitial conditions, p µ p q , p p qq “ p , q , p , q , and p , q , respectively. The angularfrequency of the limit cycles is the magnitude of the imaginary eigenvalues of therelaxation matrix R given in Eq. (30). The common fixed point is indicated by amagenta bullet at the center of the orbits. [All curves are in arbitrary units.] In Fig. 4, the trajectory corresponding to that in Fig. 3 is illustrated in blue color inthe perceptual state space spanned by µ and p , including two other time developmentsfrom different choices of initial conditions. All data were calculated using the samegenerative parameters and sensory inputs used for Fig. 3. Regardless of the initialconditions, after each transient period, the trajectories approach stationary limit cyclesabout a common fixed point, as seen in the case of static sensory inputs in Fig. 2(b).The fixed point Ψ eq and stationary frequency ω of the limit cycles are not affected bythe initial conditions, which are solely determined by the generative parameters m w , m z , and φ and the prior belief ϑ d for a given sensory input ϕ [see Eqs. (32) and (35)].In addition, we have numerically observed that the precise location of the fixed pointsis stochastic, reflecting the noise from the nonstationary sensory influx ϕ .In the framework of active inference, motor behavior is attributed to the inference ofthe causes of proprioceptive sensations (Adams et al. 2013), and in turn, the predictionerrors convey the motor signals in the closed-loop dynamics of perception and motorcontrol. In Fig. 5, we depict the sensorimotor signals ∆ ϕ p µ ; ϕ p t qq that appear as time-dependent driving terms in Eqs. (27) and (28). In both figures, the agent is assumedto be initially situated in such a way as to sense the sensory data ϕ p q “
4. After aninitial transient period elapses, the motor signals exhibit a stationary oscillation aboutaverage zero in Fig. 5 (Left), implying the successful fulfillment of the active inference ofnonstationary sensory influx matching the desired belief ϑ d “
20. The amplitude of the ayesian mechanics of perceptual inference and motor control in the brain
10 20 30 40 t - Δ φ
10 20 30 40 t - - - - Δ φ Figure 5.
Motor signals ∆ ϕ p µ ; ϕ p t qq evoked by the discrepancy between thenonstationary sensory stream and its top-down prediction [see Eq. (20)]. Here, we setthe prior belief ϑ d “
20 (Left) and ϑ d “
10 (Right). The blue and red curves representthe results from the initial condition p µ p q , p p qq “ p , q and p , q , respectively.The gray curves represent the corresponding signals from the plain perception of thestatic sensory input. All data were obtained by setting the statistical parameters as σ w “ . , σ z “ , and φ “ ´ .
8. [All curves are in arbitrary units.] motor signal shown by the blue curve is smaller than that shown by the red curve, whichis also reflected in the size of the corresponding limit cycles in Fig. 4. The prediction-error signal from the plain perception also exhibits an oscillatory feature in the graycurve, which arises from the stationary time dependence of the brain variable µ p t q . Theamplitude shows a large variation due to the significant discrepancy between the staticsensory input ϕ “ ϑ d “
20. In Fig. 5 (Right), for comparison,we have repeated the calculation with another value: ϑ d “
10. In this case, the priorbelief ϑ d about the sensory input does not accord with the stationary sensory streams.Therefore, the blue and red signals for active inference oscillate about the negativelyshifted values from average zero. In contrast to Fig. 5 (Left), the error-signal amplitudeof the static input is reduced because the difference between the sensory data and theprior belief decreases.Next, we consider the role of correlation φ in the brain’s RD, whose value islimited by the constraint | φ | ď ? σ w σ z . To this end, we chose three values of φ forthe fixed variances σ w and σ z , and integrated the RD for active inference. In Fig. 6,we present the resulting time evolution of the brain states µ for the initial condition p µ p q , p p qq “ p , q . In this figure, the conjugate momentum variables are not shown.The noticeable features in the results include the changes in the fixed point and theamplitude of the stationary oscillation with correlation. The average value of µ p t q inthe periodic oscillation corresponds to the perceptual outcome µ eq of the sensory datain the stationary limit. We remark that for all numerical data presented in this work,we have chosen only negative values for φ . This choice was made because our numericalinspection revealed that positive correlation does not yield stable solutions.As the final numerical manifestation, in Fig. 7, we show the temporal buildup of thelimit cycles in the perceptual phase space; however, this time, we fix σ w while varying σ z and φ . The resulting fixed points are located approximately at the center of each ayesian mechanics of perceptual inference and motor control in the brain
10 20 30 40 t - μ ( t ) Figure 6.
Time evolution of the brain variable µ . Here, we vary the correlation φ for fixed variances σ w “ . σ z “
10. The red, blue, and gray curves correspondto φ “ ´ . ´ .
8, and ´ .
6, respectively. For all data, the agent is environmentallysituated at x “
2, where it senses the transient sensory inputs ϕ p t q induced by themotor reflexes at the proprioceptive level. The agent’s initial cognitive state is assumedto be p µ p q , p p qq “ p , q , and the prior belief is set as ϑ d “
20. [All curves are inarbitrary units.] limit cycle, which are not shown. Similar to that in Fig. 6, it can be observed that thepositions of the fixed point and the amplitudes of oscillation are altered by the variationin the statistical parameters. Evidently, a different set of parameters, namely σ w , σ z ,and φ , which are the learning parameters encoded by the brain, result in a distinctiveBayesian mechanics of active inference.Here, we summarize the major findings from the application of our formulation tothe simple nonstationary model. The brain’s Bayesian mechanics, i.e., Eqs. (27) and(28), employ the linear generative models given in Eqs. (25) and (26).(i) The steady-state solutions of the RD turn out to be a center about which stationarylimit cycles (periodic oscillations) are formed in the perceptual phase space, whichconstitute the brain’s nonequilibrium resting states.(ii) The nonequilibrium stationarity stems from the pair of purely imaginary eigenvaluesof the relaxation matrix with opposite signs, given by Eq. (30); the magnitudespecifies the angular frequency of the periodic trajectory.(iii) The centers are determined by the generative parameters and the prior belief for agiven sensory input in the steady limit [see Eq. (32)], which represents the outcomeof active inference and the entailed prediction error.(iv) The inclusion of noise correlation between the brain’s expectation of the externaldynamics and sensory generation is consequential to ensuring a stable solution.Furthermore, based on numerical experience, a negative correlation is a prerequisitefor obtaining stable solutions using the current model. ayesian mechanics of perceptual inference and motor control in the brain - μ - - - - - p Figure 7.
Limit cycles in the perceptual phase space. Here, we have considered severalsets of σ z and φ for a fixed σ w “ .
0. The red, blue, and gray curves were obtainedfrom p σ z , φ q “ p , ´ . q , p , ´ . q , and p , ´ . q , respectively. For all data, theagent’s initial cognitive state is assumed to be p µ p q , p p qq “ p , q , and the prior beliefis set as ϑ d “
20. The agent is environmentally situated at x “
2, where it senses thetransient sensory inputs ϕ p t q induced by the motor reflexes at the proprioceptive level.[All curves are in arbitrary units.]
5. Concluding remarks
We implemented the FEP with the minimization scheme derived from the principle ofleast action to cast the promising unified biological framework to a more physicallygrounded approach. It should be noted that in our theory, the continuous time-integralof the induced IFE in the brain, not an instantaneous IFE, performs as a variationalobjective function over a finite temporal horizon. To present the novel aspects of ourformulation as well as a manifestation of its utility with a concrete model, this studyfocused on the perceptual inference of nonstationary sensory influx at the interface. Thenonstationary sensory inputs were assumed to be unknown or contingent to the neuralobserver without explicitly engaging in motor-inference dynamics in the RD; in otherwords, the agent’s sensorimotor selection was not part of our internal model. Instead, weconsidered that the motor signals are triggered by the discrepancies between the sensoryinputs at the proprioceptive level and their top-down predictions. They appeared asnonautonomous source terms in the derived RD, thus completing the sensorimotordynamics via reflex arcs or oculomotor dynamics of sampling visual stimuli. This closed-loop dynamics contrasts with the gradient-descent implementation, which involves thedouble optimization of the top-down belief propagation as well as the motor inferencein the message-passing algorithms.Using the parsimonious agent-based model, we evinced that the ensuing trajectories ayesian mechanics of perceptual inference and motor control in the brain F t given in Eq. (8) to the future expected FE (EFE) ina time series and formulates the adaptive decision-making processes in action-orientedmodels. The EFE or generalized FE essentially appears to be a discrete sum of theinformational action S in the cumulative time steps, i.e., S « ř t F t [see Eq. (12)], whileencapsulating motor planning or policies as priors in the framework of Markov decisionprocesses. The assimilation of this feature needs to be studied in depth (Millidge etal. 2020). Moreover, an inferential mechanism of motor planning for active perceptioncan be included in the continuous-time formulation as a part of the generative models(Bogacz 2020). We are currently considering a formulation of motor planning togetherwith the assimilation of extended IFEs in the scope of the least action principle. References [1] Adams RA, Shipp S, Friston KJ (2013) Predictions not commands: active inference in the motorsystem. Brain Struct Funct 218:611–643. https://doi.org/10.1007/s00429-012-0475-5[2] Balaji B, Friston K (2011) Bayesian state estimation using generalized coordinates. Proc. SPIE8050, Signal Processing, Sensor Fusion, and Target Recognition XX, 80501Y (5 May 2011);https://doi.org/10.1117/12.883513.[3] Baltieri M, Buckley CL (2019) PID control as a process of active inference with linear generativemodels. Entropy 21:257[4] Biehl M, Guckelsberger C, Salge C, Smith SC, Polani D (2018) Expanding the active inferencelandscape: More intrinsic motivations in the perception-action loop. Front. Neurorobot 12:45.doi: 10.3389/fnbot.2018.00045[5] Bogacz R (2017) A tutorial on the free-energy framework for modelling perception and learning.Journal of Mathematical Psychology 76(B):198–211. https://doi.org/10.1016/j.jmp.2015.11.003[6] Bogacz R (2020) Dopamine role in learning and action inference. eLife 9:e53262. DOI:https://doi.org/10.7554/eLife.53262 ayesian mechanics of perceptual inference and motor control in the brain [7] Buckley CL, Kim CS, McGregor S, Seth AK (2017) The free energy principle for actionand perception: A mathematical review. Journal of Mathematical Psychology 81:55–79.https://doi.org/10.1016/j.jmp.2017.09.004[8] Colombo M, Wright C (2018) First principles in the life sciences: The free-energy principle,organicism, and mechanism. Synthese. https://doi.org/10.1007/s11229-018-01932-w[9] Cover T, Thomas JA (2006) Elements of Information Theory 2nd ed. Wiley-Interscience, Hoboken[10] Da Costa L, Parr T, Sajid N, Veselic S, Neacsu V, Friston K (2020) Active inference on discretestate-spaces: a synthesis. arXiv:2001.07203 [q-bio.NC][11] de Gardelle V, Waszczuk M, Egner T, Summerfield C (2013) Concurrent repetition enhancementand suppression responses in extrastriate visual cortex. Cerebral Cortex 23(9):2235–2244.https://doi.org/10.1093/cercor/bhs211[12] Durstewitz D, Huys Q, Koppe G (2020) Psychiatric illnesses as disorders of net-work dynamics. Biological Psychiatry: Cognitive Neuroscience and Neuroimaging.https://doi.org/10.1016/j.bpsc.2020.01.001[13] Elfwing S, Uchibe E, Doya K (2016) From free energy to expected energy: Improving energy-based value function approximation in reinforcement learning. Neural Networks 84:17–27.http://dx.doi.org/10.1016/j.neunet.2016.07.013[14] Fox RF (1987) Stochastic calculus in physics. J Stat Phys 46:11451157.https://doi.org/10.1007/BF01011160[15] Friston K (2008a) Hierarchical models in the brain. PLoS Comput Biol 4(11): e1000211.doi:10.1371/journal.pcbi.1000211[16] Friston KJ (2008b) Variational filtering. NeuroImage 41:747–766[17] Friston KJ, Daunizeau J, Kiebel SJ (2009) Reinforcement learning or active inference?. PLoS ONE4(7):e6421. doi:10.1371/journal.pone.0006421[18] Friston K (2010a) The free-energy principle: a unified brain theory?. Nature Reviews Neuroscience11:127–138[19] Friston K, Stephan K, Li B, Daunizeau J (2010b) Generalized filtering. Mathematical Problemsin Engineering 2010:261670.[20] Friston KJ, Daunizeau J, Kilner J, Kiebel SJ (2010c) Action and behavior: a free-energyformulation. Biol Cybern 102(3):227260[21] Friston K, Mattout J, Kilner J (2011a) Action understanding and active inference. Biol Cybern104:137–160.[22] Friston K (2011b) What is optimal about motor control?. Neuron 72(3):488–498. DOI10.1016/j/neuron.2011.10.018[23] Friston K, Adams R, Perrinet L, Breakspear M (2012) Perceptions as hypotheses: saccades asexperiments. Frontiers in Psychology 3:151. https://doi.org/10.3389/fpsyg.2012.00151[24] Friston K (2013) Life as we know it. J R Soc Interface 10:1020130475.http://doi.org/10.1098/rsif.2013.0475[25] Friston K, Rigoli F, Ognibene D, Mathys C, Fitzgerald T, Pezzulo G (2015) Active inference andepistemic value. Cogn Neurosci 6:187–214. doi: 10.1080/17588928.2015.1020053[26] Friston KJ, Parr T, de Vries B (2017) The graphical brain: Belief propagation and active inference.Network Neuroscience 1(4):381414.[27] Huang Y, Rao RPN (2011) Predictive coding. WIREs Cogni Sci 2:580–593. DOI: 10.1002/wcs.142[28] Isomura T, Kotani K, Jimbo Y (2015) Cultured cortical neurons can perform blind sourceseparation according to the free-energy principle. PLoS Comput Biol 11(12):e1004643.doi:10.1371/journal.pcbi.1004643[29] Isomura T, Friston K (2020) Reverse engineering neural networks to characterise their costfunctions. bioRxiv 654467. doi: https://doi.org/10.1101/654467[30] Jazwinski AH (1970) Stochastic process and filtering theory. Academic Press, New York[31] Kiefer AB (2020) Psychophysical identity and free energy. J. R. Soc. Interface 17: 20200370.http://dx.doi.org/10.1098/rsif.2020.0370 ayesian mechanics of perceptual inference and motor control in the brain24