Yes, the brain runs algorithms

There are two ways to test if a set of neurons in the brain are approximating an algorithm. Either we can propose an algorithm that fits with the way an animal behaves, and then see if the activity of neurons approximates that algorithm. Or, we can measure the activity of neurons during a behaviour, and then see what algorithm this activity approximates. We have examples of both. Let’s take an example where we’ve derived an algorithm from behaviour first.

We know a lot about how animals — including us — behave when deciding between two equally dull options. Indeed there is a panoply of experimental tasks where we ask the subject to make a choice between two options based on the evidence available to them. For example, we often show primates (including us) a set of randomly moving dots, within which are embedded a few dots all moving in the same direction — either left, or right. And we ask the primate to decide which direction (left or right) those few coherent dots are moving in. So the primate stares at the screen for a while, watches the dots moving, and eventually makes a decision.

Taking many of these decisions creates specific patterns of reaction times and errors. For example, the number of errors made has a lawful dependence on the proportion of dots that were moving in the same direction — the fewer the dots, the more errors are made in judging their direction. These patterns of times and errors can be reproduced by a simple model in which the evidence that the dots are moving in each direction (left or right) is added up by two counters that compete: evidence for one direction is also used as evidence against the other direction. And these two competing counters turns out to be formally the same as a decision theory algorithm (the sequential probability ratio test).

So we’ve arrived at an algorithm derived from behaviour: what about the brain’s activity during this behaviour? When we look in the brains of monkeys making these decisions, we see neural activity that increases and decreases over time, some activity representing the correct option, which tends to increase; and some activity representing the incorrect option, which tends to decrease. Just like two competing counters of evidence for two options. And Mike Shadlen’s lab have even shown that each jump in activity reflects the amount of evidence available, exactly like the sequential probability ratio test. Here then we see neural activity that closely approximates an algorithm we arrived at from observing behaviour.

(We can even extend the decision algorithm to more than two options, in which case we have the catchily named “multiple sequential probability ratio test”. And this more complex algorithm seems to fit with neural activity in the basal ganglia [read here for way more than you wanted to know about this]).

We also have examples of where we started with the neural activity, and worked out what algorithm they seem to represent. These include a (literally) prize-winning triumph of computational neuroscience: the reward prediction error theory of dopamine. The data came first. In a series of papers, Wolfram Schultz had shown how dopamine neurons fired in response to rewards. A few features were particularly intriguing. Dopamine neurons burst excitedly when unexpected rewards were received. They then “learnt” to instead burst in response to something (like a light flash) that predicted a reward was imminent, and no longer burst in response to the reward itself. And once this link between light flash and reward was learnt, the dopamine neurons stopped firing when the reward was predicted but not received.

Based on these data, two teams (one with Read Montague and Peter Dayan, the other Jim Houk and Andy Barto) independently proposed that dopamine neurons are encoding the reward prediction error used in the algorithms of reinforcement learning theory. These algorithms are equipped with a range of options about what to do in the future, and choose an option based on the predicted value of taking each of them. Once an option is chosen, an error is computed between what was predicted and what turned out to be the actual outcome of choosing that option. This error is then used to update the predicted value of the chosen option: if the outcome was as expected, then no error occurred, and nothing needs changing; if the outcome was better than expected — a positive error — the value of the option increases; if the outcome was worse than expected — a negative error — the value of the option decreases. So this “prediction error” creates a way of turning feedback from the world into changes in behaviour.

The match between the firing of dopamine neurons and this “prediction” error was irresistible. According to Schultz’s data, dopamine neurons signal all three types of error: the absence of error when a reward is predicted, a positive error when a reward is unexpected, and a negative error when expected reward fails to materialise. A seemingly clear match between the discrete step of an algorithm and the activity of some neurons in the brain.

(Well, not quite. Reinforcement learning theory itself was inspired by decades of research on how animals’ behaviour changes as they learn from reward, and then elaborated into how to best train a computer to learn. So in truth we had behaviour -> computational algorithms -> developed far beyond behavioural observations -> then neural activity found that matches steps in these algorithms).

The AI smelters among you may be wondering: what about success of deep neural networks for doing brain-like computation? Like training neural networks to classify images, and then finding units that have properties like neurons in visual cortex? Well, AI-style neural networks are discrete time algorithms at heart. And deep neural networks throw another issue into the mix as they have discrete layers, each feeding their output into the next one along. The brain does not have discrete layers. So the success of AI-style neural networks still leaves open the question of whether these operations can be mapped to the continuous dynamics of the brain.

Another, outre, option here is that while the underlying biology of the brain works in continuous time, the effective operations of the brain are divided into discrete steps. One way this might work is through oscillations in neural activity. Brains are beset by oscillations in their activity, alternating between a short period of activity, and a short period of inactivity. If we were feeling generous, and perhaps had a little bit too much of the old vino, we could interpret each short period of activity as dividing continuous time into discrete windows. So each period of activity is one “step”. There is evidence, for example, of attention oscillating in this way, our ability to mentally engage periodically switching on and off. But oscillations of brain activity are rarely sustained over long periods, and are never nice clean runs of on then off, and on then off. And these oscillation are slow: many things the brain “computes” happen on much faster time-scales. Still, this idea that actually the brain does have discrete steps deserves holding in mind. And then not in mind. Then in mind again. Then not in mind. You can see where I’m going with this.