How far are we from the virtual world in "The Matrix"?

Will we be able to place ourselves in various fantastic virtual worlds by stimulating our brains, just like in The Matrix? Will we be able to help disabled patients walk or talk again by deciphering the electrical signals in their brains through brain-computer interfaces? Will we be able to make machines "understand" and "feel" the world like humans?

At the core of these propositions is a question that has plagued the field of brain science for decades: How does the brain encode external information? Imagine that you are staring at the screen and reading this article, and at this moment your brain is presenting all kinds of information: the screen in front of you, the text on the screen, the brightness of the lights in the room, the sounds in the distance, the thoughts you have when reading the article, your mood at this moment... We live in a world full of various information all the time, and a key task of the brain is to encode this information one by one and present it in our minds, so that we can use this information to generate memories, think, and take actions.

Computers use binary "0" and "1" as the basic units of their coding, and the brain is made up of thousands of neurons, so how do neurons encode abstract information? This is the question we are going to discuss today.

Action potential is the basic way neurons activate and communicate. Studying how the brain converts information into electrical signals of neurons is the problem of "encoding"; studying how to infer what the brain sees and thinks through the activity of neurons is "decoding". If we understand the encoding and decoding of the brain one day, it would be like becoming an excellent "brain programmer" who can use these codes, decipher codes, and even rewrite codes - this is infinitely close to the science fiction story at the beginning of the article.

Therefore, studying neural coding in the brain is undoubtedly one of the core issues in the current brain science community. Among the five major branches of neuroscience, the entire field of systems neuroscience (which is also my current research direction) is basically studying this issue.

So, back to the example of "you are reading this article on the screen right now", what specific knowledge do we have about how the brain encodes visual, environmental and other information? How do brain-computer interfaces, which are at the forefront of today, decode EEG signals? Below, we briefly review the three stages of brain science's exploration of the issue of neural coding:

Part 1

Stage 1: “Supercell”

In the 1960s, Hubel and Wiesel of Harvard University pioneered the use of precision electrodes to record the activity of single cells in the brain, and won the Nobel Prize in Physiology or Medicine for revealing how single neurons in the visual cortex encode visual stimuli.

An important idea was born: various kinds of information may be encoded by "super cells" in the brain, for example, some neurons encode "flowers", some neurons encode "grass", some encode "braised pork", and some even encode abstract content such as "my grandmother". A sensational article in Nature in 2005 supported this idea. They found that there were cells in the brains of human subjects that specifically encoded Jennifer Aniston, the actress in "Friends": this cell did not respond to other people or objects, but would be activated when seeing a photo of Aniston. Similarly, the article also found the "Sydney Opera House Cell": this cell was only activated when seeing a photo of the Sydney Opera House, and was not interested in other scenes or characters (including Aniston).

Quiroga, RQ, Reddy, L., Kreiman, G., Koch, C., &Fried, I. (2005). Invariant visual representation by single neurons in the human brain. Nature, 435(7045),1102-1107.

In the theory of "super cells", each cell has its own duties; just like in a class, there are special study committee members responsible for study, sports committee members responsible for sports games, and labor committee members responsible for duty, and everyone has their own division of labor. There are many such cells in the brain, such as the "place cells" in the hippocampus: when we walk in a room, a single cell in the hippocampus will encode a position in two-dimensional space. Whenever we walk to a specific position in the room, the corresponding cell will be activated.

The discovery of place cells also won the 2014 Nobel Prize. In the golden age of the "super cell" theory, it seemed that everyone believed that as long as they took electrodes to "pan for gold" in the brain, they would always find some magical "super cells."

However, the theory of "super cells" has a fatal problem: there are only so many cells in the brain. If each cell really encodes a kind of information, what will happen if the cells are used up? We are constantly learning new information throughout our lives. "Super cells" are indeed not an efficient encoding method, so the idea of ​​"hybrid encoding" was born.

Part 2

Phase 2: Hybrid Coding

The idea of ​​mixed coding is also very natural: the study committee, sports committee, and labor committee in the class should not only be responsible for their own tasks. For example, when the general cleaning is coming, the sports committee can help the labor committee to undertake part of the work; when the spring outing is notified, the study committee and the art committee can temporarily organize it together. In other words, although everyone has their main responsibilities, this specific division of labor is not absolute, and can be mixed according to the task situation, reflecting a kind of "mixed selectivity".

In a 2013 study at Stanford University, researchers found that neurons in the prefrontal cortex of monkeys showed mixed specificity to multiple experimental factors such as the color, motion, and context of the stimulus signal, and their degree of activation depended on the combination of these factors: for example, some cells were most sensitive to red and left-facing stimuli, while others were most sensitive to blue and right-facing stimuli.

Mante, V., Sussillo, D., Shenoy, KV, & Newsome, WT (2013). Context-dependent computation by recurrent dynamics in prefrontalcortex. nature, 503(7474), 78-84.

Hybrid specificity solves the problem of inefficient one-to-one information encoding in the "super cell" theory: cells are "multi-functional", and the same cell can encode multiple types of information at the same time, and each type of information is also spread among multiple cells. The idea of ​​hybrid coding has received increasing attention in recent years. Computer modeling studies have shown that hybrid coding is more efficient than super cell coding, and the information it can encode is also more complex.

Part 3

Stage 3: Population coding

In the above, we have focused on cells with clear encoding information, such as Aniston cells, place cells, and even mixed-specific cells. But in fact, these cells only account for a small part of all cells in the brain - there are still a large number of cells that are also firing, but it is difficult to clearly define what they are encoding. Even so, do these cells also carry information?

Based on this question, scientists proposed the idea of ​​"group coding": instead of focusing on individual coding cells, perhaps we can use data analysis methods such as machine learning to consider the neuron group as a whole, in order to read more information from more neurons. This is not difficult to understand, 100 neurons can always store more information than 10 neurons.

If we correspond to the example of class cadres mentioned above, instead of giving all the work in the class to the class cadres, why not let the whole class work together to complete it? So we can look at the whole class as a whole, and study how the class carries out the general cleaning, how to organize the spring outing, and how the group performs at the sports meeting.

The most advanced brain-computer interface is based on the idea of ​​group coding: researchers record the electrical signals of all neurons in the subject's brain, and then use these electrical signals to train a machine learning decoder. This decoder reads the electrical signals of the neuron group in real time, and uses this to predict the direction the subject wants to move and operate the robot. This innovative technology is beginning to bring new hope to the lives of some disabled patients.

Willett, FR, Avansino, DT, Hochberg, LR,Henderson, JM, & Shenoy, KV (2021). High-performance brain-to-textcommunication via handwriting. Nature, 593(7858),249-254.

For example, in 2021, the Shenoy group at Stanford University used the electrical signals of group neurons to train a recurrent neural network to help patients with spinal paralysis complete writing with an accuracy rate of nearly 95%.

In the past decade, with the development of invivo imaging technology, we have been able to record the group activity of hundreds or even thousands of neurons at the same time. With this technological advancement, people have begun to think about a new way to understand group coding: taking the electrical activity of each neuron as a dimension, then the activation of N neurons at each moment corresponds to a point in N-dimensional space, and our perception, thinking and consciousness process also corresponds to a line in this N-dimensional space, which is called a neural trajectory.

Kim, TH, & Schnitzer, MJ (2022). Fluorescenceimaging of large-scale neural ensemble dynamics. Cell, 185(1),9-41; Harvey, CD, Coen, P., & Tank, DW (2012). Choice-specific sequences in parietal cortex during a virtual-navigation decision task. Nature, 484(7392),62-68.

You can imagine that there is such an abstract N-dimensional space that corresponds to all our possible mental activities, and our thoughts are like a small fish swimming around in this N-dimensional space. This small fish will respond to external stimuli and is also driven by our own thoughts and emotions, and this small fish swims in our mind world all the time. This is a new way of thinking to understand brain coding. Studying neural trajectories can further understand the activities of neuronal groups and predict our behavior.

In a Nature article in 2012, optical imaging technology of group neurons allowed researchers to not only record the real-time neural trajectories in the mouse brain, but also predict whether the mouse would turn left or right in a few seconds - the mouse had already made the decision in its mind through the neural trajectories, and the direction of the neural trajectories determined the direction in which the mouse would move.

Part 4

Summary: Questions for the Future

The brain used to be like a black box, mysterious and unknowable. Now, we are beginning to step into this black box and try to understand the code of the brain, which is very exciting. Finally, as a member of the scientific research team who studies neural coding on a daily basis, I would like to talk about my personal outlook on this issue in the future.

How far are we from the sci-fi scenes such as brain-computer interfaces mentioned in the previous article? Very far. A common phenomenon in scientific research is that even if we collect the activation "codes" of hundreds or thousands of neurons, this code is still like a book without words, and we basically can't understand what it is saying. This is like a beginner in programming, facing a lot of "for loop" and "end" and being completely confused.

For example, when I do data analysis, facing several terabytes of data from hundreds or thousands of neurons, the first step is to blindly try all the data analysis methods I can think of, such as ANOVA, PCA, SVM, etc., to see if there are any discoveries. The reason behind this phenomenon is that we still lack some fundamental and principled understanding of the brain's encoding, so we can only make a fuss based on the surface data.

A problem that has been criticized for a long time in brain science and even the entire biological community is that we lack the understanding of the fundamental laws of biological systems like "Newton's Three Laws". This problem still exists and it seems that it will exist for a long time, but it is something we cannot avoid if we want to understand the brain more deeply.

Of course, we can still expect the development of new technologies to continue to inject fresh blood into this field. From single-cell electrode recording to optical imaging, the number of neurons we can record simultaneously has increased from 10 to 100, and now to 1,000, which continues to grow exponentially. The latest optical imaging technology allows us to record neurons in multiple brain regions or even the entire cerebral cortex of mice at the same time - the number of neurons is very close to breaking through the order of 10,000. This is thanks to the fact that many scientists with backgrounds in physics, optics, and mechanical engineering have flocked to the field of neuroscience in recent years, making these advanced electrical recording and optical imaging technologies possible.

So in the end, I prefer to fantasize that maybe decades later the plot in "The Matrix" will no longer be a fantasy.

References:

Quiroga, RQ, Reddy, L., Kreiman, G., Koch, C., &Fried, I. (2005). Invariant visual representation by single neurons in the human brain. Nature, 435(7045), 1102-1107.

Mante, V., Sussillo, D., Shenoy, KV, & Newsome, WT (2013). Context-dependent computation by recurrent dynamics in prefrontalcortex. nature, 503(7474), 78-84.

Willett, FR, Avansino, DT, Hochberg, LR,Henderson, JM, & Shenoy, KV (2021). High-performance brain-to-textcommunication via handwriting. Nature, 593(7858),249-254.

Kim, TH, & Schnitzer, MJ (2022). Fluorescenceimaging of large-scale neural ensemble dynamics. Cell, 185(1),9-41;

Harvey, CD, Coen, P., & Tank, DW (2012). Choice-specific sequences in parietal cortex during a virtual-navigation decision task. Nature, 484(7392),62-68.

https://braintour.harvard.edu/archives/portfolio-items/hubel-and-wiesel

Produced by: China Science Expo×Zhihu

Author: xcsun Zhihu answerer, PhD in Neuroscience from MIT

The article only represents the author's views and does not represent the position of China Science Expo

This article was first published in China Science Expo (kepubolan)

Please indicate the source of the public account when reprinting

Please indicate the source of the reprint. Reprinting without authorization is prohibited.

For reprint authorization, cooperation, and submission matters, please contact [email protected]