Laughing, but not quite! How does your brain read emojis?

© Freepik

Leviathan Press:

The same sentence, for example:

That's good.

That's good.

That's good.

I believe we can all feel the subtle difference, or in other words, even if you don’t know the context of the sentence, you can basically tell what the speaker wants to express through emoticons.

More importantly, emojis can convey more intriguing meanings that cannot be fully expressed in words, such as different degrees of ridicule and sarcasm. Of course, the use of emojis is somewhat ambiguous for people of different ages. For example, some people think that "" just means smiling, while many people often use this expression as a hint of helplessness, embarrassment and sadness.

Twenty years ago, Microsoft's instant messaging platform added a new feature: users could insert dozens of small icons into messages to convey happiness, surprise, confusion, or a sheep. Gradually at first, and then suddenly and with sweeping speed, emojis arrived: They spread from chat platforms to text messages, emails, social media, and even infiltrated school assignments, much to the annoyance of many teachers.

For years, I was an emoji resister. Embracing these little cartoon images felt like a violation of the virtues of words.

To a linguistically traditional soul like me, who grew up reading Jane Austen and Isaac Babel, emojis seem cheap and needlessly revealing, and I resent their creeping into written communication, which has long flowed smoothly with just the alphabet, thank goodness.

But a few years ago, my attitude shifted after I became friends with a coworker who used them profusely. Our daily banter became doubly fun because of the emotional energy they added, and the sense of connection they inspired.

At first, I timidly strung them into my digital discourse, but now they are woven into my communications with many people in my life, punctuating a phrase or standing alone as a message—a 💥 or 🔥 or 🌟 can be a complete reply.

What surprised me most was the tangible joy these icon-based interactions added to everyday interactions.

The effect is like a shot of meaning-making caffeine—an infusion of pure emotion.

Valeria Pfeifer, a cognitive scientist at the University of Arizona, is one of the researchers who study how emojis affect our thinking. She told me that my newfound joy made sense. Emojis, she said, “convey an extra, complex layer of meaning that doesn’t seem to be really conveyed through words.”

Many “word nerds” who love to find and memorize new words worry that emojis are dumbing us down and making us communicate with each other. But Pfeifer and other cognitive scientists and linguists are working to explain what’s so special about them.

In a book called The Emoji Code, British cognitive linguist Vyvyan Evans calls emojis “arguably the world’s first truly universal form of communication.” That may seem like a tall claim for an ever-expanding collection of symbols whose meanings may be fluid. But language is always evolving, and these ideograms have become the lingua franca of digital communication.

The history of emojis dates back to before the early days of instant messaging. Before these graphically detailed icons became easy to display, clunkier, character-constructed emojis took their place.

Scott Fahlman, a computer scientist at Carnegie Mellon University, is generally credited with creating the first smiley and winking emojis made up of punctuation marks.

He saw posters on early online bulletin boards getting into conflict with one another — sometimes, for example, their sarcasm was misinterpreted — so in 1982 he suggested that colleagues add “:-)” or “:-(” to their posts to indicate their tone. He thought readers would be less easily offended if posters could mark when they were joking or sarcastic.

Farman and his emojis. © WIRED

Writers and thinkers have proposed replacing expressions of emotion with punctuation for decades, though many seem to do so in jest. Early emoticons like the “;)” that Abraham Lincoln used in a handwritten note in 1862 to describe the “applause and laughter” of his audience during a speech were likely typographical errors or examples of lax punctuation norms at the time.

The intersection of language and emotion is where the magic lies.

However, it wasn’t until the fertile soil of the early internet that these uses really took root. As graphical interfaces improved, modern emoticons emerged.

© The Guardian

At first, emojis were not "emoticons for effectively conveying emotions." In 1999, Japanese artist Shigetaka Kurita developed the first set of 176 emojis for his mobile phone company. Most of them were not intended to convey a certain emotion, but were everyday symbols that he imagined people could occasionally use to replace words: houses, ears, tennis rackets, fax signs.

It wasn’t until 2011, when Apple first included a dedicated emoji keyboard on their mobile devices (Android followed suit two years later), that emojis really started to go mainstream.

By 2015, more than 90 percent of internet users had used them, and the Oxford English Dictionary named “😂” its word of the year. Today, the Unicode Consortium, the governing body for emojis, lists more than 3,500 of them.

The word "emoji" has no etymological connection with "emote".

It is a combination of the Japanese words for “image” (絵/e) and “character” (文字/moji), which is different from the other word meaning “emoticon”, which is an American invention and a combination of “emotion” and “icon”. This difference in origin and intention also influenced early scientific research on these new communication tools and their effects on the people who used them.

Perhaps the first study of how these visual representations activate the brain came at a conference in 2006.[1] Masahide Yuasa, then a computer scientist at Tokyo Denki University, and his colleagues wanted to know whether our brains could interpret abstract symbolic representations of faces—emoticons made up of punctuation marks—in the same way that they could interpret photographs of faces.

They put college students into a brain scanner (they used functional magnetic resonance imaging [fMRI]) and showed them realistic photos of happy and sad faces, as well as scrambled versions of those images, happy and sad emoticons, and some short, random punctuation combinations.

Photos activated brain regions associated with faces, while emojis did not. But they did activate a different region that is thought to be involved in how people judge something to be emotionally positive or negative. The team later expanded on this finding in a 2011 study[2], reporting that emojis at the end of sentences caused language and non-language regions of the brain to respond more strongly to written text.

“Just as prosody enriches vocal expression,” the researchers wrote in their early paper, emojis appear to add meaning and impact. The effect is like a shot of meaning-making caffeine — an infusion of pure emotion.

© Sky News

It is amazing how these faces, made of punctuation marks, convey their emotional value into the reader’s mind without first being recognized as abstract faces.

At the time, many researchers assumed that people first recognized the faces in the symbols’ lines and dots and then inferred their expressions, “as a bottom-up process,” Yuasa explained to me via email. But the results suggest that the emojis connect to something more basic than facial recognition, suggesting a primal, even deeper drive to respond to emotions in communication.

A few years later, Australian researchers reported[3] that people were much faster at recognizing the smiley emoji “:-)” as a face than they were at recognizing its reverse input, “)-:”. To lead researcher Owen Churches, these results suggest that our brains are amazingly adaptable in the face of a rapidly changing world. “Babies are not born with an innate neural response to emojis,” he told the Australian Broadcasting Corporation. “It’s a completely cultural neural response.”

© Editable Gifs

Once the more visually rich emojis began to proliferate, scientists had more concepts to explore to discern the real-time integration of language, communication, and feeling. The research quickly became interesting and nuanced.

For example: do emojis and text function similarly when trying to convey sarcasm? Sarcasm in its most basic form is about saying the opposite of what you mean in order to get a point across. The inconsistency it conveys is so cognitively satisfying precisely because it adds a dramatic layer and meaning to language.

Benjamin Weissman and Darren Tanner of the University of Illinois at Urbana-Champaign recorded participants’ brain activity patterns as they read simple sentences ending with different emojis — one that matched the meaning of the sentence, one that didn’t, and a winking emoji that clearly indicated sarcasm.

The cake she made was terrible.

In a 2018 paper wittily titled “A strong wink between verbal and emoji-based irony,”[4] they compared their findings with previous research on how the brain responds to sarcastic language and found that, as far as the brain is concerned, emojis and text have much the same effect.

“It was largely consistent,” says Weissman, now a cognitive scientist at Rensselaer Polytechnic Institute in Troy, New York. “ As long as some form of sarcasm was being delivered and interpreted, the brain’s response looked very similar to the response to traditional non-emoji sarcasm. ”

This finding is consistent with new, as-yet-unpublished research by Weissman and cognitive scientist Neil Cohn of Tilburg University in the Netherlands,[5] which explored the brain’s response to reading sentences containing a word or emoji that either matched or mismatched the intended meaning.

Meaning of the sentences in the picture: Her favorite animal is monkey (top); Her favorite animal is avocado (bottom).

Here, too, the brain’s response to emojis was very similar to its response to words: expected emojis triggered patterns of brain activity associated with successful language prediction, while unexpected emojis produced brain activity associated with processing misplaced meaning words.

Weissman said that, to some extent, it doesn't matter whether we think of a concept from a word or an icon. In general, he said, the brain builds complex meaning from the input it receives, and in this high-level cognition, it can integrate all kinds of elements, including facial expressions and intonation. Emojis are just another type of this input. "The process of building meaning can probably operate at a level independent of modality itself," he said.

Of course, anyone who has used emojis knows that they are more than just colorful, jaunty stand-ins for text. Like their 41-year-old cousins, emojis are taking on a much heavier duty. Their ability to convey emotion creates a complex interaction with language. For Pfeifer, that intersection is where their magic lies.

“ Emojis have this ability to make the same words seem more emotional or less emotional — sometimes making the text seem alarming, sometimes making it seem completely normal or even a joke, ” she said. These are important social functions. “As we moved to more text-based communication, we lost this extra layer of meaning that emojis can now provide.”

Our species and many of our spoken languages ​​emerged from the intense baptism of face-to-face communication, replete with intonation, volume, facial expression, gesture, posture, knowing glances. Even when written forms emerged (such as cuneiform more than 5,000 years ago), for most of history they were used primarily for official purposes—government, commerce, religion. Interpersonal, social, cohesive communication remained face-to-face, and mostly out of necessity; as late as 1960, less than half of the world’s population could read and write, and even in Austen’s day, when romantic love letters seemed to be everywhere, only about half of the British population over the age of 14 could write or read a letter.[6]

Positive emojis say, “Hey, I’m listening.” Negative emojis have a completely different effect.

But as we enter the brave new world of the digital age, the written word has taken over the majority of daily social and co-worker communication. Today, we use text messages and direct messages more than social visits or cell phone calls. Today, work meetings or phone calls often devolve into a quick keyboard text on Slack or Microsoft Teams.

Nature abhors a vacuum, so it fills it. Emojis seem to have arrived at a time when new forms of communication were in need of filling the void. “ When we all started texting or emailing, it felt like something was missing from the way we communicated, and emojis seemed to fill that void, ” Pfeifer said.

She and her colleagues were also interested in how different emojis influence social dynamics by adding emotional color to written statements. They found[7] that happy emojis like hearts and smiley faces add a general positive emotional effect to messages, though without a specific, specific “lift value.” Instead, these graphic cues serve more as a sign of connection.

© Cheezburger

“Positive emojis are like the blinking lights on a recording device,” she said. “Maybe we send these types of emojis to say, ‘Hey, I’m listening’ or ‘I’m interested in what you’re saying’ — just a way of affirming our relationship.” These emojis appear to help promote social cohesion.

Negative emojis, on the other hand, affect text and interpretation in different ways. Pfeifer said that in their study, subjects viewed frowns, angry faces and tears as more specific indicators of mental states, and they processed these symbols more carefully and deeply. "From the sender's perspective, sending a negative emoji says more than sending a positive one."

Furthermore, while positive emotions are the social glue that binds us together — even when they appear in emoji form — negative emotions require a deeper exploration of a relationship or shared understanding to clarify their intent. And, just as when we communicate face to face, negative emotions are more prone to misunderstanding.

© Dribbble

Emojis also fill other social gaps that have emerged as we move to a digital lifestyle. Linda Kaye, a psychologist at Edge Hill University in Ormskirk, UK, is writing a comprehensive book on emoji research, exploring how these ubiquitous symbols can provide valuable clues to someone’s personality.[8]

Interactions on social media platforms are often thought of as opaque — they lack many of the markers that humans rely on to understand and accurately judge each other. Kaye decided to test whether the way people use emojis online could help others understand what kind of people they are. She and her colleagues asked one group of Facebook users to complete a written personality questionnaire, and then showed another group screenshots of their profiles.

The researchers looked at five main personality dimensions and found that the emojis people used in their profiles helped viewers assess two of them - openness and extraversion - fairly accurately. While extraversion is easy to judge in person, openness is harder to assess. "This tells us that when we form first impressions, online environments may actually sometimes provide more behavior than offline targets to help us understand the other person's openness," Kaye said.

It turns out that a newly emoji-studded world isn’t radically different. Nor is it as bland as many might think. “ At the end of the day, communication — the purpose of communication and how we communicate — hasn’t fundamentally changed that much ,” Kaye explains. “I would say its impact has been more about expanding our boundaries.”

To me, this is an inspiring thought as I continue down my newly emoji-filled path, because it suggests that we need not scoff at this novel concoction but rather view it as a triumph of the human brain’s dazzling adaptability.

I rejoice in the fact that we humans can not only use a rich conceptual and emotional language to create a work of literature as resonant as Moby Dick, but can also translate it completely into emojis: 🐳.[9]

References:

[1]m-yuasa.net/chi2006_yuasa.pdf

[2]m-yuasa.net/pdf/brain/Brain_Activity_When_Reading_Sentences_and_Emoticons.pdf

[3]www.tandfonline.com/doi/pdf/10.1080/17470919.2013.873737#.Uv0F-EJdW52

[4]journals.plos.org/plosone/article?id=10.1371/journal.pone.0201727

[5]research.tilburguniversity.edu/en/publications/the-grammar-of-emoji-constraints-on-communicative-pictorial-seque

[6]ourworldindata.org/literacy

[7]www.sciencedirect.com/science/article/abs/pii/S0747563221003393

[8]www.sciencedirect.com/science/article/abs/pii/S0747563216302072

[9] www.emojidick.com

By Alla Katsnelson

Translated by Kushan

Proofreading/Yuba and Lean Bamboo

Original text/nautil.us/your-%F0%9F%A7%A0-on-emoji-365823/

This article is based on the Creative Commons License (BY-NC) and is published by Kushan on Leviathan

The article only reflects the author's views and does not necessarily represent the position of Leviathan