"AI Stefanie Sun" is actually more popular than the real person. Can you hear the difference? | Digital Literacy

Audit expert: Zheng Yuanpan

Professor of Zhengzhou University of Light Industry

Recently, many people have started to revisit the classics and recall the classic songs that accompanied them growing up. In addition to the singers who are still holding concerts, the "hungry" fans have come up with new tricks.

On Bilibili, "AI Stefanie Sun" has covered many classic songs by celebrities. Her works have not only accumulated tens of millions of views, but have also sparked heated discussions in the technology and music circles. (Due to copyright issues, you can go to Bilibili to search and listen)

Source | Screenshot of Bilibili website

"AI Stefanie Sun" is certainly not the Chinese pop star Stefanie Sun herself, but the songs she covered are difficult for most people to tell whether they are real or fake. Even Stefanie Sun, who has faded from the public eye for a long time, published an article online to respond, and also expressed her views on this phenomenal popularity and "re-emergence" event.

Source | Screenshot of Stefanie Sun’s response

1

How does “AI Stefanie Sun” cover songs?

The obvious word "AI" in the front has already told us that the technology and hard work in this music industry still rely on the keyword that cannot be avoided in recent years - Artificial Intelligence (AI).

The songs covered by "AI Stefanie Sun" are mainly made by multiple creators through an open source project called "so-vits-svc", and then uploaded to the video website Bilibili. In addition to "AI Stefanie Sun", the website also has songs covered by "AI JJ Lin" and "AI Faye Wong".

Source | Screenshot of Bilibili website

At present, this project has been upgraded to version 4.0. Compared with previous versions of the project, so-vits-svc is more simplified. It only needs a few experimental target audios to synthesize the audio of the target tone and train the desired acoustic model.

The trained model can not only retain the pitch and tone of the target audio, but also present the audio in different languages, and ultimately cover songs of different nationalities and singers. According to current technology, although the timbre can be basically copied 1:1, the singer's singing voice, skills and style are still difficult to completely imitate.

In addition, teaching videos about AI audio production can be found everywhere on the Internet. In simple terms, it only requires four steps: ① Download the one-click start package; ② Input appropriate dry sound (pure human voice without music); ③ Train the acoustic model (the longer the training time, the better the effect); ④ Enter the audio editing software for synthesis and post-production .

However, it is not easy to implement in practice, and it requires some technical thresholds and time. For example, the extraction of dry sound and the training of acoustic models require a lot of data collection and experiments, and the entire implementation process also requires certain algorithm-related experience and results support.

Rcell, one of the creators of "AI Stefanie Sun", said that he and his team spent about half a year collecting about 100 songs of Stefanie Sun as training data, and conducted hundreds of experiments before finally obtaining the current model.

At the same time, they also tried the voice transformation of other singers, such as Jay Chou, JJ Lin, Faye Wong, etc., but the effect was not as good as Stefanie Sun.

Source | Weibo

Another creator, Ziyu, speculated that this may be related to the recognizability of Stefanie Sun's voice. Stefanie Sun's pronunciation is relatively clear, and her timbre has a certain granularity. Such characteristics allow the current version of the open source project to train a model with the same characteristics , so it feels particularly similar.
In these "AI Stefanie Sun" cover videos, there are many comments and comments such as "I'm on my knees when she sings", "Exactly the same", "Isn't this the original singer?", and many netizens are shocked that AI technology has developed to such a level.

2

Voice Privacy and Security

Everyone's voice and appearance are unique, but AI deep synthesis technology is gradually breaking this boundary.

Long before it conquered the music industry, AI had already sparked heated discussions in other fields, such as AI face-changing in videos. At first, people just used AI to change the faces of actors in film and television works into other stars or individuals, which was a form of self-entertainment and self-cultivation for a small number of people.

However, with the development of AI technology, while it brings convenience to people and improves production efficiency, it also brings many new problems. For example, criminals use AI face-changing to carry out fraud activities, and the combination of AI face-changing and AI voice generation is even more difficult to guard against.

Source | AIGC (AI Graphics)

It is reported that in 2020, a senior executive of a company in Shanghai was defrauded of 1.5 million yuan because a scammer used AI face-changing and artificially generated voice technology to impersonate the company leader and asked the executive to transfer money.

According to professionals, AI technology is currently used in telecommunications fraud crimes in the following two aspects: First, the scammers extract someone's voice through harassing phone recordings, etc. After obtaining the material, they use AI technology to synthesize the voice , and then use the fake voice to deceive the victim. Second, the scammers use AI face-changing technology to pretend to be the victim's acquaintances, and then use video calls to confirm information and deceive trust.

Just like the copyright issues caused by the popularity of ChatGPT (Chat Generative Pre-trained Transformer), the copyright issues related to AI cover songs have been mentioned again, such as: Does AI cover song infringe the singer's voice copyright? Does the song sung by AI have its own copyright?

There are currently no clear legal provisions or judicial precedents on these issues in my country, so there are many controversies. In response to this, the Cyberspace Administration of China issued the "Regulations on the Administration of Generative Artificial Intelligence Services (Draft for Comments)" on April 11, 2023.

Source | Cyberspace Administration of China

The public consultation on this draft ended on May 10, 2023, but the specific release and effective date of the final version is still uncertain.

Legal experts said that the formulation of this method shows that the national network regulatory authorities are paying close attention to the social ethics, legal and other issues brought about by AI technology, and have begun to study the definition of the "boundaries" of AI applications.

3

What do you think of AI cover songs?

As netizens listened to the covers of these "AI Stefanie Sun"s, some simply enjoyed this rare "welfare" time, while others thought more deeply. For example, does the praise for "outdated" singers and the covers of classic old songs mean that there have been no good singers and songs in the Chinese music scene for a long time? Will singers face the problem of being replaced by AI like translators and painters?

In short, there are all kinds of different voices, and the singer Chen Shanni's performance art-like operation gave us more inspiration. In March 2023, Chen Shanni released a new song "Teach Me How to Be Your Lover". After listening to it, fans said that her singing level is still online and even better than before.

Source | Weibo

But later, Chen Shanni posted on Weibo saying that the new song was not sung by her, but by the "AI model" that she spent a lot of time and energy training, and even the single cover was generated by AI.

When the news came out, everyone was shocked: musicians actually took the initiative to accept AI and let AI "replace" themselves.

In the face of various controversies, Chen Shanni also announced her original intention for doing so, "At a time when the development of AI is hotly debated, I hope that through this song, I can think with all those who care about creation - if the era of AI is bound to come, what creators should care about may not be 'whether we will be replaced', but 'what else can we do'. "

Source | Pexels

Indeed, neither celebrities nor the general public can stop the advancement of science and technology. Instead of opposing it or fearing it, we should think about how to use and master it.