Alibaba's new AI technology turns ordinary people into dancing masters in seconds

At the beginning of 2024, social media and WeChat Moments were dominated by a series of amazing dance videos. Iron Man danced the third subject, and Musk also performed the dance steps of Internet celebrities. These videos of about 10 seconds were made with the help of large model technology, which easily turned anyone or character into a dance master, setting off a dance battle craze.

Netizens were amazed at the one-click generation capability of the AI ​​creation tool, saying that AI cured their own limb incoordination, and even the archaeological community felt the trend of Subject 3. Now, with just a photo, everyone can easily transform into a dance master, no longer needing to dance in person!

This is exactly what Alibaba's black technology, Animate Anyone, does. Since last November, this innovative tool that makes pictures move has become extremely popular on Twitter and YouTube, with related videos being played more than 100 million times, and the number of attention on GitHub has also soared, exceeding 10,000 stars. Foreign netizens and developers are full of praise for this technology and are looking forward to more opportunities to experience it.

Using Animate Anyone is also very simple. By opening the Tongyi Qianwen APP, entering "Tongyi Dance King" or "National Dance King", selecting a favorite dance template, and uploading a full-body photo, the system can generate a dance video of about 10 seconds. This technology can process pictures of real people, animation or cartoon characters, and easily realize popular dances such as Subject Three, Ghost Step Dance or Rabbit Dance. It also provides 12 popular dance templates for users to choose from, allowing everyone to become a dance master, and it is completely free.

In the past, making the character movements smooth and natural has always been a difficult problem in video production, but Alibaba's Animate Anyone technology has achieved this. It not only accurately captures every detail of the character, such as facial expressions and clothing textures, but also makes the character movements in the animation smooth and natural, looking as realistic as the original image. This is undoubtedly a major breakthrough in AI animation in the field of video generation, especially in processing character movements.

How does Animate Anyone create image animation?

In the hot field of video generation, big names such as Google, Meta and Runway are also making a splash. But the difficulty is to make the movements of the characters in the video both realistic and smooth, which has always been a technical hurdle.

Previous technologies, such as GAN-based methods, can also make images move, but there are often some problems, such as some parts of the image becoming distorted or blurred, or each frame of the animation does not look coherent enough. It's like watching a movie and finding that the characters in it suddenly deformed, or the picture suddenly jumped, which feels very strange.

This time, Alibaba's research team proposed a solution, Animate Anyone. This technology can transform any character's picture into an animated video that follows a specific sequence of poses. They used the Diffusion network design, which can process multiple frames of input, that is, it can take into account multiple frames in the video at the same time.

According to Alibaba's public paper, Animate Anyone integrates a number of innovative technologies, including the introduction of ReferenceNet, which focuses on capturing and retaining original image information and can accurately restore the appearance, expressions and clothing details of the characters. In addition, it also uses an efficient Pose Guider to ensure the accuracy and controllability of the movements; at the same time, through its time series generation module, it effectively ensures the smoothness and coherence between video frames.

Friends who are interested can go there to learn more.

Project address: https://humanaigc.github.io/outfit-anyone/

Experience address: https://huggingface.co/spaces/HumanAIGC/OutfitAnyone

Animate Anyone Framework

This technology has been trained on a dataset of more than 5,000 character video clips, and the effect is natural and realistic. It can maintain the temporal consistency of the appearance and movement of the characters in the video, and generate high-definition videos without jitter or flicker. In performance tests, Animate Anyone outperformed other models in fashion video synthesis and human dance generation without the need for additional human mask learning, which also demonstrated its strong ability to understand the relationship between foreground and background and the visual coherence of movements.

To illustrate this difference, let's take a still photo as an example.

From an intuitive point of view, DreamPose and BDMM are lacking in maintaining clothing texture details, and the continuity and flickering of movements are more obvious. In contrast, Animate Anyone performs as naturally and smoothly as a real model, and the texture of the clothes is well maintained. Even the slits of the legs are handled very accurately, and the details are more in place.

What practical applications does Animate Anyone have?

Animate Anyone technology not only promotes progress in the field of artificial intelligence research, but also crosses the boundaries of various industries. From online retail to entertainment video production, to artistic creation and virtual character development, it provides new possibilities for various application scenarios.

The virtual fitting room Outfit Anyone launched by the team is an example. When paired with Animate Anyone, this virtual fitting room technology not only makes personalized clothing matching easy, but also means that no matter who you are, no matter what style you like, you can find a virtual fitting experience that suits you. And it can also adapt to various body types, from fitness to curves, and even petite, so that everyone can find their own unique style in this virtual fitting room.

In addition, combined with Animate Anyone technology, the threshold for AI anime character drawing generation has been greatly lowered, allowing ordinary people to easily create a variety of anime characters. Users can freely match the character's face, clothing, accessories and background according to their preferences to create a two-dimensional character with personality and charm.

In addition, with the development of digital human technology and the reduction of costs, Animate Anyone technology has also shined in the field of virtual digital humans, with its applications expanding from news broadcasting to customer service and explanation. It is estimated that by 2026, the scale of China's virtual digital human market will reach 10.24 billion yuan. Users can create digital avatars that meet their needs through customization functions, further promoting the application of digital humans in a wider range of fields.

In the past year, AI technology has been like a speeding train, from text and code creation to movie-level HD production to today's video generation. AI is not only a generalist in the technology world, but also a pioneer of change.

In this technological wave, video generation technology is particularly eye-catching. From Runway's Gen-2 model to Meta's Emu Video, and then to Stability AI's Stable Video Diffusion, every step of progress is a broadening of the boundaries. Domestic ByteDance and Huawei have also demonstrated the innovative strength of Chinese technology, launching eye-catching applications and continuously broadening the boundaries of the industry.

Alibaba has also performed well in this competition, integrating Animate Anyone into the Tongyi Qianwen app, making dance video synthesis within reach. This not only represents a technological breakthrough, but also heralds a change in lifestyle. With the continuous advancement of AI, we are ushering in a new era and witnessing how generative AI will change the way we work and create. Alibaba will undoubtedly continue to play an important role in this technological change.

As a winner of Toutiao's Qingyun Plan and Baijiahao's Bai+ Plan, the 2019 Baidu Digital Author of the Year, the Baijiahao's Most Popular Author in the Technology Field, the 2019 Sogou Technology and Culture Author, and the 2021 Baijiahao Quarterly Influential Creator, he has won many awards, including the 2013 Sohu Best Industry Media Person, the 2015 China New Media Entrepreneurship Competition Beijing Third Place, the 2015 Guangmang Experience Award, the 2015 China New Media Entrepreneurship Competition Finals Third Place, and the 2018 Baidu Dynamic Annual Powerful Celebrity.