Q: In the AI ​​world, which company is the best at bulldozer production?

If someone asks you, "Which company has the best excavator technology?"

You will probably remember this famous advertising slogan: "Look for Lanxiang in Shandong, China." (No advertising, no advertising.)

What if we change the subject and ask in a different way, "In the world of artificial intelligence (AI), which bulldozer is cooler?"

What is a possible answer?

Come, let’s fight with some pictures first.

How about it? Aren’t the various bulldozers very cool and full of artistic atmosphere?

But in fact, their real-life template is just the ordinary, "Bumblebee"-colored bulldozer below.

When an ugly bulldozer meets Van Gogh's "Starry Night," something magical happens.

The "magics" who allow the bulldozer to transform itself into anything it wants come from Cornell University and Adobe Research.

It is reported that they can use artificial intelligence technology to incorporate the style of famous works of art into any 3D scene.

Moreover, this latest technology can show higher quality details than previous studies.

Figure | The new method shows fewer geometric artifacts (Source: arXiv)

In addition to the "Starry Night" style, the research team also successfully matched a pickup truck with the artistic style of Edvard Munch's "The Scream", with better integration effects both in terms of color matching and detail presentation.

(Source: arXiv)

The related research paper, titled “ARF: Artistic Radiance Fields”, has been published on the preprint website arXiv.

According to the paper, the reason for the improved conversion quality is that the research team's artificial intelligence technology can directly compare the details between the original image and the new style 3D scene. Previous artificial intelligence technologies only converted image features into a more compact statistical set for analysis, thus losing a lot of original details.

In this regard, Kai Zhang, one of the authors of the paper, said, "We tried to really capture subtle artistic styles, like the brushstrokes of Starry Night. This is very important for human perception because our eyes are very sensitive to local details."

In this work, the research team demonstrated that the nearest neighbor feature matching (NNFM) style loss is very effective in capturing style details while maintaining consistency across multiple views.

Figure｜Getting consistent free viewpoint style renderings based on NNFM (Source: arXiv)

At the same time, they also proposed a new deferred back-propagation method that uses style losses defined on full-resolution rendered images to optimize memory-intensive radiance fields.

Figure | Schematic diagram of delayed back propagation (Source: arXiv)

In addition, they showed videos of 5 different 3D scenes in 5 different artistic styles in an online survey. Compared with previous AI technologies, their AI technology was more popular, accounting for more than 86%, and obtained more points.

Creating artistic images often requires a great deal of time and special expertise, and extending artwork beyond the 2D image plane into dimensions such as time (in animation) or 3D space (in sculpture or virtual environments) presents many limitations and challenges.

The research team said that this latest style transfer technique is expected to be used in the animated film and game industries in the future to manually fine-tune style templates before use.

In addition, this technology is also applicable to photorealistic style transfers, such as converting the Statue of Liberty at noon to the Statue of Liberty at sunset.

However, this technology also has many limitations. For example, when capturing a 360-degree visual scene, it still requires the help of multiple cameras or drones that can hover and capture views from multiple angles.

Furthermore, it may take up to 20 minutes for the algorithm to successfully run once. This is a very time-consuming process as people may try different styles through trial and error.

To this end, the research team will focus on improving the efficiency of content generation in their subsequent work.

Perhaps one day in the future, this technology will be used on the smartphones that everyone uses.

References:

https://arxiv.org/abs/2206.06360

https://www.cs.cornell.edu/projects/arf/

https://github.com/Kai-46/ARF-svox2