Can China's Yongle Palace murals, which are as large as 1,000 square meters, be restored by AI?

Nowadays, artificial intelligence (AI) is not only increasingly being used in science and business, but is also beginning to make its mark in the arts.

As people marvel at the endless stream of AI technologies, a new AI mural restoration technology has emerged.

Recently, in order to solve the problem of "difficult restoration" of the Yongle Palace murals, a masterpiece of ancient Chinese murals, a research team from Shanxi Datong University, Universiti Sains Malaysia, and Dali University proposed an AI model that can repair giant murals - 3M-Hybrid.

It is reported that compared with the best model among four representative convolutional neural network (CNN) models, the model improves SSIM and PSNR by 14.61% and 4.73% respectively in the restoration of regular-sized murals. In addition, it also achieves good results in the final restoration of huge murals.

The related research paper, titled “A 3M-Hybrid Model for the Restoration of Unique Giant Murals: A Case Study on the Murals of Yongle Palace”, has been published on the preprint website arXiv.

How can AI restore large-scale murals?

The murals of Yongle Palace are located in Yongle Palace (also known as Dachunyang Wanshou Palace) in Ruicheng, Shanxi Province. The most artistically valuable murals are the exquisite large-scale murals. The entire murals cover a total area of ​​more than 1,000 square meters and are painted in the Wuji Hall, Sanqing Hall, Chunyang Hall and Chongyang Hall.

As a precious cultural heritage, the Yongle Palace murals represent artistic masterpieces in the history of Chinese painting. However, due to a long-term lack of maintenance, these unique murals have many defects, and repairing the murals has become an urgent task.

Compared with manual restoration techniques, digital restoration methods are more efficient and reversible, especially deep learning-based image restoration technology has achieved remarkable results. However, the literature on deep learning-based mural restoration mainly focuses on Dunhuang murals or other murals of regular size, and lacks research specifically on the restoration of Yongle Palace murals and similar huge murals .

Compared with other studies on mural restoration, the restoration of the huge Yongle Palace murals faces two major challenges : 1) the scarcity and unique style of the Yongle Palace murals; 2) the huge size of the murals and the limited proficiency of the models in repairing defects of different types and sizes.

Figure｜The types and scales of damage to the Yongle Palace murals vary, and the actual forms of damage are even more diverse.

According to the paper, the 3M-Hybrid model proposed in this study can effectively restore the Yongle Palace murals. Among them, "3M" refers to three key strategies: multi-frequency, multi-angle and multi-scale, and "Hybrid" refers to the hybrid CNN-VIT network.

First, the research team divided the huge mural into regular-sized parts for restoration, and then reassembled the restored parts into their original size. In order to enable the regular-sized mural restoration model to effectively handle defects of various types and sizes with limited image data, the research considered two aspects: optimizing training data and improving model structure.

To achieve better repair results, the study used separate networks specifically designed to learn and extract these high-frequency and low-frequency signals, thereby enhancing feature learning and repair capabilities within these specific frequency ranges. The frequency-based training method enables the model to effectively handle defects of different scales and types.

In terms of model structure, this study integrated a convolutional neural network (CNN) with a pre-trained visual transformer (VIT) to enhance the feature extraction capability of the model.

In addition, when repairing huge murals, basic cutting methods can cause seam gaps and structural distortion when repairing oversized defects. To solve this problem, the research team adopted a multi-angle strategy to reduce the gaps and used a multi-scale approach, combining cutting and reduction methods. This ensured accurate restoration while enhancing the extraction of the overall structure of the mural and solving the problem of multi-scale defects.

From the visualization, the model shows promising results for the restoration of free dust, free gel, and free linear masks. In addition, the restoration results for free-form block masks show preserved structural integrity and credible textures. The 3M-Hybrid model proves to be a viable approach for the restoration of these unique and huge murals .

However, the research is not perfect.

First, the proposed method relies on multiple experiments to select the best values ​​of the three-scale fusion weights. However, considering that the weight settings cover countless possibilities and the number of experiments is limited, this method may not be accurate enough. Therefore, the weight values ​​determined based on the experimental results can only guarantee relatively favorable final results.

Secondly, the evaluation indicators used in the study are not objective enough. The four evaluation indicators currently used do not comprehensively evaluate the image structure and usually cannot accurately reflect human perception and evaluation of images.

However, it is undeniable that this study explored the application of deep learning in the restoration of huge murals, with a particular focus on the use of deep learning technology to restore the Yongle Palace murals. It represents the first attempt to explore deep learning restoration methods for large-scale artworks.

In addition, in terms of improving the regular size image restoration model, this study has made comprehensive improvements from both data and structural perspectives. This provides new insights into restoring unique small datasets for future research.

Helping humans extend the life of cultural relics

In the past few years, people have also witnessed the wonderful combination of AI technology and cultural heritage history.

In 2020, a Weibo user named "Otani Spitzer" used AI technology to restore the black-and-white images of Beijing in 1920 released by the People's Daily four years ago, completing tasks such as coloring, restoring frame rate, and expanding resolution.

In 2021, Tencent Multimedia Lab also cooperated with the Dunhuang Research Institute to use deep learning methods to analyze the Dunhuang mural disease data and developed an efficient AI mural disease identification tool. At the same time, it provided immersive remote consultation technology, using 4K ultra-clear 360-degree images to display the scenes inside the caves and the details of the cultural relics, realizing barrier-free remote cultural relic consultation.

In June this year, at the theme forum "Cultural Relics Protection and Utilization and Cultural Confidence and Self-improvement" of the Heritage Day main city event held in Chengdu, Tencent demonstrated the Sanxingdui human-machine collaborative simulation splicing effect completed using AI technology.

The many applications of AI technology in the field of cultural relics restoration are exciting. In the future, we expect AI technology to go further in cultural relics protection and restoration, helping humans to extend the life of cultural relics.

Paper link:

https://arxiv.org/abs/2309.06194