Deep learning for the extraction of sketches from spectral images of historical paintings

Tools

Zhang, Q., Cui, S., Liu, L., Wang, J., Wang, J., Zhang, E., Peng, J., Kogou, S. ORCID: 0000-0003-1722-7626, Liggins, F. and Liang, H. ORCID: 0000-0001-9496-406X, 2021. Deep learning for the extraction of sketches from spectral images of historical paintings. In: H. Liang ORCID: 0000-0001-9496-406X and R. Groves, eds., Optics for arts, architecture, and archaeology VIII. Proceedings of SPIE, 11784 . Washington: SPIE. ISBN 9781510644021

Preview

Text
1454882_Liang.pdf - Published version
Download (1MB) | Preview

Official URL: https://doi.org/10.1117/12.2593680

Abstract

The sketches of painted cultural objects can be the most indicative of the style of paintings. Extraction of the sketches is an integral process used by conservators and art historians for documentation and for artists to learn historical painting styles through copying and painting. However, at present, sketch extraction is mainly manually drawn, which is not only time-consuming, but also subjective and dependent on experience. Therefore, both accuracy and efficiency need to be improved. In recent years, with the development of machine learning, a series of extraction methods based on edge detection have emerged. However, most of the existing methods can only perform successful extraction if the sketches are well preserved , but for the data with faded sketches or severe conservation issues, the extraction methods need to be improved. It is beneficial to extract the bands that accentuate the sketches while suppressing the effects of the degraded areas and the overlapping paints. We propose a sketch extraction method based on hyperspectral image and deep learning. Firstly, the hyperspectral image data is collected and the bands sensitive to the sketches are extracted by a prior knowledge of the sketches (e.g. near infrared bands will be chosen if the sketches are made of carbon ink), and a dataset including a large number of existing natural images is used to pre-train the bi-directional cascade network (BDCN). The network parameters in the model are then fine-tuned by using the images of painted cultural objects drawn by experts, so as to solve the problem of insufficient sketch dataset of painted cultural objects and enhance the generalization ability of the model. Finally, the U-net network is used to further suppress the noise, i.e. unwanted information, and make the sketch clearer. The experimental results show that the proposed method can not only effectively extract sketch from ideal data, but also extract clear sketches from data with faded sketches and even with noise interference. It is superior to the other six advanced based on edge detection methods in visual and objective comparison, and has a good application prospect. The proposed deep learning method is also compared with an unsupervised clustering method using Self-Organising Map (SOM) which is a ‘shallow learning’ method where pixels of similar spectra are grouped into clusters without the need for data labeling by experts.

Item Type:

Chapter in book

Description:

Paper presented at Optics for Arts, Architecture, and Archaeology VIII, 1-26 June 2021.

Creators:

Zhang, Q., Cui, S., Liu, L., Wang, J., Wang, J., Zhang, E., Peng, J., Kogou, S., Liggins, F. and Liang, H.

Publisher:

SPIE

Place of Publication:

Washington

Date:

June 2021