Deep learning for the extraction of sketches from spectral images of historical paintings

Zhang, Q, Cui, S, Liu, L, Wang, J, Wang, J, Zhang, E, Peng, J, Kogou, S ORCID: https://orcid.org/0000-0003-1722-7626, Liggins, F and Liang, H ORCID: https://orcid.org/0000-0001-9496-406X, 2021. Deep learning for the extraction of sketches from spectral images of historical paintings. In: Liang, H ORCID: https://orcid.org/0000-0001-9496-406X and Groves, R, eds., Optics for arts, architecture, and archaeology VIII. Proceedings of SPIE, 11784 . Washington: SPIE. ISBN 9781510644021

Preview

Text 1454882_Liang.pdf - Published version Download (1MB) | Preview

Abstract

The sketches of painted cultural objects can be the most indicative of the style of paintings. Extraction of the sketches is an integral process used by conservators and art historians for documentation and for artists to learn historical painting styles through copying and painting. However, at present, sketch extraction is mainly manually drawn, which is not only time-consuming, but also subjective and dependent on experience. Therefore, both accuracy and efficiency need to be improved. In recent years, with the development of machine learning, a series of extraction methods based on edge detection have emerged. However, most of the existing methods can only perform successful extraction if the sketches are well preserved , but for the data with faded sketches or severe conservation issues, the extraction methods need to be improved. It is beneficial to extract the bands that accentuate the sketches while suppressing the effects of the degraded areas and the overlapping paints. We propose a sketch extraction method based on hyperspectral image and deep learning. Firstly, the hyperspectral image data is collected and the bands sensitive to the sketches are extracted by a prior knowledge of the sketches (e.g. near infrared bands will be chosen if the sketches are made of carbon ink), and a dataset including a large number of existing natural images is used to pre-train the bi-directional cascade network (BDCN). The network parameters in the model are then fine-tuned by using the images of painted cultural objects drawn by experts, so as to solve the problem of insufficient sketch dataset of painted cultural objects and enhance the generalization ability of the model. Finally, the U-net network is used to further suppress the noise, i.e. unwanted information, and make the sketch clearer. The experimental results show that the proposed method can not only effectively extract sketch from ideal data, but also extract clear sketches from data with faded sketches and even with noise interference. It is superior to the other six advanced based on edge detection methods in visual and objective comparison, and has a good application prospect. The proposed deep learning method is also compared with an unsupervised clustering method using Self-Organising Map (SOM) which is a ‘shallow learning’ method where pixels of similar spectra are grouped into clusters without the need for data labeling by experts.

Item Type:	Chapter in book
Description:	Paper presented at Optics for Arts, Architecture, and Archaeology VIII, 1-26 June 2021.
Creators:	Zhang, Q., Cui, S., Liu, L., Wang, J., Wang, J., Zhang, E., Peng, J., Kogou, S., Liggins, F. and Liang, H.
Publisher:	SPIE
Place of Publication:	Washington
Date:	June 2021
Volume:	11784
ISBN:	9781510644021
Identifiers:	Number Type 10.1117/12.2593680 DOI 1454882 Other
Rights:	© 2021 SPIE
Divisions:	Schools > School of Science and Technology
Record created by:	Laura Ward
Date Added:	30 Jul 2021 12:57
Last Modified:	01 Feb 2022 16:25
URI:	https://irep.ntu.ac.uk/id/eprint/43684

Actions (login required)

Edit View

Statistics

Views

Downloads