One of the key contribution of EnHerit was new approaches to discover copies in artworks and historical documents collections. Many examples, code and details are available on the ArtMiner and SegSwap webpages.
Our initial goal was to discover repeated details in Brueghel’s workshop artworks collected by Elizabeth Honig http://janbrueghel.net. We also developed a case study on a larger temporal and spatial scale on a large database of Venus depictions with Béatrice Joyeux-Prunel and K. Bender. Since then, we used the used these algorithms successfully in many contexts, for example to analyse scientific illustrations circulations in manuscripts.
From a Computer Vision point of view, our main contribution were:
- for ArtMiner, a new unsupervised approach to learn deep features specifically for matching spatially consistent patterns across depiction styles.
- For SegSwap, a new training procedure based on synthetic data to teach recent Transformer style architecture to detect repeated patterns
We also developed RANSAC-Flow, a method to compute precise displacement flows between similar artworks, giving insights into the copy process, and allowing better the copied details.
Publications:
Discovering Visual Patterns in Art Collections with Spatially-consistent Feature Learning,
Xi Shen, Alexei Efros and Mathieu Aubry, Proceedings IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019,
PDF, Project page, code.
The Burgeoning Computer-Art Symbiosis,
Shiry Ginosar, Xi Shen, Karan Dwivedi, Elizabeth Honig, and Mathieu Aubry, XRDS: Crossroads, The ACM Magazine for Students – Computers and Art, PDF
RANSAC-Flow: generic two-stage image alignment
Xi Shen, François Darmon, Alexei Efros and Mathieu Aubry,
European Conference on Computer Vision (ECCV) 2020
PDF, webpage, code
Learning Co-segmentation by Segment Swapping for Retrieval and Discovery
X. Shen, A. Efros, A. Joulin, M. Aubry
ArXiv 2021, CVPR 2022 workshops
Download pdf | View project web page
Image Collation: Matching illustrations in manuscripts
R. Kaoua, X. Shen, A. Durr, S. Lazaris, D. Picard, M. Aubry
ICDAR 2021
Download pdf | View project web page
Spatially-consistent Feature Matching and Learning for Heritage Image Analysis
X. Shen, R. Champenois, S. Ginosar, I. Pastrolin, M. Rousselot, O. Bounou, T. Monnier, S. Gidaris, F. Bougard, P.-G. Raverdy, M.-F. Limon, C. Bénévent, M. Smith, O. Poncet, K. Bender, B. Joyeux-Prunel, E. Honig, A. Efros, M. Aubry
IJCV 2022
Download pdf | View project web page