Painting style transfer for head portraits using convolutional neural networks

Ahmed Selim, Mohamed Elgharib, Linda Doyle

Research output: Contribution to journalArticle

72 Citations (Scopus)

Abstract

Head portraits are popular in traditional painting. Automating portrait painting is challenging as the human visual system is sensitive to the slightest irregularities in human faces. Applying generic painting techniques often deforms facial structures. On the other hand portrait painting techniques are mainly designed for the graphite style and/or are based on image analogies; an example painting as well as its original unpainted version are required. This limits their domain of applicability. We present a new technique for transferring the painting from a head portrait onto another. Unlike previous work our technique only requires the example painting and is not restricted to a specific style. We impose novel spatial constraints by locally transferring the color distributions of the example painting. This better captures the painting texture and maintains the integrity of facial structures. We generate a solution through Convolutional Neural Networks and we present an extension to video. Here motion is exploited in a way to reduce temporal inconsistencies and the shower-door effect. Our approach transfers the painting style while maintaining the input photograph identity. In addition it significantly reduces facial deformations over state of the art.

Original languageEnglish
Article numbera129
JournalACM Transactions on Graphics
Volume35
Issue number4
DOIs
Publication statusPublished - 11 Jul 2016

Fingerprint

Painting
Neural networks
Graphite
Textures
Color

Keywords

  • Deformations
  • Gain maps
  • Gatys
  • NPAR
  • Painting transfer
  • Portrait
  • Spatial constraints
  • VGG
  • Video

ASJC Scopus subject areas

  • Computer Graphics and Computer-Aided Design

Cite this

Painting style transfer for head portraits using convolutional neural networks. / Selim, Ahmed; Elgharib, Mohamed; Doyle, Linda.

In: ACM Transactions on Graphics, Vol. 35, No. 4, a129, 11.07.2016.

Research output: Contribution to journalArticle

@article{8dd31bfd42574847a45616a58a2b214a,
title = "Painting style transfer for head portraits using convolutional neural networks",
abstract = "Head portraits are popular in traditional painting. Automating portrait painting is challenging as the human visual system is sensitive to the slightest irregularities in human faces. Applying generic painting techniques often deforms facial structures. On the other hand portrait painting techniques are mainly designed for the graphite style and/or are based on image analogies; an example painting as well as its original unpainted version are required. This limits their domain of applicability. We present a new technique for transferring the painting from a head portrait onto another. Unlike previous work our technique only requires the example painting and is not restricted to a specific style. We impose novel spatial constraints by locally transferring the color distributions of the example painting. This better captures the painting texture and maintains the integrity of facial structures. We generate a solution through Convolutional Neural Networks and we present an extension to video. Here motion is exploited in a way to reduce temporal inconsistencies and the shower-door effect. Our approach transfers the painting style while maintaining the input photograph identity. In addition it significantly reduces facial deformations over state of the art.",
keywords = "Deformations, Gain maps, Gatys, NPAR, Painting transfer, Portrait, Spatial constraints, VGG, Video",
author = "Ahmed Selim and Mohamed Elgharib and Linda Doyle",
year = "2016",
month = "7",
day = "11",
doi = "10.1145/2897824.2925968",
language = "English",
volume = "35",
journal = "ACM Transactions on Graphics",
issn = "0730-0301",
publisher = "Association for Computing Machinery (ACM)",
number = "4",

}

TY - JOUR

T1 - Painting style transfer for head portraits using convolutional neural networks

AU - Selim, Ahmed

AU - Elgharib, Mohamed

AU - Doyle, Linda

PY - 2016/7/11

Y1 - 2016/7/11

N2 - Head portraits are popular in traditional painting. Automating portrait painting is challenging as the human visual system is sensitive to the slightest irregularities in human faces. Applying generic painting techniques often deforms facial structures. On the other hand portrait painting techniques are mainly designed for the graphite style and/or are based on image analogies; an example painting as well as its original unpainted version are required. This limits their domain of applicability. We present a new technique for transferring the painting from a head portrait onto another. Unlike previous work our technique only requires the example painting and is not restricted to a specific style. We impose novel spatial constraints by locally transferring the color distributions of the example painting. This better captures the painting texture and maintains the integrity of facial structures. We generate a solution through Convolutional Neural Networks and we present an extension to video. Here motion is exploited in a way to reduce temporal inconsistencies and the shower-door effect. Our approach transfers the painting style while maintaining the input photograph identity. In addition it significantly reduces facial deformations over state of the art.

AB - Head portraits are popular in traditional painting. Automating portrait painting is challenging as the human visual system is sensitive to the slightest irregularities in human faces. Applying generic painting techniques often deforms facial structures. On the other hand portrait painting techniques are mainly designed for the graphite style and/or are based on image analogies; an example painting as well as its original unpainted version are required. This limits their domain of applicability. We present a new technique for transferring the painting from a head portrait onto another. Unlike previous work our technique only requires the example painting and is not restricted to a specific style. We impose novel spatial constraints by locally transferring the color distributions of the example painting. This better captures the painting texture and maintains the integrity of facial structures. We generate a solution through Convolutional Neural Networks and we present an extension to video. Here motion is exploited in a way to reduce temporal inconsistencies and the shower-door effect. Our approach transfers the painting style while maintaining the input photograph identity. In addition it significantly reduces facial deformations over state of the art.

KW - Deformations

KW - Gain maps

KW - Gatys

KW - NPAR

KW - Painting transfer

KW - Portrait

KW - Spatial constraints

KW - VGG

KW - Video

UR - http://www.scopus.com/inward/record.url?scp=84980004921&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84980004921&partnerID=8YFLogxK

U2 - 10.1145/2897824.2925968

DO - 10.1145/2897824.2925968

M3 - Article

AN - SCOPUS:84980004921

VL - 35

JO - ACM Transactions on Graphics

JF - ACM Transactions on Graphics

SN - 0730-0301

IS - 4

M1 - a129

ER -