Dr.3D:
Adapting 3D GANs to Artistic Drawings

Abstract

While 3D GANs have recently demonstrated the high-quality synthesis of multi-view consistent images and 3D shapes, they are mainly restricted to photo-realistic human portraits. This paper aims to extend 3D GANs to a different, but meaningful visual form: artistic portrait drawings. However, extending existing 3D GANs to drawings is challenging due to the inevitable geometric ambiguity present in drawings. To tackle this, we present Dr.3D, a novel adaptation approach that adapts an existing 3D GAN to artistic drawings. Dr.3D is equipped with three novel components to handle the geometric ambiguity: a deformation-aware 3D synthesis network, an alternating adaptation of pose estimation and image synthesis, and geometric priors. Experiments show that our approach can successfully adapt 3D GANs to drawings and enable multi-view consistent semantic editing of drawings.

Fast-Forward

Deformation-aware 3D Synthesis Network

Network architecture of a deformation-aware 3D synthesis network. The network consists of a deformation network, a mapping network, a feature generator and a volume rendering module. The network takes latent codes $z_d$ and $z$, and a camera pose parameter $\theta$ as inputs, and synthesizes an image in a multi-view consistent way.

Alternating Adaptation and Geometric Priors

Our approach alternatingly adapts the deformation-aware 3D synthesis network and the pose-estimation network. In order to guide the alternating adaptation process to a proper solution at early iterations, we introduce depth similarity loss $\mathcal{L}_d$, normal smoothness loss $\mathcal{L}_n$, and pose loss $\mathcal{L}_p$.

Results

Figure 1: 3D-aware drawing synthesis results by Dr.3D.

Figure 2: Drawing synthesis with different yaw angles.

Figure 3: Semantic editing of input drawings.

Video 1: Novel view synthesis of an input drawing.

Citation

 @inproceedings{Jin2022Dr3D,
  title     = {Dr.3D: Adapting 3D GANs to Artistic Drawings},
  author    = {Wonjoon Jin,Nuri Ryu,Geonung Kim,Seung-Hwan Baek,Sunghyun Cho},
  booktitle = {Proceedings of the ACM (SIGGRAPH Asia)},
  year      = {2022}}