Abstract

The task of reflection symmetry detection remains challenging due to significant variations and ambiguities of symmetry patterns in the wild. Furthermore, since the local regions are required to match in reflection for detecting a symmetry pattern, it is hard for standard convolutional networks, which are not equivariant to rotation and reflection, to learn the task. To address the issue, we introduce a new convolutional technique, dubbed the polar matching convolution, which leverages a polar feature pooling, a self-similarity encoding, and a systematic kernel design for axes of different angles. The proposed high-dimensional kernel convolution network effectively learns to discover symmetry patterns from real-world images, overcoming the limitations of standard convolution. In addition, we present a new dataset and introduce a self-supervised learning strategy by augmenting the dataset with synthesizing images. Experiments demonstrate that our method outperforms state-of-the-art methods in terms of accuracy and robustness.

Motivation: Convolutional descriptor is not suitable for reflective matching.
(a) matching with convolutional descriptor (b) matching with self-similarity descriptor

Figure 1. Reflective matching with (a) convolutional descriptors and (b) self-similarity descriptors for symmetry detection. Grids represent feature maps, and red squares denote corresponding regions. While the convolutional descriptors are required to be entirely invariant for both reflection and rotation, the self-similarity descriptor is preserved in reflection if each similarity value is invariant to reflection and roation. Note that this invariance requirement on pairwise similarities is weaker than the original invariance requirement on individual features. The red grid in (b) represents a self-similarity descriptor where each position encodes its similarity value to the center.

Overall architecture of Polar Matching Convolution Network

Figure 2. Given an input image I, a base feature F is computed by a feature encoder ENC. We transform the base feature F to a polar self-similarity descriptor P. Then, the polar matching convolutions, PMCF and PMCP, compute the symmetry scores, SF and SP, for the features, F and P, respectively. The final prediction is obtained by applying a convolutional decoder DEC after combining the scores SF and SP, and the base feature F.

Polar Matching Convolution (PMCF) and PMC with self-similarity (PMCP)

Figure 3. Illustration of the Polar Matching Convolution (PMCF). (a) The polar region descriptor ZF is sampled from the given base feature F with the number of sampling angles Mang and radii Mrad. (b) The intra-region correlation CF are computed using the polar region descriptor ZF. (c) The reflective matching kernel KF is applied to CF to compute the symmetry score tensor SF. The matching feature pairs are indicated with black dotted lines. Note that Naxi is the number of the candidate axes.

Figure 4. Illustration of PMC with self-similarity (PMCP). (a) The polar self-similarity P contains self-similarity values of the neighborhood pixels sampled with Nang and radii Nrad. The polar region descriptor ZP is then sampled from the polar self-similarity P with the sampling angles Mang and radii Mrad. (b) The reflective matching kernel KP extracts the relevant angle-wise relation pairs for Naxi candidate axes. The kernel can be decomposed to the lower-dimensional kernels to tackle the relations within the polar region descriptor and the polar self-similarity descriptor.

Comparison to the state-of-the-art methods

Figure 5. Precision-Recall curve on the SDRW [23] symmetry detection dataset. The (recall, precision) point of the maximum F1-score is indicated with dots.

Qualitative results of PMCNet

Figure 6. Qualitative results from the test splits of (a) SDRW and (b) LDRS datasets.

Acknowledgements

This work was supported by Samsung Electronics Co., Ltd. (IO201208-07822-01) and the IITP grants (No.2019-0-01906, AI Graduate School Program - POSTECH) (No.2021-0-00537, Visual common sense through self-supervised learning for restoration of invisible parts in images) funded by Ministry of Science and ICT, Korea.

Paper

Learning to Discover Reflection Symmetry via Polar Matching Convolution
Ahyun Seo*, Woohyeon Shim*, Minsu Cho
ICCV, 2021
[paper] [Bibtex]

Code

Check our GitHub repository: [GitHub]