TransforMatcher: Match-to-Match Attention for Semantic Correspondence

Figure 1. The feature maps extracted from an image pair are used to compute a multi-channel correlation map to be processed by our match-to-match attention module for refinement. The multi-level scores are used as features for each match. We construct a dense flow field from the resulting correlation map, which can be used to transfer keypoints for training with keypoint pair annotation. We formulate our training objective to minimize the average Euclidean distance between the predicted target keypoints and the ground-truth target keypoints.

Figure 3. The multi-channel correlation map is projected to query, key and value matrices, which are multiplied with rotary positional embeddings. The match-to-match attention module exploits additive addition mechanisms to aggregate query/key matrices to global vectors, which is used for element-wise product to induce global context awareness. The final output is projected to a single-width channel to be reshaped to a refined 4D correlation map. As this additive attention structure has linear complexity (in comparison to vanilla transformers which have quadratic complexity), we can effectively model long-range match-to-match interactions within feasible computational constraints.

Table 1. Higher PCK is better. All the results reported in the table uses pretrained ResNet-101 model as the feature extractor. Methods in the first group were trained with weak supervision (image pair annotations), while those in the second group were trained with strong supervision (sparse keypoint match annotations). Models with * are retrained using keypoint annotations from ANC-Net. † indicates the use of data augmentation during training. Numbers in bold indicate the best performance, followed by the underlined numbers. It can be seen that TransforMatcher trained without data augmentations outperform CATs trained with data augmentations, proving the efficacy of our 4D match-to-match attention and multi-level correlation score features.

Figure 4. The source image is TPS-transformed to the target image using the predicted correspondences. The results show the transformed images. It can be seen that TransforMatcher can establish accurate correpsondences under challenging intra-class variations.

Figure 5. Nonlocality distributions of high-dimensional convolutional kernels (left) and TransforMatcher’s attention layers (right). Convolutional layers statically transforms maches with fixed, local receptive fields. In contrast, Transformatcher layers can dynamically transform matches by adaptively deciding regions of attention for effective transformation with global receptive fields.

Figure 6. Proportion of image pair difficulty with respect to nonlocality. For all difficulty types, the proportion of hard/medium samples increase with increasing nonlocality. This trend is especially visible in types of truncation/occlusion; our model attends larger contexts to better perceive truncated/occluded parts. Therefore, the harder the image pair is to match, the more TransforMatcher relies on nonlocal global cues to establish correspondences.