Integrative Few-Shot Learning for Classification and Segmentation

Figure 1. The limitations of the conventional few-shot classification and few-shot segmentation setups. (a) & (b): FS-C presumes that the query always contains no more or less than one of the target classes in classification, while FS-S allows the presence of multiple classes but strictly assumes the query class set (C) and the support class set (C_s) are always equivalent. (c): FS-S learners are trained to segment a query image using a semantically-coupled support set thus often blindly highlight any salient objects regardless of support semantics. The proposed FS-CS learners are trained to predict the class presence as well as corresponding masks thus correctly discriminate what to segment based on the semantic relevance between the support.

Figure 2. Integrative task of few-shot classification and segmentation. FS-CS combines few-shot classification and segmentation while it generalizes them to more realistic episodes with arbitrary image pairs, where each target class may or may not be present in the query.

Figure 3. Integrative Few-Shot Learning (iFSL) for FS-CS. An iFSL learner estimates the class presence and the their corresponding class foreground maps from the class-wise shared foreground maps.

Figure 4. Overview of ASNet consisting of ASLayers. ASNet first constructs a hypercorrelation with image feature maps between a query (colored red) and a support (colored blue), where the 4D correlation is depicted as two 2D squares for demonstrational simplicity. ASNet then learns to transform the correlation to a foreground map by gradually squeezing the support dimension on each query dimension via global self-attention. Each input correlation, intermediate feature, and output foreground map has a channel dimension but is omitted in the illustration.

Table 1. Performance comparison of ASNet and others on FS-CS and Pascal-5ⁱ . All methods are trained and evaluated under the iFSL framework given strong labels, i.e., class segmentation masks, except for ASNet_w that is trained only with weak labels, i.e., class tags.

Table 2. FS-S results on 1-way 1-shot and 1-way 5-shot setups on Pascal-5ⁱ using ResNet50 (R50).

Figure 6. (a) FS-CS is extensible to a multi-class problem with arbitrary numbers of classes, while FS-S is not as flexible as FS-CS in the wild. (b) We evaluate the transferability between FS-CS, FS-C, and FS-S by training a model on one task and evaluating it on the other task.