Abstract

Robust visual recognition under adverse weather conditions is of great importance in real-world applications. In this context, we propose a new method for learning semantic segmentation models robust against fog. Its key idea is to consider the fog condition of an image as its style and close the gap between images with different fog conditions in neural style spaces of a segmentation model. In particular, since the neural style of an image is in general affected by other factors as well as fog, we introduce a fog-pass filter module that learns to extract a fog-relevant factor from the style. Optimizing the fog-pass filter and the segmentation model alternately gradually closes the style gap between different fog conditions and allows to learn fog-invariant features in consequence. Our method substantially outperforms previous work on three real foggy image datasets. Moreover, it improves performance on both foggy and clear weather images, while existing methods often degrade performance on clear scenes.

Overall Archiecture of FIFO

Figure 1. Overall pipeline of FIFO. For each iteration of training, the fog-pass filtering module and the segmentation network are updated alternately. (top) Given Gram matrices of feature maps of the segmentation network as input, the fog-pass filtering module learns to extract fog factors so that fog conditions of images are discriminated by their fog factors. (bottom) The segmentation network is trained by reducing the gap between fog factors of images with different fog conditions as well as by the segmentation loss. Note that the fog-pass filters are auxiliary modules used only in training.

Empricial Verification

1. Impact of Fog-pass Filtering Modules

Figure 2. Empirical analysis on the impact of FIFO. (a) 2D visualization of distributions of Gram matrices and their fog factors. (b) Comparison between the quality of k-means clustering of the Gram matrices and that of the corresponding fog factors in adjusted Rand index. (c) The fog-style gap between different domains before and after training with FIFO, where the gap is measured by the average Hausdorff distance between two sets of fog factors.

2. Fog-invariance Learned by FIFO

Figure 3. Images reconstructed by the baseline, a variant of FIFO closing the gap between Gram matrices, and FIFO.

Quantitative Results

1. Performance Comparison with Other Methods

Table 1. Quantitative results in mean intersection over union (mIoU) on three real foggy datasets—Foggy Zurich (FZ) test v2, Foggy Driving Dense (FDD), Foggy Driving (FD), and a clear weather dataset—Cityscapes lindau 40.

2. Ablation Studies

Table 2. (a) Analysis on the impact of domain pairs. CW, SF and RF denote clear weather, synthetic fog, and real fog, respectively. (b) Analysis on the impact of the fog style matching loss, the prediction consistency loss, and the fog-pass filtering modules.

3. Generalization to Other Weather Conditions

Table 3. (a) Quantitative results on Rainy Cityscapes (RC). (b) Performance (mIoU) versus the corruption severity. FIFO and baseline are evaluated on Frosty Cityscapes and Rainy Cityscapes. (c) Quantitative results on the ACDC dataset, following the unsupervised learning setting of the benchmark.

Qualitative Results

1. Qualitative Results on Foggy Datasets

Figure 4. Qualitative results on the real foggy datasets. (a) Input images. (b) Baseline. (c) FIFO without the fog-pass filtering. (d) FIFO. (e) Groundtruth.

2. Qualitative Results on Other Weather Conditions

Figure 5. Qualitative results under rain and frost conditions.

Acknowledgements

This work was supported by Samsung Research Funding & Incubation Center of Samsung Electronics under Project Number SRFC-IT1801-05.

Paper

FIFO: Learning Fog-invariant Features for Foggy Scene Segmentation
Sohyun Lee, Taeyoung Son, and Suha Kwak
CVPR, 2022
[arXiv] [Bibtex]

Code

Check our GitHub repository: [github]