Teaching Tailored to Talent: Adverse Weather Restoration via Prompt Pool and Depth-Anything Constraint

Sixiang Chen1    Tian Ye1    Kai Zhang1    Zhaohu Xing1    Yunlong Lin2    Lei Zhu1,3 ✉️   
1The Hong Kong University of Science and Technology (Guangzhou)    2Xiamen University   
3The Hong Kong University of Science and Technology   
European Conference on Computer Vision (ECCV), 2024, MiCo Milano
Clean Image
Degraded Image
Clean Image 2
Degraded Image 2
Left: Input Image, Right: Our Result

Abstract

Recent advancements in adverse weather restoration have shown potential, yet the unpredictable and varied combinations of weather degradations in the real world pose significant challenges. Previous methods typically struggle with dynamically handling intricate degradation combinations and carrying on background reconstruction precisely, leading to performance and generalization limitations. Drawing inspiration from prompt learning and the "Teaching Tailored to Talent" concept, we introduce a novel pipeline, T3-DiffWeather. Specifically, we employ a prompt pool that allows the network to autonomously combine sub-prompts to construct weather-prompts, harnessing the necessary attributes to adaptively tackle unforeseen weather input. Moreover, from a scene modeling perspective, we incorporate general prompts constrained by Depth-Anything feature to provide the scene-specific condition for the diffusion process. Furthermore, by incorporating contrastive prompt loss, we ensures distinctive representations for both types of prompts by a mutual pushing strategy. Experimental results demonstrate that our method achieves state-of-the-art performance across various synthetic and real-world datasets, markedly outperforming existing diffusion techniques in terms of computational efficiency.

Observations

Method Overview
Figure 1. t-SNE visualization of different feature distributions. (a). Scenes with different contents also have significant commonalities compared to degradations. And there are some differences and commonalities between degradations and degradations. (b). Degradation residuals can represent degradations to a certain extent and be distinguished from the background.
Sub-Prompt Selection
Weather-Prompt Visualization
Left Figure: The selection frequency of sub-prompts. Some similar selection frequencies reflect the network's ability to adaptively exploit common attributes in some similarity between tasks (e.g. rain and raindrop). At the same time, the unique prompt frequencies highlight the flexibility to adapt to the specific characteristics of each weather condition.
Mid Figure: t-SNE visualization of weather-prompts for different weather conditions.
Right Figure: Motivation of Depth-Anything as a constraint. Depth-Anything has degradation-independent performance, and the intermediate features have better robustness than the previous pre-trained network.

Method

Method Overview
Figure 3. The overview of proposed method. (a) showcases our pipeline, which adopts an innovative strategy focused on learning degradation residual and employs the information-rich condition to guide the diffusion process. (b) illustrates the utilization of our prompt pool, which empowers the network to autonomously select attributes needed to construct adaptive weather-prompts. (c) depicts the general prompts directed by Depth-Anything constraint to supply scene information that aids in reconstructing residuals. (d) shows the contrastive prompt loss, which exerts constraints on prompts driven by two distinct motivations, enhancing their representations.

Quantitative Comparison

Left Image

Visual Comparison

Image 1 Image 2 Image 3 Image 4 Image 5 Image 6
Scroll ➜
Scroll horizontally to view more images
Clean Image
Degraded Image
Clean Image 2
Degraded Image 2
Left: Input Image, Right: Our Result
Clean Image
Degraded Image
Clean Image 2
Degraded Image 2
Left: Input Image, Right: Our Result

Citation

@InProceedings{chen2024teaching,
    title     = {Teaching Tailored to Talent: Adverse Weather Restoration via Prompt Pool and Depth-Anything Constraint},
    author    = {Chen, Sixiang and Ye, Tian and Zhang, Kai and Xing, Zhaohu and Lin, Yunlong and Zhu, Lei}, 
    booktitle = {European conference on computer vision},
    year      = {2024},
    organization={Springer}
}