TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Models

Official PyTorch implementation of the following paper:

TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Models,

Xin Wang¹, Kai Chen¹, Jiaming Zhang², Jingjing Chen¹, Xingun Ma^1
1Shanghai Key Lab of Intell. Info. Processing, School of CS, Fudan University
²Hong Kong University of Science and Technology

Environment Setup

To set up the required environment, please follow the installation instructions provided in the Train-Time APT repository.

Data Preparation

Before training or evaluating the models, you'll need to prepare the necessary datasets. Detailed instructions on downloading, preprocessing, and organizing the data can be found in DATASETS.md.

ADV_DIR can be generated using the Train-Time APT repository.

Training and Evaluation

This project provides scripts for test-time tuning and evaluating various prompt designs. You can find all scripts in the ./scripts directory.

Example Usage

Here are examples of how to run the plain TAPT baselines with a ViT-B/16 backbone in a zero-shot setting:

• TAPT VLI (TAPT V-L Independent Prompt):

  ./scripts/VLI/TAPT_VLI_0shots_b16.sh

• TAPT VLJ (TAPT V-L Joint Prompt):

  ./scripts/VLJ/TAPT_VLJ_0shots_b16.sh

• TAPT V (TAPT Vision Prompt):

  ./scripts/V/TAPT_V_0shots_b16.sh

MoE Variants

The TAME variants are the test-time counterparts of the train-time MoE schemes (MoEAdvIVLP / MoEAdvMaPLe / MoEAdvVPT in the MMoP repo).

• TAME VLI (MoE V-L Independent Prompt):

  ./scripts/TAMEVLI/TAME_VLI_0shots_b16.sh

• TAME VLJ (MoE V-L Joint Prompt):

  ./scripts/TAMEVLJ/TAME_VLJ_0shots_b16.sh

• TAME V (MoE Vision Prompt):

  ./scripts/TAMEV/TAME_V_0shots_b16.sh

Acknowledgement

This repository is built upon PromptAlign and TPT. Thanks for those well-organized codebases.

Citation

@inproceedings{wang2025tapt,
  title={TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Models},
  author={Wang, Xin and Chen, Kai and Zhang, Jiaming and Chen, Jingjing and Ma, Xingjun},
  booktitle={Proceedings of the Computer Vision and Pattern Recognition Conference},
  pages={19910--19920},
  year={2025}
}

@article{wang2026tame,
  title={TAME: Test-Time Adversarial Prompt Tuning via Mixture-of-Experts for Vision-Language Models},
  author={Wang, Xin and Wang, Yixu and Zhang, Jiaming and Wang, Ruofan and Yu, Jiaqi and Chen, Kai and Chen, Jingjing and Ma, Xingjun and Jiang, Yu-Gang},
  journal={arXiv preprint arXiv:2605.17577},
  year={2026}
}