Fast Proxies for LLM Robustness Evaluation

Evaluating the robustness of LLMs to adversarial attacks is crucial for safe deployment, yet current red-teaming methods are often prohibitively expensive. We compare the ability of fast proxy metrics to predict the real-world robustness of an LLM against a simulated attacker ensemble. This allows us to estimate a model's robustness to computationally expensive attacks without requiring runs of the attacks themselves. Specifically, we consider gradient-descent-based embedding-space attacks, prefilling attacks, and direct prompting. Even though direct prompting in particular does not achieve high ASR, we find that it and embedding-space attacks can predict attack success rates well, achieving $r_p=0.87$ (linear) and $r_s=0.94$ (Spearman rank) correlations with the full attack ensemble while reducing computational cost by three orders of magnitude.

@article{beyer2025_2502.10487,
  title={ Fast Proxies for LLM Robustness Evaluation },
  author={ Tim Beyer and Jan Schuchardt and Leo Schwinn and Stephan Günnemann },
  journal={arXiv preprint arXiv:2502.10487},
  year={ 2025 }
}