The Promise of AI-Generated Video Designs for Survey Experiments in International Relations: An Experimental Test

Michal Smetana, Marek Vranka, Lauren Sukin, Ondřej Rosendorf

Abstract: Generative AI tools promise to make IR survey experiments more immersive by delivering treatments through realistic videos. But would these novel designs offer real methodological gains over traditional text-based vignettes? To address this question, we conducted a preregistered experiment testing the relative effectiveness of four “communication modalities”: an AI-generated video, long text with AI-generated images, plain long text, and a short-text vignette. Contrary to our expectations, the AI-generated video did not elicit the strongest affective responses nor the largest shifts in policy preferences; its effects were statistically indistinguishable from plain long-form text and even marginally weaker than the text-with-images treatment. All three long-form modalities outperformed the short vignette across all measures except attentiveness. The findings suggest that we should be cautious about the promise of AI-generated videos for IR experiments, whereas designs combining engaging text-based descriptions with AI-generated images may offer a cost-effective alternative to the traditional short plain-text vignettes.

Keywords: survey experiment, experimental design, communication modality, generative AI, foreign policy attitudes, ecological validity