Evaluating Generative AI Safeguards: Preventing Harmful Public Health Images and Videos

The research employed a cross-sectional observational study design to assess the safeguards of leading generative AI applications. Ten prominent text-to-image models and two text-to-video models were selected for evaluation. The study focused on five specific public health themes identified as prevalent and capable of causing demonstrable harm. For each theme, researchers developed ten paraphrased prompts to test the AI's response. These prompts were submitted in duplicate to image models and once to video models to account for randomness and consistency. Two independent reviewers classified the generated outputs as potentially harmful or not, with a third reviewer resolving any discrepancies. Statistical analyses, including chi-squared tests, were used to identify significant differences in output generation rates across themes and models.

The evaluation of ten text-to-image models revealed a stark contrast in their ability to prevent the generation of harmful content. Across 1000 prompt submissions, a significant 52% (521 images) were classified as potentially harmful to public health. The performance varied dramatically between models. Reve and Stability AI Dream Studio exhibited the highest rates of harmful image generation, with 98% and 89% of their submissions, respectively, resulting in harmful content. Other models like Ideogram, Midjourney, Gemini, Recraft, and Flux AI Image Generator also showed substantial rates of harmful output, ranging from 52% to 70%. In contrast, Meta AI and Adobe Firefly demonstrated lower rates of harmful image generation (21% and 9%, respectively). Notably, ChatGPT 4o was the only model that did not generate any images classified as potentially harmful, successfully refusing all 100 submitted prompts.

In addition to image generation, the study conducted exploratory evaluations of two text-to-video AI models: Sora (OpenAI) and Flow (Google). A total of 100 video prompt submissions were made across the five public health themes. The results indicated that 52% of outputs from Sora were classified as potentially harmful. Flow demonstrated a slightly better performance, with 30% of its generated videos deemed potentially harmful. While these findings are preliminary due to the exploratory nature of the video evaluation, they suggest that text-to-video models may also present significant risks for generating public health-damaging content, mirroring some of the concerns observed with image generation models.

The findings of this study have profound implications for the development and deployment of generative AI. The significant proportion of potentially harmful content generated by leading AI models underscores a critical deficiency in current AI safeguards. The variability across models and themes suggests that AI safety is not a uniform standard and requires continuous improvement and adaptation. The study highlights the urgent need for greater transparency from AI developers regarding their safety mechanisms and content moderation policies. Furthermore, it calls for robust oversight from regulatory bodies to ensure that AI technologies do not become vectors for public health misinformation, the promotion of unhealthy behaviors, or the reinforcement of harmful stereotypes. Proactive measures are essential to mitigate the potential negative impacts of generative AI on public health.