Secure Image Generation with Stable Diffusion: AWS AI Content Moderation

Generative AI solutions, while offering immense power and utility, also present vulnerabilities to manipulation and abuse. For customers employing these technologies for image generation, prioritizing content moderation is paramount to safeguarding users, the platform, and the brand's reputation. Implementing robust moderation practices is essential for creating a safe and positive user experience and for protecting the integrity of the platform or brand. This article delves into how AWS AI services, specifically Amazon Rekognition and Amazon Comprehend, can be used in conjunction with other techniques to effectively moderate content generated by models like Stable Diffusion, achieving near real-time review.

The proposed solution involves creating a RESTful proxy API for moderating images generated by Stable Diffusion. This architecture leverages SageMaker JumpStart to deploy a Stable Diffusion model (v2-1-base). The system incorporates negative prompts, text moderation solutions like Amazon Comprehend, and a rule-based filter for prompt moderation. Additionally, Amazon Rekognition is employed to moderate the generated images. If any unsafe information is detected, the RESTful API returns the generated image along with moderation warnings to the client. The workflow begins with a user submitting a prompt. An AWS Lambda function then orchestrates image generation and moderation using Amazon Comprehend, SageMaker JumpStart, and Amazon Rekognition. This includes applying rule-based conditions for forbidden words, using a custom Amazon Comprehend classifier for toxicity analysis, sending prompts to the Stable Diffusion model via a SageMaker endpoint (including negative prompts), and submitting generated image bytes to Amazon Rekognition's DetectModerationLabel API for image moderation. Finally, a response message containing image bytes and any detected warnings is sent back to the client. A sample application demonstrates this architecture, blurring generated images containing unsafe content.

A common method for text moderation involves a rule-based search to identify forbidden words or phrases from a predefined list. This approach is simple, has minimal performance impact, and is cost-effective. However, its primary limitation is its inability to detect new or modified variations of forbidden words not present in the list, and users can attempt to bypass rules through alternative spellings or special characters. To overcome these limitations, a hybrid approach combining rule-based keyword searching with ML-based toxicity detection is often adopted. This solution utilizes a custom Amazon Comprehend Classifier to train a toxicity detection model. This model analyzes prompts for potentially harmful content even when no explicit forbidden words are found, leveraging machine learning to recognize patterns indicative of toxicity. Amazon Comprehend simplifies training and inference, allowing for custom classification training and deployment in just two steps. For more specific text moderation needs, tailoring the training dataset to your specific requirements is recommended.

Fine-tuning is a common technique to adapt pre-trained models for specific tasks, such as generating images with particular objects, styles, or characters. When fine-tuning Stable Diffusion, content moderation is critical to prevent the creation of inappropriate or offensive images. This involves carefully reviewing and filtering all data that could lead to such outputs, ensuring the model learns from a broader, more representative dataset and improves accuracy while preventing the dissemination of harmful content. SageMaker JumpStart simplifies the fine-tuning of Stable Diffusion models by providing transfer learning scripts using the DreamBooth method. Users need to prepare their training data, define hyperparameters, and initiate the training job. The training dataset must reside in a single Amazon S3 bucket, including images and a configuration file (dataset_info.json) linking images to instance prompts. For large-scale projects, manually filtering images can be impractical. In such cases, automating a batch process using Amazon Rekognition's DetectModerationLabel API can centrally flag or remove images that might skew training data.

In summary, this article presented a sample solution for moderating both prompts and output images generated by Stable Diffusion using Amazon Comprehend and Amazon Rekognition. The integration of negative prompts further enhances content safety. By implementing multiple layers of moderation, the risk of producing unsafe content is significantly reduced, ensuring a safer and more reliable user experience. AWS offers further resources on content moderation and ML use cases to help streamline moderation operations.