AIGC-Based Image and Video Generation: Methods, Models, and Future Directions

Visual generation is becoming central to creative fields such as art and entertainment, and to critical domains such as medical imaging and digital publishing. Advances in generative AI promise to fundamentally change how people create and interact with visual data. This article reviews the development of AI-driven image and video generation, highlighting the models, methods, datasets, and open problems that define the current landscape.

Early visual synthesis relied on image matching or hand-crafted rules that produced only simple textures and structures. The deep learning era introduced variational autoencoders (VAEs) and generative adversarial networks (GANs), which learned typical image distributions such as faces and indoor scenes. Today, large diffusion models generate images and far more challenging videos with quality that can be difficult to distinguish from real content. Two factors drive this progress: the growth of training data, from CelebA's 200,000 low-resolution images to LAION-5B's 5.8 billion images, and advances in computing power, model architecture, and scale, which allow models to learn increasingly complex distributions and even produce visual concepts absent from the real world.

Generative models learn a mapping from a simple known distribution, such as a Gaussian, to the target data distribution, enabling sampling of new data at inference time. VAEs maximize a likelihood lower bound by encoding data into a regularized latent space and decoding samples back, with variants such as VQ-VAE using discrete latents. GANs train a generator and a discriminator adversarially; stability improvements such as LSGAN, WGAN, WGAN-GP, and spectral normalization address vanishing gradients and mode collapse. Autoregressive models factorize the joint distribution into sequential conditionals, as in PixelRNN, PixelCNN, and Transformer-based models like CogView, though their slow generation has limited visual use; VAM accelerates them with multi-scale prediction. Diffusion models progressively add Gaussian noise in a forward process and learn to reverse it. DDPM simplified training, while DDIM and DPM-Solver reduced sampling steps, and latent diffusion models (LDM) moved the process into a compressed latent space, greatly lowering computational cost. Architectures such as SDXL and DiT further improved quality and scalability.

Unconditional generation produces images from random noise and laid the groundwork for later conditional methods. DCGAN introduced fully convolutional GANs; ProgressiveGAN grew resolution gradually and added minibatch standard deviation to improve diversity; StyleGAN and its successors StyleGAN2 and StyleGAN3 introduced style-based generators with adaptive instance normalization, enabling interpretable latent spaces, interactive style mixing, and improved high-frequency detail. Class-conditional generation adds labels to control output: cGAN and ACGAN introduced conditioning, and BigGAN scaled class-conditional synthesis to 1,000 ImageNet categories. DiT replaces U-Net with a Transformer backbone and uses adaptive layer normalization to inject conditions efficiently. Text-to-image generation is the most widely adopted condition. Early GAN approaches such as GAN-INT-CLS and StackGAN generated images from text in multiple stages. Transformer-based models such as DALL-E and CogView treat text and image tokens uniformly. Diffusion-based systems, including GLIDE, Imagen, DALL-E 2, and LDM-based Stable Diffusion, use classifier-free guidance and cross-attention and have become dominant. Newer models such as SD3, PixArt, Hunyuan-DiT, and Flux.1 employ multimodal DiT architectures with further gains in fidelity and speed.

Image-to-image translation maps an input image to an output image and underlies tasks such as colorization, restoration, semantic-map-to-photo synthesis, and low-light enhancement. Supervised approaches such as pix2pix use paired data, a U-Net generator with skip connections to preserve low-level detail, a PatchGAN discriminator that focuses on local realism, and a combined adversarial and L1 loss. Because paired data is often unavailable, unsupervised methods such as CycleGAN exploit cycle consistency: an image translated to another domain and back should match the original. Recent diffusion-based methods perform controllable translation and editing by conditioning on an input image, offering higher generation quality and greater flexibility than earlier GAN pipelines.

Video generation extends image diffusion to the temporal dimension and is far more demanding in data and computation. Methods can be grouped by training data requirements. Data-driven approaches train large video diffusion models on extensive video corpora and produce high-quality, temporally coherent clips, though at high cost. One-shot approaches fine-tune a pretrained image or video model on a single video to perform editing or generation with the same motion or style. Zero-shot approaches leverage pretrained image diffusion models directly, injecting temporal consistency through attention sharing, latent alignment, or motion guidance, without any video-specific training. Each category trades off quality, controllability, data needs, and computational expense. Diffusion transformers capture complex temporal relationships and show particular promise for long-video generation and detail consistency.

Progress in visual generation depends on datasets such as CelebA for faces, ImageNet for categorical diversity, and LAION-5B for web-scale image-text pairs, along with video datasets used for training and benchmarking generative video models. Evaluation combines quantitative and qualitative measures. Common metrics include Inception Score and Fréchet Inception Distance for realism and diversity, CLIP-based scores for text-image alignment, and human preference studies for perceptual quality. Consistent evaluation protocols remain important for fair comparison across models.

Several challenges remain. Data acquisition is constrained by scale, quality, licensing, and bias in web-crawled corpora. Inference efficiency is a bottleneck for diffusion models, which require many sampling steps, though distillation and fast solvers help. Long-video generation demands sustained temporal consistency and coherent narratives over extended durations. Controllable video generation, including fine-grained control over motion, camera, and identity, is still developing. Security concerns, including deepfakes, copyright infringement, and misuse of generated content, require technical safeguards such as watermarking and detection, along with governance.

Promising directions include more efficient architectures and training recipes that lower computational cost, unified models that handle image, video, and 3D generation within a single framework, and improved temporal modeling for long, coherent videos. Better controllability through structured conditions such as layout, depth, pose, and trajectories will support practical creative and industrial workflows. Research on trustworthy generation, including detection, provenance, and privacy-preserving training, is equally important for responsible deployment.

AI-driven visual generation has advanced rapidly, moving from simple textures to photorealistic images and increasingly coherent videos. Diffusion models now lead in quality and flexibility, while GAN, autoregressive, and VAE methods continue to contribute key ideas. Addressing data, efficiency, long-horizon consistency, controllability, and security will determine how far these technologies can be responsibly extended into art, medicine, publishing, and beyond.