Overview
This work investigates modern Generative Adversarial Networks (GANs) that employ adversarial supervision on intermediate generator outputs, often interpreting this multi-stage synthesis as a coarse-to-fine hierarchical generation process. The research challenges this interpretation, positing that standard scale-wise adversarial supervision does not inherently establish a proper coarse-to-fine hierarchy.
Research Context
The core issue identified in existing GAN architectures with multi-stage adversarial supervision is that each intermediate image is independently pushed towards the real data distribution at its specific resolution. This scale-wise realism, however, does not guarantee that outputs across different generation stages represent the identical generated sample. Furthermore, the scale-specific image produced at an earlier stage is not explicitly utilized as a refinement target for subsequent stages. Consequently, an adversarial loss applied at an intermediate stage can enhance a scale-specific output without compelling later stages to maintain the same sample trajectory. This can lead to later stages deviating towards a different sample rather than refining the preceding output. This phenomenon is termed the cross-scale trajectory misalignment problem.
Approach
To address the identified cross-scale trajectory misalignment problem, the researchers propose a method called CAT, which stands for Cross-scale Aligned Transformer. CAT is designed for multi-scale adversarial generation. The architecture of CAT maintains a scale-wise discriminator, meaning each intermediate output is evaluated at its corresponding resolution. A key component of CAT is the introduction of a generator-side consistency regularization. This regularization mechanism is specifically designed to align intermediate outputs with the final output of the generative process.
Findings
The proposed CAT architecture was evaluated. On the class-conditional ImageNet-256 dataset, the CAT-H/2 variant achieved a Fréchet Inception Distance (FID-50K) score of 1.56. This result was obtained after 60 training epochs and with a one-step inference process. The performance of CAT-H/2, as measured by FID-50K, demonstrated an improvement over various one-step GAN baselines and diffusion/flow baselines.