Stanford CS236: Deep Generative Models I 2023 I Lecture 9 - Normalizing Flows

()

Generative Adversarial Networks (GANs)

GANs use an adversarial process to train a generator and a discriminator network.
The generator network aims to produce realistic samples that can fool the discriminator, while the discriminator network tries to distinguish between real data and samples generated by the generator.
GANs offer more flexibility in defining the generative model family since the training objective is not based solely on maximum likelihood.
GANs have the advantage of not requiring likelihood evaluations, making them flexible in choosing the generator architecture.
Training GANs is challenging due to the Mini-Max optimization problem and convergence issues.

Two-sample tests are used to determine whether two groups of samples come from the same probability distribution.
A test statistic is used to compare the two groups of samples, and a threshold is set to determine whether to reject the null hypothesis.
Choosing a good test statistic is important to minimize type one and type two errors.
Instead of handcrafting a test statistic to compare two probability distributions, we can train a classifier (discriminator) to distinguish between samples from the data distribution and samples from the model distribution.
The loss of the classifier can be used as a test statistic, where a high loss indicates that the two distributions are different and a low loss indicates that they are similar.

The generator is trained to fool the discriminator by minimizing the objective function V.
The optimal discriminator is the conditional probability of a point x belonging to the positive class (real samples).
The K Divergence is minimized when P(Theta) = P, but in practice, we cannot reach the global optimum.
Among suboptimal models, it may be preferable to have one that cannot fool a discriminator over one with high compression.
A likelihood-based model can be used as a discriminator, but it defeats the purpose of not needing access to the likelihood.