Autoencoders
Variational AE, Disentangled VAE
Embedded Files
Autoencoder is a type of neural networks that learn efficient representation of raw unsupervised data.
Problem it solves: dimensionality reduction problem
Solution: train encoder and decoder networks to compress input data then restore it with minimum or even no loss of information
Typical autoencoder architecture
Autoencoder use cases
Autoencoder use cases
- Denoising
Idea: Train autoencoder to denoise the images:
Given dataset, add white noise -> noisy dataset
use noisy data set to encode & decode
compute loss between AE output and initial dataset
Denoising AE architecture
Example of MNIST denoising
2. Anomaly detection
2. Anomaly detection
Idea: we pass the input data through pre-trained AE and check the loss.
If the loss is too high than usual treshhold, we consider data as anomaly.
Note: Typically anomaly detection is a part of ML pipeline to remove outliers.
Example 1: We learn to classify cats vs dogs. We are given image of a car.
We will prefer to skip this image instead of training model on it.
3. Neural inpaiting
3. Neural inpaiting
Idea: similar as Denoising, we corrupt image by croping some regiong, then we train AE to restore the missing part.
4. Vector math of patterns
4. Vector math of patterns
Latnt space allows us to encode the input into vectors and recognize the patterns.
A well trained AE can manipulate with concepts like "Smile" or "Glasses" and add or substract patterns from image using vector math.
Variational autoencoders (VAE)
Variational autoencoders (VAE)
Drawback of simple autoencoder:
bad representation of latent space
bad sampling from latent space
Idea of VAE: instead of mapping input data to a fixed vector, VAE maps input data to a distribution
AE latent space
VAE latent space
Typical VAE archietcture
Note:
Loss now has two parts:
Expectation of MSE loss
KL divergence (to ensure the distribution is close to normal)
Reparametrization trick to allow backpropagation:
we don't learn the noise (no derivative for gaussian noise)
we do learn distribution parameters (propagation)
Disentangled variational autoencoders
Disentangled variational autoencoders
Idea: different neurons in VAE are uncorrelated, each one represents some particular data feature.
Result: Perturbing one latent variable (while other are fixed) corresponds to an interpretable changes in output
Example: apply DVAE in RL to find sparse rewards by running trainings inside the DVAE-reduced latent space
Page updated
Google Sites
Report abuse