Regression

• Linear regression (MSE minimization)

  • it’s 0L neural network (perceptron) with act(x) = x

• Ridge regression = Linear + regularization2

  • L2 prevents collinear weights from unjustified mutual explosion : x1 - x2 ~= 10e5 * x1 - 10e*5 * x2 

• Logistic regression → (Likelihood minimization)

  • perceptron with act(x) = sigmoid(x)

Bootstrap aggregation

• N - lenght of train data

• K<N - lenght of samples

• n - number of bins

1. We make n bins where we sample with repetition K train objects from N. 

2. On each bin we train network. We will have n networks.

3. we average network answers (Condorcet principle)

Note: Condorcet works well even with overfitted networks

Gradient descent (GD)

• GD works poorly with high/low coeff magnitude. It might be improved by normalizing coeff

Why SGD?

1. Efficiency: GD use entire dataset for a single step

2. Faster convergence: SGD is noisier, it can lead to faster initial convergence

3. Better Generalization: the noise by SGD helps in avoiding local minima / saddle points

4. Memory: SGD batch requires only few samples in RAM

Dropout

Freeze some neurons and learn data with remaining neurons 

• Idea: averaging the answers of many networks performs better. 

• Neuron should not rely a lot on the presence/absence of other neurons

• Network should not rely a lot on the presence/absence of some particular features

• Hiding some neurons it’s like train different networks

• Other neurons should learn to work independently

• forces network to learn more stable data representation

• Train loss can be worse than test (we perturb net during learning)

• model.predict() : do not apply dropout

<<<Code block>>>

self.dropout = nn.Dropout(p=0.5)

Normalization - is a type of regularization. 

It helps prevent overfitting by keeping coeff magnitude stable.

• Equally regularize feature weights

• improve stability by reducing model parameter sensibility

• We can now use bigger learning rate => faster learning

• We don’t want to regularize intercept. It’s spread between model and data

Batch norm: all layers

• Idea

  • Normalize by Mean and Var every batch

  • Learn new Mean and Var y =x+

• model.predict()

  • when predict (forward pass) we have 1 object (batch_size=1)

  • We use then general mean/var = batch norm moving average

<<<Code block>>>

self.bn = nn.BatchNorm1d(120)

Convolutional batch norm

• Every batch is set of 3D tensors [ 🧊, 🧊, 🧊… 🧊]

• We want image pattern (cat) to be spacial independent 

  • ✔️We average by batch & height & weight 

  • ❌Meaningless : (white_pixel + cat_pixel + white_pixel + white_pixel)/4

<<<Code block>>> 

self.convbn = nn.BatchNorm2d(16)

Quality function (metrics)

• Accuracy

  • number of misclassified objects

• Precision

• Recall

• F-score

• ROC-curve ROC AUC

• PR-curve

Gradient Vanishing 

Gradient Explosion

Inductive Transfer Learning

• target task is different from source task

• the model use the knowledge from the source task

Fine-tuning Algo: 

• • Take pretrained big network

  • Replace last layer(s) architecture (final layer = number of new classes)

  • Freeze first layers (low-level, normally conv layers)

  • Train on specific (small) data set

Why not just loss function, why do we need metrics?

Use loss function as performance is not always good as we are learning to minimize loss function. 

Metrics allows us to independently check the performance 

What is the difference between Loss function and Metric?

• Loss function is used to train. Metric - to evaluate.

• Loss must be differentiable. Metric - anything.

• metric can also be loss function if differentiable.