[RL basics] Week 4. Deep Q-learning

So far, we trained Q-table on relatively simple envs. But Q-table becomes ineffective in large state space environments.

Problem: tabular methods are not scalable.

Idea: instead of training Q-table for each state ☓ action, approximate the Q-value using parametrized Q function (ex.Neural Networks)

DQN Instability tricks

DQN might suffer from instability as we combine non-linearity (NN) with bootstraping (update NN on existing estimations and not ground truth).

To help stabilize the network we apply:

1. Experience replay : stores experience tuples to be later sampled within mini-bathces (size N is a hyperparameter)

   • • reuse and learn from particular experience multiple times (without cost of new sampling)

     • reduce correlation between sequential samples (avoid forgetting previous experience, thus avoid weights overwriting )

2. Fixed Q-target : prevent moving rarget optimization:

We want to reduce the error between target and prediction. By updating weights, predictions become closer to initial target (Good), but new weights also make new target move away from initial target, thus increasing the error

2. • we use a separate "target" network to fix the target. Every C step (hyperparameter), we merge learned weights to "target" netwrok.

3. Double Deep Q-learning (read more)

Problem: Over-estimation of Q-values