Autodiff Puzzles

• by Sasha Rush - srush_nlp

Click here to get started:

This notebook contains a series of self-contained puzzles for learning about derivatives in tensor libraries. It is the 3rd puzzle set in a series of puzzles about programming for deep learning (Tensor Puzzles, GPU Puzzles).

Your goal in these puzzles is to implement the derivatives for each core function. In each case the function takes in a tensor x and returns a tensor f(x), so your job is to compute $\frac{d f(x)_o}{dx_i}$ for all indices $o$ and $i$. If you get discouraged, just remember, you did this in high school (it just had way less indexing).