Syllabus Lesson 53 of 246 · Module 4: Your own autograd engine
Module 4: Your own autograd engine

A Tensor: an array that remembers

Why it matters The engine's one data type is a Tensor: an array plus a slot for its gradient plus a note of where it came from.

A Tensor holds three things:

  • data: the NumPy array of values.
  • grad: an array of the same shape, starting at zeros, which backward will fill.
  • parents: the Tensors it was computed from (empty for inputs and weights).

Every operation makes a new Tensor and records its parents. Today, just the forward part of + and *: compute the values and remember the parents.

Tensor: in deep learning, simply an n-dimensional array. Here, an array that also carries a gradient and its history.

Words and symbols in this lesson
Tensor
here: an array plus its gradient plus the Tensors it came from
.grad
the gradient of the final loss with respect to this Tensor
unbroadcast
summing a gradient back down to the shape it was stretched from
Your turn

Fill in __add__ and __mul__ so they return a new Tensor whose data is the sum or product, and whose parents is (self, other). Plain numbers should work too, via Tensor.lift. Leave _backward alone for now.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.