An unusual intro to deep learning.
The only primitive operation that one needs to understand to get started on in Deep Learning is dot product: vectors r and x of same dimension d can be dotted to produce a scalar
⟨r,x⟩=∑i∈[d]rixilangle r,x rangle = sum_{iin [d]}{r_i x_i} ⟨r,x⟩=∑i∈[d]rixi
.
To do this operation in parallel across multiple vectors
rr r
, one can write it as a matrix vector product. One way to think about these dot product is that each vector r is a measurement of (an unknown) signal x. For example if vector r is [1, 0, 0, 0], then it is measuring the first coordinate of x. Likewise vector r [0.25, 0.25, 0.25, 0.25] is measuring the average value of coordinates of x. Notice we can make many measurements of x by doing this matrix vector multiplication where each measurement is independent (and possibly redundant) and done in parallel.
As an aside, are you wondering what exactly is a dot product geometrically? It is simply a (scaled) projection of the high dimensional signal along a “line”. The line could be x-axis (e.g. when r i s [1, 0, 0, 0]) or some 45-degree line passing through origin.
So given an “unknown” x, one is interested in asking the question, hey which direction should I measure x in to get maximum bang for buck. More on this later.
Suppose we have an image represented as a vector x. To make things concrete, imagine it is a 10×10 image with each value in [0, 1] representing the grayscale measure normalized to lie in [0, 1]. This image can be represented as a 100-dim column vector. We want to deduce if it is an image of a dog or a cat.
