So, turns out, the answer I searched was correct… It was on the next page. Well, lesson learned, I’ll finish a section from now on if possible instead of using a time measured learning system.
The answer to my previous post (Day 10)
let’s say I have these vectors
x_1 = begin{pmatrix}
1 \
2 \
-3 \
4
end{pmatrix};
x_2 = begin{pmatrix}
1 \
1 \
0 \
2
end{pmatrix};
x_3 = begin{pmatrix}
-1 \
-2 \
1 \
1
end{pmatrix} \
x1=⎝⎛12−34⎠⎞;x2=⎝⎛1102⎠⎞;x3=⎝⎛−1−211⎠⎞
To know if they’re linearly dependent:
lambda_1 x_1 + lambda_2 x_2 + lambda_3 x_3 = 0
λ1x1+λ2x2+λ3x3=0
Equals:
lambda_1 begin{pmatrix}
1 \
2 \
-3 \
4
end{pmatrix} +
lambda_2 begin{pmatrix}
1 \
1 \
0 \
2
end{pmatrix} +
lambda_3 begin{pmatrix}
-1 \
-2 \
1 \
1
end{pmatrix} = 0
λ1⎝⎛12−34⎠⎞+λ2⎝⎛1102⎠⎞+λ3⎝⎛−1−211⎠⎞=0
Sooo
If we use the RREF it’ll be:
left[begin{array}{cccc}
1 & 1 & 1 \
2 & 1 & -1 \
-3 & 0 & 1 \
4 & 2 & 1
end{array}right] to dots to
left[begin{array}{cccc}
1 & 1 & -1 \
0 & 1 & 0 \
0 & 0 & 1 \
0 & 0 & 0
end{array}right]
⎣⎡12−3411021−111⎦⎤→⋯→⎣⎡10001100−1010⎦⎤
And since every element of the column is a pivot column, there’s no trivial solution making every single column a unique column which means :
lambda_1 = lambda_2 = lambda_3 x_3 = 0
λ1=λ2=λ3x3=0
and these vectors are linearly independent from one another.
Generating Set and Basis
Finally, from a long time ago I didn’t know what a basis was and it’s going to be discussed today!
Consider the vector space:
V = (nu, +, bullet)
V=(ν,+,∙)
and a set of vectors
mathscr{A} = {x_1, dots, x_k} subseteq nu
A=x1,…,xk⊆ν
If every vector set of the vector is a part of the vector space, meaning:
nu in V
ν∈V
and can be expressed as a linear combination of A (Yes, that absolute horrendous letter is an A but using mathscr). Then A is a generating set of V
The set of all linear combination of the vectors A is called the span of A.
text{If } mathscr{A} text{ spans the vector space } V. \
text{} V = text{span} [mathscr{A}] = text{span} [x_1, dots, x_k]
If A spans the vector space V.V=span[A]=span[x1,…,xk]
Consider a vector space:
V = (nu, +, bullet) text{ and } mathscr{A} subseteq nu
V=(ν,+,∙) and A⊆ν
If there’s no smaller set, then the last generating set is called a minimalbasis.
tilde{mathscr{A}} subseteq mathscr{A} subseteq A subseteq nu text{ that spans } V \
text{Then } tilde{mathscr{A}} text{ is a minimal or basis}
A~⊆A⊆A⊆ν that spans VThen A~ is a minimal or basis
One more example
Let
V = (nu, +, bullet)
V=(ν,+,∙)
and
mathscr{B} subseteq nu, mathscr{B} not = emptyset
B⊆ν,B=∅
Then,
B is a basis of V
B is a minimal generating set
B is a maximal linearly independent set of vectors in V, i.e. adding any other vectors to this set will make it linearly dependent
Every vector
x in V text{ is a linearly combination of vector from } mathscr{B}
x∈V is a linearly combination of vector from B
and every linear combinations are unique, i.e. with:
x sigma_{i = 1}^k lambda_i b_i = sigma_{i = 1}^k psi_i b_i
xσi=1kλibi=σi=1kψibi
with
lambda_i, psi_i in reals, b_i in B \
text{ it follows that } lambda_i = psi_i, i = 1, dots, k
λi,ψi∈R,bi∈B it follows that λi=ψi,i=1,…,k
Thoughts
Honestly, today’s section is pretty great to read, since today is focused on answering the previous chapter’s question / confusion and finally learn what the hell is a basis, it almost feels like eating a dessert after a buffet, not too difficult but enough since it completes a few gaps in the previous sections.
A little TL;DR:
A generating set is a subset of the vector space that can recreate the vector subspace with linear combination
A generating set can be linearly dependent, since it’s not a smaller chunk of the vector space, but can be an entirely different vector all together, but can recreate the subspace
A basis on the other hand is a linearly independent vector subspace, since this is the smallest known generating set that the vector space can create. So removing even a single vector from this set will remove the ability to recreate a portion of the vector space.
Acknowledgement
I can’t overstate this: I’m truly grateful for this book being open-sourced for everyone. Many people will be able to learn and understand machine learning on a fundamental level. Whether changing careers, demystifying AI, or just learning in general, this book offers immense value even for fledgling composer such as myself. So, Marc Peter Deisenroth, A. Aldo Faisal, and Cheng Soon Ong, thank you for this book.
Source:
Deisenroth, M. P., Faisal, A. A., & Ong, C. S. (2020). Mathematics for Machine Learning. Cambridge: Cambridge University Press.
https://mml-book.com
