Linear algebra

Note

🚨 So why do we need linear algebra?

Mathematics is not only about numbers. It is also about structure. A large part of modern mathematics is concerned with understanding structures: what their essential properties are, how they relate to one another, and what can be learned from the transformations that preserve those properties.

The goal of linear algebra is not simply to learn how to calculate with matrices. The goal is to learn how to recognize generic structures.

The ideas developed in this chapter will reappear in increasingly sophisticated forms. In particular, the language of vector spaces and linear transformations will eventually lead us toward representations of groups and, ultimately, Galois representations.

Remember that in the introduction we said:

In mathematics, we often try to generalize concepts, theorems, and equations.

Here, the deeper idea is that seemingly unrelated mathematical problems can often be described using the same underlying structure. As we will see, we can use these shared concepts to look at certain problems from a new, often more convenient perspective.

For example, in linear algebra, we can write any vector with respect to its basis vectors:

\[ \overrightarrow{v}=c_1\overrightarrow{e_1}+c_2\overrightarrow{e_2}+c_3\overrightarrow{e_3} = \begin{pmatrix}c_1 \\ c_3 \\ c_3\end{pmatrix} \]

In the following two examples, we will do the same in the vector space of functions.

        Linear algebra
                |                
        Vector spaces
                |                
        Function spaces
                |                
            Expansion
            /          \
    Taylor          Fourier
        |                |
polynomial basis   trigonometric basis

Although Fourier analysis and Taylor expansion may seem to belong to a completely different part of mathematics, from the perspective of linear algebra, we can think of a function as an element of a vector space. This allows us to ask:

What are the right basis vectors, and what are the coordinates of our object in that basis?

Just as an ordinary vector can be written as a linear combination of basis vectors, a function can be written as a linear combination of basis functions.

✍️ Example 1: Taylor expansion

We can conceptualize the Taylor expansion as a kind of coordinate representation of a function with respect to a basis of polynomials.

Note

🚨 All concepts below will be properly addressed and explained in this very chapter on linear algebra.

Take for example a function \(f(x)\). Around \(x=0\) we can write

\[ f(x)=a_0+a_1x+a_2x^2+a_3x^3+\cdots \]

We can think of the objects

\[ 1,\quad x,\quad x^2,\quad x^3,\ldots \]

as basis vectors of a vector space of polynomials, exactly those notions that are defined in this chapter on linear algebra! The scalars \(a_0,a_1,a_2,a_3,\ldots\) are exactly the coordinates van \(f(x)\) with respect to that basis.

Note that a Taylor series is not just an arbitrary representation of \(f(x)\) in the basis \(\{1,x,x^2,x^3,\ldots\}\) as the coordinates are specifically determined by the derivatives of \(f(x)\) around the expansion point:

\[ a_n=\frac{f^{(n)}(0)}{n!}. \]

For example,

\[ e^x=1+x+\frac{x^2}{2!}+\frac{x^3}{3!}+\cdots \]

can be interpreted in the context of linear algebra as:

\[ [e^x]_{\{1,x,x^2,x^3,\ldots\}} = \left(1,1,\dfrac{1}{2!},\dfrac{1}{3!},\ldots\right). \]

👉 The Taylor series expansion provides a different basis

👨‍💻 Taylor expansion demo

We can look at it in an even more abstract way. Around a point \(a\) we have the basis

\[ 1,\quad (x-a),\quad (x-a)^2,\quad (x-a)^3,\ldots \]

with which \(f(x)\) becomes

\[ f(x) = f(a) +f'(a)(x-a) +\frac{f''(a)}{2!}(x-a)^2+\cdots. \]

Therefore, the coordinates with respect to that polynomial basis vectors are

\[ \boxed{ \left( f(a), f'(a), \frac{f''(a)}{2!}, \frac{f^{(3)}(a)}{3!},\ldots \right) } \]

So we observe the following analogy in lineaire algebra:

Linear algebra Taylor analysis
vector \(v\) function \(f\)
basis vectors \(e_1,e_2,\ldots\) \(1,(x-a),(x-a)^2,\ldots\)
coordinates \(c_1,c_2,\ldots\) \(f(a),f'(a),f''(a)/2!,\ldots\)
\(v=\sum c_i e_i\) \(f=\sum \frac{f^{(n)}(a)}{n!}(x-a)^n\)

👉 We can look at the Taylor expansion as expressing a function in coordinates along the directions \(1,(x-a),(x-a)^2,\ldots\).

There exists an even deeper connection: differentiation can be seen as a linear operator that works on these coordinates. This directly leads to a connection between Taylor series, vector spaces, and matrices/operators.

✍️ Example 2: Fourier transform

Here the connection with linear algebra becomes even more apparent, since the Fourier expansion amounts to a representation of a function in the following orthogonal directions:

\[ 1,\quad \cos x,\quad \sin x,\quad \cos 2x,\quad \sin 2x,\quad\ldots \]

Any function \(f\) can thus be built using this basis:

\[ f(x) = \frac{a_0}{2} +\sum_{n=1}^{\infty} \left(a_n\cos(nx)+b_n\sin(nx)\right). \]

The Fourier-coefficiens \(a_0,a_1,b_1,a_2,b_2,\ldots\) are the coordinats of the function \(f\) in this basis.

👨‍💻 Fourier transform demo

Comparision between Taylor and Fourier series

Orthogonality

We have seen that \[ \text{Taylor}=\text{coordinates in a polynomial basis} \]

and

\[ \text{Fourier}=\text{coordinates in a trigonometric basis}. \]

However, in the case of Fourier, we have seen yet another concept from linear algebra, namely orthogonality.

Two vectors are orthogonal when \(u\cdot v=0\). For functions this can be defined analogously:

\[ \langle f,g\rangle = \int_{-\pi}^{\pi}f(x)g(x)dx. \]

With this definition \(\cos x\) and \(\sin x\) turn out to be perpendicular to one another:

\[ \langle\cos x,\sin x\rangle = \int_{-\pi}^{\pi}\cos x\sin x dx =0. \]

The same holds for the different frequencies \(\langle\cos x,\cos2x\rangle=0\) and \(\langle\sin3x,\cos7x\rangle=0\). In fact, we have an infinite-dimensional Euclidean space where the Fourier basis vectors are perpendicular to each other.

This orthogonality is then used to find the (frequency) coordinates. Suppose in “ordinary” lineaire algebra we have:

\[ \overrightarrow{v}=3\overrightarrow{e_1}+5\overrightarrow{e_2}. \]

When \(\overrightarrow{e_1}\) and \(\overrightarrow{e_2}\) are prependicular, we can find \(3\) by projecting \(\overrightarrow{v}\) onto \(\overrightarrow{e_1}\).

Analogously we can find the Fourier coefficients \(a_n\) by projecting \(f\) onto \(\cos(nx)\). So conceptually:

\[ a_n \sim \langle f,\cos(nx)\rangle, \quad b_n \sim \langle f,\sin(nx)\rangle. \]

With exact normalisation this results in

\[ a_n=\frac1\pi \int_{-\pi}^{\pi}f(x)\cos(nx)dx, \quad b_n=\frac1\pi \int_{-\pi}^{\pi}f(x)\sin(nx),dx. \]

Geometric interpretation

We have just established that Fourier coefficients are literally projections of the function vector onto orthogonal basis vectors.

👉 This makes a direct geoemtric interpretation of the Fourier transform possible.

For a vector in 3D this geometric interpretation is trivial. For example,

\[ \overrightarrow{v}= \begin{pmatrix} 3 \\ 2 \\ 5 \end{pmatrix}. \]

tells us that \(\overrightarrow{v}\) consists of 3 units in the \(x\)-direction, 2 in the \(y\)-direction and 5 in the \(z\)-direction. Fourier tells us something similar: this function consists so much of frequencies \(1\), that much of frequencies \(2\), etc.

For example, we could have \(f(x)= 2\cos x +0.7\sin x +3\cos2x -1.2\sin3x\).

The two examples may be summarized in a table.

Taylor Fourier
Vector function \(f\) function \(f\)
Vector space functions/polynomials functions
Basis \(1,(x-a),(x-a)^2,\ldots\) \(1,\cos x,\sin x,\cos2x,\sin2x,\ldots\)
Coördinaten derivatives of \(f\) projections/integrals
Idea local behaviour frequenty-contents
Basis orthogonal? no yes
Geometric interpretation less direct very direct

🧠 Conclusion

👉 The choice of basis determines which properties of a function become apparent.

  • With Taylor local properties become apparent via the derivatives:
    • Decomposition of \(f\) in polynomial directions
    • How does \(f\) behave locally around a point \(a\)?
  • With Fourier frequencies become visible via projection onto the basis vectors:
    • Decomposition of \(f\) in frequency directions
    • From which frequencies can \(f\) be built up?

👉 This is why linear algebra can be so powerful:

  • It gives us a common language in which very different mathematical problems can reveal the same underlying structure.

  • We can do a basis transformation to look at a certain problem from a new, more convenient perspective.