How to find a least-squares solution using normal equations

A system can have more equations than we can satisfy at once. Least squares gives us a way to choose a solution by minimizing the total squared error. We’ll use normal equations to find that choice for three equations in two unknowns.

Learn Linear Algebra with meStep-by-step videos, lesson notes, and practice for matrices, vectors, and transformations.

Explore Linear Algebra →

Write the system as a matrix equation

Use $x-y=3$, $2x+y=1$, and $-x-4y=2$. The coefficient matrix and target vector are

$A=\left[\begin{array}{rr}1& -1\\ 2& 1\\ -1& -4\end{array}\right]$

$b=\left[\begin{array}{r}3\\ 1\\ 2\end{array}\right]$

Make the residual perpendicular to the columns

Call the least-squares coefficient vector $x^{*}$. We want the output $A{x}^{*}$ closest to $b$. The residual $b-A{x}^{*}$ must be perpendicular to every column of $A$, so $A^T(b-A{x}^{*})=0$. Rearranging gives the normal equations.

$A^{T}A{x}^{*}=A^{T}b$

$A^{T}A=\left[\begin{array}{rr}6& 5\\ 5& 18\end{array}\right]$

$A^{T}b=\left[\begin{array}{r}3\\ -10\end{array}\right]$

Solve the two normal equations

$\begin{array}{l}6x+5y=3\\ 5x+18y=-10\\ x=\frac{104}{83}\\ y=-\frac{75}{83}\end{array}$

The residual is $\frac{1}{83}(70,-50,-30)$. Dotting it with either column of $A$ gives zero. The columns are independent, so this least-squares solution is unique.

Full Linear Algebra course

Learn Linear Algebra with me

We’ll work through matrices, vectors, and transformations, one step at a time.

Explore my Linear Algebra course →

• Video lessons
• Lesson notes
• Practice problems

Included with membership, along with every other Krista King Math course.