Orthogonality

Inner product. Let $\mathbf{u}$ and $\mathbf{v}$ be two column vectors in $\mathbb{R}^n$ . Then we can define the inner product (also referred as “dot product”) by

$\displaystyle \langle \mathbf{u}, \mathbf{v} \rangle = \mathbf{u}^T \mathbf{v} = u_1v_1 + \cdots + u_nv_n$

The inner product is the product of $1\times n$ matrix $\mathbf{u}^T$ and $n\times 1$ matrix $\mathbf{v}$ , yielding a scalar value. Since the transpose $(\mathbf{u}^T \mathbf{v})^T$ will be the same scalar value $\mathbf{u}^T \mathbf{v}$ , we have

$\displaystyle \langle \mathbf{u}, \mathbf{v} \rangle = \mathbf{u}^T \mathbf{v} ... ...bf{v})^T = \mathbf{v}^T \mathbf{u} = \langle \mathbf{v}, \mathbf{u} \rangle$

Properties of inner product. We can summarize the properties of inner product.

$\langle \mathbf{u}, \mathbf{v} \rangle = \langle \mathbf{v}, \mathbf{u} \rangle$ (Symmetric property)
$\langle a\mathbf{u} + b\mathbf{v}, \mathbf{w} \rangle = a\langle \mathbf{u}, \mathbf{w} \rangle + b\langle \mathbf{v}, \mathbf{w} \rangle$ (Linearity property)
$\langle \mathbf{u}, \mathbf{u} \rangle \ge 0$ ; the equality holds if and only if $\mathbf{u} = \mathbf{0}$ .

When $\langle \mathbf{u}, \mathbf{v} \rangle = 0$ , $\mathbf{u}$ and $\mathbf{v}$ are said to be orthogonal.

Norm of vector. Consider the inner product of $\mathbf{v}$ with itself

$\displaystyle \Vert\mathbf{v}\Vert^2 = \langle\mathbf{v},\mathbf{v}\rangle = v_1^2 + \cdots + v_n^2$

It determines the length $\Vert\mathbf{v}\Vert$ of the vector $\mathbf{v}$ , and $\Vert\mathbf{v}\Vert$ is called the norm of $\mathbf{v}$ . It is easy to see the scalar multiple of $\mathbf{v}$ changes the norm exactly multiplied by the magnitude of the scalar.

$\displaystyle \Vert c\mathbf{v}\Vert = \sqrt{\langle c\mathbf{v},c \mathbf{v}\r... ...gle \mathbf{v},\mathbf{v}\rangle} = \vert c\vert \cdot \Vert\mathbf{v}\Vert$

A vector $\mathbf{u}$ is called a unit vector if $\Vert\mathbf{u}\Vert = 1$ .

EXAMPLE 1. Let $\mathbf{u} = \left[\begin{array}{c} 1 \\ -2 \\ -4 \end{array}\right]$ and $\mathbf{v} = \left[\begin{array}{c} 2 \\ 3 \\ -1 \end{array}\right]$ be vectors in $\mathbb{R}^3$ . Then

$\displaystyle \langle \mathbf{u}, \mathbf{v} \rangle = (1)(2) + (-2)(3) + (-4)(-1) = 0$

Thus, $\mathbf{u}$ and $\mathbf{v}$ are orthogonal. The orthogonal vectors $\mathbf{u}$ and $\mathbf{v}$ satisfy the pythagorean theorem:

$\displaystyle \Vert\mathbf{u} - \mathbf{v}\Vert^2 = \Vert\mathbf{u}\Vert^2 + \Vert\mathbf{v}\Vert^2$