[Home]

Table of contents

$\newcommand{\v}{\vec}$ $\newcommand{\col}{\mathcal C}$ Pseudoinverse

Pseudoinverse

We know that nonsingular matrices have unique inverses. This concept works for any field.

It is possible to generalise the concept of inverses to any matrix (need not even be square) if the field is ${\mathbb R}$ or ${\mathbb C}.$
Theorem For any $m\times n$ matrix $A$ there is a $n\times m$ matrix $A^+$ such that
  1. $A A^+A = A$
  2. $A^+A A^+ = A^+.$
  3. $(A^+A)$ and $AA^+$ are both Hermitian.
This matrix is called a pseudoinverse of $A.$

Proof:First we show existence by directly constructing $A^+.$

Let $A = UDV^*$ be an SVD.

If $D_{m\times n} = diag(\sigma_1,...,\sigma_r,0,...,0),$ where $\sigma_i>0,$ then define $S_{n\times m} = diag\left(\frac{1}{\sigma_1},...,\frac{1}{\sigma_r},0,...,0\right).$

Then $A^+ = VSU^*$ will trivially satsify the given conditions. [QED]

Theorem Every matrix has a unique pseudoinverse.

Proof: Suppose that $A$ has two pseudoinverses $B$ and $C.$ Then $AB=AC.$

Because: $$ AB = ACAB = (AC)^*(AB)^* = C^*A^*B^*A^* = C^*(ABA)^* = C^*A^* = (AC)^* = AC. $$
Similarly, $BA = CA.$

Hence $B = BAB = BAC = CAC = C.$ [QED]

Theorem $A A^+$ is the orthogonal projector onto $\col(A).$

Proof: We already have $A A^+$ is Hermitian and idempotent, and so an orthogonal projector.

Enough to show $\col(A A^+) = \col(A).$

Obviously $\col(A A^+)\leq \col(A).$ Now $A A^+ A = A,$ and so $r(A^+A)\geq r(A).$

Combining, $\col(A A^+) = \col(A).$ [QED]

Hence we get the following theorem:
Theorem If $A\v x = \v b$ is any system, then $A^+\v b$ is the a least squares solution. (In particular, for a consistent system, we get an exact solution).

Proof: "$\v x = A^+\v b$ is a least squares solution" means $A\v x$ is the orthogonal projection of $\v b$ onto $\col(A).$

But here $A\v x = A A^+\v b,$ which is need the orthogonal projection of $\v b$ onto $\col(A).$ [QED]

We know that in general a least squares solution may not be unique. However, $A^+\v b$ is guranteed to have minimum norm among all least squares solutions.
Theorem For any system $A\v x = \v b$ the vector $\v x_0 = A^+\v b$ provides a minimum norm least squares solution.

Proof: We have already shown that $\v x_0$ is a least squares solution. Let $\v x$ be any least squares solution. Shall show $\|\v x_0\|^2 \leq \|\v x\|^2.$

Splitting $\v x = \v x_0 +(\v x-\v x_0),$ we can see that it is enough to show that $\v x_0\perp (\v x-\v x_0).$

This is proved easily on noticing that $A^+A\v x_0 =\v x_0.$ So $$ \v x_0^* (\v x-\v x_0) = \v x_0^* A^+\underbrace{A(\v x-\v x_0)}_{\v 0} = 0. $$ [QED]

Problems

  1. Prove that $A^+ A$ is the orthogonal projector onto $\col(A^+).$
  2. Prove that if $A$ is full column rank, then $A^+ = (A^*A) ^{-1}A^*.$
  3. Prove that $(A^*)^+ = (A^+)^*.$
  4. Obtain an explicit form of $A^+$ if $A$ is full row rank.
  5. If $A$ is Hermitian, is it true that $A^+$ must be so too?
  6. Show that in general $(AB)^+$ may not equal $B^+A^+.$ But if $A$ is full column rank, and $B$ is full row rank, then $(AB)^+=B^+A^+.$
  7. Let $A = BC$ be any rank factorisation. Obtain an explicit form of $A^+$ in terms of $B$ and $C.$
  8. Find $A^+$ if $A_{m\times n}$ has all entries equal to $1.$
  9. Prove that minimum norm least squares solution must be unique. [Hint: Prove it directly. No need to use pseudoinverses.]
  10. Show that $(A^*A)^+A^* = A^*(A A^*)^+ = A^+.$
  11. Find $A^+$ if $A = \left[\begin{array}{ccccccccccc}1\\2\\\cdots\\n \end{array}\right].$
  12. If $A^*A= I$ then show that $A^+ = A^*.$ If $BA = I,$ then does it imply that $A^+ = B?$
  13. If $P$ is an orthogonal projector, then show that $P^+ = P.$
HTML Comment Box is loading comments...