Least-Squares Errors Fit


Assume you have two corresponding vectors of data:

X    =       x1,    x2,    x3,    x4,    x5,    x6,    x7,    x8,    ....    xN

Y    =       y1,    y2,    y3,    y4,    y5,    y6,    y7,    y8,    ....    yN

And assume some rule which produces estimates of yn from xn:    yn' = f(xn, parameters), where yn' is the estimate. The square of the error between the estimate and the actual value of one piece of data yn is:

En = (yn' - yn)2 = (f(xn, parameters) - yn)2.

The total square error is then defined as the sum of square errors for each piece of data:

ET = E1 + E2 + E3 + ... + EN

This may also written in terms of the vectors X and Y as:

ET(parameters) = ( Y' - Y )2 = (f(X, parameters) - Y )2

where Y' = f(X, parameters) is a vector containing the estimates for all the data elements of Y (for some values of the parameters). For a given set of data we see that the total square error is a function of the paramters of our rule.

In the least-squares-error sense, the best fit of the data to a theoretical curve is found for the set of parameters that minimizes ET.

 

Linear Least-Squares Error Fitting

When the rule mapping the data vector X to vector Y is a linear one,

Y = aX +b,

then the problem of minimizing ET as a function of the parameters 'a' and 'b' becomes a basic linear algebra problem. Although the math is fairly complicated, the solution is quite simple.
If p is the vector 
a
b
     and A is the matrix 
1 x1
1 x2
1 x3
* *
* *
1 xN
     and Y is the vector 
y1
y2
y3
*
*
yN

then the linear least-squares fit parameters 'a' and 'b' are given by the following matrix product:

p = (ATA)-1 A Y

where   -1 denotes the inverse of a matrix.

Many calcuators support linear least-square data fitting. Such a fit is also easily performed by MatLab. If the data and parameters 'a' and 'b' are put in the form above (vector p and matrix A), then p is solved in MatLab by the statement:

p = A\Y

MatLab can also perform a polynomial least-squares error fit of a vector of data Y to vector X. That is, it finds parameters of the rule,

yn = P( xn ) = p1xnm + p2xnm-1 + ... + pmxn1 + pm+1

that minimize the total error,

ET = (P(X) - Y )2.

In MatLab this is performed by the function,

p = polyfit(X,Y,m)

where the result  p = [p1p2 ... pm+1] is the vector of the polynomial coefficients and m is the order of the polynomial.

As an example, to find the linear least-squares fit (Y=aX + b) of data Y and X using the MatLab polyfit command, one would type:

p = [a, b] = polyfit(X, Y, 2)


For an example of how Dr. Erlenmeyer used linear least-squares fitting, and for copies of his matlab scripts, click on Dr. Erlenmeyer.

 

Linear Least-Squares Error Fitting to Non-Linear Formulas

How can a linear least-square fit be used when one knows that the data should follow a non-linear formula? For instance,

zn = c (xn - d)2      or      zn = c exp(d xn)2

In many cases the data can be transformed by applying the inverse non-linear operation to each side. For example, suppose we want to fit a set of data,

Z    =       z1,    z2,    z3,    z4,    z5,    z6,    z7,    z8,    ....    zN

as a function of the data,
X    =       x1,    x2,    x3,    x4,    x5,    x6,    x7,    x8,    ....    xN     ?

Answer: by making the transformation,
zn = c (xn - d)2      --->      sqrt( zn ) = sqrt(c) xn - sqrt(c) d

a linear least-squares fit can be performed on the transformed data and parameters using the linear rule,

Y = aX + b

where  Y=sqrt(Z)  (i.e., yn = sqrt zn  )     and     a = sqrt(c),     b = - sqrt(c) d .

An example using this type of transformation is given on the Dr. Erlenmeyer Least-Squares Fit page.

Similarly, by making the transformation,

zn = c exp(d xn)   --->   ln( zn ) = d xn + ln(c)

a linear least-squares fit can be performed on the transformed data and parameters using the linear rule,

Y = aX + b

where   Y=ln(Z)   (i.e., yn = ln zn  )     and     a = d,     b = ln(c)



[
Physics and Astronomy Home Page]  [Northwestern University Home Page]