TOPICS
Search

Nonlinear Least Squares Fitting


Given a function f(x) of a variable x tabulated at m values y_1=f(x_1), ..., y_m=f(x_m), assume the function is of known analytic form depending on n parameters f(x;lambda_1,...,lambda_n), and consider the overdetermined set of m equations

y_1=f(x_1;lambda_1,lambda_2,...,lambda_n)
(1)
y_m=f(x_m;lambda_1,lambda_2,...,lambda_n).
(2)

We desire to solve these equations to obtain the values lambda_1, ..., lambda_n which best satisfy this system of equations. Pick an initial guess for the lambda_i and then define

 dbeta_i=y_i-f(x_i;lambda_1,...,lambda_n).
(3)

Now obtain a linearized estimate for the changes dlambda_i needed to reduce dbeta_i to 0,

 dbeta_i=sum_(j=1)^n(partialf)/(partiallambda_j)dlambda_j|_(x_i,lambda)
(4)

for i=1, ..., m, where lambda=(lambda_1,...,lambda_n). This can be written in component form as

 dbeta_i=A_(ij)dlambda_j,
(5)

where A is the m×n matrix

 A_(ij)=[(partialf)/(partiallambda_1)|_(x_1,lambda) ... (partialf)/(partiallambda_n)|_(x_1,lambda); (partialf)/(partiallambda_1)|_(x_2,lambda) ... (partialf)/(partiallambda_n)|_(x_2,lambda); | ... |; (partialf)/(partiallambda_1)|_(x_m,lambda) ... (partialf)/(partiallambda_n)|_(x_m,lambda)].
(6)

In more concise matrix form,

 dbeta=Adlambda,
(7)

where dbeta is an m-vector and dlambda is an n-vector.

Applying the transpose of A to both sides gives

 A^(T)dbeta=(A^(T)A)dlambda.
(8)

Defining

a=A^(T)A
(9)
b=A^(T)dbeta
(10)

in terms of the known quantities A and dbeta then gives the matrix equation

 adlambda=b,
(11)

which can be solved for dlambda using standard matrix techniques such as Gaussian elimination. This offset is then applied to lambda and a new dbeta is calculated. By iteratively applying this procedure until the elements of dlambda become smaller than some prescribed limit, a solution is obtained. Note that the procedure may not converge very well for some functions and also that convergence is often greatly improved by picking initial values close to the best-fit value. The sum of square residuals is given by R^2=dbeta·dbeta after the final iteration.

NonlinearLeastSquares

An example of a nonlinear least squares fit to a noisy Gaussian function

 f(x;A,x_0,sigma)=Ae^(-(x-x_0)^2/(2sigma^2))
(12)

is shown above, where the thin solid curve is the initial guess, the dotted curves are intermediate iterations, and the heavy solid curve is the fit to which the solution converges. The actual parameters are (A,x_0,sigma)=(1,20,5), the initial guess was (0.8, 15, 4), and the converged values are (1.03105, 20.1369, 4.86022), with R^2=0.148461. The partial derivatives used to construct the matrix A are

(partialf)/(partialA)=e^(-(x-x_0)^2/(2sigma^2))
(13)
(partialf)/(partialx_0)=(A(x-x_0))/(sigma^2)e^(-(x-x_0)^2/(2sigma^2))
(14)
(partialf)/(partialsigma)=(A(x-x_0)^2)/(sigma^3)e^(-(x-x_0)^2/(2sigma^2)).
(15)

The technique could obviously be generalized to multiple Gaussians, to include slopes, etc., although the convergence properties generally worsen as the number of free parameters is increased.

An analogous technique can be used to solve an overdetermined set of equations. This problem might, for example, arise when solving for the best-fit Euler angles corresponding to a noisy rotation matrix, in which case there are three unknown angles, but nine correlated matrix elements. In such a case, write the n different functions as f_i(lambda_1,...,lambda_n) for i=1, ..., n, call their actual values y_i, and define

 A=[(partialf_1)/(partiallambda_1)|_(lambda_i) (partialf_1)/(partiallambda_2)|_(lambda_i) ... (partialf_1)/(partiallambda_n)|_(lambda_i); | | ... |; (partialf_m)/(partiallambda_1)|_(lambda_i) (partialf_m)/(partiallambda_2)|_(lambda_i) ... (partialf_m)/(partiallambda_n)|_(lambda_i)],
(16)

and

 dbeta=y-f_i(lambda_1,...,lambda_n),
(17)

where lambda_i are the numerical values obtained after the ith iteration. Again, set up the equations as

 Adlambda=dbeta,
(18)

and proceed exactly as before.


See also

Least Squares Fitting, Linear Regression, Moore-Penrose Matrix Inverse

Explore with Wolfram|Alpha

References

Bates, D. M. and Watts, D. G. Nonlinear Regression and Its Applications. New York: Wiley, 1988.

Referenced on Wolfram|Alpha

Nonlinear Least Squares Fitting

Cite this as:

Weisstein, Eric W. "Nonlinear Least Squares Fitting." From MathWorld--A Wolfram Web Resource. https://mathworld.wolfram.com/NonlinearLeastSquaresFitting.html

Subject classifications