LM演算法全稱為Levenberg-Marquard algorithm，在正式介紹該演算法之前，我們需要先研讀一下對該演算法的發展有重要意義的幾篇論文。首先，我們從LM演算法的開篇之作（Levenberg於1944年發表）開始。

A method for the solution of certain non-linear problems in least squares

------------------------------------------------------------------------------------------------------------

引言部分：

The standard method for solving least squares problems which lead to non-linear normal equations depends upon a reduction of the residuals to linear form by first order Taylor approximations taken about an initial or trial solution for the parameters. If the usual least squares procedure, performed with these linear approximations, yields new values for the parameters which are not sufficiently close to the initial values, the neglect of second and higher order terms may invalidate the process, and may actually give rise to a larger value of the sum of the squares of the residuals than that corresponding to the initial solution. This failure of the standard method to improve the initial solution has received some notice in statistical applications of least squares and has been encountered rather frequently in connection with certain engineering applications involving the approximate representation of one function by another. The purpose of this article is to show how the problem may be solved by an extension of the standard method which insures improvement of the initial solution. The process can also be used for solving non-linear simultaneous equations, in which case it may be considered an extension of Newton’s method.

解決導致非線性法向方程的最小二乘問題的標準方法取決於通過關於引數的初始或試驗解的一階泰勒近似將殘差減少到線性形式。如果使用這些線性近似執行的通常的最小二乘過程產生的引數不足以接近初始值，則忽略二階和更高階項可能使過程無效，並且實際上可能會產生更大的殘差的平方和的值大於對應於初始解的殘差的平方和。標準方法改進初始解決方案的這種失敗在最小二乘法的統計應用中已經得到了一些注意，並且經常遇到涉及一個功能的近似表示的某些工程應用。本文的目的是展示如何通過擴充套件標準方法來解決問題，從而確保改進初始解決方案。該過程也可用於求解非線性聯立方程，在這種情況下，它可以被認為是牛頓方法的擴充套件。——來源於谷歌翻譯

從上面的引言部分我們可以得到三點資訊：

1. 以往使用的標準方法存在缺陷（後面會介紹）；

2. 作者提出了一種擴充套件方法（後面會介紹）；

3. 這種方法可以被認為是牛頓法的擴充套件，從後續發展來看，確實可以認為是在牛頓法的基礎上的改進，多數講LM演算法的部落格都會先科普一下牛頓法。

------------------------------------------------------------------------------------------------------------

結論部分：

The nature of the damping which we have imposed upon the parameter variables can be given a simple geometric interpretation. For instance, if the unity weighting system is considered, the "overshooting" of the solution is prevented by damping the distance (k dimensional) from the initial solution point, since Q is then the square of this distance. By this restriction of k dimensional distance (which would appear to be a natural way to prevent overshooting), we are not obliged to decide on an arbitrary preassigned procedure restricting the variables individually, as is done, for example, by the method of Cauchy (l.c.). The greater freedom given the individual variables by the method of damped least squares may account for the fact that it has solved, with a comparatively rapid rate of convergence, types of problems which are of much greater complexity than those to which the principle of least squares is ordinarily applied.

我們對引數變數施加的阻尼的性質可以給出一個簡單的幾何解釋。例如，如果考慮統一加權系統，則通過阻尼距離（k維）到初始解點的距離（k維）來防止解的“超調”，因為q是該距離的平方。通過對k維距離的限制（這似乎是防止超調的一種自然方法），我們沒有義務決定一個任意的預先指定的程式，單獨地限制變數，例如，通過柯西（L.C.）方法。通過阻尼最小二乘法賦予單個變數更大的自由度可以解釋這樣一個事實：它以相對較快的收斂速度解決了比通常應用最小二乘原理更復雜的問題型別。——來源於百度翻譯