Reprinted from Sousa ov Marsosarica stats A APPLICATIONS Vo. 15, No 2, Aout 1966, [Ai Rahs Reserved by Acdende Pre, New Yor ard Laon Printed Bao ‘A Newton-Raphson Method for the Solution of ‘Systems of Equations Apr Bex-Israg ‘Technion: torael Institute of Techwology ond Northesters University® ‘Submitted by Richard Bellman Inrmopuction ‘The Newton-Raphson method for solving an equation, Fs) =0 w is based upon the convergence, under suitable conditions [1, 2}, of the sequence Ses) FG) ‘10.2 solution of (1), where sy is an approximate solution. A detailed discussion of the method, together with many applications, can be found in [3] Extensions to systems of equations (9 =0,1,2," @ Silene %) = or fix) =0 @. fab» ye) = 0 are immediate in cas rs [I]p where the analog of (2) is: Xo — P)f,) (= @ Extensions and applications in Banach spaces were given by Hildebrandt and Graves [4], Kantorovié [5-7], Altman [8], Stein [9], Bartle [10], Schedder [11] and others. In these works the Frechet derivative replaces J(x) in (4), yet nonsingularity is assumed throughout the iterations, T Par af the research underlying this report was undertaken forthe Office of Naval Research, Contract Nont-1228(10), Project NR 047-021, and for the U.S. Army Research’ Office-Ducham, Contract No, DA-31-124-ARO-D-222 at Northwestern University. Reproduction of thie paper in whole or in partis permitted for any pur- pose of the United States Government. "See section on notations below. 2434 BEN-ISRAEL “The modified Newion-Raphson method Son =%p— Jn) fl) (P= 0,1) © was extended in [17] to the case of singular J(x.), and conditions were given for the sequence Kon = %— FG) AG) (P= Ost to converge 0 a solution of I) fx) = 0. o In this paper the method (4) is likewise extended to the case of singular ‘Tex,), and the resulting sequence (12) is shown to converge to a stationary point of Et, fx). ) © Norarioss Let BF denote the A-dimensional (complex) vector space of vectors x, with the Euclidean norm ||x || =(s, x)". Let B= denote the space of ‘m Xn complex matrices with the norm \|A | = max (Vi: Aan eigenvalue of 444}, -A* being the conjugate transpose of 4 “These norms satisfy [12] Ax || S| Allixll forevery xek, Ae Bm, Let R(A), N(A) denote the range resp. mull space of A, and A* the generalized inverse of A, [13]. Fora € B* and r > O let S(iy,7) ={x eB: ix —uli <7} denote the open ball of radius r around w. ‘The components of a function f: B"—> B" are denoted by f(x), (= 1, ,,m). The Jacobian of fat x © B* is the m x n matrix 1m) Ga) For an open set $C B*, the function f: E* > Bm is in the class C(S) ifthe mapping E*-> 2 given by x— j(x) is continuous for every x € S [4]. Joy = (42[A.NEWTON-RAPHSON METHOD 24s Resuuts ‘Puwoneat 1. Let f: 8" B® be a function, xy a point in Bs, andr >0 bbe such that f & C'(S(x 5”) Let M, N be positive constants such that for all u,v in S(xy,) sith uve RW): {1Je) (a = ¥) — f(a) + fv) E be defined by efx) =x — J*(x) f(x). (4) Equation (12) now becomes: Xo = 8%) (P= 0,1, -~) as) ‘We prove now that KES) (P=L2~) (16) For p= 1, (16) is guaranteed by (11). Assuming (16) is true forall sub- scripts 4 — ele) I< Ue — pps ll + | aC5) ~ eC) | Sllxy Xp + Ally — a 5) where the right-hand side of (25) tends to zero as p—» co. But Hy = sb) 26) is equivalent, by (14), to JQ) £4) 7) which is equivalent to Fre) fe) = 0 (28) singe N(A*) = N(A*) for every Ae B™ [14]. QED. Rewancs, (a) If m—m and the matrices J(x,) aré nonsingular, (p= 0, 1, -), then (12) reduces to (4), which converges to a solution of (3). In this case (13) and (3) are indeed equivalent because N(J*(.)) = (0}- () From Ff) = Land & ‘fe(s)) 9)[A NEWTON-RAPHSON METHOD 247 it follows that the limit x, of the sequence (12) is a stationary point of SAA), which by Theorem 1 exists in S(x7), where (8)-(11) are satis- fed. Even when (3) has a solution in Six, 7); the sequence (12) doce not necessarily converge to it, but to stationary point of P(x) which may be a least squares solution of (3), or a saddle point of the function D7. f.*(x), te. Darmsrion. A point x is an itolated point forthe function fin the linear manifold L if there is a neighborhood U of x such that yeUnl, yex=fy)#fix). ‘Tuono 2. Let the function f : B®» B be in the clas C*(S(1)). Then ui an iolated pont for fin the linear manifold (a + RJ") Poor. Suppose the theorem is false, ‘Then there is a sequence xe € S(u,7) fa + RI*(a)} (Rm 1,2) G0) such that mou and fix) = fu) = 1,2,--) @) From f © C'(S(q,)) it follows that hs (ua + 24) — f(a) — Ju) xe [| <8 ze ID Hl ze I (32) , psu ta @) and +0 as 0. 4) Combining (31) and (32) it follows that | s09) PEG] sole. @5) ‘The sequence {z,}/| 2». (t = 1, 2, *~), consists of unit vectors which by (30) and (33) lie in R(J*(uw)), and by (35) and (34) converge toa vector in N(J(4)), 2 contradiction. QED. Rasanxs (a) In case the linear manifold (u + RJ*%(u))} is the whole space B*, (ie, N(J(w)) = {0}, ies the columns of J(u) are linearly inde- pendent), Theorem 2 is due to Rodnyanski [15] and was extended to Banach spaces by S. Kurepa [16], whose idea is used in our proof. (&)_ Using Theorem 2, the following ean be ssid about the solution x ‘of (13), obtained by (12):248 ‘BEN ISRAEL COROLLARY. The limit x of the sequence (12) isan iolated zero for the function JC) (x) in the near manifold (xg + RCI), alse J(,) = 0 the sero matri Paoor. xe is in the interior of S(xy 7), therefore f C'(S(ay »r4)) for some ry > 0. Assuming the corollary t0 be fale it follows, asin (35), that Frese) Sten) 3 (U2 I) i Tee) {I (36) where ty = x4 +22, (= 1,2,-), is a sequence in {xa + RU*(E4))) converging to x, and such that 0 = J*(x) fxs) = J*s) fay). Excluding the trivial case: J(x,) =0 (Gee remark below), it follows that {all ze), (R= 1, 2, ), is a sequence of unit vectors in R(J*(x)). which by (36) converges to a vector in N(J*(x») J(e)) = N(U(e)), a contradic- tion. QED. Rewark: The definition of an isolated point is vacuous if the linear ‘manifold L is zero-dimensional. Thus if (xz) =0, then RU*(x.)) = (0) and every x 2" is a zero of J*(x4) (3): Exanres ‘The following examples were solved by the iterative method: Kewsbit = Ke — Jn) fRaesads @) where «> 0 is an integer a1 if aso # if a=0 and P=O1~ For a = 1: (37) reduces to (12), a= ‘of {17}, and for a > 2: wis the number of iterations with the modified method [17], (with the Jacobian j(x,), 6 = 0, 1, ->), between successive computa tions of the Jacobian J(x,,) and its generalized inverse, In all the examples worked out, convergence (up to the desired accuracy) required the smallest number of iterations for « = 1; but often for higher values of « less computa- tions (and time) were required on account of computing J and J* onlyA NEWTON-RAPHSON METHOD 249 ‘once in « iterations. The computations were carried out on Philco 2000. ‘The method of [18] was used in the subroutine of computing the generalized inverse J*. Exampue 1. The system of equations is Ailes ym) = a8 + m2 fx) = {is =m =0 les») Equation (13) is eto) eS an hove solutions are (0, 0) a saddle point of F fi4x) and (1,1), (— 1, — 1) the solutions of f(x) = 0. In applying (37), derivatives were replaced by differences with A¥ = 0.001 ‘Some results are pth ° 1 2 3 _ Sono | usmmias | tasiisi | asso 2.000000 1,355469, 1.199107 | 1.118148 Axa) 11,0000 2.327834 ose1s | 0.585672 Lowoco | oazuer4 | oseme | staat soon | fibers | asses | oDsatse veromo | 67eao | 3022 | oaasrsr mth | Saw | hgoss90 | Laponsi|—1.000118 | 1.00000 1.008365 | 1.000980 | 1.000118 | 1.000000 firms) | 003361 | 00382 | oan | a.00000 | Sonec2s | .meceo | o.oote00 | o.ate0 faoiesss | Qooisez | Govease | c00000 Eye | oaoiais | 0.000019 | 000000 | 0.000000 Note the sharp improvement for each change of Jacobian (iterations: 1,4, and 7).250 BEN-ISRAL, 5 path ° s 6 | 8 9 nee | 3.000000 | 1.050657 | 1.001078) 1.000002 | 1.000000 2.000000 | 1.083431 1.001078, 1.000002 | 1.000000, 1.00000 | 0.007225 | 0.000000) 0.900000 | 0.000000 s.00000 | a.0v6289 | 0.002157 vam | cme forms) | 1.00000 | 0.192630 | 0.004315} 0.000009 | 0.000000 Efe | 147.0000 | 0.046430 | o.ot02) 000000 ‘The Jacobian and its generalized inverse were twice calculated (iterations 1, 6), whereas for a = 3 they were caleulated 3 times. Kit 30 | 1.003686 | 1.000008 | 1.000000 20 1.003559 | 1.000008 | 1.000000 flame) | 110 0.14516 | 0.000032 | 0.000000 | ro .000128 | 0.000000 | 0.000000 | so 0.007258 | 0.000016 | 0.000000 Be | wo 0.000263 | 0.000000 | 0.000000 Exanece 2. Sil» %2) Has.) [houe) = G.— Itt —9 <0. This is an inconsistent system of equations, whose least squares solutions are (1.000000, 1.914854) and (1.000000, — 1.914854). Applying (37) with « and exact derivatives, resulted in the sequence:ee ee = se Se) es Efex) | 684232,.0 61515.80 3695.23 229.6009 x, 1.000000 1.000000 | 1.000000 +1,000000 Eye) | 48114030 | 42.691255 | 42.666667 | 42.6667 an xm 10 sma — I = \Abam) £9) Thowisd ‘The solutions of this system are (2,8) and (8,2). However, applying (12) with an initial x, on the line: x, — x , results in the whole sequence being on the ‘same line. Indeed, Tors (1, yt) sothafor xy = (1) Jed=(,)). Peod-aa lt and consequently x; is also on the line: =, =, . Thus confined to: =, =, the sequence (12) will, depending on the choice of xy, converge to either (4.057646, 4.057646) or (— 3.313982, — 3.313982). These are the 2 least squares solutions on the line: x, = =.252 ‘BEN-ISBABL AckowLEDGMENT “The author gratefully acknowledges the computing facilites made available to him by the SWOPE Foundation. Reveences 1, A'S. Hovsenotnes, “Principles of Numerical Analysis, 2. ALM. Osmmowat. “Solution of Equstions and Systems of Equations,” Academic Press, New York, 1960. 3, Rc Bruasas anb BR. Katana “Quasilinearzation and Nonlinear Boundary-Value Problems.” American Elsevier, New York 196. 4. TAL Hiconmeanor aNo L, M. Graves. Implicit functions and cheirdiferentials in general analysis, Trant. Amer. Math. Soc, 29 (1920), 127-183. 5. LV, Kanrorovté. On Newton's method for Funetional equations, Dokl. Akad. ‘Nauk SSSR (NS, 59 (1948), 1237-12405 (Math, Rev. 9-537), 1948. 6. LV, Kantorovie. Functional analysis and applied mathematics, Uspehi Mat Nauk (N.S. 3, No. 6 (28), (1948), 89-185 (Math. Rev. 10-380), 1949. 7. LV, Kusronovie, On Newton's method. Trudy Mat. Int. Steklo. 28 (1949), 104-148 (Math. Reo, 12-419), 1951 8 Mu Autauan. A generalization of Newton's method, Bull, Acad. Polon. Sei (1955), 189-193, 19. M. 1, Stain. Sufficient conditions for the convergence of Newton's method in ‘complex Banach spaces. Proc, Amer. Math, Soc. 3 (1952), 858-863. 10. RG. Bawras, Newton's method in Banech spaces. Proc. Amer. Math. Soe. 6 (1955), 827-891. 11. J, Sewxdom, Uber das Newtonsche Verfahren. Arch, Rat, Mech, Anal. 1 1987), 154-190. 12, ALS. Housenoupen, “Theory of Matrices in Numerical Analysis.” Blaisdell, 1964 {3 Rc Punmose, A genceaized inverse for matrices. Proc. Cambridge Phil. Soc. 51 (1955), 406-413. 14, A Br-lanast. axp A. Cuarsses. Contributions to the theory of generalized inver~ fee J. Soe Indust, Appl. Math, 11 (1963), 667-699. 15, AM, Roowvavcxst. On continuous and differentiable mappings of open sets of Buclidean space. Mot, Sb. (N.S.) 42 (84), (1957), 179-196, 16, 5. Kunars, Remack on the (F)-diflerentiabe functions in Banach spaces” Glas ‘Mat-Fis, Artrmam, Ser, 14 (1959), 213-217; (Math. Rev. 24-A1030), 1962 17, A BascTatarn, A modified Newton-Raphson method for the solution of systems of equations. Fae! J. Math. 3 (1965), 94-98. 18, As Bev-louam. an. J. Wena, An climination method for computing the gene- feof un arbitrary eomplex matrix. J Assoc. Comp. Mach, 10 (1963), ees eee‘A Newton-Raphson Method for the Solution of, Systems of Equations Apr Bew-Isram. ‘Tecbnion-oral Institute of Technology and Northwestern University ‘Submited by Richard Bellman Reprinted fron Jou OF MATHENATIE AKALY AND APLINTIONS V5, No 2 Aug 1966 Ped [AI Rigs Revere! by Academie Pen New Wot ened Landon Balm