Detecting Potential Instabilities of Numerical Algorithms
aa r X i v : . [ c s . NA ] S e p Detecting Potential Instabilities of Numerical Algorithms
Yao Yang
Abstract
It has been the standard teaching of today that backward stability analysis is taughtas absolute, just as in Newtonian physics time is taught as absolute time. We will prove that itis not true in general. It depends on algorithms. We will prove that forward and mixed stabilityanalysis are absolutely invalid stability analysis in the sense that they have absolutely wrongreference points for detecting huge element growth of any algorithms (if any), even an ”ideal”or ”desirable” backward stability analysis is not so ”ideal” or ”desirable” in general. Any offorward stable, backward stable and mixed stable algorithms as in Demmel, Kahan and Parlettand others’ papers and text books [6], and [8] may not be really stable at all because they mayfail to detect and expose any potential instabilities of the algorithm in corresponding stabilityanalysis. Therefore, it is impossible to prove an algorithm is stable according to the standardteaching of today, just as it is impossible to prove a mathematical equation(s)(Maxwell’s,) is alaw of physics according to the standard teaching in Newtonian physics.
Keywords: detecting potential instabilities. mixed stability analysis. forward stability analysis.backward stability analysis. wrong reference point for detecting potential instabilities.
Given an interesting numerical algorithm, mathematicians would like to prove that the algorithmis stable. Berkeley’s math professors Kahan. Demmel and Parlett have been using forward,backward and mixed stability analysis for proving their algorithms are stable in three differentstability analysis as in [1], [2] and [4]. The standard teaching in today’s text books teachesbackward stabilitt analysis is ”desirable” or ”ideal” as in [6],p.5 or [5],p.872 and it is taught asabsolute stability analysis just as in classical physics in which time is taught as absolute time.Math professors such as Demmel, Kahan and Parlett at Berkeley teach their students if one canderive a backward error bound for an algorithm, then she/he can claim that algorithm is provenbackward stable and it is ”ideal” case scenario. In all of their accurate singular value computingpapers such as award winning paper [1] and [4], they proved their algorithms are forward stableor mixed stable. Are these forward stable, backward stable and mixed stable algorithms reallystable ? We will prove that the answer is negative in general. We will prove backward stabilityanalysis is not absolute. It depends on algorithms, see Theorem 3 and discussion right before it.Therefore, it is impossible to prove that an algorithm is stable according to standard teaching oftoday just as it is impossible to prove mathematical equations(e.g. Maxwell) is a law of physicsaccording to standard teaching in classical physics.In 1993, the author as a student at Berkeley observed in [3] that given an extremely simpleand straightforward algorithm dqd or dqds, his math professors Kahan, Demmel and Parlett atBerkeley could not determine if the algorithm has potential instabilities or not in [4],p.2 which1redicts that dqds has no potential instabilities at all with any shifts. Just take a look at dqdsone can see easily that with a right choice of shift, cancellation can occur and huge elementgrowth can occur. ‘ Any testing with dqds would indicate that dqds has infinitely potentialinstabilities due to excessive element growth with general input vectors. It follows that in 1994they could not be sure if dqds has potential instabilities or not in any numerical computingeven after they had derived tiny relative error bound. Any perturbation result can not helpthem a bit with the matter because it obviously can not determine if an algorithm has potentialinstabilities or not either. It means that in 1994 they could not prove that dqds is really stablebased on their mixed stability analysis despite they have proved it is mixed stable(as opposedto forward stable or backward stable). Whether or not an algorithm has potential instabilitiesor not can not be determined by deriving relative mixed error bound with or without anyperturbation result. This statement could be generalized to any error analysis as follows: Sinceone of three stability analyses, their mixed stability analysis fails totally to be able to detectpotential instabilities of an algorithm, dqds viewed as general vector iterations. It is possible thatthe other two stability analyses, backward stability analysis and forward stability analysis mayalso fail to detect potential instabilities due to excessive element growth unless proven otherwise.In 1994, the author asked them if they had such a proof, the answer was negative(this paperwill prove such a proof does not exist). Otherwise, in 1994 they would have had a proof to showthe author that any of the other two stability analyses is able to detect potential instabilities.Since 1994, they probably have been trying to give such a proof. This paper will prove thatsuch a proof does not exist, see Theorem 2, 3. Therefore, the student proved that they werenot sure if any algorithm has potential instabilities or not in 1994. It implies that his threefamous Berkeley math professors were not sure they could compute any digit correctly in anynumerical computing( a testing of the author in [3] shows that no correct digit can be expectedif instabilities occur due to excessive element growth). This paper will prove that even ”ideal”or ”desirable” backward stability analysis (see [5],p.872, or [6],p.5) is not able to detect excessiveelement growth in general.There was a second way to prove in the author’s Ph.D thesis [3] that the three famous Berkeleymath professors Demmel, Kahan and Parlett were not sure in 1994 they could compute any digitcorrectly in any numerical computing in theory. Given an extremely simple and straightforwardalgorithm dqd or dqds, the author observed and conjectured at Berkeley in his thesis [3],p.24that they would never be able to detect potential instabilities of dqds.(I am sure it is true,easy to check) in their standard perspectives. As a consequence, their paper [4] contradictsto a student’s Ph.D thesis in which his backward stability analysis predicts dqds as a generalvector iteration has infinitely many potential instabilities due to excessive element growth, beingconsistent with their own testing and the author’s and consistent with their own teaching butcontradicting completely to their mixed stability analysis in [4],p.2 which predicts that dqdshas no potential instabilities at all, what so ever. That is, the author proved that their papercontradicts to their own teaching and to testing with dqds viewed as general vector iterations.Again they did not have a proof that any of the other two stability analyses, backward stabilityanalysis and forward stability analysis is able to detect potential instabilities. It follows thatthey were not sure in 1994 any algorithm has potential instabilities or not. This contradictsto their claims that singular values can be computed highly accurately, including tiny ones in[4] and award winning paper [1]. Therefore, the author proved in 1994 automatically rightafter he solved the open problem of accurate singular value computing for deriving a backwardstability analysis for dqds that they were not sure that they could compute any digit correctlyin any numerical computing in theory.(they did not have a proof that any of the other two erroranalyses, forward and backward error analysis was able to detect potential instabilities )2he purpose of listing contradictions above is to prove they have not understood accuratesingular values yet up to today. To the best knowledge of the author, these contradictions stillexist today. The author’s conjecture in [3] is still true today that they would never be able todetect and expose potential instabilities of dqd or dqds. And the author has shown in his Ph.Dthesis [3] that a backward error analysis can expose all infinitely many potential instabilitiesof dqd and dqds but they have neve been able to derive one in terms of their perspectives.Therefore, they still can not determine if dqds has potential instabilities or not in any numericalcomputing. And they still do not have a proof that any of the other two stability analyses is ableto detect potential instabilities. This paper will prove such a proof does not exist in general.In mathematics, the self contradictions of theirs usually proves that the three Berkeley mathprofessors have not understood the subject yet.This paper presents a new perspective that is different from the two proofs listed above as inhis Ph.D thesis [3] that in all of their accurate singular value computing papers [1] and [4]and numerous other scientific computing papers of theirs, the famous Berkeley math professorsKahan, Demmel and Parlett have been using forward stability analysis and mixed stabilityanalysis, we will prove forward stable algorithm and mixed stable algorithm may still havepotential instabilities due to excessive element growth in general just as dqds in [4],p.2 viewedas general vector iterations. More surprisingly, we will prove even an algorithm that has beenproven backward stable based on ”ideal” or ”desirable” backward stability analysis may stillhave potential instabilities due to excessive element growth. That is, no error analysis is able todetect potential instabilities due to excessive element growth. In particular, backward stabilityanalysis is not absolute. It depends on algorithms, contrary to the standard teaching of today.It follows that it is impossible to prove that an algorithm is stable according to the standardteaching of today as in their text book [6],p.5 at Berkeley. According to Wilkinson, derivingan error bound itself is usually the least important. Much more important is to expose(detect)potential instabilities, if any, of an algorithm such as dqd or dqds. The following Theorem 1 isconsistent with Wilkinson’s perspectives as in [7],p.64.
Theorem 1
Any true stability analysis must satisfy a necessary condition that it is able to detectpotential instabilities of an algorithm due to excessive element growth. Excessive element growthcan cause potential instabilities due to potential loss of information in any numerical computingand is too important to be ignored in a true stability analysis.Proof: Otherwise the stability proof or analysis of an algorithm based on the stability analysiswill be inconsistent with computing experience. Excessive element growth can cause potentialinstabilities of an algorithm due to potential loss of information and is too important to beignored. Proof done.An extremely simple and straightforward dqd or dqds is an example that they have never beenable to prove it is stable in any numerical computing, in particular, in singular value computingbased on their mixed stability analysis for a simple reason: they can not determine if dqdshas potential instabilities or not based on their mixed stability analysis or any mathematicalanalysis of theirs. Their mixed stability analysis for dqds viewed as general vector iterationssuch as [6],p;.247 or originally in [4],p.2 have proved that dqds has no potential instabilitieswhat so ever. Any simple testing can verify that dqds viewed as general vector iterations hasinfinitely many potential instabilities due to excessive element growth.One can find forward stability analysis and mixed stability analysis in numerous papers ofDemmel, Kahan and Parlett such as in [1], [4], etc and in all of their singular value computingpapers. It follows that Kahan and his team Demmel and Parlett and others are not sure if any3ingular value computing has potential instabilities or not. See Higham [8].
Theorem 2
Forward or mixed stability analysis or any post priori error analysis is absolutelyinvalid stability analyses for any algorithm in the sense that they are absolutely invalid analysesfor detecting potential instabilities of any algorithm due to excessive element growth. Anyperturbation theory can not detect potential instabilities of an algorithm either.Proof: Forward error analysis or forward stability analysis depends on forward error measure-ment abs(y’ - y) which measures how much a point y’ moves away from a reference point y,where y’ is the computed solution for exact y=f(x). It has a wrong reference point y whichcould be moving with y’ in the same direction during a computation. It is therefore invalid fordetecting potential instabilities of any algorithm due to excessive element growth in y’ relativeto x which is fixed during computation for y. In contrast, backward stability analysis has theright reference point x. Similar argument can be applied to mixed stability analysis. That is,the mixed error analysis has a wrong reference point too. Proof done.In the proof of Theorem 2, it is pointed out that forward or mixed stability analysis is absolutelyinvalid stability analysis in the sense that they are absolutely invalid for detecting potentialinstabilities due to excessive element growth because they have absolutely wrong reference points.In contrast, backward error measurement abs(x - x’) has a right reference point x. But havinga right reference point itself is not sufficient. According to the standard teaching, if a backwarderror bound is derived for an algorithm f, then the algorithm f is said to be stable(”ideal” casescenario) and backward stability analysis is absolute, meaning it does not depends anything. Itis not true in general. When y’, the computed output for exact y=f(x) grows excessively largerelative to x during a computation, its inverse image x’ defined by y’=f(x’) may or may notgrow with it relative to x. It is possible that it may stay bounded relative to x despite y’ growsexcessively large relative to x in which case, a backward stability analysis fails to detect potentialinstabilities due to excessive element growth. One dimension vector iteration y=f(x)=1/x is suchan example. For an input x0 approaching to zero, a computed data y’=1/x0(1+u) is arbitrarilylarge, where abs(u) is less than machine precision. f inversely transforms y’ to x’ defined byy’=f(x’) so that x’ = x0/(1+u) = x0(1+u+ u +...). Therefore, (x’-x0)/x0 = u +O( u ). We seethe computed data y’ can be arbitrarily large relative to x0 but it’s inverse image x’ definedby y’=f(x’) relative to x0 stays bounded. Backward stability analysis fails totally to be able todetect excessive element growth of f when x0 approaches to zero. See the following Theorem 3. Theorem 3
Backward error analysis is not a stability analysis in general in the sense thatit can not detect potential instabilities due to excessive element growth. But for a subclass ofalgorithms f satisfying inversely increasing condition, that is, whenever y’ is increasing in a normexcessive large relative to x, where y’ is computed solution for exact y=f(x), then its inverseimage x’ defined by y’=f(x’) is also increasing in the norm excessively large relative to x, thena backward error analysis often known as backward stability analysis or a priori error analysisis able to detect potential instabilities of f due to excessive element growth for this sub class ofalgorithms.Proof: Self evident by the definition of backward error analysis.
Corollary
QR and Jacobi’s method clearly belong to this subclass of algorithms because theyinversely preserve a distance.
Theorem 4
No error analysis is able to detect potential instabilities of an algorithm dueto excessive element growth in general. Nor is any perturbation result. Whether or not an4lgorithm has potential instabilities or not can not be determined by deriving any relative errorbound with or without any perturbation result.Proof: combining Theorem 2,3 above. Any perturbation result can not be applied to analyzeany algorithm to detect and expose potential instabilities of an algorithm. Proof done.
Theorem 5 : No one is sure that she/he can compute any digit correctly in any numerical com-puting according to standard teaching of today that backward stability analysis is an absolutestability analysis for a generic algorithm as in [6],p.5 or based on forward or mixed stabilityanalysis with or without a perturbation theory.Proof: Any error analysis is not able to detect potential instabilities due to excessive elementgrowth in general according to Theorem 2, 3 above. Any matrix perturbation can not determineif an algorithm has potential instabilities or not either. It does not have any information aboutany algorithm. Therefore, no body is sure if any algorithm has potential instabilities or notaccording to the standard teaching of today. And a testing in [3] shows that no correct digit canbe expected in a computed solution in general if instabilities due to excessive element growthoccur.In conclusion, backward stability analysis is not an absolute stability analysis. It depends onalgorithms. It has been taught as absolute stability analysis, just as time is taught as absolutetime in classical physics. We have proved that it is impossible to prove that an algorithm isstable according to the standard teaching of today,just as it is impossible to prove mathematicalequations(e.g. Maxwell) is a law of physics according to standard teaching in classical physics.None of backward stable or forward stable or mixed stable algorithm as in [2],[1] and [4] is trulystable in general. It may have potential instabilities due to excessive element growth.
References [1] J. Demmel and W. Kahan(1990) ”Accurate Singular Values of Bidiagonal Matrices.”
SIAM J. Sci.Sta. Comput. , 873-912.[2] J. Demmel and K.Veselic (1992) ”Jacobi’s method is more accurate than QR.” SIAM J. Sci. Sta.Comput. , 873-912.[3] Y. Yang(1994.) Error Analysis of the qds and dqds Algorithms.
Phd thesis, Univerity of Californiaat Berkeley.[4] K.V. Fernando and B.N. Parlett(1994). “Accurate Singular Values and Differential qd Algorithms,”
Numerische Mathematik. ,191-229,[5] I.J. Dhillon and B.N. Parlett(1994). “Orthogonal Eigenvectors and Relative Gaps,” SIAM J.MatrixAnal.Appl. , No. 3, pp 858-899,[6] J.W. Demmel (1997). “Applied Numeical Linear Algebra” SIAM Press. .[7] G.H. Golub and C.F.Van Loan(1996).
Matrix Computations.
Johns Hopkins University Press, Bal-timore.[8] Nicholas J. Higham
Accuracy and Stability of Numerical Algorithms , 2rd edition. SIAM, 2rd edition. SIAM