choosing regularization parameters in iterative methods for ... - DRUM

CHOOSINGREGULARIZATIONPARAMETERSINITERATIVE METHODSFORILL-POSEDPROBLEMS
andwepresentnumericalexamples. (projectionontoanitedimensionalsubspace)followedbyregularization.Ifthediscreteproblem hashighdimension,though,typicallywecomputeanapproximatesolutionbyprojectionontoan evensmallerdimensionalspace,viaiterativemethodsbasedonKrylovsubspaces.Inthisworkwe presentecientalgorithmsthatregularizeafterthissecondprojectionratherthanbeforeit.We provesomeresultsontheapproximateequivalenceofthisapproachtootherformsofregularization Abstract.Numericalsolutionofill-posedproblemsisoftenaccomplishedbydiscretization MISHAE.KILMERyANDDIANNEP.O'LEARYz
curve,Tikhonov,TSVD,projection,Krylovsubspace Keywords.ill-posedproblems,regularization,discrepancyprinciple,iterativemethods,L-
65R30,65F20 RunningTitle:ChoosingRegularizationParameters 1.Introduction.Linear,discreteill-posedproblemsoftheform
andoccurinavarietyofapplications.Weshallassumethatthefull-rankmatrixA ismn,withmnin(2)andm=nin(1).Indiscreteill-posedproblems,Aisill-
arise,forexample,fromthediscretizationofrst-kindFredholmintegralequations (1)
min xkAxbk2;orequivalently,AAx=Ab Ax=b
toolargetosolveexactly.Inthatcase,wetypicallycomputeanapproximatesolution solution.Regularizationmethodsreplacetheoriginaloperatorbyabetter-conditioned butrelatedoneinordertodiminishtheeectsofnoiseinthedataandproducea regularizedsolutiontotheoriginalproblem.Sometimesthisregularizedproblemis usearegularizationmethodtodetermineasolutionthatapproximatesthenoise-free (1)or(2)haslittlerelationshiptothenoise-freesolutionandisworthless.Instead,we noise,incombinationwiththeill-conditioningofA,meansthattheexactsolutionof handsidebcontainsnoiseduetomeasurementand/orapproximationerror.This conditionedandthereisnogapinthesingularvaluespectrum.Typically,theright
thenewoperatorisapoorapproximationtoA.Asmallregularizationparameter basedonKrylovsubspaces. byprojectionontoanevensmallerdimensionalspace,perhapsviaiterativemethods
CCR-97-32022andbytheArmyResearchOce,MURIGrantDAAG55-97-1-0013. generallyyieldsasolutionveryclosetothenoise-contaminatedexactsolutionof(1) or(2),andhenceitsdistancefromthenoise-freesolutionalsocanbelarge.Thus, conditionedproblem,butitssolutionmaybefarfromthenoise-freesolutionsince parametersspecictothemethod.Alargeregularizationparameteryieldsanewwell-
ThisworkwassupportedbytheNationalScienceFoundationunderGrantsCCR95-03126and Theconditioningofthenewproblemiscontrolledbyoneormoreregularization
(mkilmer@ece.neu.edu) land,CollegePark,MD20742(oleary@cs.umd.edu). yDept.ofComputerandElectricalEngineering,NortheasternUniversity,Boston,MA02115 zDept.ofComputerScienceandInstituteforAdvancedComputerStudies,UniversityofMary-
1

akeyissueinregularizationmethodsistochoosearegularizationparameterthat
balancestheerrorduetonoisewiththeerrorduetoregularization.
Awisechoiceofregularizationparameterisobviouslycrucialtoobtaininguseful
approximatesolutionstoill-posedproblems.ForproblemssmallenoughthatarankrevealingfactorizationorsingularvaluedecompositionofAcanbecomputed,there
arewell-studiedtechniquesforcomputingagoodregularizationparameter.These
techniquesincludetheDiscrepancyPrinciple[8],generalizedcross-validation(GCV)
[9],andtheL-curve[15].Forlargerproblemstreatedbyiterativemethods,though,
theparameterchoiceismuchlessunderstood.Ifregularizationisappliedtothe
projectedproblemthatisgeneratedbytheiterativemethod,thenthereareessentially
tworegularizationparameters:oneforthestandardregularizationalgorithms,such
asTikhonovortruncatedSVD,andonecontrollingthenumberofiterationstaken.
Onesubtleissueisthatthestandardregularizationparameterthatiscorrectforthe
discretizedproblemmaynotbetheoptimaloneforthelower-dimensionalproblem
actuallysolvedbytheiteration,andthisobservationleadstotheresearchdiscussed
inthispaper.Atrstglance,therecanappeartobealotofworkassociatedwith
theselectionofagoodregularizationparameter,andmanyalgorithmsproposedin
theliteratureareneedlesslycomplicated.Butbyregularizingafterprojectionbythe
iterativemethod,sothatweareregularizingthelowerdimensionalproblemthatis
actuallybeingsolved,muchofthisdicultyvanishes.
Thepurposeofthispaperistopresentparameterselectiontechniquesdesigned
toreducetheregularizationworkforiterativemethodssuchasKrylovsubspacetechniques.Ourpaperisorganizedasfollows.Inx2,wewillgiveanoverviewofthe
regularizationmethodswewillbeconsidering,andwefollowupinx3bysurveying
somemethodsforchoosingthecorrespondingregularizationparameters.Inx4,we
showhowparameterselectiontechniquesfortheoriginalproblemcanbeappliedinsteadtoaprojectedproblemobtainedfromaniterativemethod,greatlyreducingthe
costwithoutmuchdegradationinthesolution.Wegiveexperimentalresultsinx5
andconclusionsandfutureworkinx6.
2.Regularizationbackground.Inthefollowingweshallassumethatb=
btrue+e,wherebtruedenotestheunperturbeddatavectorandedenoteszero-mean
whitenoise.WewillalsoassumethatbtruesatisesthediscretePicardcondition;
thatis,thespectralcoecientsofbtruedecayfaster,onaverage,thanthesingular
values.
Undertheseassumptions,itiseasytoseewhytheexactsolutionto(1)or(2)is
hopelesslycontaminatedbynoise.Let^U^Vdenotethesingularvaluedecomposition
ofA,wherethecolumnsof^Uand^Varethesingularvectors,andthesingularvalues
areorderedas12:::n.Thenthesolutionto(1)or(2)isgivenby
x=nXi=1^uib
i^vi=nXi=1^uibtrue
i+^uie
i^vi:
(3)
Asaconsequenceofthewhitenoiseassumption,j^uiejisroughlyconstantforalli,
whilethediscretePicardconditionguaranteesthatj^uibtruejdecreaseswithifaster
thanidoes.ThematrixAisill-conditioned,sosmallsingularvaluesmagnifythe
correspondingcoecients^uieinthesecondsum,anditisthislargecontributionof
noisefromtheapproximatenullspaceofAthatrenderstheexactsolutionxdened
in(3)worthless.Thefollowingregularizationmethodstryindierentwaystolessen
thecontributionofnoisetothesolution.Forfurtherinformationonthesemethods,
see,forexample,[17]. 2

2.1.Tikhonovregularization.OneofthemostcommonmethodsofregularizationisTikhonovregularization[34].Inthismethod,theproblem(1)or(2)is
replacedwiththeproblemofsolving
min
xkAxbk2+2kLxk2
(4)
whereLdenotesamatrix,oftenchosentobetheidentitymatrixIoradiscrete
derivativeoperator,andisapositivescalarregularizationparameter.Foreasein
notation,wewillassumethatL=I.Solving(4)isequivalenttosolving
(AA+2I)x=Ab:
(5)
Inanalogywith(3)wehave
x=nXi=1i^uibtrue
2i+2+i^uie
2i+2^vi:
(6)
Inthissolution,thecontributionsfromnoisecomponents^uieforvaluesofin`.
Analternative,yetrelated,approachtoTSVDisanapproachintroducedbyRust
[31]wherethetruncationstrategyisbasedonthevalueofeachspectralcoecient
^uibitself.Thestrategyistoincludeinthesum(3)onlythosetermscorrespondingto
aspectralcoecient^uibwhosemagnitudeisgreaterthanorequaltosometolerance
,whichcanberegardedastheregularizationparameter.
2.3.Projectionanditerativemethods.Solving(5)or(7)canbeimpractical
ifnislarge,butfortunately,regularizationcanbeachievedthroughprojectiononto
asubspace;see,forexample,[7].ThetruncatedSVDisanexampleofonesuch
projection:thesolutionisconstrainedtolieinthesubspacespannedbythesingular
vectorscorrespondingtothelargestn`singularvalues.Otherprojectionscanbe
moreeconomical.Ingeneral,weconstrainourregularizedsolutiontolieinsome
k-dimensionalsubspaceofCn,spannedbythecolumnsofannkmatrixQ(k).For
example,wechoosex(k)
reg=Q(k)y(k)wherey(k)solves
min
y2CkkAQ(k)ybk2
(8) 3

orequivalently (Q(k))AAQ(k)y=(Q(k))Ab:
(9)
Theideaisthatwithanappropriatelychosensubspace,theoperator(Q(k))AAQ(k)
willbebetterconditionedthantheoriginaloperatorandhencethatx(k)
regwillapproximatextruewellonthatsubspace.
Thisprojectionisoftenachievedthroughtheuseofiterativemethodssuchasconjugategradients,GMRES,QMR,andotherKrylovsubspacemethods.Thematrix
Q(k)thencontainsorthonormalcolumnsgeneratedviaaLanczostridiagonalizationor
bidiagonalizationprocess[27,1].Inthiscase,Q(k)isabasisforsomek-dimensional
Krylovsubspace(i.e.,thesubspaceKk(c;K)spannedbythevectorsc;Kc;:::;Kk1c
forsomematrixKandvectorc).Theregularizedsolutionsx(k)
regaregeneratediterativelyasthesubspacesarebuilt.KrylovsubspacealgorithmssuchasCG,CGLS,
GMRES,andLSQRtendtoproduce,atearlyiterations,solutionsthatresemblextrue
inthesubspacespannedby(right)singularvectorsofAcorrespondingtothelargest
singularvalues.Atlateriterations,however,thesemethodsstarttoreconstructincreasingamountsofnoiseintothesolution.Thisisduetothefactthatforlargek,
theoperator(Q(k))AAQ(k)approachestheill-conditionedoperatorAA.Therefore,thechoiceoftheregularizationparameterk,thestoppingpointfortheiteration
andthedimensionofthesubspace,isveryimportant.1
2.4.Hybridmethods:projectionplusregularization.Anotherimportant
familyofregularizationmethods,oftenreferredtoashybridmethods[17],wasintroducedbyO'LearyandSimmons[27].Thesemethodscombineaprojectionmethod
withadirectregularizationmethodsuchasTSVDorTikhonovregularization.The
problemisprojectedontoaparticularsubspaceofdimensionk,buttypicallythe
restrictedoperatorin(9)isstillill-conditioned.Therefore,aregularizationmethod
isappliedtotheprojectedproblem.Sincethedimensionkisusuallysmallrelative
ton,regularizationoftherestrictedproblemismuchlessexpensive.Yet,withan
appropriatelychosensubspace,theendresultscanbeverysimilartothoseachieved
byapplyingthesamedirectregularizationtechniquetotheoriginalproblem.Wewill
becomemorepreciseabouthow\similar"thesolutionsareinx4.5.BecausetheprojectedproblemsareusuallygeneratediterativelybyaLanczosmethod,thisapproach
isusefulwhenAissparseorstructuredinsuchawaythatmatrix-vectorproducts
canbehandledecientlywithminimalstorage.
3.Existingparameterselectionmethods.Inthissection,wediscussasamplingoftheparameterselectiontechniquesthathavebeenproposedintheliterature.
Theydierintheamountofaprioriinformationrequiredaswellasinthedecision
criteria.
3.1.TheDiscrepancyPrinciple.Ifsomeextrainformationisavailable{for
example,anestimateofthevarianceofthenoisevectore{thentheregularization
parametercanbechosenrathereasily.Morozov'sDiscrepancyPrinciple[25]saysthat
ifistheexpectedvalueofkek2,thentheregularizationparametershouldbechosen
sothatthenormoftheresidualcorrespondingtotheregularizedsolutionxregis;
thatis, kAxregbk2=;
(10)
1Usually,smallvaluesoftheregularizationparametercorrespondtoaclosersolutiontothenoisy
equation,butdespitethis,wewillcallk,ratherthan1=k,theregularizationparameter.
4

10 3
parameterselectionmethoddoesnotdependonaprioriknowledgeaboutthenoise Othermethodsbasedonknowledgeofthevariancearegiven,forexample,in[12,5]. regularizationtotheprobleminExample2 where>1issomepredeterminedrealnumber.Notethatas!0;xreg!xtrue. 3.2.GeneralizedCross-Validation.TheGeneralizedCross-Validation(GCV) Fig.1.ExampleofatypicalL-curve.ThisparticularL-curvecorrespondstoapplyingTikhonov
10 2
10 1
10 −3 10 −2 10 −1 10 0 10 1
|| r λ
||
solutionx.InTikhonovregularization,forexample,A]is variance.ThisideaofGolub,Heath,andWahba[9]istondtheparameterthat minimizestheGCVfunctional
GCVchoosesaregularizationparameterthatisnottoodependentonanyonedata (11) whereA]denotesthematrixthatmapstherighthandsidebontotheregularized G()=k(IAA])bk2 (AA+2I)1A: (trace(IAA]))2;
measurement[11,12.1.3]. erroranderrorduetonoiseistoplotthenormoftheregularizedsolutionversus 3.3.TheL-Curve.Onewaytovisualizethetradeobetweenregularization
2
sinceforvalueshigherthanthis,theresidualincreaseswithoutreducingthenorm increasesrapidlywithoutmuchdecreaseinresidual.Inpractice,onlyafewpoints ontheL-curvearecomputedandthecornerislocatedbyapproximatemethods, ofthesolutionmuch,whileforvaluessmallerthanthis,thenormofthesolution thecorrespondingresidualnormforeachofasetofregularizationparametervalues.
estimatingthepointofmaximumcurvature[19]. TheresultistheL-curve,introducedbyLawsonandpopularizedbyHansen[15]. SeeFigure1foratypicalexample.Astheregularizationparameterincreases,noise Intuitively,thebestregularizationparametershouldlieonthecorneroftheL-curve, isdamped,sothatthenormofthesolutiondecreaseswhiletheresidualincreases.
dicultproblem,andeachmethodhassevereaws. atechoiceofregularizationparameter{especiallyforprojectionalgorithms{isa pendonspecicknowledgeaboutthenoisevector. 3.4.Disadvantagesoftheseparameterchoicealgorithms.Theappropri-
LikeGCV,thismethodofdeterminingaregularizationparameterdoesnotde-
5
|| x λ
|| 2

egularizationmethodswithvariousparameterselectiontechniques.Notation: Summaryofadditionalopsneededtocomputetheregularizationparameterforeachfour Tikhonov Rust'sTSVDO(mn2)O(mlogm)O(mlogm)O(mlogm) ProjectionBasiccost O(mn2)O(p(m+n))O(p(n+m))O(p(m+n))
O(m) Disc. AddedCost
qisthecostofmultiplicationofavectorbyA. O(qk) Table1 0 O(m) GCV O(q) O(m+n) L-curve
kisthedimensionoftheprojection. pisthenumberofdiscreteparametersthatmustbetried; mandnareproblemdimensions. O(q)
correctestimateofthevariance,thesolutionstendtobeoversmoothed[20,pg.96] itdiculttodeterminetheoptimalnumerically[35]. orastheerrornormgoestozero[6].Allmethodsthatassumenoknowledgeofthe nonideal.Thesolutionestimatesfailtoconvergetothetruesolutionasn!1[36] (seealsothediscussioninx6.1of[15]). knowinginformationthatisoftenunavailableorincorrectlyestimated.Evenwitha TheL-curveisusuallymoretractablenumerically,butitslimitingpropertiesare OnenoteddicultywithGCVisthatGcanhaveaveryatminimum,making TheDiscrepancyPrincipleisconvergentasthenoisegoestozero,butitrelieson
thesesystemsusingprojectionmethods. glance,itappearsthatforTikhonovregularization,multiplesystemsoftheform (5)mustbesolvedinordertoevaluatecandidatevaluesoffortheDiscrepancy PrincipleortheL-curve.Techniqueshavebeensuggestedintheliteratureforsolving errornorm{includingGCV{havethislatterproperty[6]. ThecostofthesemethodsistabulatedinTable1. Forfurtherdiscussionandreferencesaboutparameterchoicemethods,see[5,17].
matricesmatricesC()=AA+I,andtheysuggestsolvingthesystemssimultaneouslyusingaGalerkinprojectionmethodonasequenceof\seed"systems.Although
thisiseconomicalinstorage,itcanbeunnecessarilyexpensiveintimebecausethey ChanandNg[4],forexample,notethatthesystemsinvolvethecloselyrelated 3.5.Previousworkonparameterchoiceforhybridmethods.Atrst
redundancyingeneratingtheKrylov-subspacesforeachi.Thesecondmethod forlaterproblems.LikeChanandNg,thisalgorithmdoesnotexploitanyofthe theiterativeprocessearlyforlargeandusingprevioussolutionsasstartingguesses theypropose,however,doesexploittheredundancysothattheCGiteratesforallk donotexploitthefactthatforeachxedk,theKrylovsubspaceKk(Ab;C())is
systemscanbeupdatedsimultaneouslywithnoextramatrix-vectorproducts.They thesameforallvaluesof.
stoptheir\shiftedcg"algorithmwhenkAxbk2foroneoftheirvalues. whichtheyuseconjugategradientstosolveksystemsoftheform(5),truncating satisestheDiscrepancyPrinciple(10).Therstisa\truncatedcg"approachin FrommerandMaass[8]proposetwoalgorithmsforapproximatingthethat
forthisparticularsystemtoconverge.Wenotethatwhilethealgorithmswepropose inx4forndingagoodvalueofarebasedonthesamekeyobservationregarding Thusthenumberofmatrix-vectorproductsrequiredistwicethenumberofiterations 6

theKrylovsubspace,ourmethodswillusuallyrequirelessworkthantheshiftedcg algorithm. generatedbythematricesC()usingaLanczosbidiagonalizationprocess.Fromthis, proachfortheTikhonovL-curveproblem.Theirmethodismorecomplicatedthanthe problemthatisactuallybeingused. izationontheprojectedproblem,sincethisistheapproximationtothecontinuous theyapproximatethebestparameterforTikhonovregularizationwithoutprojection. Inx4,wechooseinsteadtoapproximatethebestparameterforTikhonovregular-
methodsweproposebecausetheymaintainnonnegativityconstraintsonthevariables. Calvetti,Golub,andReichel[3]computeupperandlowerboundsontheL-curve KaufmanandNeumaier[21]suggestanenvelopeguidedconjugategradientap-
[27].Forexample,inplaceof(4),wesolve soweimplicitlyassumethattheproblemhasbeenleft-preconditionedor\ltered" numberofiterationsisunnecessary.Thatviewpointisechoedinthiscurrentwork, viewpointthattheproblemshouldbepreconditionedappropriatelysothatamassive pointfortheprojectedSVD.Theiremphasisisonstablewaystomaintainanaccurate problems.Bjorck,Grimme,andvanDooren[2]useGCVtodeterminethetruncation factorizationwhenmanyiterationsareneeded,andtheyusefullreorthogonalization andimplicitrestartstrategies.O'LearyandSimmons[27]takeasomewhatdierent SubstantialworkhasalsobeendoneonTSVDregularizationoftheprojected
rightpreconditioning,whichamountstosolving,intheTikhonovcase, forasquarepreconditionerM.See[14,26,24,23]forpreconditionersappropriatefor certaintypesofill-posedproblems.Notethatwecouldalternatelyhaveconsidered min xkM1AxM1bk2+2kxk2
eectivelychangesthebalancebetweenthetwotermsintheminimization. forythensettingx=M1y.Notethateitherleftorrightpreconditioning ofPaigeandSaunders[29]andtheGMRESalgorithmofSaadandSchultz[33]. proachestoregularizationusingKrylovmethods.ManyKrylovmethodshavebeen proposed;foreaseofexpositionwefocusonjusttwoofthese:theLSQRalgorithm 4.Regularizingtheprojectedproblem.Inthissectionwedevelopnineap-
min ykAIM1yb0k;
onalizationintroducedbyGolubandKahan[10].Givenavectorb,thealgorithmis asfollows[29,Alg.Bidiag1]: TheLSQRalgorithmofPaigeandSaunders[29]iterativelycomputesthebidiag-
Computeascalar1andavectoru1oflengthonesothat1u1=b.
algorithmcanberewritteninmatrixformbyrstdeningthematrices Thevectorsui;viarecalledtheleftandrightLanczosvectorsrespectively.The Fori=1,2,... Similarly,determine1andv1sothat1v1=ATu1. Endfor Leti+1ui+1=Aviiuiandi+1vi+1=ATui+1i+1vi, wherethenon-negativescalarsi+1andi+1arechosen sothatui+1andvi+1havelengthone. Uk[u1;:::;uk];Vk[v1;:::;vk]; 7

Witheidenotingtheithunitvector,thefollowingrelationscanbeestablished: Bk2641
(12) 22
(13) AVk=Uk+1Bk; b=1u1=1Uk+1e1; 3... ...k k+1 3 75:
(14) (15) wherethesubscriptonIdenotesthedimensionoftheidentity. (16) whereSdenotesthek-dimensionalsubspacespannedbytherstkLanczosvectors Nowsupposewewanttosolvemin V kVk=Ik; ATUk+1=VkBTk+k+1vk+1eTk+1;
vi.Thesolutionweseekisoftheformx(k)=Vky(k)forsomevectory(k)oflengthk. Dener(k)=bAx(k)tobethecorrespondingresidual.Fromtherelationsabove, x2SkbAxk2 Uk+1Uk+1=Ik+1;
observethatinexactarithmetic
(18) (17) Therefore,theprojectedproblemwewishtosolveis SinceUk+1has,inexactarithmetic,orthonormalcolumns,wehave r(k)=1u1AVky(k) kr(k)k2=k1e1Bky(k)k2:
y(k)k1e1Bky(k)k2: =Uk+1(1e1Bky(k))
(19) Typicallykissmall,soreorthogonalizationtocombattheeectsofinexactarithmetic volvingthebidiagonalmatrix:BkBky(k)=1Bke1: Solvingthisminimizationproblemisequivalenttosolvingthenormalequationsin-
ofitssingularvaluesapproximatesomeofthesmallsingularvaluesofA.Therefore discussoptionsindetailbelow.Asalludedtoinx4,theideaistogeneratey(k) mightormightnotbenecessary.ThematrixBkmaybeill-conditionedbecausesome regularizedsolutionto(18),andthentocomputearegularizedsolutionto(16)as solvingtheprojectedproblemmightnotyieldagoodsolutiony(k).However,we canuseanyofthemethodsofSection3toregularizethisprojectedproblem;we x(k) relationships. relations.Here,though,theUandVmatricesareidenticalandtheBmatrixis upperHessenbergratherthanbidiagonal.Conjugategradientswouldyieldsimilar reg=Vky(k) IfweusedthealgorithmGMRESinsteadofLSQR,wewouldderivesimilar Forcostcomparisonsforthesemethods,seeTables1and2.Storagecomparisons reg. reg,the
aregiveninTables3and4. 8

4.1.Regularizationbyprojection.Asmentionedearlier,ifweterminatethe
iterationafterksteps,wehaveprojectedthesolutionontoakdimensionalsubspace
andthishasaregularizingeectthatissometimessucient.Determiningthebest
valueofkcanbeaccomplished,forinstance,byoneofourthreemethodsofparameter
choice:
1.DiscrepancyPrinciple.
Inthiscase,westoptheiterationforthesmallestvalueofkforwhichkrkk
.BothLSQRandGMREShaverecurrencerelationsfordeterminingkrkk
usingscalarcomputations,withoutcomputingeitherrkorxk[29,32].
2.GCV.
Fortheprojectedproblems(seex4.1)denedbyeitherLSQRorGMRES,
theoperatorAA]isgivenbyUk+1BkBykUk+1
whereBykisthepseudo-inverseofthematrixBk.Thusfrom(11),theGCV
functionalis[17] G(k)=kr(k)k2
(mk)2:
WenotethatthereareinfacttwodistinctdenitionsforBykandhencetwo
denitionsforthedenominatorinG(k);forsmallenoughk,thetwoare
comparable,andthedenitionweusehereislessexpensivetocalculate[18,
x7.4].
3.L-Curve.
TodeterminetheL-curveassociatedwithLSQRorGMRES,estimatesof
krkk2andkxkk2areneededforseveralvaluesofk.Usingeitheralgorithm,
wecancomputekrkk2withonlyafewscalarcalculations.PaigeandSaunders
giveasimilarmethodforcomputingkxkk2[29],but,withGMRES,thecost
forcomputingkxkk2isO(k2).InusingthismethodorGCV,onemustgoa
fewiterationsbeyondtheoptimalkinordertoverifytheoptimum[19].
4.2.RegularizationbyprojectionplusTSVD.Ifprojectionalonedoesnot
regularize,thenwecancomputetheTSVDregularizedsolutiontotheprojected
problem(19).WeneedtheSVDofthe(k+1)kmatrixBk.ThisrequiresO(k3)
operations,butcanalsobecomputedfromtheSVDofBk1inO(k2)operations[13].
Clearly,westillneedtousesometypeofparameterselectiontechniquetonda
goodvalueof`(k).First,noticethatitiseasytocomputethenormsoftheresidual
andthesolutionresultingfromneglectingthe`smallestsingularvalues.Ifjkisthe
componentofe1inthedirectionofthej-thleftsingularvectorofBk,andifjis
thej-thsingularvalue(orderedlargesttosmallest),thentheresidualandsolution
2-normsare10@k+1
X
j=k`(k)+12jk1A1=2and10@k`(k)
Xj=1jk
j21A1=2:
(20)
Usingthisfact,wecanuseanyofourthreesamplemethods:
1.DiscrepancyPrinciple.
Letr(k)
`denotethequantitybAx(k)
`andnotethatby(13)andorthonormality,kr(k)
`k2isequaltotherstquantityin(20).Therefore,wechoose
`(k)tobethelargestvalueforwhichkr(k)
`k,ifsuchavalueexists.
9

2.GCV.
Anotheralternativeforchoosing`(k)istouseGCVtocompute`(k)for
theprojectedproblem.TheGCVfunctionalforthekthprojectedproblem
isobtainedbysubstitutingBkforAandB]kforA],andsubstitutingthe
expressionoftheresidualin(20)forthenumeratorin(11):
Gk(`)=21Pk+1
j=k`+12jk
(`+1)2:
3.L-Curve.
WenowhavemanyL-curves,oneforeachvalueofk.Thecoordinatevalues
in(20)formthediscreteL-curveforagivenk,fromwhichthedesiredvalue
of`(k)canbechosenwithoutformingtheapproximatesolutionsorresiduals.
Askincreases,thevalue`(k)chosenbytheDiscrepancyPrinciplewillbemonotonicallynondecreasing.
4.3.RegularizationbyprojectionplusRust'sTSVD.AsinstandardTSVD,
touseRust'sversionofTSVDforregularizationoftheprojectedproblemrequires
thatwecomputetheSVDofthe(k+1)kmatrixBk.Usingthepreviousnotation,
Rust'sstrategyistoset y(k)
=X
i2I(k)
ik
iq(k)
i
whereq(k)
jaretherightsingularvectorsofBkandI(k)
=fig.We
focusonthreewaystodetermine:
1.DiscrepancyPrinciple.
Usingthenotationfromtheprevioussection,thenormoftheregularizedsolutionisgivenby1(Pi62I(k)
2ik)1=2=kr(k)
k2:Accordingtothediscrepancy
principle,wemustchoosesothattheresidualislessthan.Inpractice,
thiswouldrequirethattheresidualbeevaluatedbysortingthevaluesjikj
andaddingtermsinthatorderuntiltheresidualnormislessthan.
2.GCV.
Letusdenotebycard(I(k)
)thecardinalityofthesetI(k)
.From(11),itis
easytoshowthattheGCVfunctionalcorrespondingtotheprojectedproblem
forthisregularizationtechniqueisgivenby
Gk()=21Pi2I(k)
2ik
(k+1card(I(k)
))2:
Inpractice,foreachkwerstsortthevaluesjikj;i=1;:::;kfromsmallest
tolargest.Thenwedenekdiscretevaluesjtobeequaltothesevalues
with1beingthesmallest.Weset0=0.Notethatbecausethevaluesof
j;j=1;:::;karethesortedmagnitudesoftheSVDexpansioncoecients,
wehaveGk(j)=21(j(k+1);kj2+Pji=12j)
(j+1)2 ;j=0;:::;k:
Finally,wetaketheregularizationparametertobethejforwhichGk(j)
isaminimum. 10

3.L-Curve.
AswithstandardTSVD,wenowhaveoneL-curveforeachvalueofk.For
xedk,ifwedenethej;j=0;:::;kaswedidforGCVaboveandwe
reordertheiinthesamewaythatthejikjwerereorderedwhensorted,
thenwehave
kx(k)
jk2=21kX
i=j+1i
i2;kr(k)
jk2=21(j(k+1);kj2+jXi=12j)j=0;:::;k:
Whenthesesolutionandresidualnormsareplottedagainsteachotheras
functionsof,thevalueofjcorrespondingtothecornerisselectedasthe
regularizationparameter.
4.4.RegularizationbyprojectionplusTikhonov.Finally,letusconsider
usingTikhonovregularizationtoregularizetheprojectedproblem(18)forsomeintegerk.Thus,foragivenregularizationparameter,wewouldliketosolve
min
yk1e1Bkyk2+2kyk2;
(21)
or,equivalently, min
yk1e1
0Bk
Iyk2:
(22)
Thesolutiony(k)
toeitherformulationsatises
(BkBk+2I)y(k)
=1Bke1:
(23)
Using(13)and(15),weseethaty(k)
alsosatises
(V
kAAVk+2I)y(k)
=V
kAb:
(24)
Therefore, y(k)
=argminykAIVkyb0k2:
Usingx(k)
=Vky(k)
,wehave
x(k)
=argminx2SkAxbk2+2kxk2:
Thusask!n,thebackprojectedregularizedsolutionx(k)
approachesthesolution
to(4).
Weneedtoaddresshowtochooseasuitablevalueof.
1.DiscrepancyPrinciple.
Notethatinexactarithmetic,wehave
r(k)
=bAx(k)
=Uk+1(1e1Bky(k)
):
(25)
HencekBky(k)
1e1k2=kr(k)
k2.Therefore,tousetheDiscrepancyPrinciplerequireswechoosesothatkr(k)
k2,withpdiscretetrialvaluesj.
Foragivenk,wetaketobethelargestvaluejforwhichkr(k)
k2

2.GCV.
Letusdene(Bk)ytobetheoperatormappingtherighthandsideofthe
projectedproblemontotheregularizedsolutionoftheprojectedproblem:
(Bk)y=(BkBk+2I)1Bk:
GiventheSVDofBkasabove,thedenominatorintheGCVfunctional
denedfortheprojectedproblem(referto(11))is
0@k+1kXj=12j
2j+21A2:
Thenumeratorissimplykr(k)
k2.Forvaluesofkn,itisfeasibletocompute
thesingularvaluesofBk.
3.L-Curve.
TheL-curveiscomprisedofthepoints(kBky(k)
1e1k2;ky(k)
k2).Butusing
(25)andtheorthonormalityofthecolumnsofVk,weseethesepointsare
precisely(kr(k)
k2;kx(k)
k2).Forpdiscretevaluesof,i;1ip,the
quantitieskr(k)
ik2andkx(k)
ik2canbeobtainedbyupdatingtheirrespective
estimatesatthe(k1)stiteration.2
4.5.CorrespondencebetweenDirectRegularizationandProjection
PlusRegularization.Inthissection,wearguewhytheprojectionplusregularizationapproachescanbeexpectedtoyieldregularizedsolutionsnearlyequivalentto
thedirectregularizationcounterpart.Thefollowingtheoremestablishesthedesired
resultforthecaseofTikhonovvs.projectionplusTikhonov.
Theorem4.1.Fix>0anddenex(k)
tobethekthiterateofconjugate
gradientsappliedtotheTikhonovproblem
(AA+2I)x=Ab:
Lety(k)
betheexactsolutiontotheregularizedprojectedproblem
(BkBk+2I)y=Bk(e1)
whereBk;VkarederivedfromtheoriginalproblemAA=Ab,andsetz(k)
=Vky(k)
.
Thenz(k)
=x(k)
.
Proof:Bythediscussionatthebeginningofx4.4andequations(23)and(24),it
followsthaty(k)
solvesV
k(AA+2I)Vky=V
kAb:
NowthecolumnsofVkaretheLanczosvectorswithrespecttothematrixAAand
right-handsideAb.ButthesearethesameastheLanczosvectorsgeneratedwith
respecttothematrixAA+2Iandright-handsideAb.ThereforeVky(k)
isprecisely
thekthiterateofconjugategradientsappliedto(AA+2I)x=Ab[11,pg.495].
Hencez(k)
=x(k)
.2
2Thetechnicaldetailsoftheapproacharefoundin[28,pp.197-198],fromwhichweobtain
kr(k)
k=qkr(k)
k2+2kx(k)
k2.Theimplementationdetailsforestimatingkx(k)
kandkr(k)
kwere
takenfromthePaigeandSaundersalgorithmathttp://www.netlib.org/linalg/lsqr.
12

Projectionplus{Disc.GCVL-curve
Tikhonov O(pk)O(k3)O(pk)
TSVD O(k3)O(k3)O(k3)
Rust's O(k3)O(k3)O(k3)
Table2
Summaryofopsforprojectionplusinnerregularizationwithvariousparameterselection
techniques,inadditiontotheO(qk)opsrequiredforprojectionitself.Herekisthenumberof
iterations(ie.thesizeoftheprojection)takenandpisthenumberofdiscreteparametersthatmust
betried.
LetusturntothecaseofTSVDregularizationappliedtotheoriginalproblem
vs.theprojectionplusTSVDapproach.Directcomputationconvincesusthatthe
twomethodscomputethesameregularizedsolutionifk=nandarithmeticisexact.
Anapproximateresultholdsinexactarithmeticwhenwetakekiterations,with
n`=j

parameterselectiontechniques.Theoriginalmatrixismnwithqnonzeros,pisthenumberof Summaryofadditionalstorageforeachoffourregularizationmethodsundereachofthree Tikhonov Rust'sTSVDO(^q)O(m)O(m)O(m) ProjectionBasiccost O(kn)O(1)O(k)O(k) O(^q)O(1)O(p)O(p) O(^q)O(1)O(m)O(m) Table3Disc.GCVL-curve
AddedCost
assumedtotake^qstorage. discreteparametersthatmustbetried,kiterationsareusedinprojection,andthefactorizationsare
izationapproach,variousparameterselectiontechniques.Herepdenotesthenumberofdiscrete parameterstried.EachoftheseregularizationmethodsalsorequiresustosavethebasisVorelse regenerateitinordertoreconstructx. Summaryofstorage,notincludingstorageforthematrix,forprojectionplusinnerregular-
Projectionplus{Disc.GCVL-curve Tikhonov Rust'sTSVDO(k)O(k+p)O(k+p) O(1)O(p)O(p) O(1)O(k)O(k)
experimentswerecarriedoutusingMatlabandHansen'sRegularizationTools[16], 5.Numericalresults.Inthissection,wepresenttwonumericalexamples.All Table4
normdierencebetweentheregularizedsolutionsandtheexactsolutions.Inboth tobethemagnitudesofthespectralcoecientsofbsortedinincreasingorder. solutionswereknowninbothexamples,weevaluatedthemethodsusingthetwo-
exampleswhenweappliedRust'smethodtotheoriginalproblem,theiweretaken withIEEEdoubleprecisionoatingpointarithmetic.Sincetheexact,noise-free
techniqueswereappliedtogetherwithoneofthefourregularizationmethodsonthe valuesoftheregularizationparameterschosenwhenthethreeparameterselection generatedbtrue=Axtrueandthencomputedthenoisyvectorbasb+e,whereewas weregeneratedusingthefunctionbaartinHansen'sRegularizationToolbox.We generatedusingtheMatlabrandnfunctionandwasscaledsothatthenoiselevel, kbtruek,was103.TheconditionnumberofAwasontheorderof1019.
Manyvaluesofweretested:log10=6;5:9;:::;2.Table5displaysthe 5.1.Example1.The200200matrixAandtruesolutionxtrueforthisexample
originalproblem.Sincekek2=5:3761E4,wesetthatdenesthediscrepancy principleastheverycloseapproximation5:5E4. solution.Therelativeerrorvaluesforregularizedsolutionscorrespondingtothe regularizedsolutionwiththeminimumrelativeerrorwhencomparedagainstthetrue correspondingtosolutionnormsgreaterthan106fortheTSVDmethodswerenot weredeterminedusingHansen'slcornerfunction,withthemodicationthatpoints parametersinTable5aregiveninTable6.NotethatusingGCVtodeterminea regularizationparameterforRust'sTSVDresultedinanextremelynoisysolution withhugeerror. Thelastcolumninthetablegivesthevalueoftheparameterthatyieldeda ThecornersoftheL-curvesfortheTikhonov,projection,andTSVDmethods 14

Tikhonov1:259E31:995E42:512E4 TSVD` Rust'sTSVD1:223E49:645E71:223E41:259E4or1:223E4 Projectionk Disc. 197 197 196 5:012E5 optimal
Tikhonov.1330.1110.1084.0648 Example1:parametervaluesselectedforeachmethod. 4Disc.GCVL-curveoptimal
4Table56 196
Rust'sTSVD.12137E+14.1213.1213 .1663.1213.1663.12136
theoriginalproblem,wheretheregularizationmethodwaschosenusingmethodsindicated. oneofthethreeregularizationmethodsconsidered.Foreachofthethreemethods, considered(otherwise,afalsecornerresulted). Example1:comparisonofkxtruexregk2=kxtruek2foreachof4regularizationmethodson Next,weprojectedusingLSQRandthenregularizedtheprojectedproblemwith Projection.1134.1207.1134.1134 Table6
wecomputedregularizationparametersfortheprojectedproblemusingDiscrepancy, GCV,andL-curve,thencomputedthecorrespondingregularizedsolutions;theparametersthatwereselectedineachcaseatiterations10and40aregiveninTables7
andprojectionplusRust'sTSVD.WeshouldalsonotethatforTikhonov,withand viaprojectionplusTikhonovforprojectionsizeof10usingeithertheDiscrepancy thosetechniqueshadbeenusedwithTikhonovontheoriginalproblemtodetermine aregularizedsolution.SimilarstatementscanbemadeforprojectionplusTSVD PrincipleortheL-curvetondtheregularizationparametergivesresultsasgoodasif and9respectively.Asbefore,thelcornerroutinewasusedtodeterminethecorners oftherespectiveL-curves. withoutprojection,noneoftheerrorsinthetablesisoptimal;thatis,noparameter selectiontechniquesevergavetheparameterforwhichtheerrorwasminimal. ComparingTable6and8,weobservethatcomputingtheregularizedsolution
was1:65107. truesolutionvectorxtrueisdisplayedasthetoppictureinFigure2.Wegenerated Toeplitzmatrixwithbandwidth16andexponentialdecayacrosstheband.3The 103.ThevectorbisshowninthebottomofFigure2.TheconditionnumberofA btrue=Axtrueandthencomputedthenoisyvectorbasb+e,whereewasgenerated usingtheMatlabrandnfunctionandwasscaledsothatthenoiselevel,kek 5.2.Example2.The255255matrixAforthisexamplewasasymmetric
rorof9:48E2,andseveralmoreiterationswereneededfortheL-curvemethodto principletobe8:00E2. noisevectorwas7:16E2,sowetookthevalueofthatdenesthediscrepancy Wegeneratedourdiscreteiusinglog10=5;4:9;:::;1.Thenormofthe Inthisexample,ittook61iterationsforLSQRtoreachaminimumrelativeer-
kbtruek,was
sigma=5andt=[exp(([0:band1]:^2)=(2sigma^2));zeros(1;Nband)]withband=16. 3ItwasgeneratedusingtheMatlabcommandA=(1=(2pisigma))toeplitz(t);where 15

TSVD,andRust'sTSVD. Example1,iteration10:regularizationparametersselectedforprojectionplusTikhonov, Tikhonov(k)1:259E31:995E31:995E45:012E5 TSVD`(k) Rust'sTSVD(k)1:679E41:773E41:679E51:679E5 Disc. 2 GCVL-curveoptimal
Tikhonov.1330.1486.1084.0648 Disc.GCVL-curveoptimal Table73 2 2
TSVD,andRust'sTSVD. Example1,iteration10:comparisonofkxtruexregk2=kxtruek2forprojectionplusTikhonov,
Rust'sTSVD.1213.1663.1213.1213 Disc. .1663.3451.1663.1213 Table8
TSVD,andRust'sTSVD. Example1,iteration40:regularizationparametersselectedforprojectionplusTikhonov, Tikhonov(k)1:259E31:995E31:995E45:012E5 TSVD`(k) Rust'sTSVD(k)9:201E51:225E49:201E59:201E5 10 GCVL-curveoptimal
Tikhonov.1330.1486.1084.0648 Disc.GCVL-curveoptimal Table913 8 9
TSVD,andRust'sTSVD. Example1,iteration40:comparisonofkxtruexregk2=kxtruek2forprojectionplusTikhonov,
Rust'sTSVD.1162.1162.1162.1162 .1679.1986.1206.1165 Table10
exact solution
12
10
8
6
4
2
0
−2
6
4
2
0
−2
Fig.2.Example2:Top:exactsolution.Bottom:noisyrighthandsideb. 16
50 100 150 200 250
b
50 100 150 200 250

preconditionedsystem. Example2:parametervaluesselectedforeachmethod.Theprojectionwasperformedonaleft Tikhonov1:259E21:259E21:995E33:9811E3 TSVD` Rust'sTSVD2:183E22:586E61:477E21:527E2 Projectionk Disc. 216 2Table11
GCVL-curveoptimal 254 201 5 201
Tikhonov9:909E29:909E21:050E29:394E2
1:102E18:121E19:074E29:0744E2 Disc. GCVL-curveoptimal 5
theoriginalproblem. estimateastoppingparameter.Likewise,thedimensionkoftheprojectedproblem hadtobearound60toobtaingoodresultswiththeprojection-plus-regularizationapproaches,andmuchlargerthan60fortheL-curveappliedtotheprojected,Tikhonov
Projection1:030E19:85E21:15E19:479E2 Table12 Example2:comparisonofkxtruexregk2=kxtruek2foreachof4regularizationmethodson Rust'sTSVD1:025E122:671:011E11:011E1
selectiontechniqueswereappliedtogetherwithoneofthefourregularizationmethods ontheoriginalproblemaregiveninTable11.Thelastcolumninthetablegivesthe tionerwastakentobem=50. valueoftheparameterthatgavearegularizedsolutionwiththeminimumrelative toworkwithaleftpreconditionedsystem(refertothediscussionattheendofx3.5). Ourpreconditionerwaschosenasin[22]wheretheparameterdeningtheprecondi-
regularizedoriginalproblem.Therefore,fortheprojectionbasedtechniques,wechose regularizedproblemtogiveagoodestimateofthecornerwithrespecttotheTikhonov
errorovertherangeofdiscretevaluestested,withrespecttothetruesolution.The ineective. relativeerrorsthatresultedfromcomputingsolutionsaccordingtotheparametersin Table11areinTable12.WenotethatGCVwithTSVDandRust'sTSVDwere Thevaluesoftheregularizationparameterschosenwhenthethreeparameter
weresymmetric,wecouldhaveusedMINRESasin[22])andthenregularizedthe projectedproblemwithoneofthethreemethodsconsidered.Foreachofthethree crepancy,GCV,andL-curve,thencomputedthecorrespondingregularizedsolutions; methods,wecomputedregularizationparametersfortheprojectedproblemusingDis-
correspondingtothelargestsolutionnormsfortheTSVDmethodswerenotconsidered(otherwise,afalsecornerwasdetectedbythelcornerroutine).
weredeterminedusingHansen'slcornerfunction,withthemodicationthatpoints Next,weprojectedusingLSQR(notethatsincethematrixandpreconditioner ThecornersoftheL-curvesfortheTikhonov,projection,andTSVDmethods
theparametersthatwereselectedineachcaseatiterations15and25aregiveninTables13and15,respectively.Therelativeerrorsoftheregularizedsolutionsgenerated
accordinglyaregiveninTables14and16. L-curves,exceptinthecaseofRust'sTSVDmethod.Inthelattercase,therewas Again,weusedthelcornerroutinetodeterminethecornersoftherespective 17

TSVD,andRust'sTSVD.Disc. Example2,iteration15:regularizationparametersselectedforprojectionplusTikhonov, Tikhonov(k)2:512E21:585E21:9953E33:981E3 TSVD`(k) Rust'sTSVD(k)3:558E23:558E23:558E23:558E2 Disc. 5 GCVL-curveoptimal
Tikhonov1:0001E19:9511E21:061E19:530E2 Table13 GCVL-curveoptimal 5 4 1
TSVD,andRust'sTSVD. alwaysaverysharpcornerthatcouldbepickedoutvisually. Example2,iteration15:comparisonofkxtruexregk2=kxtruek2forprojectionplusTikhonov,
Rust'sTSVD1:004E11:004E11:004E11:004E1 9:595E19:595E11:004E19:357E2
byapplyingtheL-curvemethodtoprojected-plus-Tikhonovproblemwasthesame ComparingTable11withTables13and15,weseethattheparameterchosen Table14
nalproblemwiththatsameparameter.Similarresultsareshownfortheothercases, withtheexceptionthatthediscrepancyprincipledidnotworkwellfortheprojectionplus-TSVDproblems,andGCVwasnoteectivefortheprojectedproblemswheparisonofTable12withTables14and16showsthatrelativeerrorsoftheregularized
parameterchosenbyapplyingtheL-curvetotheoriginalproblem.Moreover,acom-
solutionscomputedaccordinglyarecomparabletoapplyingTikhonovtotheorigiularizationparameterandregularizedsolutiontotheoriginalproblembasedonregularizingaprojectedproblem.Theproposedapproachofapplyingregularization
andparameterselectiontechniquestoaprojectedproblemiseconomicalintime k=25. andstorage.Wepresentedresultsthatinfacttheregularizedsolutionobtainedby equivalenttoapplyingTSVDorTikhonovtotheoriginalproblem,where\almost" backprojectingtheTSVDorTikhonovsolutiontotheprojectedproblemisalmost dependsonthesizeofk.Theexamplesindicatethepracticalityofthemethod,and illustratethatourregularizedsolutionsareusuallyasgoodasthosecomputedusing 6.Conclusions.Inthisworkwehavegivenmethodsfordeterminingthereg-
theoriginalsystemandcanbecomputedinafractionofthetime,usingafractionof thestorage.WenotethatsimilarapproachesarevalidusingotherKrylovsubspace methodsforcomputingtheprojectedproblem. astheiterationsprogress.Inthisdiscussion,wedid,however,assumethateitherk wasnaturallyverysmallcomparedtonorthatpreconditioninghadbeenappliedto parameterschosenviatheprojection-regularizationandthecorrespondingregularized solutionswerecomparabletothosechosenandgeneratedfortheoriginaldiscretized enforcethiscondition.Possiblyforthisreason,wefoundthatformodestk,round-o didnotappeartodegradeeithertheLSQRestimatesoftheresidualandsolution normsorthecomputedregularizedsolutioninthefollowingsense:theregularization problem. Inthiswork,wedidnotaddresspotentialproblemsfromlossoforthogonality
FortheTikhonovapproachinthispaper,wehaveassumedthattheregularization 18

TSVD,andRust'sTSVD.Disc. Example2,iteration25:regularizationparametersselectedforprojectionplusTikhonov, Tikhonov(k)2:512E21:259E21:995E33:982E3 TSVD`(k) Rust'sTSVD(k)4:828E27:806E34:828E24:828E2 Disc. 9 GCVL-curveoptimal
Tikhonov1:000E19:909E21:061E19:530E2 Table15 GCVL-curveoptimal 9 8 3
Rust'sTSVD1:004E12.7281:004E11:004E1 9:164E19:595E11:004E19:145E2
IfLisnottheidentitybutisinvertible,wecanrstimplicitlytransformtheproblem ustoecientlycomputekr(k) TSVD,andRust'sTSVD. operatorLwastheidentityorwasrelatedtothepreconditioningoperator;thisallowed Example2,iteration25:comparisonofkxtruexregk2=kxtruek2forprojectionplusTikhonov, kandkx(k) kformultiplevaluesofecientlyforeachk. Table16
to\standardform"[17].WithA=AL1,x=Lx,wecansolvetheequivalentsystem problem.Clearlytheprojectionbasedschemeswillbeusefulaslongassolvingsystems involvingLcanbedoneeciently.REFERENCES Thentheprojectionplusregularizationschemesmaybeappliedtothistransformed minx=kAxbk2+2kxk2:
[5]L.DesbatandD.Girard,The`minimumreconstructionerror'choiceofregularizationparameters:Somemoreecientmethodsandtheirapplicationtodeconvolutionproblems,
Report96-31,ComputationalandAppliedMathematicsDept.,UCLA,LosAngeles,CA, 1996. ization,Tech.ReportSCCM-97-12,ScienticComputingandComputationalMathematics Program,ComputerScienceDepart.,StanfordUniversity,1997. ill-posedsystems,BIT,34(1994),pp.510{534. linearequations,BIT,28(1988),pp.659{670. [4]T.ChanandM.Ng,Galerkinprojectionmethodforsolvingmultiplelinearsystems,Tech. [3]D.Calvetti,G.Golub,andL.Reichel,EstimationoftheL-curveviaLanczosbidiagonal-
[1]A.Bjorck,Abidiagonalizationalgorithmforsolvinglargeandsparseill-posedsystemsof [2]A.Bjorck,E.Grimme,andP.V.Dooren,Animplicitshiftbidiagonalizationalgorithmfor
[10]G.GolubandW.Kahan,Calculatingthesingularvaluesandpseudo-inverseofamatrix, [6]H.W.EnglandW.Grever,UsingtheL-curvefordeterminingoptimalregularizationparameters,Numer.Math.,69(1994),pp.25{31.
imagereconstructionproblems,Lin.Alg.andApplics.,130(1990),pp.133{150. SIAMJ.Numer.Anal.,2(SeriesB)(1965),pp.205{224. SIAMJ.Sci.Comput.,(toappear). agoodridgeparameter,Technometrics,21(1979),pp.215{223. [7]H.E.Fleming,Equivalenceofregularizationandtruncatediterationinthesolutionofill-posed [8]A.FrommerandP.Maass,FastCG-basedmethodsforTikhonov-Phillipsregularization, [9]G.Golub,M.Heath,andG.Wahba,Generalizedcross-validationasamethodforchoosing SIAMJ.Sci.Comput.,16(1995),pp.1387{1403.

[11]G.GolubandC.V.Loan,MatrixComputations,TheJohnsHopkinsUniversityPress,Baltimore,MD,1989.
[12]W.Groetsch,TheoryofTikhonovRegularizationforFredholmequationsoftheFirstKind,
PitmanPublishingLimited,1984.
[13]M.GuandS.Eisenstat,Astableandfastalgorithmforupdatingthesingularvaluedecomposition,Tech.ReportRR-939,YaleUniversity,DepartmentofComputerScience,New
Haven,CT,1993.
[14]M.Hanke,J.Nagy,andR.Plemmons,Preconditionediterativeregularizationforill-posed
problems,NumericalLinearAlgebraandSci.Computing,(1993),pp.141{163.L.Reichel,
A.Ruttan,andR.S.Varga,editors.
[15]P.C.Hansen,Analysisofdiscreteill-posedproblemsbymeansoftheL-curve,SIAMReview,
34(1992),pp.561{580.
[16]P.C.Hansen,Regularizationtools:aMatlabpackageforanalysisandsolutionofdiscrete
ill-posedproblems,Numer.Algo.,6(1994),pp.1{35.
[17],Rank-DecientandDiscreteIll-PosedProblems,SIAMPress,Philadelphia,1998.
[18],RankDecientandDiscreteIll-PosedProblems,PhDthesis,TechnicalUniversityof
Denmark,July1995.UNICReportUNIC-95-07.
[19]P.C.HansenandD.P.O'Leary,TheuseoftheL-curveintheregularizationofdiscrete
ill-posedproblems,SIAMJ.Sci.Comput.,14(1993),pp.1487{1503.
[20]B.Hofmann,RegularizationforAppliedInverseandIll-PosedProblems,Teubner-Texte
Mathe.,Teubner,Leipzig,1986.
[21]L.KaufmanandA.Neumaier,Regularizationofill-posedproblemsbyenvelopeguidedconjugategradients,JournalofComputationalandGraphicalStatistics,(1997).
[22]M.Kilmer,SymmetricCauchy-likepreconditionersfortheregularizedsolutionof1-dill-posed
problems,Tech.ReportCS-TR-3851,UniversityofMaryland,CollegePark,1997.
[23],Cauchy-likepreconditionersfor2-dimensionalill-posedproblems,SIAMJ.Matrix
Anal.,(1998).toappear.
[24]M.KilmerandD.P.O'Leary,PivotedCauchy-likepreconditionersforregularizedsolution
ofill-posedproblems,SIAMJ.Sci.Stat.Comput.,(1998).toappear.
[25]V.A.Morozov,Onthesolutionoffunctionalequationsbythemethodofregularization,Soviet
Math.Dokl.,7(1966),pp.414{417.citedin[17].
[26]J.Nagy,R.Plemmons,andT.Torgersen,Iterativeimagerestorationusingapproximate
inversepreconditioning,IEEETrans.ImageProc.,5(96),pp.1151{1163.
[27]D.P.O'LearyandJ.A.Simmons,Abidiagonalization-regularizationprocedureforlargescale
discretizationofill-posedproblems,SIAMJ.Sci.Stat.Comput.,2(1981),pp.474{489.
[28]C.C.PaigeandM.A.Saunders,Algorithm583,LSQR:Sparselinearequationsandleast
squaresproblems,ACMTransactionsonMathematicalSoftware,8(1982),pp.43{71.
[29],LSQR:Analgorithmforsparselinearequationsandsparseleastsquares,ACMTransactionsonMathematicalSoftware,8(1982),pp.43{71.
[30]B.N.Parlett,TheSymmetricEigenvalueProblem,Prentice-Hall,EnglewoodClis,NJ,
1980.
[31]B.W.Rust,Truncatingthesingularvaluedecompositionforill-posedproblems,Tech.Report
NISTIR6131,MathematicalandComputationalSciencesDivision,NationalInstituteof
StandardsandTechnology,Gaithersburg,MD,1998.
[32]Y.Saad,IterativeMethodsforSparseLinearSystems,PWSPublishingCompany,Boston,
1996.
[33]Y.SaadandM.H.Schultz,GMRES:Ageneralizedminimalresidualalgorithmforsolving
nonsymmetriclinearsystems,SIAMJ.Sci.Stat.Comput.,7(1986),pp.856{869.
[34]A.N.TikhonovandV.Y.Arsenin,SolutionsofIll-PosedProblems,Wiley,NewYork,1977.
[35]J.M.Varah,Pitfallsinthenumericalsolutionoflinearill-posedproblems,SIAMJ.Sci.Stat.
Comput.,4(1983),pp.164{176.
[36]C.R.Vogel,Non-convergenceoftheL-curveregularizationparameterselectionmethod,InverseProblems,12(1996),pp.535{547.
20