comp.lang.idl-pvwave archive - RDF feed
type: channel
description: Messages from Usenet group comp.lang.idl-pvwave, compiled by Paulo Penteado
items: MPfit question
items: Re: MPfit question
items: Re: MPFIT question

MPfit question
type: item
description: Hi all, I'm using the wonderfull mpfit routine from Craig Markwardt. Today I was having problems with convergence and I was looking in the code what caused it. I was using parameter limits and as far as I can tell, the parameter incrementation is done like this: xnew = x + alpha * dx where dx are the parameter increments and alpha = 1, except when xnew would exceed the limits, in which case alpha < 1. So if one parameter is close to the limit, this effects the increments for all other parameters. If alpha is too small, the fit stops. Why isn't dx adapted, so that pegged parameters won't effect the convergence for other parameters? Wout
creator: Wox
date: 2008-09-30T14:38:20-00:00

Re: MPfit question
type: item
description: Wox <nomail@hotmail.com> writes: > Hi all, > > I'm using the wonderfull mpfit routine from Craig Markwardt. Today I > was having problems with convergence and I was looking in the code > what caused it. I was using parameter limits and as far as I can tell, > the parameter incrementation is done like this: > > xnew = x + alpha * dx > > where dx are the parameter increments and alpha = 1, except when xnew > would exceed the limits, in which case alpha < 1. So if one parameter > is close to the limit, this effects the increments for all other > parameters. If alpha is too small, the fit stops. > > Why isn't dx adapted, so that pegged parameters won't effect the > convergence for other parameters? Hello, thanks for your message. The original MINPACK fortran code did not have parameter limits, so limits are my own bolt-on. If you look at the code, the value of ALPHA is adjusted so that, at the next iteration, a parameter will exactly touch its boundary, within a small tolerance. At that point, the parameter will be considered fixed, and will no longer enter into the calculation of the value of ALPHA. [*] Thus, the step *is* adaptive, it just doesn't happen in a single iteration. If your constraints are pathalogical enough that the fitter is ping-ponging between two constraints continuously, I don't have much to say other than to offer my pity. It's not an easy problem to solve. Craig [*] - the convergence criteria are also pro-rated because ALPHA is smaller than 1. -- ------------------------------------------------------------ -------------- Craig B. Markwardt, Ph.D. EMAIL: cbmarkwardt+usenet@gmail.com ------------------------------------------------------------ --------------
creator: Craig Markwardt
date: 2008-09-30T15:49:46-00:00

Re: MPfit question
type: item
description: On 30 Sep 2008 11:49:46 -0400, Craig Markwardt <craigmnet@REMOVEcow.physics.wisc.edu> wrote: > If you look at the code, the value of ALPHA is adjusted so that, at > the next iteration, a parameter will exactly touch its boundary, > within a small tolerance. At that point, the parameter will be > considered fixed, and will no longer enter into the calculation of the > value of ALPHA. [*] Thus, the step *is* adaptive, it just doesn't > happen in a single iteration. I'm sorry, but I don't see how it does this. ALPHA is adjusted and immediatly used (see below). In the next iteration, the increments are calculated again by mpfit_lmpar and used again to calculate ALPHA, whether the param was at the limit in the previous iteration or not. The thing is, my problem is solved when I adjust the increments themselves and leave ALPHA=1. I was just wondering whether I introduce some errors by doing this. INNER_LOOP: ; mpfit_lmpar: ; solve A.wa1=B and sqrt(par).D.wa1=0 ; where wa1 is the parameter increment ... ; When some param goes outside boundary, make alpha smaller so it touches the boundary: dwa1 = abs(wa1) GT MACHEP0 whl = where(dwa1 AND qllim AND (x + wa1 LT llim), lct) if lct GT 0 then $ alpha = min([alpha, (llim[whl]-x[whl])/wa1[whl]]) whu = where(dwa1 AND qulim AND (x + wa1 GT ulim), uct) if uct GT 0 then $ alpha = min([alpha, (ulim[whu]-x[whu])/wa1[whu]]) ; Apply increment! xnew = x + alpha * wa1 ; When the step puts us on a boundary, make sure it is exact ... ; Convergence tests + succesfull inner loop test ... goto INNER_LOOP
creator: Wox
date: 2008-10-01T08:31:54-00:00

Re: MPfit question
type: item
description: Wox <nomail@hotmail.com> writes: > On 30 Sep 2008 11:49:46 -0400, Craig Markwardt > <craigmnet@REMOVEcow.physics.wisc.edu> wrote: > >> If you look at the code, the value of ALPHA is adjusted so that, at >> the next iteration, a parameter will exactly touch its boundary, >> within a small tolerance. At that point, the parameter will be >> considered fixed, and will no longer enter into the calculation of the >> value of ALPHA. [*] Thus, the step *is* adaptive, it just doesn't >> happen in a single iteration. > > I'm sorry, but I don't see how it does this. ALPHA is adjusted and > immediatly used (see below). In the next iteration, the increments are > calculated again by mpfit_lmpar and used again to calculate ALPHA, > whether the param was at the limit in the previous iteration or not. That is not correct. Please search for 'zeroing the derivatives of pegged parameters'. Once a parameter is pegged at a boundary in the previous iteration, it no longer contributes to the congugate gradiate solution because its derivatives have been zeroed. > The thing is, my problem is solved when I adjust the increments > themselves and leave ALPHA=1. I was just wondering whether I introduce > some errors by doing this. Probably your best bet is to see which convergence criterium is satisfied when ALPHA < 1, and go from there. Craig -- ------------------------------------------------------------ -------------- Craig B. Markwardt, Ph.D. EMAIL: cbmarkwardt+usenet@gmail.com ------------------------------------------------------------ --------------
creator: Craig Markwardt
date: 2008-10-01T17:16:10-00:00

Re: MPfit question
type: item
description: On 01 Oct 2008 13:16:10 -0400, Craig Markwardt <craigmnet@REMOVEcow.physics.wisc.edu> wrote: > > Wox <nomail@hotmail.com> writes: > >> On 30 Sep 2008 11:49:46 -0400, Craig Markwardt >> <craigmnet@REMOVEcow.physics.wisc.edu> wrote: >> >>> If you look at the code, the value of ALPHA is adjusted so that, at >>> the next iteration, a parameter will exactly touch its boundary, >>> within a small tolerance. At that point, the parameter will be >>> considered fixed, and will no longer enter into the calculation of the >>> value of ALPHA. [*] Thus, the step *is* adaptive, it just doesn't >>> happen in a single iteration. >> >> I'm sorry, but I don't see how it does this. ALPHA is adjusted and >> immediatly used (see below). In the next iteration, the increments are >> calculated again by mpfit_lmpar and used again to calculate ALPHA, >> whether the param was at the limit in the previous iteration or not. > > That is not correct. Please search for 'zeroing the derivatives of > pegged parameters'. Once a parameter is pegged at a boundary in the > previous iteration, it no longer contributes to the congugate gradiate > solution because its derivatives have been zeroed. Ah yes, I see now. Thanks for your comments. I must say it's quite an impressive and complicated piece of code. I'm having a hard time understanding the convergence criteria and the calculation of the LM-step. Can you recommend some reference where I can find more details? I sure have some books explaining NLLS-refinement but they don't seem to mention some of the criteria I see in your code. Also the calculation of the LM-step is done differently each time I open a new book :-).
creator: Wox
date: 2008-10-02T11:39:19-00:00

Re: MPfit question
type: item
description: Wox <nomail@hotmail.com> writes: > > Ah yes, I see now. Thanks for your comments. I must say it's quite an > impressive and complicated piece of code. I'm having a hard time > understanding the convergence criteria and the calculation of the > LM-step. Can you recommend some reference where I can find more > details? I sure have some books explaining NLLS-refinement but they > don't seem to mention some of the criteria I see in your code. Also > the calculation of the LM-step is done differently each time I open a > new book :-). Hi, the code is based on MINPACK-1, so the convergence criteria come from that library. The references section of mpfit.pro contains two citations by one of the primary authors of MINPACK. Unfortunately they are hard to find. :-( I am suspecting that for your problem, MPFIT is claiming the problem converged when it didn't, if the "actual" change in chi^2 is much smaller than "predicted". The only real way I can see this happening is if you started a parameter *very close* to a boundary but not exactly on the boundary. The fix for that should be obvious. Craig -- ------------------------------------------------------------ -------------- Craig B. Markwardt, Ph.D. EMAIL: cbmarkwardt+usenet@gmail.com ------------------------------------------------------------ --------------
creator: Craig Markwardt
date: 2008-10-02T15:59:50-00:00

Re: MPFIT question
type: item
description: On Jan 13, 2:33 pm, "j.coe...@gmail.com" <j.coe...@gmail.com> wrote: > I'm fitting data to a gamma variate function using Craig Markwardt's > MPFIT. This has been working great except for a chronic error message > that occurs once every 50 fits or so: > > MPFIT: Error detected while calling mpfitfun_eval: > MPFIT: Array dimensions must be greater than 0. > MPFIT: Error condition detected. Returning to MAIN level. > MPFITFUN: Error detected while calling mpfitfun_eval: Array dimensions > must be greater than 0. > Attempt to subscript P with <INT ( 1)> is out of range. > > My five parameters are getting lost. The parameters are first passed > to MPFITFUN via the parinfo structure because some are constrained. > Then the parameters are passed along by MPFITFUN to the user-supplied > model function as a double array, p. When the error occurs, the p > array has shrunk from five doubles to just one NaN, as you can see > from the abbreviated output reproduced at the end of this post. The > subscripting error happens when the user-supplied model function tries > to subscript the suddenly nonexistent second element of p, which is > supposed to have five parameters/elements (and had five elements at > all previous iterations). This can happen during any MPFIT iteration, > but usually around iteration 4. > > Does anyone know what's going on? I checked to make sure that there > are no NaN values in the data, and that my gamma variate model > function is not producing any NaN values at any iteration. I have the > latest version of the MPFIT library. I've just been catching the > error and fitting the problematic data with IDL's routine, but MPFIT > does a much better job when it works for me. Hopefully someone else > who has encountered this issue knows what I am doing wrong. Thanks. > > Iter 1 CHI-SQUARE = 9182.7891 DOF = 353 > P DOUBLE = Array[5] > . > . > Iter 2 CHI-SQUARE = 6448.4258 DOF = 353 > P DOUBLE = Array[5] > . > . > Iter 3 CHI-SQUARE = 8402.1122 DOF = 353 > P DOUBLE…
creator: Craig Markwardt
date: 2009-01-14T06:41:17-00:00

Re: MPFIT question
type: item
description: Thank you for the reply. I also assume it is my user function. I did put a help comment at the very beginning -- that's what's generating the p description in the output text in my first post. Also, the user function is very simple, and I just can't see where it could be changing p, especially since the "Help, p" statement is the very first line. I'll step through line by line and if I can't track the problem down, I'll send you a reproducible case. Thanks again.
date: 2009-01-14T15:54:24-00:00

Re: MPFIT question
type: item
description: On Jan 14, 10:54 am, "j.coe...@gmail.com" <j.coe...@gmail.com> wrote: > Thank you for the reply. I also assume it is my user function. I did > put a help comment at the very beginning -- that's what's generating > the p description in the output text in my first post. Also, the user > function is very simple, and I just can't see where it could be > changing p, especially since the "Help, p" statement is the very first > line. I'll step through line by line and if I can't track the problem > down, I'll send you a reproducible case. Thanks again. Right, but remember I also suggested putting a HELP statement at the end of your function as well! Craig
creator: Craig Markwardt
date: 2009-01-14T16:23:44-00:00

Re: MPFIT question
type: item
description: The user function has help statements at its beginning and end now, and I tried the other suggestions, but I'm still stumped. I'm doing something foolish. Below is the problem code. Most of it is just the 353-element data array. Without the data, it's only two short functions, 20 lines of code or so total. I hope it's not bad netiquette to post so many lines. I can't figure out why these data crash the program. I can fit most other data acquired in this manner -- I have fit hundreds of thousands of these curves with this method, but the error crops up once every 50 cases or so (usually for data that don't fit the model very well, but it still shouldn't crash?). The data are being fit to a gamma variate function. Two things I'm not sure about: (1) I'm passing t0 as a fitting parameter, and I'm not sure if this is acceptable. (2) Worse, I don't have a good idea of how to estimate error, so I'm just using the standard deviation of the baseline (pre-arrival) curve, which is probably wrong. Suggestions? The code crashes after just two MPFIT iterations. Just type 'MPFIT_problem' to run. Thanks for any help. ------------------------------------ function gamma_variate, x, p print, 'p start...' help, p baseline = p[0] t0 = p[1] tmax = p[2] ; time of peak signal smax = p[3] ; signal max in mVolts alpha = p[4] ; shift time, apply gamma variate only to data after t0 wh = Where(x GE t0, ct) t = (x - t0)[wh] tmax = tmax - t0 ; gamma variate function s = baseline + (smax) * (tmax^(-alpha)) * exp(alpha) * (t^alpha) * exp ((-alpha) * t / tmax) ; add the pre-t0 data (baseline) n = n_elements(x) pre = make_array(n - ct, Value = baseline) s = [pre, s] print, 'p end...' help, p return, s end;-------------------------------------------------------- -- pro mpfit_problem !except = 2 data = [ $ 64.1682 , $ 66.3804 , $ 73.4243 , $ 64.1682 , $ 64.1682 , $ 55.9551 , $ 47.0046 , $ 52.2084 , $ 48.6855 , $ 52.2084 , $ 57.9162 , $ 75.9137 , $ 64.1682 , $ 75.9137 , $ 86.6244 , $ 89.4994 , $ 89.4994…
date: 2009-01-14T21:37:13-00:00

Re: MPFIT question
type: item
description: j.coenia@gmail.com wrote: > The user function has help statements at its beginning and end now, > and I tried the other suggestions, but I'm still stumped. I'm doing > something foolish. > > Below is the problem code. Most of it is just the 353-element data > array. Without the data, it's only two short functions, 20 lines of > code or so total. I hope it's not bad netiquette to post so many > lines. I can't figure out why these data crash the program. I can > fit most other data acquired in this manner -- I have fit hundreds of > thousands of these curves with this method, but the error crops up > once every 50 cases or so (usually for data that don't fit the model > very well, but it still shouldn't crash?). > > The data are being fit to a gamma variate function. Two things I'm > not sure about: (1) I'm passing t0 as a fitting parameter, and I'm not > sure if this is acceptable. (2) Worse, I don't have a good idea of > how to estimate error, so I'm just using the standard deviation of the > baseline (pre-arrival) curve, which is probably wrong. Suggestions? > > The code crashes after just two MPFIT iterations. Just type > 'MPFIT_problem' to run. > > Thanks for any help. > > ------------------------------------ > > function gamma_variate, x, p > > print, 'p start...' > help, p > > baseline = p[0] > t0 = p[1] > tmax = p[2] ; time of peak signal > smax = p[3] ; signal max in mVolts > alpha = p[4] > > ; shift time, apply gamma variate only to data after t0 > > wh = Where(x GE t0, ct) I think that you may have a problem here when ct EQ 0 and wh EQ -1. (that is, all x<t0). Ciao, Paolo > t = (x - t0)[wh] > tmax = tmax - t0 > > ; gamma variate function > > s = baseline + (smax) * (tmax^(-alpha)) * exp(alpha) * (t^alpha) * exp > ((-alpha) * t / tmax) > > ; add the pre-t0 data (baseline) > > n = n_elements(x) > pre = make_array(n - ct, Value = baseline) > s = [pre, s] > > print, 'p end...' > help, p > > return, s > > end;-----------------------------------------------------…
creator: pgrigis
date: 2009-01-14T22:08:48-00:00

Re: MPFIT question
type: item
description: On Jan 14, 4:37 pm, "j.coe...@gmail.com" <j.coe...@gmail.com> wrote: > The user function has help statements at its beginning and end now, > and I tried the other suggestions, but I'm still stumped. I'm doing > something foolish. > > Below is the problem code. Most of it is just the 353-element data > array. Without the data, it's only two short functions, 20 lines of > code or so total. I hope it's not bad netiquette to post so many > lines. I can't figure out why these data crash the program. I can > fit most other data acquired in this manner -- I have fit hundreds of > thousands of these curves with this method, but the error crops up > once every 50 cases or so (usually for data that don't fit the model > very well, but it still shouldn't crash?). > > The data are being fit to a gamma variate function. Two things I'm > not sure about: (1) I'm passing t0 as a fitting parameter, and I'm not > sure if this is acceptable. (2) Worse, I don't have a good idea of > how to estimate error, so I'm just using the standard deviation of the > baseline (pre-arrival) curve, which is probably wrong. Suggestions? > > The code crashes after just two MPFIT iterations. Just type > 'MPFIT_problem' to run. > > Thanks for any help. > > ------------------------------------ > > function gamma_variate, x, p > > print, 'p start...' > help, p > > baseline = p[0] > t0 = p[1] > tmax = p[2] ; time of peak signal > smax = p[3] ; signal max in mVolts > alpha = p[4] > > ; shift time, apply gamma variate only to data after t0 > > wh = Where(x GE t0, ct) > t = (x - t0)[wh] > tmax = tmax - t0 > > ; gamma variate function > > s = baseline + (smax) * (tmax^(-alpha)) * exp(alpha) * (t^alpha) * exp > ((-alpha) * t / tmax) > > ; add the pre-t0 data (baseline) > > n = n_elements(x) > pre = make_array(n - ct, Value = baseline) > s = [pre, s] Thanks for your complete example, very useful! Your function is crashing at the MAKE_ARRAY stage. This is because N EQ CT, so you are asking for an empty array. Afte…
creator: Craig Markwardt
date: 2009-01-15T07:52:17-00:00

Re: MPFIT question
type: item
description: Thanks both of you. I knew it would be something easy for someone. I was thinking that the lower constraint on t0 in parinfo was good enough to ensure that the WHERE statement found at least some times above t0, so that the found count (ct) would never be 0 as Paolo describes... unfortunately t0 can go all the way down to 0, in which case ct = n, which leads to the problem Craig describes: there is no pre-arrival baseline to prepend to the gamma variate if t0 is 0! Simply changing the GE to GT in the WHERE statement is a quick fix, but I'll implement better validity checking and MPFIT error trapping (which I didn't really understand until Craig explained that the initial MAKE_ARRAY error is being intercepted). So my code was wrong, but there's nothing theoretically wrong with trying to fit t0 here, or any piecewise continuous function, using Levenberg-Marquardt? And my method for guessing error is not specious? Sorry for the naive questions -- it takes a few days to run all these fits, so I don't want to do everything wrong from the start.
date: 2009-01-15T16:17:56-00:00

Re: MPFIT question
type: item
description: On Jan 15, 11:17 am, "j.coe...@gmail.com" <j.coe...@gmail.com> wrote: > So my code was wrong, but there's nothing theoretically wrong with > trying to fit t0 here, or any piecewise continuous function, using > Levenberg-Marquardt? There's definitely no requirement that the model function is smooth. Craig
creator: Craig Markwardt
date: 2009-01-17T06:10:30-00:00