Function Fitting: Problems 11.1, 11.2, 11.3, 11.4

Fit a line, and compare error estimates from analytical formula given underlying variance, bootstrap estimation, and brute force estimates. Python script

Write a Levenberg-Marquardt routine to fit a sinusoid. Compare convergence for fixed and adaptive stepping. Python script

As expected, fixed stepping is useful only when we start not too far from the solution. Otherwise, the value of lambda is either to aggressive and we shoot off somewhere, or it is too conservative and we spend too long taking small steps. I implemented the adaptive step using multiplicative increases and decreases based on whether our step reduced the error in the estimate. This greatly increased the range of starting positions from which we could reach the solution, but for some starting positions (including [1,1]), it was still tricky to avoid catching local minima. I didn't have time to sort this out.