Problem Set 11 (Density Estimation)

1

Generate a data set by sampling 100 points from $y = \tanh(x)$ between $x = −5$ to 5, adding random noise with a magnitude of 0.25 to $y$. Fit it with cluster-weighted modeling, using a local linear model for the clusters, $y = a + bx$. Plot the data and the resulting forecast $\langle y | x \rangle$, uncertainty $\langle \sigma_y^2 | x \rangle$, and cluster probabilities $p(x|c_m) p(cm)$ as the number of clusters is increased (starting with one), and animate their evolution during the EM iterations.

My code lives here.

Here’s a video of 50 EM iterations with three clusters. I’m displaying the local linear models over a $4 \sigma$ range. (I don’t indicate the output variance in any way, would be nice to add this.)

Not surprisingly, for tanh three clusters seems to work best. Any less and the fit degrades, any more and some clusters end up with needlessly similar local models.