Information Theory

Problem #1

Initially, I ran into a dead end using Jensen's inequality. After some poking around on the web, I stumbled upon the suggestion to take an f(x) ≠ h(x) when substituting into the inequality equation. From there, it came out a bit cleaner.

From Jensen's Inequality: E[f(x)] ≥ f(E(x))
and let f(x) = -log(x) = log 1/x
and let x = 1/p(x)

H(X) = -E[ f(1/p(x)) ] ≤ -f( E(1/p(x)) )
E(1/p(x)) = ∑ p(x)/p(x) = ∑ 1 = N
H(X) ≤ - log 1/N = log N
H(X) ≤ log N

From Jensen's Inequality: Equality only holds when when all x's are equal. So for equality to be reached:
all p(x_i) are equal. Since ∑ p(x_i) = 1, p(x_i) = 1/N for all i.

Problem #3

I implemented the Kalman filter simulation in Java. Use the slider on the right to adjust the standard deviation of the process noise which in this case can be used to tune the filter. Linear Algebra calculations are being carried out with JAMA , a java matrix package developed by MathWorks and NIST.
[Web Start]
[Source Code]