Setup

Let $X_1,\dots ,X_n$ be a sample from a distribution indexed by an unknown parameter $\theta \in \Theta$. Write $P_{\theta }$ for the probability measure under parameter $\theta$ and $E_{\theta }$ for the corresponding expectation.

An estimator $T(X_1,\dots ,X_n)$ is a measurable function of the sample. It is an unbiased estimator of $\theta$ when

where $T$ denotes the estimator, $\theta$ denotes the unknown parameter, and the expectation $E_{\theta }$ is taken under the law $P_{\theta }$.

A statistic $S=S(X_1,\dots ,X_n)$ is a sufficient statistic for $\theta$ when the conditional distribution of the sample given $S$ does not depend on $\theta$. Equivalently:

Here $S$ denotes the sufficient statistic, $s$ ranges over its support, and the independence of the right-hand side from $\theta$ is the defining property.

The theorem

Rao-Blackwell. Let $T$ be an unbiased estimator of $\theta$ with finite variance, and let $S$ be a sufficient statistic for $\theta$. Define

$$T^{\ast }:=E[T\mid S].\text{\hspace{1em}(3)}$$

where $T^{\ast }$ denotes the Rao-Blackwellised estimator obtained by conditioning $T$ on $S$. Then $T^{\ast }$ is an unbiased estimator of $\theta$, and

$${\mathrm{Var}}_{\theta }(T^{\ast })\le {\mathrm{Var}}_{\theta }(T)\text{for every }\theta ,\text{\hspace{1em}(4)}$$

with equality iff $T$ is already a function of $S$ (up to a $P_{\theta }$-null set).

Proof sketch

Two ingredients suffice.

Unbiasedness. By the tower property of conditional expectation,

where the inner expectation $E[T\mid S]$ is the conditional expectation of $T$ given $S$, and the outer expectation is taken under $P_{\theta }$.

Variance. Apply the variance decomposition with $G=\sigma (S)$:

Here $\sigma (S)$ denotes the σ-algebra generated by $S$, $\mathrm{Var}(T\mid S)$ denotes the conditional variance of $T$ given $S$, and the two right-hand terms are both non-negative.

Substituting $T^{\ast }=E[T\mid S]$ from $(3)$:

Since $E_{\theta }[\mathrm{Var}(T\mid S)]\ge 0$, we conclude ${\mathrm{Var}}_{\theta }(T^{\ast })\le {\mathrm{Var}}_{\theta }(T)$, with equality iff $\mathrm{Var}(T\mid S)=0$ almost surely — i.e. iff $T$ is determined by $S$.

The same chain holds for any convex loss $L$ via Jensen's inequality:

where $L$ denotes a convex loss function and the variance case in $(4)$ is the special case $L(t,\theta )=(t-\theta {)}^2$.

Worked example

Let $X_1,\dots ,X_n$ be i.i.d. Poisson with mean $\lambda$, and let $\tau (\lambda )=e^{-\lambda }=P_{\lambda }(X_1=0)$.

A trivial unbiased estimator:

where $1\{\cdot \}$ denotes the indicator. Then $E_{\lambda }[T]=P_{\lambda }(X_1=0)=e^{-\lambda }=\tau (\lambda )$, so $T$ is unbiased for $\tau (\lambda )$.

The sample sum $S=X_1+\cdots +X_n$ is sufficient for $\lambda$. Given $S=s$, the conditional distribution of $(X_1,\dots ,X_n)$ is $\mathrm{Multinomial}(s;1/n,\dots ,1/n)$ — independent of $\lambda$. Therefore:

where $T^{\ast }$ is the Rao-Blackwellised estimator. By $(4)$, ${\mathrm{Var}}_{\lambda }(T^{\ast })\le {\mathrm{Var}}_{\lambda }(T)$, with strict inequality for $n>1$ since $T$ is not a function of $S$ alone.

Why this matters

The recipe is constructive: take any unbiased estimator $T$, project onto the σ-algebra of a sufficient statistic $S$, and you get $T^{\ast }$ with no greater variance. Combined with completeness of $S$ (Lehmann-Scheffé, 1950), this gives the minimum-variance unbiased estimator uniquely — there is essentially one Rao-Blackwellisation up to almost-sure equality, and it dominates every other unbiased estimator simultaneously.

References

C. R. Rao, "Information and accuracy attainable in the estimation of statistical parameters," *Bulletin of the Calcutta Mathematical Society*, 37:81-91, 1945.

D. Blackwell, "Conditional expectation and unbiased sequential estimation," *Annals of Mathematical Statistics*, 18(1):105-110, 1947.

E. L. Lehmann and H. Scheffé, "Completeness, similar regions, and unbiased estimation — Part I," *Sankhyā*, 10:305-340, 1950.