Felipe Pérez

Felipe Pérez

Mathematician and Data Scientist

Problem 5: Unbiased and consistent estimators

10 minute read

Published:

What does it mean for an estimator to be unbiased? What about consistent? Give examples of an unbiased but not consistent estimator, as well as a biased but consistent estimator.

If you like my content, consider following my linkedin page to stay updated.

Concise answer:

An unbiased estimator is such that its expected value is the true value of the population parameter. A consistent estimator is such that it converges in probability to the true value of the parameter as we gather more samples.

Long answer:

When we have a population $S$ with a parameter $\theta$ we want to estimate, we propose estimators $\hat\theta$ . These estimators are random variables and as such, they have a distribution. If the expected value of this estimator is equal to the true value of the parameter, we say that the estimator is unbiased, otherwise we say it is biased. Mathematically, this can be written as $\mathbb{E}(\hat\theta)=\theta$. On the other hand, whether the estimators expected value is the true value of the paramter, it may still converge in probability to the true parameter. When this is the case, we say that the estimator is consistent.

Examples:

An unbiased estimator which is not consistent: suppose $X_1,\dots,X_n$ are i.i.d. samples with mean $\mu$ and consider the estimator of $\mu$ given by $\hat\mu = X_1$. Then $\mathbb{E}(\hat\mu) =\mu$ since $X_n$ has the same distribution for all $n$. On the other hand, $\hat\mu$ is not consistent:

\[\mathbb{P}(|X_1 - \mu| > \epsilon) > 0\]

and is independent of $n$, so it will not converge to zero.

A biased estimator which is consistent: suppose $X_1,\dots,X_n$ are i.i.d. samples with mean $\mu$ and variance $\sigma^2$, and consider the estimator of $\mu$ given by

\[\hat\mu = \dfrac{1}{n} + \dfrac{1}{n}\sum_{i=1}^n X_i .\]

Then $\mathbb{E}(\hat\mu)=\mu + \frac{1}{n}\neq\mu$ for all $n$, while by Markov’s inequality

\[\mathbb{P}\left\{\left( \dfrac{1}{n} + \dfrac{1}{n}\sum_{i=1}^n X_i - \mu\right)^2 > \epsilon^2\right\}\leq \dfrac{\mathbb{E}( \frac{1}{n} + \frac{1}{n}\sum_{i=1}^n X_i - \mu)^2}{\epsilon^2},\]

so we need to estimate the expectation of the square. Note that for this we can do

\[\mathbb{E}\left( \dfrac{1}{n} + \dfrac{1}{n}\sum_{i=1}^n X_i - \mu\right)^2 \leq \dfrac{1}{n^2}+\dfrac{2}{n}\mathbb{E} \left(\dfrac{1}{n}\sum_{i=1}^n X_i - \mu\right) + \mathbb{E}\left(\dfrac{1}{n}\sum_{i=1}^n X_i - \mu \right)^2.\]

The first term obviously converges to zero, while for the second we can see that the expectation of $X_i$ is $\mu$ so the whole term vanishes. For the last term, we can expand the square and see that

\[\newcommand{\var}{var} \mathbb{E}\left(\dfrac{1}{n}\sum_{i=1}^n X_i - \mu \right)^2= \var\left(\dfrac{1}{n}\sum_{i=1}^n X_i - \mu \right)=\var\left(\dfrac{1}{n}\sum_{i=1}^n X_i \right) =\dfrac{1}{n}\sigma^2\]

from which consistency follows.

Another nice example.

Leave a Comment

You May Also Enjoy

Your SQL is probably WRONG

22 minute read

Published:

Yes, the title is a bit click-baity, but I hope that this catches your attention for a big warning of a very subtle behavior of most SQL dialects. The main idea is that when you perform a left (or right) join that has clauses on the right (left) hand side table, it is very easy to mess up the integrity of the join and end up with results that are very different to what you are expecting. This behavior is very easy to miss and many people fall into this trap without even noticing, as it is basically a silent failure.

Is overfitting… good?

44 minute read

Published:

Conventional wisdom in Data Science/Statistical Learning tells us that when we try to fit a model that is able to learn from our data and generalize what it learned to unseen data, we must keep in mind the bias/variance trade-off. This means that as we increase the complexity of our models (let us say, the number of learnable parameters), it is more likely that they will just memorize the data and will not be able to generalize well to unseen data. On the other hand, if we keep the complexity low, our models will not be able to learn too much from our data and will not do well either. We are told to find the sweet spot in the middle. But is this paradigm about to change? In this article we review new developments that suggest that this might be the case.

PCA and supervised learning

21 minute read

Published:

The situation is this: you have been given data, with several variables $x_1,\dots,x_d$ and a response $y$ that we want to predict using such variables. You perform some basic statistical analysis on your variables, see their averages, ranges, distribution. Then you look at the correlation between these variables, and find that there is some strong correlation between some of them. You decide to perform principal components analysis (PCA) to reduce the dimension of your features to $w_1,\dots,w_m$, with $m < d$. Now you fit your model, and you find that it gives terrible results, even though your PCA variables are capable of explaining most of the variance of the features. What went wrong?

Martingales 0

37 minute read

Published:

I want to talk about martingales, but unfortunately in order to do that properly, we need to talk first about sigma-algebras and conditional expectations, subjects which can be a bit harsh at first. These concepts are essential, and while we could just work with them just as formal objects with certain properties, it is fundamental to have a deeper understanding of them so we do not get lost in formalism and we are able to capture the intuition behind this theory.