When we have a population X of data with dimension N, we are normally provided with a set (or vector) of parameters θ (for a generic parameter, we will use the notation θ) which describes some statistical characteristics of that population (namely, the mean μ). However, it is more common to deal with subsets of population, called samples. First, because we might not be able to collect the entire population’s dataset. Second, because sample sets are less heavy and easier to handle.
For each sample, we can compute some estimations θ̂ of population parameters θ. Estimations differs from parameters in the fact that they are computed using only a small portion of the whole population, hence they are conceptually different. Namely, if we have a sample of size n, x1, x2 …, xn, the sample mean will be:
Ideally, we’d like our sample to be representative of our population, hence following the same distribution. Furthermore, we don’t want our sample to be biased, that means, not really representing the whole population. That’s why we aim at finding estimators which are unbiased: it means that expected value of our estimator is equal to the real value of the parameter we are estimating. So we define the bias as:
And we say that an estimator is unbiased if:
To recall the previous example, we can demonstrate that the sample mean is an unbiased estimator. Indeed:
What about the variance? Starting from the general variance and sample variance formula:
Can we say that the expected value of that estimator is equal to the real variance? The answer is no, unless we use a correction factor, called Bessel’s correction. Indeed, if we change our formula as follows:
Where the green term is the Bessel’s correction, we can demonstrate that the expected value of our estimator is equal to the variance parameter.
Let’s demonstrate it (assuming that our observations are IID – identical and independently distributed). Knowing that:
We can see that: