Compute sample variance of data stream

Tue, 06 Oct 2015 19:39:59 +0000

Sample variance of a data stream $\left \{ X_1, X_2, \cdots, X_N, X_{N+1}, \cdots \right \}$ can be computed without saving the data points individually, but only sample mean and sample variance values for the current sample size $N$:

$$ \begin{align} s² N &= \frac{1}{N} \sum {i=1}^N \left( X i - \bar{X} N \right)² \\ &= \frac{1}{N} \sum {i=1}^N \left( X i² - \bar{X} _N² \right)~, \end{align} $$

where

$$ \begin{equation} {\bar{X}} N = \frac{1}{N} \sum {i=1}^N X _i~. \end{equation} $$

Store $N$, $\bar{X} N$, and $s N²$. When datum $X _{N+1}$ is obtained, the values can be updated as the following:

$$ \begin{equation} \bar{X} {N+1} = \frac{1}{N+1} \left( N \bar{X} N + X _{N+1} \right)~, \end{equation} $$

$$ \begin{align} s² {N+1} &= \frac{1}{N+1} \sum {i=1}^{N+1} \left( X i² - \bar{X} {N+1}^2\right) \\ &= \frac{1}{N+1} \sum {i=1}^{N+1}X i² - \bar{X} {N+1}^2 \\ &= \frac{1}{N+1} \left(N s N² + \bar{X} N² + X {N+1}^2 \right)- \bar{X} _{N+1}^2~. \end{align} $$

pages tagged probability

Compute sample variance of data stream