CCD data reduction: Noise

Noise

Every measurement suffers from noise. So does your image taken with a CCD camera, and data reduction is meant to retain control of it. The understanding of the data reduction process and the noise propagation is essential for optimal results. Let's start with some basics, I'll keep it simple.

Sky noise and Poisson distribution

The emission of light is a discrete process (n photons are emitted during a fixed time span) and underlies the laws of quantum physics. For a source of (on average) constant brightness, the probability of having n photons within some time is described by the Poisson distribution. For such, the standard deviation (or the uncertainty, or noise) of having n photons in a given time is

Thus, if we take an exposure of the night sky with an idealised noise-free camera, the noise in the resulting image will be equal to the square root of the average brightness recorded. This noise is called the sky noise. If the exposure returns a sky brightness of 900 ADU, then the sky noise will be 30 ADU.

Averaging out the noise

The sum of m Poisson distributions is again a Poisson distribution. The expectation value of the sum is equal to the sum of the expectation values of the individual distributions. If the expectation value of one distribution is equal to B and is the same for the other distributions, then we have

Similarly, for the noise of the sum we then have

Therefore, the signal-to-noise ratio (commonly referred to as S/N) of an individual exposure is

And the signal-to-noise of m exposures becomes

As a result we see that in a series of m added images

  • the signal increases linearily with m
  • the noise increases more slowly, with sqrt(m)

If we take the mean (the average) of the m exposures, the noise of the mean is a factor of sqrt(m) smaller than the noise in the individual image. That is, the mean image is a better estimate of the real flux distribution on the sky than an individual exposure.

In other words, if you take 4 times as many images, then your S/N increases by a factor of 2, and if you take 9 times as many images, then your S/N improves by a factor of 3. There is no whatsoever difference between the sum of m images, and the average of these m images. The S/N ratio in both is exactly the same.

Outlier rejection

CCD images suffer from cosmics, pixel defects and other disturbances such as satellite trails. If the mean of a series of images is calculated, the amplitude of these effects are reduced in the mean image, but they do not vanish. To this end one needs an outlier rejection, which is obtained by calculating the standard deviation (aka "root mean square (rms)", aka "noise", aka "1 sigma") of the pixels in a stack. If the pixel values follow a Gaussian distribution (which is approximately the case for high enough illumination levels), then one will find 67% of all pixels within an interval of +/-1 sigma around the mean value. The 2-sigma interval includes about 95% of all values, and the 3-sigma interval contains 99.7%. During image combination, all pixels in the stack which are more than a few sigmas away from the mean are not taken into account. Hence, the mean image will be free of cosmics and other defects.

However, the presence of an outlier in the stack biases the calculation of the rms to larger values, in particular if only a small number of images is used. Hence, this can be unreliable. The proper way of doing it is to calculate the rms from all but the highest value in the stack, no matter if the highest pixel is a bad pixel or not.

Mean and median

Another way of combining a series of m images is the median, which is the pixel value that separates the higher half from the lower half. The median yields a more stable result than the mean for a small number of images, as it is less affected by the presence of e.g. a cosmic. For a small number of images, the median is still biased towards higher values, which can be seen in the stacked image with steeply scaled contrast. Hence, even for the median the highest pixel in the stack should be discarded before the calculation of the median.
However, if the pixel values are Gaussian distributed (which is to a good approximation the case), then the mean is the optimal method for combination. It can be shown (see e.g. the statistics section in Nick Kaiser's Elements of Astrophysics) that in this case the median has roughly a factor 1.25 higher noise level than the mean. A more intuitive explanation is that the mean uses all the available numeric information, whereas the median just sorts the numbers according to their value and then picks the middle one. If you replace the lower half of numbers by an entirely different set of "lower half numbers", then the median would not change. But the mean would take this into account.
To achieve the noise level of the mean with the median combination, you would have to expose 1.252 = 1.6 times as long. Sloppily spoken, if you use the median combination, then you throw away 60% of the exposure time. Hence, only use it when you have to.

Calibration noise

Sky noise is not the only source of noise in your images. The camera electronics contributes readout noise, and the detector adds dark current which is noisy by itself. The detector's pixels have intrinsically different sensitivities, and also suffer from an inhomogeneous illumination pattern. These are systematic effects, which can be removed by means of master dark frames and master flat fields constructed from a series of individual darks and flats. However, they are noisy themselves as they are not made of an infinitely large number of such exposures. Therefore, when you subtract a dark from your image and divide it by a flat, you take out the systematic effects, but at the same time you copy the noise from the calibration images into your exposure. This is called calibration noise.