In L2011_25128 I noticed that data of certain antennas and certain subbands were corrupted (or probably swapped with others). This seems to affect only the raw beam-formed data but neither the incoherent Stokes nor the visibility data.

By comparing autocorrelations derived from the raw voltage data with autocorrelations from the visibility data sets, I established which data are corrupted. In this analysis I assume that the visibility data are correct.

Bad raw data are:

• 0=FR606 (X: subband 0 bad, Y: subband 1 bad; all bad in first second)
• 3=UK608 (all bad)

The other antennas are fine.

The same is probably true for L2011_25134. At the same time, a bug was discovered in the software that writes the beam-formed data. This would repeat the first three beams (i.e. one antenna X and Y and the next only X). Even though this is not entirely consistent with the findings in this data set, I postpone a deeper analysis to the next data set where the bug should be fixed.

In this data set we had 6 stations and 60 subbands, much more than before. They were recorded on the CEP2 cluster, which may cause additional problems.

Subbands 24-27 of the visibilities are missing. The same is true for the raw voltage file L25531_B003_S0_P000_bf.raw and some incoherent Stokes files. All the missing files should have been on locus007 which had technical problems during the observations.

Only the first 20 seconds were inspected for the following analysis.

For the first comparison I integrated autocorrelations derived from the raw voltage data over 1 sec intervals, compared with the visibility data and visualised the differences per subband. The plots can be found here. There are two plots (one per page) for each antenna, one for XX and one for YY. The horizontal axis shows frequency (subband number), the vertical axis shows the deviations between the raw-voltage results and the visibility results. The green points (relative errors) are generally more meaningful.

This is what we find (only bad data are listed):

• DE603 XX: sb 0 (sb 24-27 missing, see above)
• DE603 YY: sb 1
• FR606 XX: sb 2
• FR606 YY: sb 3
• RS106 XX: sb 4
• RS106 YY: sb 5
• RS208 XX: missing (see above)
• RS208 YY: all but sb 12,19,29,49,55
• RS307 XX: all but sb 12,19,29,49,55
• RS307 YY: all but sb 12,19,29,49,55
• UK608 XX: all but sb 12,19,29,49,55
• UK608 YY: all but sb 12,19,29,49,55

The logical antenna numbers are 0=DE603,1=FR606,2=RS106,3=RS208,4=RS307,5=UK608.

At a closer inspection I found that “bad” subbands are not always entirely bad. This can be seen in one of the following files:

• This file shows time on the horizontal and frequency (channel/subband numbers) on the vertical axis. Note that the missing subbands are neglected in the numbering scheme. Channels 0 are set to zero in all subbands; high deviations are therefore expected in these cases.
• This file is similar to the first plots, but it shows all subbands. Frequency is on the horizontal axis (missing subbands not neglected here), deviations on the vertical axis.

We find that for RS208,RS307,UK608, the bad subbands are actually good in the first 32 channels. Only the upper halves are bad. This can be seen better in these versions of the plots that only show subbands 0-24 (but the behaviour is the same for higher subbands):

• 2d
• points

I also checked whether “bad” data blocks in raw voltages end up somewhere else in the visibilities (other antenna/polarisation/subband/channel) or vice versa, but this does not seem to be the case.

Looking only at the raw voltage data themselves, I found that some combinations of antenna/pol/subband/channel have exactly the same voltages as other combinations of the same block (of about 1sec) and integration. These “duplicates” show up in regular patterns that are the same for every 1-sec block. This is only of concern (and included in the analysis) as long as the voltages are not exactly 0 (i.e. flagged) as happens particularly in the first two blocks.

When interpreting the following analysis, keep in mind that a duplicate (by definition) always appears at least twice. At least one of each pair is corrupted, but other data points may be corrupted as well.

This plot shows at which integrations (horizontal axis) and station/pol (vertical axis) duplicates appear:

There seem to be two classes: (1) the regular “grid” ones happening at almost regular intervals in time, (2) the bad “bunches” closer to the end of the block that affect a full range of integrations

Here are histograms of the integration numbers at which duplicates appear:

The “grid duplicates”, appear at intervals of 98 or 99:

The upper panel shows the integration number divided by interval number as function of the latter, the lower one shows at which intervals bad integrations appear. The pattern is only semi-regular.

The following plot shows at which stations/pols pairs of duplicates appear:

These are the pairs of subbands that are affected:

We see that data are only duplicated within the same subband but never between different subbands. Subbands 12,19,29,49,55 do not seem to be affected at all. (Compare with what was found above!)

The pairs of channels are plotted here:

The patterns seem to depend on the integration number.

Here is the same as the first plot above, but this time separately for duplicates within one channel and between channels:

— Olaf Wucknitz 30-Jun-2011 16:51

After modifications of the software, Jan David Mol made this test observation on 8th July 2011. My first analysis shows the following:

preliminary: all stations, subbands and polarisations seem to be bad

preliminary: The header of the incoherent Stokes files seems to be incorrect. The sequency number is not in bytes 0-3 but in 4-7!

Blocks 0-12 look fine, all other blocks (with the exception of a few toward the end) have a ratio of incoherent/visib of exactly 4/5.

— Olaf Wucknitz 10-Jul-2011 16:47

L32955 is a new test observation with 8 stations in HBA low, 1sec integrations, 32 subbands, 16 channels/SB with raw voltages, incoherent Stokes and visibilities. Target was again the Crab.

In contrast to previous observations, we now have several (4) data blocks per correlator integration time of ~1sec. Comparison of products of integrated raw voltages with visibility data shows that the raw voltages seem to be okay now!

However, there may be some problems with the visibilities: Whenever there are bad samples (voltages set to zero) within an integration, the visibilities do not exactly correspond to the integrated raw-voltage-products.

This happens for several antennas at the beginning and end of the observations, where it becomes quite clear that the result derived from the voltages is more plausible. In addition, antenna 4 (DE602) always has a significant fraction of bad samples and deviating visibilities. Mostly the errors are within a few percent, but there are outliers which are off by factors of a few.

The definition of the visibilities seems to be the following:

V(ant1,ant2) = < conjugate (voltage1) * voltage2 >

Polarisation product XY means X for antenna2 and Y for antenna1, opposite to what I had expected! Note that the ordering of antennas within the baseline is ant1-ant2 with ant1>=ant2. This was different in older data sets (at least until MJD 55742.6, 29th June 2011), but true for all newer data sets (at least since MJD 55748.4, 5th July 2011).

The scaling is such that the visibility is the V(ant1,ant2) product averaged over the good samples, multiplied by the number of (good and bad) samples and divided by 10^6.

The following plots are about the comparison of RAW data with visibilities. Only autocorrelations were used here, but the general picture would be the same.

WEIGHT_SPECTRUM in MS vs. fraction of good samples in RAW:

We see that the WEIGHT_SPECTRUM numbers are mostly correct, but not always. They should be equal to the fraction of good samples.

Ratio of MS visibilities (only ACF) to value from RAW:

We see that the MS visibilities are not entirely consistent with the RAW data for blocks with bad samples.

Ratio of MS visibilities to value from RAW, only fully good blocks:

We see that MS and RAW are entirely consistent for blocks with only good samples. The scatter results from numerical errors.

Fraction of good samples:

All antennas but 4 (DE602) are good, with the exception of short periods towards the beginning and end. Antenna 4 has a high fraction of bad samples all the time.

A comparison of raw voltages with the incoherent Stokes sum is not possible, because some raw-voltage files are empty.

— Olaf Wucknitz 31-Oct-2011 13:14

John Romein suggested that the discrepancies may be due to the fact that the correlator first averages in blocks of 0.25sec which are then combined to form the 1sec visibilities. This last averaging steps does not seem to use the weights of the 0.25sec blocks. Using the same averaging scheme myself, the discrepancies disappear, with the exception of some freak points at the start and end of the observation. These are not entirely understood yet. They are also the points where the WEIGHT_SPECTRUM does not correspond to the number of good samples.

— Olaf Wucknitz 02-Nov-2011 12:23

After correcting the LofarStMan on 2/3 Nov 2011 for the incorrectly swapped antennas (ant1>=ant2 instead of ant1⇐ant2), the ordering problems explained above are solved.

The definition of the visibilities is now the following:

V(ant1,ant2) = < conjugate (voltage1) * voltage2 > (as before)

Polarisation product XY now means X for antenna1 and Y for antenna2, as it should be.

See here or more specifically here (pdf) for definitions in the LOFAR MS.