Data Processing Pipeline
The data processing pipeline describes the work flow from the data acqusition
to the final data repository in the archives and substantial processing steps. The methods and
procedures are explained in the sections below:
At Observation Time
|-- expose to constant "density"||vary exposure
time to adapt for equal average intensities in AOI in each frame|
|-- frame selection on pixel contrast||calculate standard
deviation of intensities in AOI and select frame with max. std. dev. within a certain
number of frames|
|--» A. Raw Level Data||
FITS, normally not available|
After Observation Processing
|-- discard corrupted images||check for reasonable disk radii and intensities in all quadrants|
|-- cadence reduction||select "best of hour" by the Optimum Window Method,
or keep all during high activity periods|
|-- determination of actual disk center coord. and radii|
|-- shift disk to center of FoV||only by integer number of pixel
to avoid any data averaging|
|-- calculate properties and parameters for achive level data||e. g.
angle Θ, image scale,...|
|-- mask outside disk = 0||set pixels far outside the disk to zero for
a better data compression|
| --» B. Archive Level Data||
FITS, optional JPEG available|
|-- application of a gain table (flatfield)||"Burlov"-method,
which assumes and reduces the image to a circular symmetry on large scales and delivers
also a CLV profile|
|-- calculate properties and parameters for synoptic level data
||e. g. P, B0, L0 and SOHO standard FITS keys for
|-- contrast enhancement and derotation||unsharp masking,
intensity range rescaling, rotated by P and Θ to have solar N up|
| --» C1 Synoptic Level Data||
denominated as "low contrast", FITS and JPEG available.|
|-- normalization||divide by CLV (quiet Sun map)|
|-- contrast enhancement and derotation||unsharp masking,
linear intensity range rescaling, rotated by P and Θ to have solar N up|
| --» C2 Synoptic Level Data - normalized
||denominated as "high contrast", FITS and JPEG available.|
Determination of the solar limb is crucial
for deriving geometrical information from the solar images. Properties of the disk
(center coordinates and radius) are needed for the shifting of the disks towards the
center of the field of view and the actual image scale. They are given in the FITS
headers of archives and synoptic level data.
The solar limb is commonly defined as the inflexion point of a
radial intensity profile. A stable method which works also on images which show only
a part of the disk (cf fig. 10) is to find limb points on a set of disk profiles
parallel to the CCD y-axis. A profile is smoothed, the first derivative is calculated
and again smoothed. The max and the min of the derivative should indicate the position
of the inflexion point of the intensity profile.
A circle (x0, y0, r) representing the disk is
derived by fitting this set of limb points with the least squares method.
Figure 7: Position of the intensity
profiles used for limb detection. Starting from column 200 to col. 1800 each
ten pixel columns a profile is selected.
Figure 8: Limb profile of a continuum image.
The dots represent the individual pixel values along the radial profile (in this plot
parallel to the CCD x-axis), the solid line is a running average over 15 pixel values.
The dashed line is the smoothed derivative (again running average over 15) of the
intensity profile. The vertical line shows the max of the derivative and therefore
the position of the solar limb.
In a second step the individual radial distances of the set of
limb points to the fitted circle from the above described first approximation is
calculated, limb points with distances above a certain threshold (e.g. 10 pix) are
neglected for a new estimation of the limb circle. With this iteration we yield
usually standard errors in radius (from the least squares method) of < 1
In order to derive heliophysical coordinates of solar features on
solar images one has to know the orientation of a pixel axis of
the CCD with respect to celestial coordinates. The equatorial mounting of the telescope has the
advantage of no image rotation during the diurnal movement of the Sun across the celestial
sphere. Apart from small errors due to non-perfect telescope setup or bending of the optical
axis (on the changing direction of the weight of the telescope during a day) the angle Θ
between the pixel axis and e.g. the celestial E-W-direction should be constant as long as
the system will not be disassembled for maintenance. If the tracking system of the
telescope is switched off and one neglects the change of the solar declination and of
the refraction during a few minutes the track of a solar feature or of the disk center
will represent the celestial E-W-direction.
Figure 9: The principle of the determination of
the E-W direction in the images. Inclination of the Solar rotation axis (P and B0)
is reckoned from celestial North which is perpendicular to the derived E-W track.
Therefore we obtain from time to time a series of about 20
images with no tracking having the solar disk moving across the field of view.
Applying an edge detection filter, fitting circles through the solar limb points
and calculating the center coordinates yield finally a track of the disk
centers and the angle Θ.
Figure 10, top: Drift of the solar disk
centers of an images series taken at 2007-08-01 around 6:50 UT. The + mark the
calculated disk center coordinates of the individual images. Θ is derived by a
linear fit from these positions y = tanΘ.x + d.
Bottom: The image series taken at 2007-08-01 around 6:50 UT covers a range where
about a half disk is visible at the eastern limb of the field of view until still a
half disk is visible at the opposite end of the CCD field.
The standard error of this procedure was determined of about 0.02°
for Θ and the daily variation due to the above described imperfections is in the
order of 0.05°. The standard error for the deviation of individual limb pixels from
the calculated circles is within one image less than 1 pixel, the observed variation of
2...3 pixels typically in the disk radii of about 930 pixels in image series is rather an
influence of the seeing than an error in the fitting procedure. So the individual
calculated disk radii have an error in the order 0.1% and we can estimate a
reliability of better than 1° in heliographic positions in our images.
Figure 11: The derived radii of the images (gain from the series taken
at 2007-08-01 around 6:50 UT) are relatively stable even when only a part of the disk is visible,
the variation is mainly due to seeing effects.
Figure 12: The variation of Θ
from 3 images series taken on 2007-08-01. The observation time (middle of each series)
is indicated by the hour angle of the telescope at the observation time. We assume that
the variation should depend on the geometrical position i. e. the hour angle of the
parallactic telescope mounting. The dashed line is a second order polynomial fit
which should be symmetric to noon.
CCDs have basically a linear relation between the
produced charge (and the output voltage) and the amount of the incident light, but there
exists a level of saturation which limits the linear range. Therefore the amount of light
during exposure has to be limited below that level.
The TM-4100CL allows to set the gain and the lower level of the A/D converter, so that the
full dynamic range (which corresponds to pixel value 0...1023) can be mapped into the linear part
of the CCDs charge vs. incident light relation. The zero level is set properly when
some noise floor (dark current) is visible. Signal noise limits the
dynamic resolution of the pixels (the number of useful bits).
For setting and checking of these properties we used a simple procedure which was proposed
by ESO (see Deiries, S.: 1995, ESO Doc. No. VLT-INS-ESO-13670-0001).
Figure 13, left: Single dark current frame of the TM4100CL.
The mean of the pixel values is dc = 4.3 counts with σ = 2.8.
The read-out noise, defined as the standard deviation of the pixel differences of 2 dark current frames,
is rmsnoise = 2.77 which yields a dynamic range of 51 dB or 8.5 significant bits.
Right: Smoothed average of 17 dark current frames with integration times from less than 1 ms to about 10 ms.
There are variations in the dark current of the individual taps of the dual tap CCD array visible,
however we noticed that a general subtraction of such a smoothed average dark current frame
did not improve the images.
In the lab the cam was illuminated with a stable and uniform light source
(e.g. a LED with diffusor) and a set of paired frames (with identical exposure times) were
taken in a wide range of exposure times. Sums and differences of individual pixel values of
the 2 frames of each pair were used to calculate statistical properties (mean, standard
deviation - for details see the ESO paper and the image captions). These statistics were
made on sub-fields (e.g. of size 256x256 pixels) to avoid averaging over potentially
non-uniform areas: The means of the pixel values as a function of the amount of incident
light (i. e. of the exposure time) showed a non-linearity of 0.2%. The standard
deviations of the difference of the equally exposed frame pairs give the noise in the data.
The dark current proved to be fairly independent from the exposure time and is noise
dominated (see fig. 13 and 14), probably produced by the electronics (read-out noise).
The ratio between full scale and noise yields finally the useful dynamic range and can be
expressed in the effective number bits.
Figure 14: Parameters for the TM4100 cam with
the finally selected gain and lower level for the A/D converter, derived from the statistical
evaluation of the pairs of frames uniformly illuminated as described in the text. The plots
show the numbers for a 256x256 pixels area starting from pixel (1024,768), values for
other sub-fields do not differ substantially, even on fields with double linear dimensions.
Top left the variances of the individual pixel differences of the frame pairs which can
be used for an estimation of the conversion factor (data units per absorbed electron/photon).
Top, right: Linearity of the output signal as a function of the incident light, which is
proportional to the exposure/integration time if we assume a stable light source. Bottom
row: Dark current (mean and standard deviation) of the individual non-illuminated
frames but with a variation of integration times. Right: The standard deviations of
the individual pixel differences of the frame pairs of identical integration times.
Both plots show that the values are integration-time independent and of the same order,
therefore the dark current is rather a noise than a bias. As noise is unpredictable,
it is an uncertaincy in the signal and cannot be subtracted. The methods are explained in
detail in the mentioned ESO paper.
A flatfield frame can be considered as a
multiplicative map of non-uniform gain of the individual pixels. It may origin from a
real non-uniform gain or sensitivity of the pixels or may be a result of non-uniform
illumination of the telescope, due to e.g. tilt lenses. This non-uniformity was
investigated and calculated with 2 methods which deliver such maps and a center-to-limb
variation/darkening (CLV) estimate as well:
- Method A) - denominated as Kuhn-Lin method - by a set 8-10 consecutively
taken solar image frames which assume to show the same object but at shifted positions
(see Kuhn, J. R., Lin, H., & Loranz, D.: 1991, Publ. Astr. Soc. Pacific, 103, 1097),
any variation of the solar image in the set must origin from non-uniformity.
- Method B) - denominated as Burlov method - by assuming circular symmetry in a
(spotless) solar disk image. The disk is split into concentric rings. The intensity function
of each ring and therefore potential asymmetries are mapped by fitting polynomials. These
polynomial represent the non-uni-formity and can be used to calculate a flatfield map and
correct the images. Real asymmetries in the images, e. g. caused by active regions, are of
smaller scale and should not be corrected. This can be handled by limiting the order of the
polynomials. This method was basically developed by K. Burlov-Vasiljev and P. N. Brandt at
KIS Freiburg in 1996 for the processing of RISE/PSPT images but never published.
Figure 15: Comparison of flatfield methods described in
the text, left method A) and right method B), bottom row is a TV-representation where higher
values are brighter. Clearly visible that in method A) the noise which is from nature variable
from frame to frame produces more small-scale non-uniformity than any visible large scale
variation. Method B) uses only a single frame and no differences, assumes large scale circular
symmetry and stops compensation of the variation at a certain scale.
Both methods showed that the effect of noise dominates over non-uniformity
which is in the single percent range (Fig. 15). Fig. 16 shows for comparison two very high contrast
solar images of a spotless Sun where limb darkening is compensated by the CLV function which
is a by-product of the both methods. It is clearly visible that both methods fail at the
extreme limb regions, but method B) seems to be a little bit better (see left part
of the disk), but it has to be further checked on images with sunspots.
Figure 16: Very high contrast images of the
spotless Sun from 2007-08-01 to check the quality of the flat fielding methods. Left
processed with method A) and right with method B). Limb darkening is compensated by
applying the CLV function derived from the flat-fielding procedures. Between the
black and white level there is only an intensity variation of 1%.
In practice the method B) has advantage that it can be applied
as a post-processing on any archived image without any prerequisite at observation
time, but needs some computing power for the calculation of the rings and the fitting
of the polynomials. Method A) however needs a recording of the set of displaced
images on a clear sky from time to time followed by a calculation of the flat-field
frame, but allows then to apply this frame to a series of subsequently taken images.