Enhanced Flicker Removal through Stationary Block Re-examination
KIN KEONG WONG, AMITABHA DAS and MAN NANG CHONG
School of Computer Engineering
Nanyang Technological University
SINGAPORE
Abstract: - Flicker, the unnatural temporal fluctuations in perceived image intensity, is a common artifact in old film sequences. Flicker causes much viewing discomfort and is thus much desirable to be removed in old films restoration. In this paper, we propose a robust flicker removal method based on flicker parameters estimation with enhancement through stationary block re-examination. We tested the proposed method on sequences with artificially added global and local flicker, as well as on original flicker film material. The results obtained showed significant improvement over similar methods established earlier.
Key-Words: Flicker Parameters Estimation, Roosmalen Method, Stationary Block Re-examination
1 Introduction
Human eyes are very sensitive to flicker which is a common artifact in old film materials. Such image intensity fluctuations are normally caused by factors such as film aging, photographic process, and twin-lens telecines mechanisms used in film production in the early days. Some research work has been done in recent years to address the flicker removal problem. A method based on the mean gray level of a frame to define an overall intensity adjustment was proposed by Richardson [1]in 1995. In 1997 Roosmalen [2] originated an idea (henceforth known as the “Roosmalen method”) which was based on correctly estimating two flicker parameters that could be used for intensity correction. The Roosmalen method works well for local flicker, but it is not very reliable in correcting flicker in motion regions. This is due to the difficulty in choosing suitable threshold values to accurately identify motion regions in a frame.
In this paper, an improved approach based on the Roosmalen method is proposed which has the ability to greatly improve the flicker correction quality by accurately detecting the motion regions. This is made possible using a novel stationary block re-examination process to be described in detailed in section 2.3.
2 Flicker Parameters Estimation
2.1 Model for Flicker Sequence
A noisy frame model given by equation (1) was used in the Roosmalen method:
Y(x,y,t) =(x,y,t)I(x,y,t) + (x,y,t) + (x,y,t) (1)
where x, y are discrete spatial coordinates and t indicates the frame number. Y(x,y,t) and I(x,y,t) indicate the observed and original image intensities, (x,y,t) and (x,y,t) the flicker gain and flicker offset parameters (collectively known as “flicker parameters”), and (x,y,t) the flicker-independent noise, which includes collectively all non-flicker related noise like grain noise, additive Gaussian noise etc.
In this paper, we make the assumption that pre-processing has been done on the noisy frames first to remove all the non-flicker noise. As a result, equation (1) is simplified to:
Y(x,y,t) =(x,y,t)I(x,y,t) + (x,y,t) (2)
A flicker-free image is thus represented by (x,y,t) and (x,y,t) values of 1.0 and 0 respectively.
2.2 Flicker Parameters Estimation In Stationary Regions
Flicker correction requires estimation of flicker parameters (x,y,t) and (x,y,t) based on the model given by equation (2). Since the presence of motion impedes the accuracy in estimating these flicker parameters, stationary scene will beassumed in thederivation of equations for estimating (x,y,t) and (x,y,t).
Assuming that (x,y,t) and (x,y,t) are spatially smooth and if only a small image region is taken into consideration, then the following is true:
(x,y) (3)
Equation (2) can now be re-written as:
Y(x,y,t) = (t)I(x,y,t) + (t) (4)
Equation (4) is used to form the basis of flicker correction derivation. Since Y(x,y,t) is the observed intensity, the original intensity I(x,y,t) can be obtained (implying the image can be corrected) if ways can be found to determine the flicker parameters (t) and (t).
The flicker parameters (t) and (t) can be estimated as follows. The expected value and variance of Y(x,y,t) in equation (4) for x,y in a spatial aspect are respectively given by:
E[Y(x,y,t)] = (t)E[I(x,y,t)] + (t) (5)
2[Y(x,y,t)] = 2(t) 2[I(x,y,t)] (6)
(t) and (t) can now be readily obtained from equations (5) and (6):
(7)
(8)
Equations (7) and (8) cannot be used for flicker correction because they are both in terms of I(x,y,t), the unknown intensity to be determined. We can find a solution to this problem by using the previously corrected frame as a reference for x,y , i.e.,
(9)
(10)
where denotes the intensity of the corrected t-1 frame for x,y .
Hence (t) and (t) can now be put in terms of parameters that can all be determined:
(11)
(12)
2.3 Motion Regions Detection
Since flicker parameters (t) and (t) are respectively the flicker gain and flicker offset, they form ideal candidates that can be used for motion detection. It is a known fact that motion leads to significant change in intensity mean and variance in temporal statistics, and consequently also results in significant change in the values of flicker parameters, i.e., large deviation from 1.0 for (t) and 0 for (t) [2]. Hence threshold values of the form 1±T and ±T can be used as criteria to check respectively on (t) and (t) for motion regions. When the threshold values are surpassed, then the region is flagged as a motion region.
The quality of flicker correction depends largely on the accuracy in estimating flicker parameters, and it is a known fact that motion hampers their estimation. In the Roosmalen method, motion regions are flagged when the flicker parameters surpass some pre-selected threshold values. Unfortunately there are no reliable ways to select the threshold values properly, and hence the accuracy of motion regions detection cannot be guaranteed.
To address this problem, we introduce an additional motion regions checking mechanism in this paper to improve the accuracy in identifying the motion regions. The details are illustrated as follows.
A two-tier checking process is adopted in this new approach. As before, motion regions are first detected using some pre-selected threshold values of flicker parameters. Stationary regions(i.e., non-motion regions) that have been identified can then be corrected based on equation 2.14. However someof these stationary regions identified at this stage could well be motion regions as it is known that they were not very reliable. These preliminary stationary regions will now be subjected to a re-examination. The mean square error (MSE) in intensity between the current corrected stationary block and the corresponding block at the same location in the previous corrected frame is first computed:
(13)
where S is the block size.
If the MSE exceeds certain threshold value, then this block that has been identified as “stationary” earlier will now be re-labeled as a motion block after the re-examination. This re-examination process will be carried out on all blocks that were previously identified as stationary and at the end of this process all “true” motion blocks can then be identified.
2.4Flicker Parameters of Motion Blocks
Once all the motion blocks are identified, their flicker parameter values can be obtained via iterative interpolation. Figure1 shows an example of how this process is carried out.
In figure 1(a), the black areas indicate stationary areas with flicker parameters computed, and the white areas are motion blocks identified at the end of the two-tier checking process. Flicker parameters of each white block can be estimated provided it is bordering with least two blocks of known parameters, by taking the average value of the flicker parameters of these neighbors. The shaded blocks in figure 1(b) show motion blocks with flicker parameters estimated after the first round of interpolation. By repeating this dilation process, flicker parameters of all motion blocks can be successfully estimated.
(a) (b)
(c) (d)
Stationary block with computed flicker gain
Motion block with unknown flicker gain
Motion block whose flicker gain value is estimated after 1st round of interpolation
Motion block whose flicker gain value is estimated after 2nd round of interpolation
Motion block whose flicker gain value is estimated after 3rd round of interpolation
Fig.1 Iterative interpolation procedure
2.5Correcting Image Flicker
With flicker parameters for all regions (both stationary and motion) determined, they can now be used to correct the flicker frame based on equation (2), which is now re-written in the form of equation (14) below:
(14)
where (x,y,t) and (x,y,t) are the flicker parameters estimated.
3. Experimental Results and Analysis
The proposed method in this paper was tested with both synthetic sequence and real film sequence and the results are presented below.
Initial test was carried out using synthetic global flicker sequences of low, medium and high frequencies created by adding flicker to a flicker-free “James McDonald” sequence of 20 frames (0th to 19th frame). Test results show that the proposed method has consistently shown better performance than that of the Roosmalen method. This is demonstrated by the higher peak-signal-to-noise-ratio (PSNR) obtained throughout. Figure 3 shows the PSNRresults corresponding to the low-frequency flicker sequence as depicted in figure 2. The definition of PSNR is given in equation (15) below:
(15)
where , in which MxN is the frame size, x and y are pixel intensity before processing (i.e. original good frame) and after processing (i.e. corrected frame) respectively.
It can be seen from the curves in figure 3 that the proposed method produces a higher PSNR value than that of the Roosmalen method. In fact the former has an average PSNR of 41.52 dB compared to 37.15dB of the latter; this amounts to approximately 4dB improvement.
The improvement of PSNR in the proposed method can be explained by the two-tier motion block checking process. Figure 4(a) and figure 4(b) respectively show the stationary blocks (black regions) and motion blocks (white regions) detected by the Roosmalen method and the proposed method in frame 12 of the low-frequency flicker sequence. It can be seen that a significant number of black blocks in figure 4(a) are re-colored as white in figure 4(b) This is due to the fact that those stationary blocks identified in the Roosmalen method are actually motion blocks after the re-examination process. Improvement in the accuracy of motion blocks detection thus leads to a higher PSNR value.
Similar to that of the Roosmalen method, the intensity mean obtained for the corrected sequence using the proposed method comes close to that of the original good frame sequence, implying good flicker reduction; this is illustrated in figure 2.
Fig.2 Intensity mean curves
Fig.3 PSNR comparison
(a) (b)
Fig.4: Stationary blocks (black regions) and motion blocks (white regions) in frame 12 of the low-frequency flicker sequence by (a) Roosmalen method, (b) Proposed method
Further test was carried out using a synthetic local flicker sequence created by adding local flicker to a 10-frame (0th to 9th frame) “Taxi” flicker-free sequence. Local flicker generation is based on the flicker model given by equation (2) where (x,y,t) and (x,y,t) are respectively generated from a normal distribution of N(1,0.5) and N(0,20).
The results obtained are very similar to those obtained earlier for global flicker. The PSNR values obtained are once again better than those of the Roosmalen method, with a maximum difference of about 13 dB; this is shown in figure 5. The intensity mean curve for the corrected sequence also follows that of the original good frames closely, as depicted in figure 6.
Finally test was carried out using a 90-frame real flicker film sequence known as the “Lady” sequence. Figures 7(a) and 7(b) respectively show the much smoother mean and variance curves of the corrected frame sequence compared to those of the original corrupted sequence, implying significant reduction of flicker. This is confirmed when checking the corrupted and corrected sequence visually.
Fig.5 PSNR comparison
Fig.6 Intensity mean curves
(a)
(b)
Fig.7: (a) Means and (b) Variances of the “Lady” flicker sequence before and after processing using the proposed method
4. Conclusion
An improved flicker removal method based on stationary block re-examination was proposed in this paper. The proposed method improves the accuracy in motion blocks detection in the Roosmalen method, thus producing better quality flicker correction. This is confirmed by the test results obtained when the proposed method was tested with both synthetic and natural flicker frame sequences.
References:
[1] P.Richardson and D. Suter, “Restoration of historic film for digital compression: A case study”, Proc. ICIP-95, volume 2, pages 49-52, Washington DC, 1995.
[2] P.M.B. van Roosmalen, R.L. Lagendijk and J.Biemond, “Flicker reduction in old film sequences”, Time-varying Image Processing and Moving Object Recognition, pages 9-18, 1997.