Scalable video coding extension of HEVC (S-HEVC)

Submitted By:

Aanal Desai

1001103728

List of Acronyms and Abbreviations:

· AVC – Advanced Video Coding

· BL – Base Layer

· CABAC – Context Adaptive Binary Arithmetic Coding

· CTB – Coding Tree Block

· CTU – Coding Tree Unit

· CU – Coding Unit

· DASH – Dynamic Adaptive Streaming over HTTP

· EL – Enhancement Layer

· FPS – Frames per second

· HD – High Definition

· HEVC – High Efficiency Video Coding

· HLS – High Level Syntax

· HTTP – Hyper Text Transfer Protocol

· ILR – Inter Layer Reference

· JCTVC – Joint Collaborative Team on Video Coding

· Mbps – Megabits per second

· MPD – Media Presentation Description

· MPEG – Moving Picture Experts Group

· JPEG- Joint Picture Experts Group

· MV – Motion Vector

· PSNR – Peak Signal to Noise Ratio

· PU – Prediction Unit

· SAO – Sample Adaptive Offset

· SHVC – Scalable High Efficiency Video Coding

· SNR – Signal to Noise Ratio

· SPIE – Society of Photo-Optical Instrumentation Engineers

· TU – Transform Unit

· UHD – Ultra High Definition

· URL – Uniform Resource Locator


Overview:

Due to the increased efficiency of video coding technology and the developments of network

infrastructure, storage capacity, and computing power, digital video is used in more and more application areas, ranging from multimedia messaging, video telephony and video conferencing over mobile TV, wireless and Internet video streaming to standard- and high- definition TV broadcasting. On the one hand, there is an increasing demand for video streaming to mobile devices such as smartphones, tablet computers, or notebooks and their broad variety of screen sizes and computing capabilities stimulate the need for a scalable extension. On the other hand, modern video transmission systems using the Internet and mobile networks are typically characterized by a wide range of connection qualities, which are a result of the used adaptive resource sharing mechanisms. In such diverse environments with varying connection qualities and different receiving devices, a flexible adaptation of once-encoded content is necessary[2]. Scalable video coding is a key to the challenges modeled by the characteristics of modern video applications. The objective of a scalable extension for a video coding standard is to allow the creation of a video bitstream that contains one or more sub-bitstreams, that can be decoded by themselves with a complexity and reconstruction quality comparable to that achieved using single-layer coding with the same quantity of data as that in the sub-bitstream[2].

SHVC provides a 50% bandwidth reduction for the same video quality when compared to the current H.264/AVC standard. SHVC further offers a scalable format that can be readily adapted to meet network conditions or terminal capabilities. Both bandwidth saving and scalability are highly desirable characteristics of adaptive video streaming applications in bandwidth-constrained, wireless networks[3].The scalable extension to the current H.264/AVC [4] video coding standard (H.264/SVC) [8] provided resources of readily adapting encoded video stream to meet receiving terminal's resource constraints or prevailing network conditions. Several H.264/SVC solutions have been proposed for video stream adaptation to meet bandwidth and power consumption constraints in a diverse range of network scenarios including wireless networks [10]. But, while addressing issues of network reliability and bandwidth resource allocation, they do not address the important issue of the ever- increasing volume of video traffic. HEVC reduces the bandwidth requirement of video stream by approximately 50% without degrading the video quality. So, HEVC can significantly lessen the network congestion by reducing the bandwidth required by the growing volume of the video traffic. The JCT-VC is now developing the scalable extension (SHVC) [5] to HEVC in order to bring similar benefits in terms of terminal constraint and network resource matching as H.264/SVC does, but with a significantly reduced bandwidth requirement[3].

Introduction:

There are normally three types of scalabilities: Temporal, Spatial and SNR Scalabilities. Spatial scalability and temporal scalability defines cases in which a sub-bitstream represents the source content with a reduced picture size (or spatial resolution) and frame rate (or temporal resolution), respectively[1] . Quality scalability, which is also referred to as signal-to-noise ratio (SNR) scalability or fidelity scalability, the sub-bitstream delivers the same spatial and temporal resolution as the complete bitstream, but with a lower reproduction quality and, thus, a lower bit rate[2].

In this perspective, scalability refers to the property of a video bitstream that allows removing parts of the bitstream in order to adjust it to the needs of end users as well as to the capabilities of the receiving device or the network conditions, where the resulting bitstream remains compatible to the used video coding standard. It should, however, be noted that two or more single layer bitstreams can also be transmitted using the method of simulcast, which delivers similar functionalities as a scalable bitstream. Additionally, the adaptation of a single layer bitstream can be accomplished by transcoding. Scalable video coding has to compete against these alternatives. In particular, scalable coding is only useful if it offers a higher coding efficiency than simulcast[2].

Initial standards for the transmission of HEVC streams over loss-prone wireless and wired networks were established in a testbed environment in [7], which presented the effects of packet loss and bandwidth reduction on the quality of HEVC video streams. An equivalent work [6] provided a smaller set of largely similar benchmarks that were obtained by simulation rather than the testbed approach used in [7]. The authors of [7] have also proposed a scheme [9] to alleviate packet loss in HEVC by prioritizing and selectively dropping packets in response to a network resource constraint[3].

BLOCK DIAGRAM OF ENCODER:

The design of HEVC certainly enables temporal scalability when a hierarchical temporal prediction structure is used. Therefore the proposed scheme concentrates on spatial and SNR scalability cases. A multi-loop decoding structure is employed to support these functionalities. Inside the framework of multi-loop decoding, all the information in the base layer (BL), including reconstructed pixel samples and syntax elements, is available for coding the enhancement layer (EL) in order to attain high coding efficiency[1].

Fig 1. High-Level block diagram of the proposed encoder.[1]

(Figure1) above shows the block diagram of the proposed scalable video encoder for spatial scalability. For SNR(Quality) scalability, the up-sample step is not essential.

1.Inter-layer Intra prediction:- A block of the enhancement layer is predicted using the

reconstructed (and upsampled) base layer signal.

-Inter-layer motion prediction:- The motion data of a block are completely inferred

using the (scaled) motion data of the co-located base layer blocks, or the (scaled)

motion data of the base layer are used as an additional predictor for coding the

enhancement layer motion.

-Inter-layer residual prediction:- The reconstructed (and upsampled) residual signal of

the co-located base layer area is used for predicting the residual signal of an inter-picture coded block in the enhancement layer, while the motion compensation is

applied using enhancement layer reference pictures[2].

At the first look the scalable encoder comprises of two encoders, one for each of the layer. In spatial scalable coding, the input video is downsampled and fed into the base layer encoder, whereas the input video of the original size represents the input of the enhancement layer encoder. In quality scalable coding, both the encoders use the same input signal. The base layer encoder adapts to a single-layer video coding standard, so that the backwards compatibility with single-layer coding is achieved; the enhancement layer encoder generally contains additional coding features. The outputs of both encoders are multiplexed to form the scalable bitstream[2] .The inter and intra prediction modules of the enhancement layer encoder are altered to accommodate the base layer pixel samples in the prediction process. The base layer syntax elements containing motion parameters and intra modes are used to predict the corresponding enhancement layer syntax elements and to decrease the overhead for coding syntax elements. The transform/quantization and inverse transform/inverse quantization modules (denoted as T/Q and IT/IQ) respectively, in Figure 1 are developed such that additional DCT and DST transforms may be applied to inter-layer prediction residues for better energy compaction. The offered codec is designed to deliver a good balance between coding efficiency and implementation complexity[1]. In order to improve the coding efficiency, the data of the base layer must to be employed for an efficient enhancement layer coding by so-called inter-layer prediction methods[2]. The lower level processing modules from the single layer codec such as loop filtering, transforms, quantization and entropy coding are virtually unchanged in the enhancement layer. The changes are mainly focused in the prediction process[1]. The proposed codec was submitted as a response [11] to the joint call for proposals issued by MPEG and ITU-T on HEVC scalable extension [12]. It achieved the highest coding efficiency in terms of RD performance among all responses [13].

2 Inter-layer texture prediction:

H.264/AVC-SVC [14] presented inter-layer prediction for spatial and SNR scalabilities by using intra-BL and residual prediction under the constraint of a single-loop decoding structure. Hong et al [15] proposed a scalable video coding scheme for HEVC, where the residual prediction process is extended to both intra and inter prediction modes within a multi-loop decoding framework. In this paper, the multi-loop residual prediction is further improved by using generalized weighted residual prediction. In addition to the intra-BL and residual prediction, a combined prediction mode, which uses the average of the EL prediction and the intra-BL prediction as the final prediction, and multi- hypothesis inter prediction, which produces additional predictions for EL block using BL block motion information, are also presented.

2.1 Intra-BL prediction

To utilize reconstructed base layer information, two Coding Unit (CU) level modes,

namely intra-BL and intra-BL skip, are introduced[1]. For an enhancement layer CU, when ilpred_type indicates the IntraBL mode, the prediction signal is formed by copying or, for spatial scalable coding, upsampling the co-located base layer reconstructed samples. Since the final reconstructed samples from the base layer are used, multi-loop decoding architecture is essential[2].

When a CU in the EL picture is coded by using the intra-BL mode, the pixels in the collocated block of the up-sampled BL are used as the prediction for the current CU.

For CUs using the intra-BL skip mode, no residual information is signaled[1]. Procedure for the up-sampling is decribed later in the paper. The operation is similar to the inter-layer intra prediction in the scalable extension of H.264| MPEG-4 AVC, except that it is likely to use the samples of both intra and inter predicted blocks from the base layer[2].

2.2 Intra residual prediction:

In the intra residual prediction mode, as shown in Figure 2, the difference between the

intra prediction reference samples in the EL and collocated pixels in the up-sampled BL is generally used to produce a prediction, denoted as difference prediction, based on the intra prediction mode. The generated difference prediction is further added to the collocated block in the up-sampled BL to form the final prediction.

Fig 2. Intra Residual Prediction. [1]

In the offered codec, the intra prediction method for the difference signal remains

unchanged with respect to HEVC excluding the planar mode. For the planar mode, after intra prediction is performed, the bottom-right portion of the difference prediction is set to zero. Now the bottom-right portion refers to each position (x, y) satisfying the

condition (x + y) >= N-1, [where N is the width of the current block.]

Because of the high frequency nature of the difference signals, the HEVC mode dependent reference sample smoothing process is disabled in the EL intra residual prediction mode[1].

2.3 Weighted Intra prediction:

[Fig 3 Weighted intra prediction mode. The (upsampled) base layer reconstructed samples are combined with the spatially predicted enhancement layer samples to predict an enhancement layer CU to be coded.] [2]

In this mode, the (upsampled) base layer reconstructed signal constitutes one component for prediction. Another component is acquired by regular spatial intra prediction as in HEVC, by using the samples from the causal neighborhood of the current enhancement layer block. The base layer component is low pass filtered and the enhancement layer component is high pass filtered and the results are added to form the prediction. In our implementation, both low pass and high pass filtering happen in the DCT domain, as illustrated in Figure 3. First, the DCTs of the base and enhancement layer prediction signals are computed and the resulting coefficients are weighted according to spatial frequencies. The weights for the base layer signal are set such that the low frequency components are taken and the high frequency

components are suppressed, and the weights for the enhancement layer signal are set vice versa. The weighted base and enhancement layer coefficients are added and an inverse DCT is computed to obtain the final prediction[2].

2.4 Difference prediction modes:

The principle in difference prediction modes is to lessen the systematic error when using the (upsampled) base layer reconstructed signal for prediction. It is accomplished by reusing the previously corrected prediction errors available to both

encoder and decoder. To this end, a new signal, denoted as the difference signal, is derived using the difference amongst already reconstructed enhancement layer samples and (upsampled) base layer samples. The final prediction is made by adding a component from the (upsampled) base layer reconstructed signal and a component from the difference signal [17].This mode can be used for inter as well as intra prediction cases[2].

Fig 4.[ Inter difference prediction mode. The (upsampled) base layer reconstructed signal is combined with the motion compensated difference signal from a reference picture to predict the enhancement layer CU to be coded.] [2]

In inter difference prediction shown above in Fig 4, the (upsampled) base layer reconstructed signal is added to a motion-compensated enhancement layer difference signal equivalent to a reference picture to obtain the final prediction for the current enhancement layer block. For the enhancement layer motion compensation, the same inter prediction technique as in single-layer HEVC is used, but with a bilinear interpolation filter[2].

Intra Prediction:

Fig 5. [Intra difference prediction mode. The (upsampled) base layer reconstructed signal is combined with the intra predicted difference signal to predict the enhancement layer block to be coded.] [2]

In the intra difference prediction, the (upsampled) base layer reconstructed signal constitutes one component for the prediction. Another component is derived by spatial intra prediction using the difference signal from the underlying neighborhood of the current enhancement layer block. The intra prediction modes that are used for spatial intra prediction of the difference signal are coded using the regular HEVC syntax. As Shown in the Fig 5 above,The final prediction signal is made by adding the (upsampled) base layer reconstructed signal and the spatially predicted difference signal[2].