Project Proposal: HEVC Lossless Coding and Improvements

Topics in Signal Processing

Course ID: EE5359

Project Proposal: HEVC Lossless Coding and Improvements

SUBMITTED BY: SUJATHA GOPALAKRISHNAN

STUDENT ID: 1001024145

Table of Contents

ACRONYMS:

HEVC

Block Diagram HEVC

HEVC Lossless Coding

Block Diagram HEVC Lossless Coding

Basic Definitions

H.264[1][2]

Inter Frame

Intra Frame

Loop Filters

De-blocking Filter

Sample Adaptive Offset

Block-Based Angular Intra Prediction

Sample-Based Angular Intra Prediction

Coding Tools

LCU/CTU

Parallel Processing

Entropy Coding

Motion Estimation

Motion Compensation

Algorithm of Sample Based Angular Intra Prediction

Proposed Project

Projected Enhancement in the Project

References

ACRONYMS:

2D / Two dimension
3D / Three dimension
ACM MoVid / Association for Computer Machinery Mobile Video
AHG / Ad Hoc Groups
AIF / Adaptive Interpolation Filter
ALF / Adaptive Loop Filter
AMVP / Advanced Motion Vector Prediction
APIF / Adaptive Pre-Interpolation Filter
ASIC / Application-Specific Integrated Circuit
AVC / Advanced Video Coding
AVS / Audio Video Standard
BBC / British Broadcasting Corporation
BD / Bjontegaard Distortion
BL / Base Layer
bpp / Bits per pixel
BS / Boundary Strength
CU / Coding Unit
CI / Confidence Interval
CABAC / Context Adaptive Binary Arithmetic Coding
CPU / Central Processing Unit
CRA / Clean Random Access
CSVT / Circuits and Systems for Video Technology
CU / Coding Unit
DCT / Discrete Cosine Transform
DCTIF / Discrete Cosine Transform Interpolation Filters
DDCT / Directional Discrete Cosine Transform
DSP / Digital Signal Processing
DST / Digital Sine Transform
EC / Error Concealment
FIR / Finite Impulse Response
FPGA / Field Programmable Gate Array
fps / Frames per second
GPU / Graphics Processing Unit
HDR / High Definition Range
HEVC / High efficiency video coding
HEVStream / High Efficiency Video Stream
HTTP / Hyper Text Transfer Protocol
ICIEA / IEEE Conference on Industrial Electronics and Applications
IEEE / Institute of Electrical and Electronics Engineers
INTDCT / Integer Discrete Cosine Transform
intra HE / Intra high efficiency
IPTV / Internet Protocol Television
IS & T / Information Systems and Technology
ISO / International Organization for Standardization
ITU-T / Telecommunication Standardization Sector of the International Telecommunications Union
IVMSP / Image, Video, and Multidimensional Signal Processing
JCTVC / Joint Collaborative Team on Video Coding
JM / Joint Model
JPEG / Joint Photographic Experts Group
JPEG-XR / JPEG extended range
JSVM / Joint Scalable Video Model
JTC / Joint Technical Committee
LR / Low Resolution
Mbit/s / Megabit per second
MC / Motion Compensation
MDDCT / Modified Directional Discrete Cosine Transform
MDDT / Mode-Dependent Directional Transform
ME / Motion Estimation
MJPEG / Motion JPEG
MMSP / Multimedia Signal Processing
MPEG / Moving Picture Experts Group
Mpixel / Megapixel
Mpm / Most Probable Modes
MV / Motion Vector
NAB / National Association of Broadcasters
NAL / Network Abstraction Layer
PCM / Pulse Code Modulation
PSNR / Peak-to-peak signal to noise ratio
PU / Prediction Unit
QP / Quantizer parameter
RD / Rate Distortion
RDOQ / Rate-distortion optimized quantization
RDPCM / Residual Differential Pulse Code Modulation
ROT / Rotational Transform
RTP / Real-time Transport Protocol
SAO / Sample adaptive offset
SHVC / Scalable High Efficiency Video Coding
SVC / Scalable Video Coding
SELC / Sample based weighted prediction for Enhancement Layer Coding
SIP / Signal and Image Processing
SSVC / Spatially Scalable Video Coding
TB / Transform Block
TU / Transform Unit

HEVC

• High Efficiency Video Coding(HEVC)[1][2] is avideo compression standard, a successor toH.264/MPEG-4 AVC[22]. HEVC is said to double thedata compressionratio compared to H.264/MPEG-4 AVC[1] at the same level of video quality[2].
• The design of most video coding standards is primarily aimed at having the highest coding efficiency
• HEVC benefits from the use of largerCoding Tree Unit(CTU) sizes.
• The HEVC video coding layer uses the same "hybrid" approach used in all modern video standards, starting fromH.261[1], in that it uses inter-/intra-picture prediction and 2D transform coding.
• The main goal of the HEVCstandardization effort is to enable significantly improvedcompression performance relative to existing standards, in therange of 50% bit rate reduction for equal perceptual videoquality [10] [11].

Block Diagram HEVC

Figure 1: HEVC Encoder [2]

Figure 2: HEVC Decoder Block Diagram [3]

Some differences in HEVC[1][2] are coding tree units instead of macro blocks, single entropy coding methods-Context Adaptive Binary Arithmetic Coding (CABAC)[15] method and features like tiles , wave front parallel processing and dependent slices to enhance parallel processing.

HEVC Lossless Coding

• The lossless coding mode of HEVC main profile bypasses transform quantization and in-loop filters as shown in the fig.2[4][19].
• Comparing it with non-lossless coding mode, it has smallest quantization parameter value.
• Lossless coding mode provides perfect fidelity and average bit rate reduction.
• Outperforms the existing lossless compression solution such as JPEG-2000[22] and JPEG-LS[22].
• It employs Sample Angular-based Intra-Prediction (SAP)[4].
• Same prediction mode signaling method.
• Same interpolation method of HEVC.
• Uses adjacent neighbors as reference shown in fig.7.

Block Diagram HEVC Lossless Coding

Figure 3: HEVC lossless Algorithm Block Diagram [4]

The blocks that are marked bypass are not being used when implementing a HEVC[1][2] lossless algorithm, thereby providing average bit rate reduction.

Basic Definitions

H.264[1][2]

H.264/MPEG-4 AVC[1][2][22] is a block-oriented,motion-compensationbased video compression standard.

Inter Frame

Aninter frameis a frame in avideo compressionstream which is expressed in terms of one or more neighboring frames. The "inter" part of the term refers to the use ofInter frame prediction.

Intra Frame

The term intra-frame refers to the various lossless andlossy compressiontechniques that happens relative to information which is contained only within thecurrent frame andnot relativeto any other frame in the video sequence.

Loop Filters

HEVC[1] specifies two loop filters that are applied sequentially; thede-blocking filter(DBF)[4] applied first and the sample adaptive offset (SAO) filter applied afterwards.Both loop filters are applied in the inter-picture prediction loop, i.e. the filtered image is stored in the decoded picture buffer (DPB) as a reference for inter-picture prediction.

De-blocking Filter

• The DBF is similar to the one used by H.264/MPEG-4 AVC[1][2], but with a simpler design and better support for parallel processing.
• DBF first apply horizontal filtering for vertical edges to the picture and only after that does, it apply vertical filtering for horizontal edges to the picture.This allows for multiple parallel threads to be used for the DBF[1].

Sample Adaptive Offset

The SAO filter is applied after the DBF and is designed to allow for better reconstruction of the original signal amplitudes by applying offsets stored in alookup tablein the bit stream.

Block-Based Angular Intra Prediction

It is a method of computing predicted samples produced by PU when lossless coding is not enabled. It is defined to exploit spatial sample redundancy in intra coded CUs. As shown inthe fig.4, a total of 33 angles are defined for the angular prediction, which can be categorized into two classes: vertical and horizontal angular predictions as illustrated [14].

Figure 4: Block Based Angular Intra Prediction in HEVC [4]

Sample-Based Angular Intra Prediction

It is a method of computing predicted samples produced by PU when lossless coding is enabled. It is explained detail in the following.

Coding Tools

Coding efficiency is the ability to encode video at the lowest possible bit rate while maintaining a certain level of video quality. This could be achieved with the following coding tools.

LCU/CTU

Coding tree unit(CTU) is the basic processing unit of theHEVC video standard and conceptually corresponds in structure tomacroblockunits, which were used in several previous video standards. CTU is also referred to aslargest coding unit(LCU)[1][2].In the HEVC, one frame is divided into a series of non-overlapped Coding Tree Unit (CTU) [9][12].

Figure 5: Division of a CTB into CBs and transform blocks TB [2]

Parallel Processing

A picture is divided into tiles. Main purpose of these tiles is that, they can be decoded /encoded individually in a simultaneous way called parallel processing. Parallel computing is basically a technique in which multiple computation tasks are assigned to multiple processes and process the job simultaneously. The basic approach for parallel processing is to break the task into multiple smaller tasks and further assign each task to each of the thread which performs required operations in parallel. Parallelization can sometimes get complicated due to race conditions, data dependency, synchronization and communication among different threads [13].

Entropy Coding

HEVC uses acontext-adaptive binary arithmetic coding(CABAC) algorithm that is fundamentally similar to CABAC in H.264/MPEG-4 AVC.CABAC is the only entropy encoder method that is allowed in HEVC while there are two entropy encoder methods allowed by H.264/MPEG-4 AVC.CABAC and the entropy coding of transform coefficients in HEVC are designed for a higher throughput than H.264 while maintaining higher compression efficiency for larger transform block sizes relative to simple extensions [15][16]. These techniques include reducing context coded bins, grouping bypass bins, grouping bins with the same context, reducing context selection dependencies, reducing total bins, and reducing parsing dependencies. It also describes reductions to memory requirements that benefit both throughput and implementation costs.

Motion Estimation

Motion estimation[5] is an essential process in many video coding standards like MPEG-2, H.264/AVC and HEVC[1][2]. Motion estimation has been used at the encoder. Motion Estimation itselfconsumes more than 50% coding complexity or time to encode.To reduce the computation time, many fast motion estimationAlgorithms were proposed and implemented [5].HEVC allows for two MV modes which are Advanced Motion Vector Prediction (AMVP) and merge mode.AMVP uses data from the reference picture and can also use data from adjacent prediction blocks. The merge mode allows for the MVs to be inherited from neighboring prediction blocks.Merge mode in HEVC is similar to "skipped" and "direct" motion inference modes in H.264.

Motion estimation process in HEVC consumes more than 50% coding complexity or time to encode with equal perceptual quality [6][7]. Many block based motion estimation algorithms [8] [9] are proposed and also implemented to reduce the computation time.

Figure 6: Illustration of Motion Estimation process [5]

Motion Compensation

The interpolation of fractional luma sample positions HEVC uses separable application of one-dimensional half-sample interpolation, with an 8-tap filter or quarter-sample interpolation with a 7-tap filter [5]. While H.264/MPEG-4 AVC[1][2] uses a two-stage process that first derives values at half-sample positions, using separable one-dimensional 6-tap interpolation followed by integer rounding; then applieslinear interpolationbetween values at nearby half-sample positions to generate values at quarter-sample positions.HEVC has improved precision due to the longer interpolation filter and the elimination of the intermediate rounding error.As in H.264/MPEG-4

AVC[1][2], a scaling and offset operation may be applied to the prediction signal(s) in a manner

known as weighted prediction[1].

Algorithm of Sample Based Angular Intra Prediction

The SAP[4] is designed to better exploit the spatial redundancy in the lossless coding mode by generating intra prediction samples from adjacent neighbors.The design principle here is very similar to the sample-based DPCM in[21][4] H.264/MPEG-4 AVC [20][4]lossless coding, but SAP[4] is fully harmonized with the HEVC block-based angular intra prediction, and can be applied to all the angular intra prediction modes specified in HEVC [4].

As shown inthe fig.7 SAP is performed sample by sample. The adjacent neighboring samples,of the current samplein the current PU are used for prediction. That is, the reference samples used for prediction are not limited to those boundary reference samples from the left and upper neighboring PUs. The SAP has to be processed in a predefined order to ensure the availability of these adjacent neighbors for prediction.

Figure 7: Algorithm of SAP [4]

Proposed Project

The aim of this proposal is to analyze HEVC lossless coding. Hence if a scheme is designed that will reduce the computation time using HEVC lossless coding it will further increase its performance and reduce encoding time without much increase in complexity [21].

The simulation will be conducted using HM Software 16.0[18][26], with different video sequences [29], search range, block sizes and number of frames using GPU multicore computing. Bit rate differences, Compression ratio, Distribution of intra-coded ratio (ratio between intra-coded and coded blocks), and results based on it, will be calculated for different algorithms using various search patterns.

Projected Enhancement in the Project

Performance analysis of HEVC Lossless Mode using HM16.0 [18] software.

Performance analysis of SAP[4] using HM16.0[18] software.

Complete Study of HEVC Lossless Coding[4].

References

[1] G.J. Sullivan et al, “Overview of the high efficiency video coding (HEVC) standard”, IEEE Trans. CSVT, vol. 22, pp.1649-1668, Dec 2012.

[2] G.J. Sullivan et al, “Standardized Extensions of High Efficiency Video Coding (HEVC)”, IEEE Journal of selected topics in Signal Processing, vol.7, pp.1001-1016, Dec 2013.

[3] C. Fogg, “Suggested figures for the HEVC specification”, ITU-T/ISO/IEC Joint Collaborative Team on Video Coding (JCT-VC) document JCTVC- J0292r1, July 2012.

[4] M. Zhou et al, “HEVC lossless coding and improvements”, IEEE Trans. CSVT, vol.22, pp.1839-1843, Dec2013.

[5] N.Purnachand et al, "Fast Motion Estimation Algorithm for HEVC", IEEE Second International Conference onConsumer Electronics-Berlin (ICCE-Berlin), vol.11, pp.34-37, Sep 2012.

[6] P. Hanhart et al, “ Subjective quality evaluation of the upcoming HEVC video compression standard”, SPIE Optical Engineering+ Applications. International Society for Optics and Photonics, vol. 8499, pp.84990v-84990v, Aug 2012.

[7] M. Horowitz et al, “Informal subjective quality comparison of video compression performance of the HEVC and H.264/MPEG - 4 AVC standards for low delay applications”, SPIE Optical Engineering+ Applications. International Society for Optics and Photonics, vol.84990, pp.84990w-84990w, Aug 2012.

[8] A.Abdelazim,W.Masri and B.Noaman., "Motion estimation optimization tools for the emerging high efficiency video coding (HEVC)", IS&T/SPIE Electronic Imaging. International Society for Optics and Photonics, vol. 9029,pp. 902905-902905,Feb 17 2014.

[9] M. Jakubowski and G. Pastuszak, “Block-based motion estimation algorithms-a survey”, Journal of Opto-Electronics Review, vol. 21, pp.86-102, Mar 2013.

[10] B. Bross et al, “High Efficiency Video Coding (HEVC) Text Specification Draft 10”, Document JCTVC-L1003, ITU-T/ISO/IEC Joint Collaborative Team on Video Coding (JCT-VC), Jan. 2013, available on

​

[11] J. Ascenso et al, "Improving Frame Interpolation with Spatial Motion Smoothing for Pixel Domain Distributed Video Coding", 5th EURASIP Conference on Speech and Image Processing, Multimedia Communications and Services, pp.1-6, Smolenice, Slovak Republic, July 2005.

[12] X. Wang et al,“Paralleling Variable Block Size Motion Estimation of HEVC on Multicore CPU plus GPU platform”, IEEE International Conference on Image Processing (ICIP),vol.22, pp. 1836-1839, Sep.2013.

[13] Introductionto parallel computing

[14] L. Zhao et al, “Group-Based Fast mode decision algorithm for intra prediction in HEVC”, IEEE Eighth international Conference on Signal Image Technology and Internet based Systems. Article no.6115979, pp. 225-229, Nov 2011.

[15] V. Sze and M. Budagavi, "High Throughput CABAC Entropy Coding in HEVC", IEEE Transactions on Circuits and Systems for Video Technology, vol.22,no.12, pp.1778-1791, Dec 2012.

[16] T.Nguyen et al,"Transform Coding Techniques in HEVC", IEEE Journal of Selected Topics in Signal Processing, vol.7, pp.978–989, Dec 2013.

[17] HEVC tutorial by I.E.G. Richardson:

[18] HEVC Reference Software HM16.0.

[19]B. Bross et al,“High Efficiency Video Coding (HEVC)Text Specification Draft 8”, JCT-VC document, JCTVC-J1003, Stockholm, Sweden, Jul. 2012.

[20]Joint Video Team,“Advanced Video Coding for Generic Audiovisual Services”, ITU-T Rec. H.264 and ISO/IEC, 14496-10 (MPEG-4) AVC, pp.H.100-H.869, Feb 2014.

[21] Y.L. Lee et al, "Improved lossless intra coding for H.264/MPEG-4 AVC", IEEE Trans. Image Process., vol.15, no.9, pp.2610-2615, Sep 2006.

[22]K.R. Rao,D.N. Kim and J.J Hwang, “High Efficiency Video Coding (HEVC) Revised/Updated Chapter from the book Video Coding Standards”–Springer 2014.

[23] ITU-T website:

[24] JCT-VC documents are publicly available at and

[25]V.Sze, M.Budagavi, and G.J. Sullivan “High Efficiency Video Coding (HEVC)”Springer, 2014.

[26] Software reference manual for HM:

[27] M. Wien, “High efficiency video coding: Tools and specification”, Springer, 2015.

[28]I.E. Richardson, “Coding video: A practical guide to HEVC and beyond”, Wiley, 11 May 2015

[29] Video Sequences: