ISO/IECJTC1/SC29/WG1N3915

March 17, 2006

ISO/IEC JTC1/SC29/WG1
(ITU-T SG8)
Coding of Still Pictures
JBIG JPEG
Joint Bi-level Image Joint Photographic
Experts Group Experts Group

TITLE:Preliminary call for Advanced Image Coding (AIC) evaluation methodologies

SOURCE:WG1

PROJECT:proposed new work item AIC

STATUS:Draft

REQUESTED

ACTION:For discussion and feed-back

DISTRIBUTION:WG 1 members and public at large

Contact:

ISO/IEC JTC 1/SC 29/WG 1 Convener - Dr. Daniel T. Lee

Yahoo!, Rm 2802, Sunning Plaza, 10 Hysan Avenue, CausewayBay, Hong Kong

Tel: +1 408 992 7051/+852 2882 3898, Fax: +1 253 830 0372, E-mail:

INTERNATIONAL ORGANISATION FOR STANDARDISATION

ORGANISATION INTERNATIONALE DE NORMALISATION

ISO/IEC JTC1/SC29/WG1

CODING OF STILL PICTURES

ISO/IEC JTC 1/SC 29/WG 1 N3915

Date: 2006-03-17

Title:Preliminary call for Advanced Image Coding (AIC) evaluation methodologies

Source: Ad Hoc Group on Requirements (Touradj Ebrahimi, chair)

1. Background and rationale

WG1 has initiated an activity to study potential technologies for future generation image compression systems. This activity which is referred to as Advanced Image Coding (AIC), could become a new work item for standardization of a new image compression system if any potential technologies can be identified which significantly would improve the performance of current image compression standards.

A first step of such an effort is to define the evaluation methodologies, including test data, and anchors to be compared to. As such methodologies to evaluate image compression algorithms can be useful for technical and scientific community beyond the members of WG1, it is felt that the evaluation methodologies and the associated test data and anchors should become a standard document (such as an ISO Technical Report) hence allowing all interested parties to use the evaluation methodologies to evaluate their image compression systems.

This document is an attempt to create a call for contributions of evaluation methodologies and associated test materials and anchors.

2. Evaluation issues

Evaluation is a fundamental notion in any system design and performance assessment. In the context of Image Compression Systems, the following issues have been identified as key to a successful assessment.

2.1 Test material

The test material to be used for evaluation of an image compression system would comprise the following types and sizes which seem to reflect the type and size of images in potential applications for which an Advanced Image Compression system will be used.

  • 2D (spatial)
  • Size (from 4K X 4K)
  • 3D (volumetric)
  • Size (512 X 512 X 100 to 4K X 4K X 2K)
  • Video (2D + t)
  • Size (Typically 8K X 4K X 10 sec @ 60 fr/sec)
  • 4D (3D + t)
  • Size (Typically 128 X 128 X 100 X 10 sec @ 5 fr/sec)
  • Multiview (2D (+t) X N)
  • Size (Typically 2K X 2K X (10 sec @ 30 fr/sec) X 64 Views)

Likewise, the type of content for test material should reflect the potential applications of an AIC. The following provides some content examples:

  • Natural scenes
  • Compound (multi-layer)
  • Photo-realistic synthetic
  • Graphics and animations
  • Scanned content
  • Content with application specific noise patterns
  • Sensor array images

It is highly recommended and preferable that the test material gathered for use in the evaluation of AIC is freely redistributable and freely usable for research purposes in order to allow the research community to progress in an efficient, comparable, and verifiable manner in its explorations for future image compression systems.

2.2 Evaluation criteria

The traditional approach to evaluate performance of lossless image compression systems has been to verify the size of the compressed test material when compared to an anchor. Other issues such as the complexity have also been considered in an ad hoc manner.

In lossy image compression systems, the method of choice is often the comparison of rate-distortion performance where rate is the size of the compressed test material and distortion refers to the amount of loss brought by compression when compared to the originals. The most popular distortion metrics are Mean Squared Error (MSE) and PSNR. In some cases, such as the competitive phase of JPEG 2000 standardization, subjective test are performed to obtain the Mean Opinion Score (MOS) of the quality of the compressed test material, by a team of human observers.

In addition to the above traditional approaches, others can be envisaged in order to better grasp the performance of a given image compression system.

The following summarizes such evaluation criteria:

  • Quality metrics
  • Subjective
  • Objective
  • Full reference
  • Reduced reference
  • No reference
  • Distortion metrics
  • Subjective
  • Objective
  • X-ness (blurriness, blockiness, noisiness, …)
  • Complexity metrics
  • Computational complexity
  • Memory requirements
  • Parallelizability
  • Functionality assessment
  • Scalability (spatial, temporal, quality, complexity, component, granularity)
  • Latency
  • Error resiliency
  • Compressed domain processing capabilities
  • …
  • Architectural properties
  • Extensibility
  • Modularity
  • Compatibility with JPEG 2000 and JPEG
  • …

2.3Evaluation tools

Two types of evaluation tools are envisaged to become part of the AIC evaluation methodologies Technical Report:

  • Software to estimate the evaluation metrics mentioned in section 2.2
  • Software to simulate operational conditions/environments
  • Network/channel simulators

2.4Evaluation protocols

Evaluation protocols refer to processes, best practices, and mathematical tools that extract from various data gathered from test material, criteria and tools in sections 2.1, 2.2 and 2.3, quantifiable output which would allow to draw conclusions in terms of performance of a given image compression system.

Two main directions have been identified:

  • Statistical analysis methods for assessment of data from evaluation criteria
  • Application dependent/independent evaluation protocols

3. Requested action

The individuals or organizations with expertise in either image compression, and/or evaluation methodologies for the assessment of their performance are encouraged to provide inputs to WG1 committee and to attend its 39th meeting in Perugia, Italy, from July 10th, to July 14th, 2006. Interested parties and individuals who wish to attend this meeting are requested to contact WG1 Convener, their National Body or an organization in liaison with WG1. Further information about WG1 can be obtained from or by email to

1