1
WTSA16/11(Rev.1)-E
/ World Telecommunication Standardization Assembly (WTSA-16)Hammamet, 25 October - 3 November 2016 /
PLENARY MEETING / Revision 1 to
Document11-E
July 21 October 2016
Original: English
ITU-T Study Group 12
Performance, QoS and QoE
REPORT OF ITU-T SG12 TO THE WORLD TELECOMMUNICATION STANDARDIZATION ASSEMBLY (WTSA-16), PART I: GENERAL
Abstract: / This contribution contains the report of ITU-T Study Group 12 to WTSA-16 concerning its activities during the 2013-2016 study period. /
Note by the TSB:
The report of Study Group 12 to the WTSA-16 is presented in the following documents:
Part I:Document 11 – General
Part II:Document 12 – Questions proposed for study during the study period 2017-2020
CONTENTS
Page1Introduction
2Organization of work
3Results of the work accomplished during the 2013-2016 study period
4Observations concerning future work
5Updates to the WTSA Resolution 2 for the 2017-2020 study period
ANNEX 1 List of Recommendations, Supplements and other materials produced or deleted during the study period
ANNEX 2 Proposed updates to the Study Group 12 mandate and Lead Study Group roles
1Introduction
1.1Responsibilities of Study Group 12
Study Group 12 was entrusted by the World Telecommunications Standardization Assembly (Dubai, 2012) with the study of 17 Questions in the area of performance, quality of service (QoS) and quality of experience (QoE), covering the full spectrum of terminals, networks and services, from speech over fixed circuit-based networks to multimedia applications over networks that are mobile and packet based.
1.2Management team and meetings held by Study Group 12
Study Group 12 met six times in Plenary and two three times in Working Partiesin the course of the study period[1] (see Table 1) under the chairmanship of Mr Kwame BAAH-ACHEAMFUOR (Ghana), assisted by Vice-Chairmen Mr Paul BARRETT (United Kingdom), Mr Vincent BARRIAC (France), Mr Gamal Amin ELSAYED (Sudan), Mr Hyung-Soo (Hans) KIM (Korea (Rep. of)), Mr Al MORTON (United States), Mr Feng QI (China), Mr Akira TAKAHASHI (Japan), and Mr Hassan TALIB(Morocco). Mr José Guadalupe Rojas RAMÍREZ (Mexico) did not participate in any SG12 meetings.
In addition many Rapporteurs' meetings took place during the study period in different locations, see Table 1-bis.
TABLE 1
Meetings of Study Group 12 and its Working Parties
SG/WP 12 / Geneva, 19-28 March 2013 / COM 12 –R 1 to R 4
SG12RG-AFR / Ouagadougou, 19 July 2013 / COM 12 RGAFR – R 1
SG/WP 12 / Geneva, 3-12 December 2013 / COM 12 – R 5 to R 8
WP 2/12 / Solothurn, 25 March 2014 / COM 12 – R 9
SG12RG-AFR / Kampala, 26 June 2014 / COM 12 RGAFR – R 2
SG/WP 12 / Geneva, 2-11 September 2014 / COM 12 – R 10 to R 13
WP 1/12 / Herzogenrath, 12 December 2014 / COM 12 – R 14
SG12RG-AFR / Dakar, 23 March 2015 / COM 12 RGAFR – R 3
SG/WP 12 / Geneva, 5-14 May 2015 / COM 12 – R 15 to R 18
SG/WP 12 / Geneva, 12-21 January 2016 / COM 12 – R 19 to R 22
SG12RG-AFR / Livingstone, 18 March 2016 / COM 12 RGAFR – R 4
SG/WP 12 / Geneva, 7-16 June 2016 / COM 12 – R 23 to R 26
WP 2/12, 3/12 / Munich, 20 October 2016 / COM 12 – R 27 to R 28
TABLE 1-bis
Rapporteur meetings organized under Study Group 12 during the study period
2013-01-15
to
2013-01-16 / France [Lannion]/Orange / 5/12 / Rapporteur group meeting for Q5/12
2013-07-01
to
2013-07-02 / China [Shenzhen]/Huawei Technologies / 9/12 / Rapporteur group meeting for Q9/12
2013-07-15
to
2013-07-17 / Switzerland [Geneva] / 4/12 / Rapporteur group meeting for Q4/12
2013-09-04
to
2013-09-06 / Austria [Vienna]/FTW / 13/12
14/12
17/12 / Rapporteur group meeting for Q13, 14 and 17/12
2013-10-07
to
2013-10-08 / Switzerland [Geneva] / 5/12 / Rapporteur group meeting for Q5/12
2014-02-26
to
2014-02-28 / Sweden [Stockholm]/Ericsson / 13/12
14/12
17/12 / Rapporteur group meeting for Q13, 14 and 17/12
2014-03-24
to
2014-03-25 / Switzerland [Solothurn]/SwissQual/Rohde & Schwarz / 9/12 / Rapporteur group meeting for Q9/12
2014-05-20
to
2014-05-22 / United States [Mountain View, CA]/Audience / 9/12 / Rapporteur group meeting for Q9/12
2014-06-03
to
2014-06-05 / Germany [Chemnitz]/Chemnitz University of Technology / 14/12
17/12 / Rapporteur group meeting for Q14 and 17/12
2014-06-11
to
2014-06-13 / United States [Detroit, Michigan]/QNX Software Systems Inc. / 4/12 / Rapporteur group meeting for Q4/12
2014-06-17
to
2014-06-18 / Sweden [Stockholm]/Ericsson / 10/12 / Rapporteur group meeting for Q10/12
2014-06-25
to
2014-06-26 / Sweden [Lund]/Sony / 5/12 / Rapporteur group meeting for Q5/12
2014-12-10
to
2014-12-12 / Germany [Herzogenrath]/HEAD acoustics / 4/12 / Rapporteur group meeting for Q4/12
2015-02-02
to
2015-02-03 / Switzerland [Geneva] / 5/12 / Rapporteur group meeting for Q5/12
2015-03-10
to
2015-03-12 / United States [Brighton, MI] / 4/12 / Rapporteur group meeting for Q4/12
2015-10-07
to
2015-10-08 / United States [Detroit, Michigan]/General Motors / 4/12 / Rapporteur group meeting for Q4/12
2015-11-11
to
2015-11-13 / Germany [Berlin]/T-Labs / 13/12
14/12 / Rapporteur group meeting for Q13, 14/12
2016-04-27
to
2016-04-29 / Germany [Berlin]/T-Labs / 13/12
14/12
17/12 / Rapporteur group meeting for Q13, 14 and 17/12
2016-09-14
to
2016-09-15 / United States [Detroit, Michigan]/HEAD acoustics / 4/12 / Rapporteur group meeting for Q4/12
2016-10-17
to
2016-10-19 / Germany [Munich]/Huawei Technologies / 13/12
14/12
17/12 / Rapporteur group meeting for Q13, 14 and 17/12
2Organization of work
2.1Organization of studies and allocation of work
2.1.1At its first meeting of the study period, Study Group 12 decided to establish three Working Parties. At the same meeting in March 2013, Study Group 12 received the Deliverables of Focus Group on Driver Distraction (FG Distraction) which had successfully concluded its work.
2.1.2Table 2 shows the number and title of each Working Party, together with the number of Questions assigned to it and the name of its Chairman.
2.1.3Table 3 lists other groups created by Study Group 12 during the study period. It also lists groups under the responsibility of Study Group 12 or with active Study Group 12 participation.
Quality of Service Development Group (QSDG)
Launched in the 1980s, under responsibility of Study Group 12 since WTSA08, the primary aim of the Quality of Service Development Group (QSDG) is to improve the quality of the international service, to the benefit both of the subscribers and administrations. The QSDG held four meetings during the study period. It was instrumental in initiating new work items in Study Group 12 and bringing the work of the SG12 to a wider audience through workshops, symposia and forums on QoS and QoE.
Intersector Rapporteur Group on Audiovisual Quality Assessment (IRG-AVQA)
The Intersector Rapporteur Group on Audiovisual Quality Assessment (IRG-AVQA) was established in accordance with Annex C of WTSA-12 Resolution 18 and the corresponding provisions in Resolution ITU-R 6 in order to study topics related to video and audiovisual quality assessment among ITU-R Study Group 6, ITU-T Study Group 9 and ITU-T Study Group 12.
IRG-AVQA held six meetings during the study period–all of them in colocation with either ITU-R WP6C, ITU-T SG9, ITU-T SG12, or the Video Quality Experts Group (VQEG).
SG12 has requested the other participating groups for approval for an extension of the IRG-AVQA mandate in the next study period.
2.1.4In line with WTSA-12 Resolution 54, the Regional Group on QoS for the Africa Region (SG12 RG-AFR), launched by Study Group 12 in May 2008,continued to operate in study period 2013-2016. It held meetings during the SG12 plenary meetings in Geneva, as well as four meetings in Africa. Thanks to an increased footprint in Africa, the number of contributions to SG12 from African countries increased in the course of the study period: from one contribution in March 2013 to six contributions in June 2016.
TABLE 2
Organization of Study Group 12
and Vice-Chairmen
PLEN / Q1/12; Q2/12; QSDG; / — / —
WP1/12 / Q3/12; Q4/12; Q5/12; Q6/12; Q7/12; Q10/12; / Terminals and multimedia subjective assessment / Mr Nielsen Lars Birger (Chairman)
Mrs BerndtssonGunilla (Vice-chairman)
WP2/12 / Q8/12; Q9/12; Q14/12; Q15/12; Q16/12; / Objective models and tools for multimedia quality / Mr Barrett Paul (Chairman)
Mr Barriac Vincent (Vice-chairman)
WP3/12 / Q11/12; Q12/12; Q13/12; Q17/12; / Multimedia QoS and QoE / Mr Coverdale Paul (Chairman)
Mr Takahashi Akira (Vice-chairman)
TABLE 3
Other Groups
Quality of Service Development Group (QSDG) / Mrs Yvonne UMUTONI (Rwanda) / Ms StavroulaBOUZOUKI (Greece), Mr WenyanJIN (China), Mrs Louisa SOSU (Ghana), Mr Jacob MUNODAWAFA (Mozambique), Mr Mohammad QasimNASIMEE (Afghanistan), Mr Arvind CHAWLA (India), Mr Tiago SOUSA PRADO (Brazil), Mr Mehmet ÖZDEM (Turkey)
Regional Group on QoS for the Africa Region (SG12 RG-AFR) / Gamal Amin ELSAYED (Sudan) / Mr Robert ECHEDA (Uganda), Mr Seyni Malan FATY (Senegal), Mr Hassan TALIB (Morocco)
Intersector Rapporteur Group Audiovisual Quality Assessment (IRG-AVQA) / Chulhee LEE (Korea, Rep of),
QUAN Huynh-Thu, (Australia),
Jens BERGER (Germany) / N/A
2.2Questions and Rapporteurs
2.2.1WTSA-12 assigned to Study Group 12 the 17 Questions listed in Table 4.
2.2.2The Questions listed in Table 5 have been adopted during this period.
2.2.3The Questions listed in Table 6 have been deleted during this period.
TABLE 4
Study Group 12 – Questions assigned by WTSA-12 and Rapporteurs
1/12 / SG 12 work programme and QoS/QoE coordination in the ITU-T / PLEN / Mr Baah-Acheamfuor Kwame (Rapporteur)
Mr Echeda Robert (Rapporteur)
Mr Kim Hyung-Soo (Hans) (Rapporteur)
Mr Pomy Joachim (Rapporteur)
2/12 / Definitions, guides and frameworks related to QoS/QoE / PLEN / Mr Pomy Joachim (Rapporteur)
3/12 / Speech transmission characteristics of communication terminals for fixed circuit-switched, mobile and packet-switched (IP) networks / 1/12 / Mr Yi Gaoxiong (Rapporteur)
4/12 / Hands-free communication and user interfaces in vehicles / 1/12 / Mr Gierlich Hans Wilhelm (Rapporteur)
5/12 / Telephonometric methodologies for handset and headset terminals / 1/12 / Mr Nielsen Lars Birger (Rapporteur)
6/12 / Analysis methods using complex measurement signals including their application for speech enhancement techniques and hands-free telephony / 1/12 / Mr Gierlich Hans Wilhelm (Rapporteur)
7/12 / Methods, tools and test plans for the subjective assessment of speech, audio and audiovisual quality interactions / 1/12 / Mr Sharpley Alan (Rapporteur)
Mr Usai Paolo (Rapporteur)
8/12 / E-Model extension in wideband transmission and future telecommunication and application scenarios / 2/12 / Mr Möller Sebastian (Rapporteur)
9/12 / Perceptual-based objective methods for voice, audio and visual quality measurements in telecommunication services / 2/12 / Mr Berger Jens (Rapporteur)
10/12 / Conferencing and telemeeting assessment / 1/12 / Mrs BerndtssonGunilla (Rapporteur)
Mr SkowronekJanto (Rapporteur)
11/12 / Performance interworking and traffic management for Next Generation Networks / 3/12 / Mr Pomy Joachim (Rapporteur)
12/12 / Operational aspects of telecommunication network service quality / 3/12 / Mr Talib Hassan (Rapporteur)
13/12 / QoE, QoS and performance requirements and assessment methods for multimedia / 3/12 / Ms García Marie-Neige (Rapporteur)
Mr Takahashi Akira (Rapporteur)
14/12 / Development of parametric models and tools for multimedia quality assessment / 2/12 / Mr GustafssonJörgen (Rapporteur)
Mr Raake Alexander (Rapporteur)
15/12 / Objective assessment of speech and sound transmission performance quality in networks / 2/12 / Mr Barriac Vincent (Rapporteur)
Mr Pomy Joachim (Rapporteur)
16/12 / Framework for diagnostic functions and their interaction with external objective models predicting media quality / 2/12 / Mr MalfaitLudovic (Rapporteur)
17/12 / Performance of packet-based networks and other networking technologies / 3/12 / Mr Morton Al (Rapporteur)
TABLE 5
Study Group 12 – New Questions adopted and Rapporteurs
None
TABLE 6
Study Group 12 – Questions deleted
None
3Results of the work accomplished during the 2013-2016 study period
3.1General
During the study period[2], Study Group 12 examined 395 contributions and generated a large number of TDs and liaison statements. It also:
–drew up 23 27 new Recommendations;
–revised 25 26 existing Recommendations;
–issued 17 Amendments and 2 Corrigenda;
–developed 2 Supplements and 8 Implementer's Guides.
3.2Highlights of achievements
The main results achieved on the various Questions assigned to Study Group 12 are briefly summarized below. Formal replies to the Questions are given in a synoptic table in Annex 1 of this report.
a)Achievements of Questions reporting to the SG12 plenary
Q1/12 - SG12 work programme and QoS/QoE coordination in the ITU-T (Rapporteurs Mr Kwame Baah-Acheamfuor, Mr Robert Echeda, Mr Hyung-Soo (Hans) Kim, Mr Joachim Pomy)
As in previous study periods, Q1/12 fulfilled the role to co-ordinate QoS/QoE-related activities in the ITU-T and collaboration on QoS/QoE with other standards bodies. Q1/12 provided a temporary home to work item proposals, when they were not directly related to existing Questions. As the lead study group on QoS/QoE, SG12 used Q1to foster consistency on QoS/QoE matters within the ITU, and with related external organizations (e.g. 3GPP, IETF). In addition, any business concerning the Regional Group on QoS for the Africa Region (SG12 RG-AFR) was discussed under the stewardship of Q1.
Q2/12 - Definitions, guides and frameworks related to QoS/QoE (Rapporteur Mr Joachim Pomy)
Q2/12 is in charge of the development and maintenance of ITU-T Recommendations providing the definitions needed to support new or revised Recommendations developed by the other Questions under study in SG12. The responsibility of Q2 includes the study of new or revised definitions to be included in Recommendation ITU-T P.10/G.100 "Vocabulary for performance and quality of service".
A major milestone was achieved towards the end of the study period when SG12, under the leadership of Q2, withdrew the old definition of QoE from P.10/G.100, and inserted three new terms and definitions, and one additional bibliographical reference as Draft Amendment5 "New definitions for inclusion in Recommendation ITU-T P.10/G.100":
Quality of Experience (QoE) is the degree of delight or annoyance of the user of an application or service.
Other key achievements included the revisions of landmark Recommendations ITU-T P.800.1 and P.800.2 on mean opinion score (MOS) terminology, interpretation and reporting.
Responding to gaps identified by the ITU-T Focus Group on IMT-2020 and related contributions presented in SG12, Q2 became the home of a work item describing a QoS Framework for 5G/IMT2020.
During the study period, Q2/12 activities led to the consent of 4 revised Recommendations, 2Amendments, and to the publication of 1Implementer's Guide.
b) Achievements of Working Party 1 (WP1/12) - Terminals and multimedia subjective assessment
WP1/12 worked on transmission characteristics of terminals for fixed circuit-switched, mobile and packet-switched (IP) networks and the related telephonometric methodologies, as well as analysis methods using complex measurement signals. Hands-free communication in vehicles was an important area of study. Also addressed were methods, tools and test plans for the subjective assessment of speech, audio and audiovisual quality interactions.
During the study period, WP1/12 activities led to the consent of 10 new Recommendations, 7revised Recommendations, 7 Amendments, and to the publication of 2Implementer's Guides.
Q3/12 - Speech transmission characteristics of speech terminals for fixed circuit-switched, mobile and packet-switched (IP) network (Rapporteur Mr Yi Gaoxiong)
The work of Q3/12 was a continuation of the three previous study periods. It was primarily concerned with the electroacoustic performance of terminals that are used in the network, as well as the associated objective test methods.
During the study period, Q3/12 developed Recommendations aiming at providing electroacoustic test procedures for headsets and headphones as well as test procedures for characterisation of the electrical interfaces in terminals supporting the use of headsets and headphones. Specifically, work was progressed / completed on the items listed below:
–P.313 "Transmission characteristics for cordless and mobile digital terminals" (revised)
–P.381 "Technical requirements and test methods for the universal wired headset or headphone interface of digital mobile terminals" (revised)
–P.382 "Technical requirements and test methods for multi-microphone wired headset or headphone interfaces of digital wireless terminals" (new)
–"Technical requirements and test methods for the digital wired or wireless headset interface of mobile terminals" (new work item)
Q4/12 - Hands-free communication in vehicles (Rapporteur Mr Hans-Wilhelm Gierlich)
The work of Q4/12 was a continuation of the three previous study periods. It was primarily concerned with objective test methods for evaluation of hands-free terminals in vehicles.
During thestudy period, Q4/12 developed Recommendations aiming at providing test procedures for evaluation of hands-free systems and sub-systems in vehicles as well as test procedures for eCall (emergency call) systems. Specifically, work was progressed / completed on the items listed below:
–P.1130 "Sub-system Requirements for Automotive Speech Services" (new)
–P.1140 "Speech communication requirements for emergency calls originating from vehicles" (new)
Q5/12 - Telephonometric methodologies for handset and headset terminals (Rapporteur Mr Lars Birger Nielsen)
The work of Q5/12 was a continuation of the three previous study periods. It was primarily concerned with objective test procedures and test setups for evaluation of handset and headset terminals.
During thestudy period, Q5/12 developed Recommendations aiming at updating test procedures for using HATS (head and torso simulator), as well as test setups for creating artificial noise-fields. Furthermore, the development of a new loudness model was initiated. Specifically, work was progressed / completed on the items listed below:
–P.58 "Head and torso simulator for telephonometry" (revised)
–P.581 "Use of head and torso simulator (HATS) for hands-free and handset terminal testing" (revised)
–P.Loudness "Calculation of receiving loudness for terminals" (new work item)
–P.TBN "Artificial noise-fields in laboratory conditions" (new work item)
Q6/12 - Analysis methods using complex measurement signals including their application for speech enhancement techniques and hands-free telephony (Rapporteur Mr Hans-Wilhelm Gierlich)
The work of Q6/12 was a continuation of the three previous study periods. It was primarily concerned with objective test methods as well as test signals for evaluation of hands-free terminals.
During thestudy period, Q6/12 developed new test signals utilizing real human speech. Specifically, work was progressed / completed on the items listed below:
–P.501/Amd.2 "Test signals for use in telephonometry – Amendment 2: New Annex C" (new)
–P.501/Amd.3 "Test signals for use in telephonometry – Amendment 3: New Annex D" (new)
–P.340/Amd.1 "Transmission characteristics and speech quality parameters of hands-free terminals – Amendment 1: New Annex B" (new)
Q7/12 - Methods, tools and test plans for the subjective assessment of speech, audio and audiovisual quality interactions (Rapporteur Mr Paolo Usai)
The work of Q7/12 was a continuation of the three previous study periods. It was primarily concerned with subjective test methods that can be used for assessment of speech, audio and audiovisual quality of terminals.
During thestudy period, Q7/12 developedRecommendations aiming at providing subjective test procedures for evaluation of speech quality degradation, speech intelligibility, as well as perceived quality of web browsing. Specifically, work was progressed / completed on the items listed below:
–P.806 "A Subjective Quality Test Methodology using Multiple Rating Scales" (new)
–P.807 "Subjective Test Methodology for Assessing Speech Intelligibility" (new)
–P.1501 "Subjective testing Methodology for web browsing" (new)
Q10/12 - Conferencing and telemeetingassessment” (Rapporteurs MrsGunillaBerndtsson and Mr JantoSkowronek)
The work of Q10/12 was a continuation of the three previous study periods. It was primarily concerned with objective test methods as well as subjective test methods for assessment of conferencing and telemeetings.
During thestudy period, Q10/12 developedRecommendations aiming at providing test methods for determining the effect of delays on telemeetings quality, intelligibility of multiple concurrent talkers, as well as the effectiveness of multiparty telemeetings. Specifically, work was progressed / completed on the items listed below: