Summary of Plenary Meeting of WG2 Task Force “Crowdsourcing”
The WG2 TF “Crowdsourcing” had a 1h meeting in the Qualinet plenary on Monday, March 11, 2013, from 13:45-14:45.
The goal of the meeting was a) to present the current status, but also knowledge gathered in the TF, b) to extend the scope of the TF, and c) to attract other Qualinet members to actively participate in the TF activities. Therefore, five presentations were given in the plenary TF meeting. First, Tobias Hossfeld (University of Würzburg) presented a short overview (10min) on the achieved results, the current activities, the next steps and the extension of the scope of the TF. Dennis Guse (TU Berlin) introduced the current crowdsourcing activities (10min) at TU Berlin which aim at developing a crowdsourcing platform being open for Qualinet members and at investigating the impact of motivation and platform itself on data quality and reliability. Christian Hoene (Symonics) presented (15min) the MUSHRA test for quality assessment of the IETF RFC 6716 Opus speech and audio codec. The intention was to find a Qualinet partner for conducting subjective tests with crowdsourcing and to compare the results with lab user ratings. Then, Matthias Hirth (University of Würzburg) introduced (15min) best practices for implementing variable payment schemes depending on the test subjects’ performance and reliability as a concrete incentive mechanism. Further, a joint Qualinet worker information database was suggested to gather relevant information for crowdtesting. In general the term “Crowdtesting” refers to user experiments which are conducted via crowdsourcing, e.g. to test a software, e.g. to evaluate the user perceived quality of Internet applications. In the context of this task force, crowdtesting is typically meant to execute subjective user studies for QoE assessment of Internet applications like video streaming or web browsing. Finally, Tobias Hossfeld (University of Würzburg) introduced (10min) the idea of a joint Qualinet crowdsourcing experiment in order 1) to attract other Qualinet members to actively participate in the TF activities, 2) to approach the TF objective on comparing crowdsourcing and laboratory QoE assessment, and 3) to develop statistical mechanisms relevant for crowdtesting. Due to the strong interest (more than 15 interested members) in the Qualinet plenary on the joint crowdsourcing experiment (JOC), it was decided to have a break out session on JOC on Wednesday, March 13, 2013 from 10:00-10:30.
The main outcome of the Crowdsourcing TF plenary meeting was the following. First, the TF will focus on the analysis and identification of the impact of crowdsourcing environment (e.g. due to incentives like payments, e.g. due to reliability checks). Second, due to strong activity and joint work in TF, several goals of the TF are already successfully approached. The progress allows to extend the scope of the TF into novel scientific aspects of crowdtesting. Third, the vehicular to achieve the scientific goals and to further increase the collaboration is the JOC.
The objectives of the Crowdsourcing TF were extended (with respect to the last objectives from the Zagreb 2012 Qualinet meeting) and concretized to answer particular research questions. The updated objectives in the TF description are given below and underlined and marked in italics. The goal of this WG2 task force is
· to identify the scientific challenges and problems for QoE assessment via crowdsourcing but also the strengths and benefits,
· to derive a methodology and setup for crowdsourcing in QoE assessment,
· to challenge crowdsourcing QoE assessment approach with usual "lab" methodologies,
· to identify and analyze the impact of the crowdsourcing environment on QoE assessment,
· to analyze the impact of incentives on data quality of QoE assessments,
· to develop mechanisms and statistical approaches for identifying reliable ratings from remote crowdsourcing users and for model validation,
· to define requirements onto crowdsourcing platforms for improved QoE assessment.
As a result, a common roadmap for this task force is envisioned and joint activities among the members of Qualinet are to be stimulated.
All documents and presentations from this meeting are uploaded to the Qualinet document repository at www.qualinet.eu and the crowdsourcing TF wiki page at https://www3.informatik.uni-wuerzburg.de/qoewiki/qualinet:crowd.