Crowdsourcing

2017

Incollections

Egger-Lampl, S., Redi, J., Hoßfeld, T., Hirth, M., Möller, S., Naderi, B., Keimel, C., Saupe, D.: "Crowdsourcing Quality of Experience Experiments", Archambault, Daniel; Purchase, Helen; Hossfeld, Tobias (Ed.): Evaluation in the Crowd. Crowdsourcing and Human-Centered Experiments, 10264 , pp. 154-190, Springer International Publishing, 2017, ISBN: 978-3-319-66435-4.
[Abstract] [BibTeX] [DOI]

Crowdsourcing enables new possibilities for QoE evaluation by moving the evaluation task from the traditional laboratory environment into the Internet, allowing researchers to easily access a global pool of workers for the evaluation task. This makes it not only possible to include a more diverse population and real-life environments into the evaluation, but also reduces the turn-around time and increases the number of subjects participating in an evaluation campaign significantly, thereby circumventing bottle-necks in traditional laboratory setups. In order to utilise these advantages, the differences between laboratory-based and crowd-based QoE evaluation are discussed in this chapter.
Download citation as [.bib File]
@incollection{EggerLampl-SpringerCrowdsourcingBook2017,
title = {Crowdsourcing Quality of Experience Experiments},
author = {Sebastian Egger-Lampl and Judith Redi and Tobias Hoßfeld and Matthias Hirth and Sebastian Möller and Babak Naderi and Christian Keimel and Dietmar Saupe},
editor = {Daniel Archambault and Helen Purchase and Tobias Hossfeld},
doi = {10.1007/978-3-319-66435-4_7},
isbn = {978-3-319-66435-4},
year = {2017},
date = {2017-01-01},
booktitle = {Evaluation in the Crowd. Crowdsourcing and Human-Centered Experiments},
volume = {10264},
pages = {154-190},
publisher = {Springer International Publishing},
series = {Lecture Notes in Computer Science},
abstract = {Crowdsourcing enables new possibilities for QoE evaluation by moving the evaluation task from the traditional laboratory environment into the Internet, allowing researchers to easily access a global pool of workers for the evaluation task. This makes it not only possible to include a more diverse population and real-life environments into the evaluation, but also reduces the turn-around time and increases the number of subjects participating in an evaluation campaign significantly, thereby circumventing bottle-necks in traditional laboratory setups. In order to utilise these advantages, the differences between laboratory-based and crowd-based QoE evaluation are discussed in this chapter.},
howpublished = {Full text available from publisher},
keywords = {},
pubstate = {},
tppubtype = {incollection}
}

2015

Journal Articles

Volk, T., Keimel, C., Moosmeier, M., Diepold, K.: "Crowdsourcing vs. laboratory experiments – QoE evaluation of binaural playback in a teleconference scenario", Computer Networks, 90 , pp. 99-109, 2015, ISSN: 1389-1286.
[Abstract] [PDF] [BibTeX] [DOI]

Abstract Experiments for the subjective evaluation of multimedia presentations and content are traditionally conducted in a laboratory environment. In this respect common procedures for the evaluation of teleconference systems are no different. The strictly controlled laboratory environment, however, often gives a rather poor representation of the actual use case. Therefore in this study we crowdsourced the evaluation of a teleconference system to perform the evaluation in a real-life environment. Moreover, we used the unique possibilities of crowdsourcing to employ two different demographics by hiring workers from Germany on the one hand and the US and Great Britain on the other hand. The goal of this experiment was to assess the perceived Quality of Experience (QoE) during a listening test and compare the results to results from a similar listening test conducted in the controlled laboratory environment. In doing so, we observed not only intriguing differences in the collected QoE ratings between the results of laboratory and crowdsourcing experiments, but also between the different worker demographics in terms of reliability, availability and efficiency.
Download citation as [.bib File]
@article{Volk-ComNET2015,
title = {Crowdsourcing vs. laboratory experiments – QoE evaluation of binaural playback in a teleconference scenario},
author = {Thomas Volk and Christian Keimel and Michael Moosmeier and Klaus Diepold},
doi = {10.1016/j.comnet.2015.05.021},
issn = {1389-1286},
year = {2015},
date = {2015-10-01},
journal = {Computer Networks},
volume = {90},
pages = {99-109},
abstract = {Abstract Experiments for the subjective evaluation of multimedia presentations and content are traditionally conducted in a laboratory environment. In this respect common procedures for the evaluation of teleconference systems are no different. The strictly controlled laboratory environment, however, often gives a rather poor representation of the actual use case. Therefore in this study we crowdsourced the evaluation of a teleconference system to perform the evaluation in a real-life environment. Moreover, we used the unique possibilities of crowdsourcing to employ two different demographics by hiring workers from Germany on the one hand and the US and Great Britain on the other hand. The goal of this experiment was to assess the perceived Quality of Experience (QoE) during a listening test and compare the results to results from a similar listening test conducted in the controlled laboratory environment. In doing so, we observed not only intriguing differences in the collected QoE ratings between the results of laboratory and crowdsourcing experiments, but also between the different worker demographics in terms of reliability, availability and efficiency.},
note = {Crowdsourcing},
keywords = {},
pubstate = {},
tppubtype = {article}
}

2014

Journal Articles

Hoßfeld, T., Keimel, C., Timmerer, C.: "Crowdsourcing Quality-of-Experience Assessments", Computer, 47 (9), pp. 98-102, 2014, ISSN: 0018-9162.
[Abstract] [BibTeX] [DOI]

Crowdsourced quality-of-experience (QoE) assessments are more cost-effective and flexible than traditional in-lab evaluations but require careful test design, innovative incentive mechanisms, and technical expertise to address various implementation challenges.
Download citation as [.bib File]
@article{Hossfeld-Computer2014,
title = {Crowdsourcing Quality-of-Experience Assessments},
author = {Tobias Hoßfeld and Christian Keimel and Christian Timmerer},
doi = {10.1109/MC.2014.245},
issn = {0018-9162},
year = {2014},
date = {2014-09-01},
journal = {Computer},
volume = {47},
number = {9},
pages = {98-102},
abstract = {Crowdsourced quality-of-experience (QoE) assessments are more cost-effective and flexible than traditional in-lab evaluations but require careful test design, innovative incentive mechanisms, and technical expertise to address various implementation challenges.},
howpublished = {Full text available from publisher},
keywords = {},
pubstate = {},
tppubtype = {article}
}

Hoßfeld, T., Keimel, C., Hirth, M., Gardlo, B., Habigt, J., Diepold, K., Tran-Gia, P.: "Best Practices for QoE Crowdtesting: QoE Assessment With Crowdsourcing", Multimedia, IEEE Transactions on, 16 (2), pp. 541-558, 2014, ISSN: 1520-9210.
[Abstract] [PDF] [BibTeX] [DOI]

Quality of Experience (QoE) in multimedia applications is closely linked to the end users' perception and therefore its assessment requires subjective user studies in order to evaluate the degree of delight or annoyance as experienced by the users. QoE crowdtesting refers to QoE assessment using crowdsourcing, where anonymous test subjects conduct subjective tests remotely in their preferred environment. The advantages of QoE crowdtesting lie not only in the reduced time and costs for the tests, but also in a large and diverse panel of international, geographically distributed users in realistic user settings. However, conceptual and technical challenges emerge due to the remote test settings. Key issues arising from QoE crowdtesting include the reliability of user ratings, the influence of incentives, payment schemes and the unknown environmental context of the tests on the results. In order to counter these issues, strategies and methods need to be developed, included in the test design, and also implemented in the actual test campaign, while statistical methods are required to identify reliable user ratings and to ensure high data quality. This contribution therefore provides a collection of best practices addressing these issues based on our experience gained in a large set of conducted QoE crowdtesting studies. The focus of this article is in particular on the issue of reliability and we use video quality assessment as an example for the proposed best practices, showing that our recommended two-stage QoE crowdtesting design leads to more reliable results.
Download citation as [.bib File]
@article{Hossfeld-TOM2014,
title = {Best Practices for QoE Crowdtesting: QoE Assessment With Crowdsourcing},
author = {Tobias Hoßfeld and Christian Keimel and Matthias Hirth and Bruno Gardlo and Julian Habigt and Klaus Diepold and Phuoc Tran-Gia},
doi = {10.1109/TMM.2013.2291663},
issn = {1520-9210},
year = {2014},
date = {2014-01-01},
journal = {Multimedia, IEEE Transactions on},
volume = {16},
number = {2},
pages = {541-558},
abstract = {Quality of Experience (QoE) in multimedia applications is closely linked to the end users' perception and therefore its assessment requires subjective user studies in order to evaluate the degree of delight or annoyance as experienced by the users. QoE crowdtesting refers to QoE assessment using crowdsourcing, where anonymous test subjects conduct subjective tests remotely in their preferred environment. The advantages of QoE crowdtesting lie not only in the reduced time and costs for the tests, but also in a large and diverse panel of international, geographically distributed users in realistic user settings. However, conceptual and technical challenges emerge due to the remote test settings. Key issues arising from QoE crowdtesting include the reliability of user ratings, the influence of incentives, payment schemes and the unknown environmental context of the tests on the results. In order to counter these issues, strategies and methods need to be developed, included in the test design, and also implemented in the actual test campaign, while statistical methods are required to identify reliable user ratings and to ensure high data quality. This contribution therefore provides a collection of best practices addressing these issues based on our experience gained in a large set of conducted QoE crowdtesting studies. The focus of this article is in particular on the issue of reliability and we use video quality assessment as an example for the proposed best practices, showing that our recommended two-stage QoE crowdtesting design leads to more reliable results.},
keywords = {},
pubstate = {},
tppubtype = {article}
}

Incollections

Hoßfeld, T., Keimel, C.: "Crowdsourcing in QoE Evaluation", Möller, Sebastian; Raake, Alexander (Ed.): Quality of Experience, pp. 315-327, Springer International Publishing, 2014, ISBN: 978-3-319-02680-0.
[Abstract] [BibTeX] [DOI]

Crowdsourcing enables new possibilities for QoE evaluation by moving the evaluation task from the traditional laboratory environment into the Internet, allowing researchers to easily access a global pool of subjects for the evaluation task. This makes it not only possible to include a more diverse population and real-life environments into the evaluation, but also reduces the turn-around time and increases the number of subjects participating in an evaluation campaign significantly by circumventing bottle-necks in traditional laboratory setup. In order to utilise these advantages, the differences between laboratory-based and crowd-based QoE evaluation must be considered and we therefore discuss both these differences and their impact on the QoE evaluation in this chapter.
Download citation as [.bib File]
@incollection{Hossfeld-SpringerQoEBook2014,
title = {Crowdsourcing in QoE Evaluation},
author = {Tobias Hoßfeld and Christian Keimel},
editor = {Sebastian Möller and Alexander Raake},
doi = {10.1007/978-3-319-02681-7_21},
isbn = {978-3-319-02680-0},
year = {2014},
date = {2014-01-01},
booktitle = {Quality of Experience},
pages = {315-327},
publisher = {Springer International Publishing},
series = {T-Labs Series in Telecommunication Services},
abstract = {Crowdsourcing enables new possibilities for QoE evaluation by moving the evaluation task from the traditional laboratory environment into the Internet, allowing researchers to easily access a global pool of subjects for the evaluation task. This makes it not only possible to include a more diverse population and real-life environments into the evaluation, but also reduces the turn-around time and increases the number of subjects participating in an evaluation campaign significantly by circumventing bottle-necks in traditional laboratory setup. In order to utilise these advantages, the differences between laboratory-based and crowd-based QoE evaluation must be considered and we therefore discuss both these differences and their impact on the QoE evaluation in this chapter.},
howpublished = {Full text available from publisher},
keywords = {},
pubstate = {},
tppubtype = {incollection}
}

Inproceedings

Hoßfeld, T., Hirth, M., Korshunov, P., Hanhart, P., Gardlo, B., Keimel, C., Timmerer, C.: "Survey of web-based crowdsourcing frameworks for subjective quality assessment", Multimedia Signal Processing (MMSP), 2014 IEEE 16th International Workshop on, pp. 1-6, 2014.
[Abstract] [PDF] [BibTeX] [DOI]

The popularity of the crowdsourcing for performing various tasks online increased significantly in the past few years. The low cost and flexibility of crowdsourcing, in particular, attracted researchers in the field of subjective multimedia evaluations and Quality of Experience (QoE). Since online assessment of multimedia content is challenging, several dedicated frameworks were created to aid in the designing of the tests, including the support of the testing methodologies like ACR, DCR, and PC, setting up the tasks, training sessions, screening of the subjects, and storage of the resulted data. In this paper, we focus on the web-based frameworks for multimedia quality assessments that support commonly used crowdsourcing platforms such as Amazon Mechanical Turk and Microworkers. We provide a detailed overview of the crowdsourcing frameworks and evaluate them to aid researchers in the field of QoE assessment in the selection of frameworks and crowdsourcing platforms that are adequate for their experiments.
Download citation as [.bib File]
@inproceedings{Hossfeld-MMSP2014,
title = {Survey of web-based crowdsourcing frameworks for subjective quality assessment},
author = {Tobias Hoßfeld and Matthias Hirth and Pavel Korshunov and Phillipe Hanhart and Bruno Gardlo and Christian Keimel and Christian Timmerer},
doi = {10.1109/MMSP.2014.6958831},
year = {2014},
date = {2014-09-01},
booktitle = {Multimedia Signal Processing (MMSP), 2014 IEEE 16th International Workshop on},
pages = {1-6},
abstract = {The popularity of the crowdsourcing for performing various tasks online increased significantly in the past few years. The low cost and flexibility of crowdsourcing, in particular, attracted researchers in the field of subjective multimedia evaluations and Quality of Experience (QoE). Since online assessment of multimedia content is challenging, several dedicated frameworks were created to aid in the designing of the tests, including the support of the testing methodologies like ACR, DCR, and PC, setting up the tasks, training sessions, screening of the subjects, and storage of the resulted data. In this paper, we focus on the web-based frameworks for multimedia quality assessments that support commonly used crowdsourcing platforms such as Amazon Mechanical Turk and Microworkers. We provide a detailed overview of the crowdsourcing frameworks and evaluate them to aid researchers in the field of QoE assessment in the selection of frameworks and crowdsourcing platforms that are adequate for their experiments.},
keywords = {},
pubstate = {},
tppubtype = {inproceedings}
}

Technical Reports

Hoßfeld, T., Hirth, M., Redi, J., Mazza, F., Korshunov, P., Naderi, B., Seufert, M., Gardlo, B., Egger, S., Keimel, C.: "Best Practices and Recommendations for Crowdsourced QoE - Lessons learned from the Qualinet Task Force ''Crowdsourcing''", 2014.
[Abstract] [PDF] [BibTeX]

Crowdsourcing is a popular approach that outsources tasks via the Internet to a large number of users. Commercial crowdsourcing platforms provide a global pool of users employed for perform-ing short and simple online tasks. For quality assessment of multimedia services and applications, crowdsourcing enables new possibilities by moving the subjective test into the crowd resulting in larger diversity of the test subjects, faster turnover of test campaigns, and reduced costs due to low reimbursement costs of the participants. Further, crowdsourcing allows easily addressing additional features like real-life environments. Crowdsourced quality assessment however is not a straight-forward implementation of existing subjective testing methodologies in an Internet-based environment. Additional challenges and differences to lab studies occur, in conceptual, technical, and motivational areas [9, 25, 26]. For example, the test contents need to be transmitted to the user over the Internet; test users may have low resolution screens influencing the user experience; also users may not understand the test or do not execute the test carefully resulting in unreliable data. This white paper summarizes the recommendations and best practices for crowdsourced qual-ity assessment of multimedia applications from the Qualinet Task Force on "Crowdsourcing". The European Network on Quality of Experience in Multimedia Systems and Services Qualinet (COST Action IC 1003, see www.qualinet.eu) established this task force in 2012. Since then it has grown to more then 30 members. The recommendation paper resulted from the experience in designing, implementing, and conducting crowdsourcing experiments as well as the analysis of the crowdsourced user ratings and context data. For understanding the impact of the crowdsourcing environment on QoE assessment and to derive a methodology and setup for crowdsourced QoE assessment, data from traditional lab experiments were compared with results from crowdsourc-ing experiments. Within the crowdsourcing task force, several different application domains and scientific questions were considered, among others: • video and image quality in general, • QoE for HTTP streaming [31, 32] and HTTP adaptive streaming [19, 30], • selfie portrait images perception in a recruitment context [10], • privacy in HDR images and video [39, 20, 36], • compression of HDR images [37] [38], • evaluation of 3D video [38], • image recognizability and aesthetic appeal [12, 13], • multidimensional modeling of web QoE [14], • QoE factors of cloud storage services [21], • enabling eye tracking experiments using web technologies [41]. From a crowdsourcing perspective, the following mechanisms and approaches were investigated which are relevant to understand for crowdsourced quality assessment.
Download citation as [.bib File]
@techreport{Hossfeld-QualinetCrowdsourcing-WHP,
title = {Best Practices and Recommendations for Crowdsourced QoE - Lessons learned from the Qualinet Task Force ''Crowdsourcing''},
author = {Tobias Hoßfeld and Matthias Hirth and Judith Redi and Filippo Mazza and Pavel Korshunov and Babak Naderi and Michael Seufert and Bruno Gardlo and Sebastian Egger and Christian Keimel},
url = {https://hal.archives-ouvertes.fr/hal-01078761},
year = {2014},
date = {2014-01-01},
abstract = {Crowdsourcing is a popular approach that outsources tasks via the Internet to a large number of users. Commercial crowdsourcing platforms provide a global pool of users employed for perform-ing short and simple online tasks. For quality assessment of multimedia services and applications, crowdsourcing enables new possibilities by moving the subjective test into the crowd resulting in larger diversity of the test subjects, faster turnover of test campaigns, and reduced costs due to low reimbursement costs of the participants. Further, crowdsourcing allows easily addressing additional features like real-life environments. Crowdsourced quality assessment however is not a straight-forward implementation of existing subjective testing methodologies in an Internet-based environment. Additional challenges and differences to lab studies occur, in conceptual, technical, and motivational areas [9, 25, 26]. For example, the test contents need to be transmitted to the user over the Internet; test users may have low resolution screens influencing the user experience; also users may not understand the test or do not execute the test carefully resulting in unreliable data. This white paper summarizes the recommendations and best practices for crowdsourced qual-ity assessment of multimedia applications from the Qualinet Task Force on "Crowdsourcing". The European Network on Quality of Experience in Multimedia Systems and Services Qualinet (COST Action IC 1003, see www.qualinet.eu) established this task force in 2012. Since then it has grown to more then 30 members. The recommendation paper resulted from the experience in designing, implementing, and conducting crowdsourcing experiments as well as the analysis of the crowdsourced user ratings and context data. For understanding the impact of the crowdsourcing environment on QoE assessment and to derive a methodology and setup for crowdsourced QoE assessment, data from traditional lab experiments were compared with results from crowdsourc-ing experiments. Within the crowdsourcing task force, several different application domains and scientific questions were considered, among others: • video and image quality in general, • QoE for HTTP streaming [31, 32] and HTTP adaptive streaming [19, 30], • selfie portrait images perception in a recruitment context [10], • privacy in HDR images and video [39, 20, 36], • compression of HDR images [37] [38], • evaluation of 3D video [38], • image recognizability and aesthetic appeal [12, 13], • multidimensional modeling of web QoE [14], • QoE factors of cloud storage services [21], • enabling eye tracking experiments using web technologies [41]. From a crowdsourcing perspective, the following mechanisms and approaches were investigated which are relevant to understand for crowdsourced quality assessment.},
note = {Lessons learned from the Qualinet Task Force ''Crowdsourcing'' COST Action IC1003 European Network on Quality of Experience in Multimedia Systems and Services (QUALINET)},
keywords = {},
pubstate = {},
tppubtype = {techreport}
}

2013

Incollections

Keimel, C.: "Crowdsourcing of Multimedia QoE Subjective Experiments", Hoßfeld, Tobias; Tran-Gia, Phuoc; Vukovic, Maja (Ed.): Dagstuhl Reports - Crowdsourcing: From Theory to Practice and Long-Term Perspectives (Dagstuhl Seminar 13361), 3 (9), pp. 1–33, Schloss Dagstuhl--Leibniz-Zentrum fuer Informatik, Dagstuhl, Germany, 2013, ISSN: 2192-5283.
[Abstract] [PDF] [BibTeX] [DOI]

Download citation as [.bib File]
@incollection{Hossfeld-Dagstuhl2013,
title = {Crowdsourcing of Multimedia QoE Subjective Experiments},
author = {Christian Keimel},
editor = {Tobias Hoßfeld and Phuoc Tran-Gia and Maja Vukovic},
url = {http://drops.dagstuhl.de/opus/volltexte/2013/4354},
doi = {10.4230/DagRep.3.9.1},
issn = {2192-5283},
year = {2013},
date = {2013-01-01},
booktitle = {Dagstuhl Reports - Crowdsourcing: From Theory to Practice and Long-Term Perspectives (Dagstuhl Seminar 13361)},
journal = {Dagstuhl Reports},
volume = {3},
number = {9},
pages = {1--33},
publisher = {Schloss Dagstuhl--Leibniz-Zentrum fuer Informatik},
address = {Dagstuhl, Germany},
keywords = {},
pubstate = {},
tppubtype = {incollection}
}

Inproceedings

Redi, J., Hoßfeld, T., Korshunov, P., Mazza, F., Povoa, I., Keimel, C.: "Crowdsourcing-based Multimedia Subjective Evaluations: A Case Study on Image Recognizability and Aesthetic Appeal", Proceedings of the 2nd ACM International Workshop on Crowdsourcing for Multimedia, pp. 29-34, ACM, Barcelona, Spain, 2013, ISBN: 978-1-4503-2396-3.
[Abstract] [PDF] [BibTeX] [DOI]

Research on Quality of Experience (QoE) heavily relies on subjective evaluations of media. An important aspect of QoE concerns modeling and quantifying the subjective notions of 'beauty' (aesthetic appeal) and 'something well-known' (content recognizability), which are both subject to cultural and social effects. Crowdsourcing, which allows employing people worldwide to perform short and simple tasks via online platforms, can be a great tool for performing subjective studies in a time and cost-effective way. On the other hand, the crowdsourcing environment does not allow for the degree of experimental control which is necessary to guarantee reliable subjective data. To validate the use of crowdsourcing for QoE assessments, in this paper, we evaluate aesthetic appeal and recognizability of images using the Microworkers crowdsourcing platform and compare the outcomes with more conventional evaluations conducted in a controlled lab environment. We find high correlation between crowdsourcing and lab scores for recognizability but not for aesthetic appeal, indicating that crowdsourcing can be used for QoE subjective assessments as long as the workers' tasks are designed with extreme care to avoid misinterpretations.
Download citation as [.bib File]
@inproceedings{Redi-CrowdMM2013,
title = {Crowdsourcing-based Multimedia Subjective Evaluations: A Case Study on Image Recognizability and Aesthetic Appeal},
author = {Judith Alice Redi and Tobias Hoßfeld and Pavel Korshunov and Filippo Mazza and Isabel Povoa and Christian Keimel},
doi = {10.1145/2506364.2506368},
isbn = {978-1-4503-2396-3},
year = {2013},
date = {2013-10-01},
booktitle = {Proceedings of the 2nd ACM International Workshop on Crowdsourcing for Multimedia},
pages = {29-34},
publisher = {ACM},
address = {Barcelona, Spain},
series = {CrowdMM '13},
abstract = {Research on Quality of Experience (QoE) heavily relies on subjective evaluations of media. An important aspect of QoE concerns modeling and quantifying the subjective notions of 'beauty' (aesthetic appeal) and 'something well-known' (content recognizability), which are both subject to cultural and social effects. Crowdsourcing, which allows employing people worldwide to perform short and simple tasks via online platforms, can be a great tool for performing subjective studies in a time and cost-effective way. On the other hand, the crowdsourcing environment does not allow for the degree of experimental control which is necessary to guarantee reliable subjective data. To validate the use of crowdsourcing for QoE assessments, in this paper, we evaluate aesthetic appeal and recognizability of images using the Microworkers crowdsourcing platform and compare the outcomes with more conventional evaluations conducted in a controlled lab environment. We find high correlation between crowdsourcing and lab scores for recognizability but not for aesthetic appeal, indicating that crowdsourcing can be used for QoE subjective assessments as long as the workers' tasks are designed with extreme care to avoid misinterpretations.},
keywords = {},
pubstate = {},
tppubtype = {inproceedings}
}

Keimel, C., Pangerl, C., Diepold, K.: "Comparison of lossless video codecs for crowd-based quality assessment on tablets", Seventh International Workshop on Video Processing and Quality Metrics for Consumer Electronics - VPQM 2013, pp. 37-41, 2013.
[Abstract] [PDF] [BibTeX]

Video quality evaluation with subjective testing is both time consuming and expensive. A promising new approach to traditional testing is the so-called crowdsourcing, moving the testing effort into the Internet. The advantages of this approach are not only the access to a larger and more diverse pool of test subjects, but also the significant reduction of the financial burden. Extending this approach to tablets, allows us not only to assess video quality in a realistic environment for an ever more important use case, but also provides us with a well-defined hardware platform, eliminating on of the main drawbacks of crowdsourced video quality assessment. One prerequisite, however, is the support of lossless coding on the used tablets. We therefore examine in this contribution the performance of lossless video codecs on the iPad platform. Our results show, that crowdbased video testing is already feasible for CIF-sized videos on tablets, but also that there may be limits for higher resolution videos.
Download citation as [.bib File]
@inproceedings{Keimel-VPQM2013,
title = {Comparison of lossless video codecs for crowd-based quality assessment on tablets},
author = {Christian Keimel and Christopher Pangerl and Klaus Diepold},
url = {http://www.vpqm.org/},
year = {2013},
date = {2013-01-01},
booktitle = {Seventh International Workshop on Video Processing and Quality Metrics for Consumer Electronics - VPQM 2013},
pages = {37-41},
abstract = {Video quality evaluation with subjective testing is both time consuming and expensive. A promising new approach to traditional testing is the so-called crowdsourcing, moving the testing effort into the Internet. The advantages of this approach are not only the access to a larger and more diverse pool of test subjects, but also the significant reduction of the financial burden. Extending this approach to tablets, allows us not only to assess video quality in a realistic environment for an ever more important use case, but also provides us with a well-defined hardware platform, eliminating on of the main drawbacks of crowdsourced video quality assessment. One prerequisite, however, is the support of lossless coding on the used tablets. We therefore examine in this contribution the performance of lossless video codecs on the iPad platform. Our results show, that crowdbased video testing is already feasible for CIF-sized videos on tablets, but also that there may be limits for higher resolution videos.},
keywords = {},
pubstate = {},
tppubtype = {inproceedings}
}

2012

Inproceedings

Keimel, C., Habigt, J., Diepold, K.: "Challenges in crowd-based video quality assessment", Quality of Multimedia Experience (QoMEX), 2012 Fourth International Workshop on, pp. 13-18, 2012, ISBN: 978-1-4673-0725-3.
[Abstract] [PDF] [BibTeX] [DOI]

Video quality evaluation with subjective testing is both time consuming and expensive. A promising new approach to traditional testing is the so-called crowdsourcing, moving the testing effort into the Internet. The advantages of this approach are not only the access to a larger and more diverse pool of test subjects, but also the significant reduction of the financial burden. Recent contributions have also shown that crowd-based video quality assessment can deliver results comparable to traditional testing in some cases. In general, however, new problems arise, as no longer every test detail can be controlled, resulting in less reliable results. Therefore we will discuss in this contribution the conceptual, technical, motivational and reliability challenges that need to be addressed, before this promising approach to subjective testing can become a valid alternative to the testing in standardized environments.
Download citation as [.bib File]
@inproceedings{Keimel-QoMEX2012-ChallengesCrowdsourcing,
title = {Challenges in crowd-based video quality assessment},
author = {Christian Keimel and Julian Habigt and Klaus Diepold},
doi = {10.1109/QoMEX.2012.6263866},
isbn = {978-1-4673-0725-3},
year = {2012},
date = {2012-07-01},
booktitle = {Quality of Multimedia Experience (QoMEX), 2012 Fourth International Workshop on},
pages = {13-18},
abstract = {Video quality evaluation with subjective testing is both time consuming and expensive. A promising new approach to traditional testing is the so-called crowdsourcing, moving the testing effort into the Internet. The advantages of this approach are not only the access to a larger and more diverse pool of test subjects, but also the significant reduction of the financial burden. Recent contributions have also shown that crowd-based video quality assessment can deliver results comparable to traditional testing in some cases. In general, however, new problems arise, as no longer every test detail can be controlled, resulting in less reliable results. Therefore we will discuss in this contribution the conceptual, technical, motivational and reliability challenges that need to be addressed, before this promising approach to subjective testing can become a valid alternative to the testing in standardized environments.},
keywords = {},
pubstate = {},
tppubtype = {inproceedings}
}

Keimel, C., Habigt, J., Horch, C., Diepold, K.: "Video quality evaluation in the cloud", Packet Video Workshop (PV), 2012 19th International, pp. 155-160, 2012, ISBN: 978-1-4673-0299-9.
[Abstract] [PDF] [BibTeX] [DOI]

Video quality evaluation with subjective testing is both time consuming and expensive. An interesting new approach to traditional testing is the so-called crowdsourcing, moving the testing effort into the internet. The QualityCrowd framework allows codec independent, crowd-based video quality assessment with a simple web interface, usable with common web browsers. However, due to its codec independent approach, the framework can pose high bandwidth requirements on the coordinating server. We therefore propose in this contribution a cloud-based extension of the QualityCrowd framework in order to perform subjective quality evaluation as a cloud application. Moreover, this allows us to access an even larger pool of potential participants due to the improved connectivity. We compare the results from an online subjective test using this framework with the results from a test in a standardized environment. This comparison shows that QualityCrowd delivers equivalent results within the acceptable inter-lab correlation.
Download citation as [.bib File]
@inproceedings{Keimel-PV2012,
title = {Video quality evaluation in the cloud},
author = {Christian Keimel and Julian Habigt and Clemens Horch and Klaus Diepold},
doi = {10.1109/PV.2012.6229729},
isbn = {978-1-4673-0299-9},
year = {2012},
date = {2012-05-01},
booktitle = {Packet Video Workshop (PV), 2012 19th International},
pages = {155-160},
abstract = {Video quality evaluation with subjective testing is both time consuming and expensive. An interesting new approach to traditional testing is the so-called crowdsourcing, moving the testing effort into the internet. The QualityCrowd framework allows codec independent, crowd-based video quality assessment with a simple web interface, usable with common web browsers. However, due to its codec independent approach, the framework can pose high bandwidth requirements on the coordinating server. We therefore propose in this contribution a cloud-based extension of the QualityCrowd framework in order to perform subjective quality evaluation as a cloud application. Moreover, this allows us to access an even larger pool of potential participants due to the improved connectivity. We compare the results from an online subjective test using this framework with the results from a test in a standardized environment. This comparison shows that QualityCrowd delivers equivalent results within the acceptable inter-lab correlation.},
keywords = {},
pubstate = {},
tppubtype = {inproceedings}
}

Keimel, C., Habigt, J., Horch, C., Diepold, K.: "QualityCrowd -- A framework for crowd-based quality evaluation", Picture Coding Symposium (PCS), 2012, pp. 245-248, 2012, ISBN: 978-1-4577-2048-2.
[Abstract] [PDF] [BibTeX] [DOI]

Video quality assessment with subjective testing is both time consuming and expensive. An interesting new approach to traditional testing is the so-called crowdsourcing, moving the testing effort into the internet. We therefore propose in this contribution the QualityCrowd framework to effortlessly perform subjective quality assessment with crowdsourcing. QualityCrowd allows codec independent quality assessment with a simple web interface, usable with common web browsers. We compared the results from an online subjective test using this framework with the results from a test in a standardized environment. This comparison shows that QualityCrowd delivers equivalent results within the acceptable inter-lab correlation. While we only consider video quality in this contribution, QualityCrowd can also be used for multimodal quality assessment.
Download citation as [.bib File]
@inproceedings{Keimel-PCS2012,
title = {QualityCrowd -- A framework for crowd-based quality evaluation},
author = {Christain Keimel and Julian Habigt and Clemens Horch and Klaus Diepold},
doi = {10.1109/PCS.2012.6213338},
isbn = {978-1-4577-2048-2},
year = {2012},
date = {2012-01-01},
booktitle = {Picture Coding Symposium (PCS), 2012},
pages = {245-248},
abstract = {Video quality assessment with subjective testing is both time consuming and expensive. An interesting new approach to traditional testing is the so-called crowdsourcing, moving the testing effort into the internet. We therefore propose in this contribution the QualityCrowd framework to effortlessly perform subjective quality assessment with crowdsourcing. QualityCrowd allows codec independent quality assessment with a simple web interface, usable with common web browsers. We compared the results from an online subjective test using this framework with the results from a test in a standardized environment. This comparison shows that QualityCrowd delivers equivalent results within the acceptable inter-lab correlation. While we only consider video quality in this contribution, QualityCrowd can also be used for multimodal quality assessment.},
keywords = {},
pubstate = {},
tppubtype = {inproceedings}
}

2011

Technical Reports

Horch, C., Keimel, C., Diepold, K.: "QualityCrowd - Crowdsourcing for Subjective Video Quality Tests", Institute for Data Processing, Technische Universität München 2011.
[Abstract] [PDF] [BibTeX]

Despite continuing research on the development of better quality metrics, subjective tests are still indispensable for the assessment of video quality. These tests are both time-consuming and expensive and require installing a suitable laboratory that fulfills the corresponding ITU recommendations. In this thesis the use of crowdsourcing in conjunction with the internetbased performing of such tests shall be examined comparing the results of such a test and the results of conventional laboratory tests. For performing this test the web-based software QualityCrowd was developed, which allows the simple planning and conducting of subjective tests. The software uses Amazon's crowdsourcing platform Mechanical Turk to assign the assessment of the videos to the crowd. Amazon provides the infrastructure for distibuting large numbers of almost any task and paying the workers afterwards. Another aspect is the evaluation of the technical issues that arise from an internet-based video test. In particular, the problems concerning the compression, delivery and playback of the videos in the participants' browsers are discussed. After considering the various possibilities, a decision in favour of lossless compression using H.264/AVC and playback with Adobe's Flash Player is taken. The gathered data show very high correlation with the data from the laboratories they are compared with. Although there are also some significant deviations, the results in general are quite promising and indicate the suitability of the use of crowdsourcing for subjective video tests. Even though the test could not be conducted publicly and the workers be paid, the costs of a test like this one are estimated. It shows that - compared to conventional laboratory tests - a clear cut in costs can be achieved.
Download citation as [.bib File]
@techreport{Horch-TR-QualityCrowd-2011,
title = {QualityCrowd - Crowdsourcing for Subjective Video Quality Tests},
author = {Clemens Horch and Christian Keimel and Klaus Diepold},
url = {https://mediatum.ub.tum.de/node?id=1082600},
year = {2011},
date = {2011-01-01},
institution = {Institute for Data Processing, Technische Universität München},
abstract = {Despite continuing research on the development of better quality metrics, subjective tests are still indispensable for the assessment of video quality. These tests are both time-consuming and expensive and require installing a suitable laboratory that fulfills the corresponding ITU recommendations. In this thesis the use of crowdsourcing in conjunction with the internetbased performing of such tests shall be examined comparing the results of such a test and the results of conventional laboratory tests. For performing this test the web-based software QualityCrowd was developed, which allows the simple planning and conducting of subjective tests. The software uses Amazon's crowdsourcing platform Mechanical Turk to assign the assessment of the videos to the crowd. Amazon provides the infrastructure for distibuting large numbers of almost any task and paying the workers afterwards. Another aspect is the evaluation of the technical issues that arise from an internet-based video test. In particular, the problems concerning the compression, delivery and playback of the videos in the participants' browsers are discussed. After considering the various possibilities, a decision in favour of lossless compression using H.264/AVC and playback with Adobe's Flash Player is taken. The gathered data show very high correlation with the data from the laboratories they are compared with. Although there are also some significant deviations, the results in general are quite promising and indicate the suitability of the use of crowdsourcing for subjective video tests. Even though the test could not be conducted publicly and the workers be paid, the costs of a test like this one are estimated. It shows that - compared to conventional laboratory tests - a clear cut in costs can be achieved.},
keywords = {},
pubstate = {},
tppubtype = {techreport}
}