Christian Keimel

Publications

On this page, all my publications are listed in chronological order, grouped by publication type. Wherever possible, a full-text PDF version of the paper is provided or if not possible a link to the publisher’s online version of the paper via DOI or URL is given. In addition for each paper, the abstract and a citation in BibTeX format are provided.

For an always up-to-date citation count of the publications, please visit my Google Scholar profile.

2017

Incollections

Egger-Lampl, S., Redi, J., Hoßfeld, T., Hirth, M., Möller, S., Naderi, B., Keimel, C., Saupe, D.: "Crowdsourcing Quality of Experience Experiments", Archambault, Daniel; Purchase, Helen; Hossfeld, Tobias (Ed.): Evaluation in the Crowd. Crowdsourcing and Human-Centered Experiments, 10264 , pp. 154-190, Springer International Publishing, 2017, ISBN: 978-3-319-66435-4.
[Abstract] [BibTeX] [DOI]

Crowdsourcing enables new possibilities for QoE evaluation by moving the evaluation task from the traditional laboratory environment into the Internet, allowing researchers to easily access a global pool of workers for the evaluation task. This makes it not only possible to include a more diverse population and real-life environments into the evaluation, but also reduces the turn-around time and increases the number of subjects participating in an evaluation campaign significantly, thereby circumventing bottle-necks in traditional laboratory setups. In order to utilise these advantages, the differences between laboratory-based and crowd-based QoE evaluation are discussed in this chapter.
Download citation as [.bib File]
@incollection{EggerLampl-SpringerCrowdsourcingBook2017,
title = {Crowdsourcing Quality of Experience Experiments},
author = {Sebastian Egger-Lampl and Judith Redi and Tobias Hoßfeld and Matthias Hirth and Sebastian Möller and Babak Naderi and Christian Keimel and Dietmar Saupe},
editor = {Daniel Archambault and Helen Purchase and Tobias Hossfeld},
doi = {10.1007/978-3-319-66435-4_7},
isbn = {978-3-319-66435-4},
year = {2017},
date = {2017-01-01},
booktitle = {Evaluation in the Crowd. Crowdsourcing and Human-Centered Experiments},
volume = {10264},
pages = {154-190},
publisher = {Springer International Publishing},
series = {Lecture Notes in Computer Science},
abstract = {Crowdsourcing enables new possibilities for QoE evaluation by moving the evaluation task from the traditional laboratory environment into the Internet, allowing researchers to easily access a global pool of workers for the evaluation task. This makes it not only possible to include a more diverse population and real-life environments into the evaluation, but also reduces the turn-around time and increases the number of subjects participating in an evaluation campaign significantly, thereby circumventing bottle-necks in traditional laboratory setups. In order to utilise these advantages, the differences between laboratory-based and crowd-based QoE evaluation are discussed in this chapter.},
howpublished = {Full text available from publisher},
keywords = {},
pubstate = {},
tppubtype = {incollection}
}

Inproceedings

Ziegler, C., Keimel, C., Ramdhany, R., Vinayagamoorthy, V.: "On Time or Not on Time: A User Study on Delays in a Synchronised Companion-Screen Experience", Proceedings of the ACM International Conference on Interactive Experiences for TV and Online Video, pp. 105-114, ACM, 2017.
[Abstract] [PDF] [BibTeX] [DOI]

One major challenge in creation of compelling companion screen experiences, are the time delays between the presentation of content on the TV compared to the presentation of content on the companion screen. Through the use of a synchronised, interactive textbook application, we conducted a user study to evaluate the potential influence of different delays, between the TV and the companion screen, on how users experience watching a Shakespearean play on the TV. Our results indicate that although users do not notice delays of up to 1000 ms, for the kind of experience tested, they feel significantly more distracted by the tablet content for increasingly higher delays. We discuss the implications of our findings with regards to the time delay tolerances users might have when using a synchronised text accompaniment to these kinds of TV programmes.
Download citation as [.bib File]
@inproceedings{Ziegler-TVX2017,
title = {On Time or Not on Time: A User Study on Delays in a Synchronised Companion-Screen Experience},
author = {Christoph Ziegler and Christian Keimel and Rajiv Ramdhany and Vinoba Vinayagamoorthy},
doi = {10.1145/3077548.3077557},
year = {2017},
date = {2017-06-01},
booktitle = {Proceedings of the ACM International Conference on Interactive Experiences for TV and Online Video},
pages = {105-114},
publisher = {ACM},
series = {TVX ' 17},
abstract = {One major challenge in creation of compelling companion screen experiences, are the time delays between the presentation of content on the TV compared to the presentation of content on the companion screen. Through the use of a synchronised, interactive textbook application, we conducted a user study to evaluate the potential influence of different delays, between the TV and the companion screen, on how users experience watching a Shakespearean play on the TV. Our results indicate that although users do not notice delays of up to 1000 ms, for the kind of experience tested, they feel significantly more distracted by the tablet content for increasingly higher delays. We discuss the implications of our findings with regards to the time delay tolerances users might have when using a synchronised text accompaniment to these kinds of TV programmes.},
keywords = {},
pubstate = {},
tppubtype = {inproceedings}
}

Arndt, S., Räty, V., Nieuwenhuis, T., Keimel, C., Ibáñez, F., Perkis, A.: "Enhancing Use of Social Media in TV Broadcasting", Adjunct Proceedings of the ACM International Conference on Interactive Experiences for TV and Online Video, pp. 51-56, ACM, 2017.
[Abstract] [PDF] [BibTeX] [DOI]

Traditional linear TV is decreasing in popularity and the broadcast industry has identified the need to change communication with their audience as a way to counteract on this. Especially younger generations are using social media twenty-four-seven and would like to continue doing so while watching TV. The VisualMedia project accepts this challenge by enriching the TV experience with elements and information from social media in real-time during live broadcasts. It enables broadcasters to select and distribute posts and stats from various social media sources in a fast and reliable way. With the help of VisualMedia, posts and stats can be shown within live programs using enhanced graphical representations or on the second screen, with minimal latency. The goal of the project is to deliver a framework that combines all necessary steps from retrieving posts to delivering them into a live TV show. This gives broadcasters the opportunity to react fast on their audiences, and facilitates audiences to interact with their broadcaster.
Download citation as [.bib File]
@inproceedings{Arndt-TVX2017,
title = {Enhancing Use of Social Media in TV Broadcasting},
author = {Sebastian Arndt and Veli-Pekka Räty and Taco Nieuwenhuis and Christian Keimel and Francisco Ibáñez and Andrew Perkis},
doi = {10.1145/3084289.3089923},
year = {2017},
date = {2017-01-01},
booktitle = {Adjunct Proceedings of the ACM International Conference on Interactive Experiences for TV and Online Video},
pages = {51-56},
publisher = {ACM},
series = {TVX ' 17},
abstract = {Traditional linear TV is decreasing in popularity and the broadcast industry has identified the need to change communication with their audience as a way to counteract on this. Especially younger generations are using social media twenty-four-seven and would like to continue doing so while watching TV. The VisualMedia project accepts this challenge by enriching the TV experience with elements and information from social media in real-time during live broadcasts. It enables broadcasters to select and distribute posts and stats from various social media sources in a fast and reliable way. With the help of VisualMedia, posts and stats can be shown within live programs using enhanced graphical representations or on the second screen, with minimal latency. The goal of the project is to deliver a framework that combines all necessary steps from retrieving posts to delivering them into a live TV show. This gives broadcasters the opportunity to react fast on their audiences, and facilitates audiences to interact with their broadcaster.},
keywords = {},
pubstate = {},
tppubtype = {inproceedings}
}

2016

Books

Keimel, C.: "Design of Video Quality Metrics with Multi-Way Data Analysis: A data driven approach", Springer, Singapore, 2016, ISBN: 978-3-319-02680-0.
[Abstract] [BibTeX] [DOI]

This book proposes a data-driven methodology using multi-way data analysis for the design of video-quality metrics. It also enables video- quality metrics to be created using arbitrary features. This data- driven design approach not only requires no detailed knowledge of the human visual system, but also allows a proper consideration of the temporal nature of video using a three-way prediction model, corresponding to the three-way structure of video. Using two simple example metrics, the author demonstrates not only that this purely data- driven approach outperforms state-of-the-art video-quality metrics, which are often optimized for specific properties of the human visual system, but also that multi-way data analysis methods outperform the combination of two-way data analysis methods and temporal pooling.
Download citation as [.bib File]
@book{Keimel-Springer2016,
title = {Design of Video Quality Metrics with Multi-Way Data Analysis: A data driven approach},
author = {Christian Keimel},
doi = {10.1007/978-981-10-0269-4},
isbn = {978-3-319-02680-0},
year = {2016},
date = {2016-01-01},
publisher = {Springer},
address = {Singapore},
series = {T-Labs Series in Telecommunication Services},
abstract = {This book proposes a data-driven methodology using multi-way data analysis for the design of video-quality metrics. It also enables video- quality metrics to be created using arbitrary features. This data- driven design approach not only requires no detailed knowledge of the human visual system, but also allows a proper consideration of the temporal nature of video using a three-way prediction model, corresponding to the three-way structure of video. Using two simple example metrics, the author demonstrates not only that this purely data- driven approach outperforms state-of-the-art video-quality metrics, which are often optimized for specific properties of the human visual system, but also that multi-way data analysis methods outperform the combination of two-way data analysis methods and temporal pooling.},
howpublished = {Full text available from publisher},
keywords = {},
pubstate = {},
tppubtype = {book}
}

2015

Journal Articles

Volk, T., Keimel, C., Moosmeier, M., Diepold, K.: "Crowdsourcing vs. laboratory experiments – QoE evaluation of binaural playback in a teleconference scenario", Computer Networks, 90 , pp. 99-109, 2015, ISSN: 1389-1286.
[Abstract] [PDF] [BibTeX] [DOI]

Abstract Experiments for the subjective evaluation of multimedia presentations and content are traditionally conducted in a laboratory environment. In this respect common procedures for the evaluation of teleconference systems are no different. The strictly controlled laboratory environment, however, often gives a rather poor representation of the actual use case. Therefore in this study we crowdsourced the evaluation of a teleconference system to perform the evaluation in a real-life environment. Moreover, we used the unique possibilities of crowdsourcing to employ two different demographics by hiring workers from Germany on the one hand and the US and Great Britain on the other hand. The goal of this experiment was to assess the perceived Quality of Experience (QoE) during a listening test and compare the results to results from a similar listening test conducted in the controlled laboratory environment. In doing so, we observed not only intriguing differences in the collected QoE ratings between the results of laboratory and crowdsourcing experiments, but also between the different worker demographics in terms of reliability, availability and efficiency.
Download citation as [.bib File]
@article{Volk-ComNET2015,
title = {Crowdsourcing vs. laboratory experiments – QoE evaluation of binaural playback in a teleconference scenario},
author = {Thomas Volk and Christian Keimel and Michael Moosmeier and Klaus Diepold},
doi = {10.1016/j.comnet.2015.05.021},
issn = {1389-1286},
year = {2015},
date = {2015-10-01},
journal = {Computer Networks},
volume = {90},
pages = {99-109},
abstract = {Abstract Experiments for the subjective evaluation of multimedia presentations and content are traditionally conducted in a laboratory environment. In this respect common procedures for the evaluation of teleconference systems are no different. The strictly controlled laboratory environment, however, often gives a rather poor representation of the actual use case. Therefore in this study we crowdsourced the evaluation of a teleconference system to perform the evaluation in a real-life environment. Moreover, we used the unique possibilities of crowdsourcing to employ two different demographics by hiring workers from Germany on the one hand and the US and Great Britain on the other hand. The goal of this experiment was to assess the perceived Quality of Experience (QoE) during a listening test and compare the results to results from a similar listening test conducted in the controlled laboratory environment. In doing so, we observed not only intriguing differences in the collected QoE ratings between the results of laboratory and crowdsourcing experiments, but also between the different worker demographics in terms of reliability, availability and efficiency.},
note = {Crowdsourcing},
keywords = {},
pubstate = {},
tppubtype = {article}
}

2014

Journal Articles

Hoßfeld, T., Keimel, C., Timmerer, C.: "Crowdsourcing Quality-of-Experience Assessments", Computer, 47 (9), pp. 98-102, 2014, ISSN: 0018-9162.
[Abstract] [BibTeX] [DOI]

Crowdsourced quality-of-experience (QoE) assessments are more cost-effective and flexible than traditional in-lab evaluations but require careful test design, innovative incentive mechanisms, and technical expertise to address various implementation challenges.
Download citation as [.bib File]
@article{Hossfeld-Computer2014,
title = {Crowdsourcing Quality-of-Experience Assessments},
author = {Tobias Hoßfeld and Christian Keimel and Christian Timmerer},
doi = {10.1109/MC.2014.245},
issn = {0018-9162},
year = {2014},
date = {2014-09-01},
journal = {Computer},
volume = {47},
number = {9},
pages = {98-102},
abstract = {Crowdsourced quality-of-experience (QoE) assessments are more cost-effective and flexible than traditional in-lab evaluations but require careful test design, innovative incentive mechanisms, and technical expertise to address various implementation challenges.},
howpublished = {Full text available from publisher},
keywords = {},
pubstate = {},
tppubtype = {article}
}

Hoßfeld, T., Keimel, C., Hirth, M., Gardlo, B., Habigt, J., Diepold, K., Tran-Gia, P.: "Best Practices for QoE Crowdtesting: QoE Assessment With Crowdsourcing", Multimedia, IEEE Transactions on, 16 (2), pp. 541-558, 2014, ISSN: 1520-9210.
[Abstract] [PDF] [BibTeX] [DOI]

Quality of Experience (QoE) in multimedia applications is closely linked to the end users' perception and therefore its assessment requires subjective user studies in order to evaluate the degree of delight or annoyance as experienced by the users. QoE crowdtesting refers to QoE assessment using crowdsourcing, where anonymous test subjects conduct subjective tests remotely in their preferred environment. The advantages of QoE crowdtesting lie not only in the reduced time and costs for the tests, but also in a large and diverse panel of international, geographically distributed users in realistic user settings. However, conceptual and technical challenges emerge due to the remote test settings. Key issues arising from QoE crowdtesting include the reliability of user ratings, the influence of incentives, payment schemes and the unknown environmental context of the tests on the results. In order to counter these issues, strategies and methods need to be developed, included in the test design, and also implemented in the actual test campaign, while statistical methods are required to identify reliable user ratings and to ensure high data quality. This contribution therefore provides a collection of best practices addressing these issues based on our experience gained in a large set of conducted QoE crowdtesting studies. The focus of this article is in particular on the issue of reliability and we use video quality assessment as an example for the proposed best practices, showing that our recommended two-stage QoE crowdtesting design leads to more reliable results.
Download citation as [.bib File]
@article{Hossfeld-TOM2014,
title = {Best Practices for QoE Crowdtesting: QoE Assessment With Crowdsourcing},
author = {Tobias Hoßfeld and Christian Keimel and Matthias Hirth and Bruno Gardlo and Julian Habigt and Klaus Diepold and Phuoc Tran-Gia},
doi = {10.1109/TMM.2013.2291663},
issn = {1520-9210},
year = {2014},
date = {2014-01-01},
journal = {Multimedia, IEEE Transactions on},
volume = {16},
number = {2},
pages = {541-558},
abstract = {Quality of Experience (QoE) in multimedia applications is closely linked to the end users' perception and therefore its assessment requires subjective user studies in order to evaluate the degree of delight or annoyance as experienced by the users. QoE crowdtesting refers to QoE assessment using crowdsourcing, where anonymous test subjects conduct subjective tests remotely in their preferred environment. The advantages of QoE crowdtesting lie not only in the reduced time and costs for the tests, but also in a large and diverse panel of international, geographically distributed users in realistic user settings. However, conceptual and technical challenges emerge due to the remote test settings. Key issues arising from QoE crowdtesting include the reliability of user ratings, the influence of incentives, payment schemes and the unknown environmental context of the tests on the results. In order to counter these issues, strategies and methods need to be developed, included in the test design, and also implemented in the actual test campaign, while statistical methods are required to identify reliable user ratings and to ensure high data quality. This contribution therefore provides a collection of best practices addressing these issues based on our experience gained in a large set of conducted QoE crowdtesting studies. The focus of this article is in particular on the issue of reliability and we use video quality assessment as an example for the proposed best practices, showing that our recommended two-stage QoE crowdtesting design leads to more reliable results.},
keywords = {},
pubstate = {},
tppubtype = {article}
}

Seybold, T., Knopp, M., Keimel, C., Stechele, W.: "Beyond Standard Noise Models: Evaluating Denoising Algorithms with Respect to Realistic Camera Noise", International Journal of Semantic Computing, 08 (02), pp. 145-167, 2014.
[Abstract] [PDF] [BibTeX] [DOI]

The development and tuning of denoising algorithms is usually based on readily processed test images that are artificially degraded with additive white Gaussian noise (AWGN). While AWGN allows us to easily generate test data in a repeatable manner, it does not reflect the noise characteristics in a real digital camera. Realistic camera noise is signal-dependent and spatially correlated due to the demosaicking step required to obtain full-color images. Hence, the noise characteristic is fundamentally different from AWGN. Using such unrealistic data to test, optimize and compare denoising algorithms may lead to incorrect parameter tuning or suboptimal choices in research on denoising algorithms.

In this paper, we therefore propose an approach to evaluate denoising algorithms with respect to realistic camera noise: we describe a new camera noise model that includes the full processing chain of a single sensor camera. We determine the visual quality of noisy and denoised test sequences using a subjective test with 18 participants. We show that the noise characteristics have a significant effect on visual quality. Quality metrics, which are required to compare denoising results, are applied, and we first evaluate the performance of 12 full-reference metrics. As no-reference metrics are especially useful for parameter tuning, we additionally evaluate five no-reference metrics with our realistic test data. We conclude that a more realistic noise model should be used in future research to improve the quality estimation of digital images and videos and to improve the research on denoising algorithms.
Download citation as [.bib File]
@article{Seybold-IJSC2014,
title = {Beyond Standard Noise Models: Evaluating Denoising Algorithms with Respect to Realistic Camera Noise},
author = {Tamara Seybold and Marion Knopp and Christian Keimel and Walter Stechele},
doi = {10.1142/S1793351X14400029},
year = {2014},
date = {2014-01-01},
journal = {International Journal of Semantic Computing},
volume = {08},
number = {02},
pages = {145-167},
abstract = {The development and tuning of denoising algorithms is usually based on readily processed test images that are artificially degraded with additive white Gaussian noise (AWGN). While AWGN allows us to easily generate test data in a repeatable manner, it does not reflect the noise characteristics in a real digital camera. Realistic camera noise is signal-dependent and spatially correlated due to the demosaicking step required to obtain full-color images. Hence, the noise characteristic is fundamentally different from AWGN. Using such unrealistic data to test, optimize and compare denoising algorithms may lead to incorrect parameter tuning or suboptimal choices in research on denoising algorithms.

In this paper, we therefore propose an approach to evaluate denoising algorithms with respect to realistic camera noise: we describe a new camera noise model that includes the full processing chain of a single sensor camera. We determine the visual quality of noisy and denoised test sequences using a subjective test with 18 participants. We show that the noise characteristics have a significant effect on visual quality. Quality metrics, which are required to compare denoising results, are applied, and we first evaluate the performance of 12 full-reference metrics. As no-reference metrics are especially useful for parameter tuning, we additionally evaluate five no-reference metrics with our realistic test data. We conclude that a more realistic noise model should be used in future research to improve the quality estimation of digital images and videos and to improve the research on denoising algorithms.},
keywords = {},
pubstate = {},
tppubtype = {article}
}

Incollections

Hoßfeld, T., Keimel, C.: "Crowdsourcing in QoE Evaluation", Möller, Sebastian; Raake, Alexander (Ed.): Quality of Experience, pp. 315-327, Springer International Publishing, 2014, ISBN: 978-3-319-02680-0.
[Abstract] [BibTeX] [DOI]

Crowdsourcing enables new possibilities for QoE evaluation by moving the evaluation task from the traditional laboratory environment into the Internet, allowing researchers to easily access a global pool of subjects for the evaluation task. This makes it not only possible to include a more diverse population and real-life environments into the evaluation, but also reduces the turn-around time and increases the number of subjects participating in an evaluation campaign significantly by circumventing bottle-necks in traditional laboratory setup. In order to utilise these advantages, the differences between laboratory-based and crowd-based QoE evaluation must be considered and we therefore discuss both these differences and their impact on the QoE evaluation in this chapter.
Download citation as [.bib File]
@incollection{Hossfeld-SpringerQoEBook2014,
title = {Crowdsourcing in QoE Evaluation},
author = {Tobias Hoßfeld and Christian Keimel},
editor = {Sebastian Möller and Alexander Raake},
doi = {10.1007/978-3-319-02681-7_21},
isbn = {978-3-319-02680-0},
year = {2014},
date = {2014-01-01},
booktitle = {Quality of Experience},
pages = {315-327},
publisher = {Springer International Publishing},
series = {T-Labs Series in Telecommunication Services},
abstract = {Crowdsourcing enables new possibilities for QoE evaluation by moving the evaluation task from the traditional laboratory environment into the Internet, allowing researchers to easily access a global pool of subjects for the evaluation task. This makes it not only possible to include a more diverse population and real-life environments into the evaluation, but also reduces the turn-around time and increases the number of subjects participating in an evaluation campaign significantly by circumventing bottle-necks in traditional laboratory setup. In order to utilise these advantages, the differences between laboratory-based and crowd-based QoE evaluation must be considered and we therefore discuss both these differences and their impact on the QoE evaluation in this chapter.},
howpublished = {Full text available from publisher},
keywords = {},
pubstate = {},
tppubtype = {incollection}
}

Inproceedings

Horch, C., Habigt, J., Keimel, C., Diepold, K.: "Evaluation of video quality fluctuations using pattern categorisation", Quality of Multimedia Experience (QoMEX), 2014 Sixth International Workshop on, pp. 117-122, 2014.
[Abstract] [PDF] [BibTeX] [DOI]

Fluctuations of video quality over time can have a significant influence on the overall perceived quality as represented by the QoE. Existing methodologies for subjective video quality assessment, however, are often not suitable for the evaluation of these quality fluctuations, especially if they occur within very small time frames. In this contribution, we therefore propose a new method, VIQPAC, which addresses this shortcoming by using a pattern categorisation approach. Instead of requiring the subjects to provide a continuous quality evaluation, the subjects assess the overall quality impression and the strength of the quality fluctuation, combined with a categorisation of the encountered fluctuation pattern. This allows us to determine the fluctuation dependent temporal changes in the quality. The results show that VIQPAC is able to capture the pattern and strength of quality fluctuations, allowing for a proper description of the temporal quality changes within a video sequence.
Download citation as [.bib File]
@inproceedings{Horch-QoMEX2014,
title = {Evaluation of video quality fluctuations using pattern categorisation},
author = {Clemens Horch and Julian Habigt and Christian Keimel and Klaus Diepold},
doi = {10.1109/QoMEX.2014.6982306},
year = {2014},
date = {2014-09-01},
booktitle = {Quality of Multimedia Experience (QoMEX), 2014 Sixth International Workshop on},
pages = {117-122},
abstract = {Fluctuations of video quality over time can have a significant influence on the overall perceived quality as represented by the QoE. Existing methodologies for subjective video quality assessment, however, are often not suitable for the evaluation of these quality fluctuations, especially if they occur within very small time frames. In this contribution, we therefore propose a new method, VIQPAC, which addresses this shortcoming by using a pattern categorisation approach. Instead of requiring the subjects to provide a continuous quality evaluation, the subjects assess the overall quality impression and the strength of the quality fluctuation, combined with a categorisation of the encountered fluctuation pattern. This allows us to determine the fluctuation dependent temporal changes in the quality. The results show that VIQPAC is able to capture the pattern and strength of quality fluctuations, allowing for a proper description of the temporal quality changes within a video sequence.},
keywords = {},
pubstate = {},
tppubtype = {inproceedings}
}

Hoßfeld, T., Hirth, M., Korshunov, P., Hanhart, P., Gardlo, B., Keimel, C., Timmerer, C.: "Survey of web-based crowdsourcing frameworks for subjective quality assessment", Multimedia Signal Processing (MMSP), 2014 IEEE 16th International Workshop on, pp. 1-6, 2014.
[Abstract] [PDF] [BibTeX] [DOI]

The popularity of the crowdsourcing for performing various tasks online increased significantly in the past few years. The low cost and flexibility of crowdsourcing, in particular, attracted researchers in the field of subjective multimedia evaluations and Quality of Experience (QoE). Since online assessment of multimedia content is challenging, several dedicated frameworks were created to aid in the designing of the tests, including the support of the testing methodologies like ACR, DCR, and PC, setting up the tasks, training sessions, screening of the subjects, and storage of the resulted data. In this paper, we focus on the web-based frameworks for multimedia quality assessments that support commonly used crowdsourcing platforms such as Amazon Mechanical Turk and Microworkers. We provide a detailed overview of the crowdsourcing frameworks and evaluate them to aid researchers in the field of QoE assessment in the selection of frameworks and crowdsourcing platforms that are adequate for their experiments.
Download citation as [.bib File]
@inproceedings{Hossfeld-MMSP2014,
title = {Survey of web-based crowdsourcing frameworks for subjective quality assessment},
author = {Tobias Hoßfeld and Matthias Hirth and Pavel Korshunov and Phillipe Hanhart and Bruno Gardlo and Christian Keimel and Christian Timmerer},
doi = {10.1109/MMSP.2014.6958831},
year = {2014},
date = {2014-09-01},
booktitle = {Multimedia Signal Processing (MMSP), 2014 IEEE 16th International Workshop on},
pages = {1-6},
abstract = {The popularity of the crowdsourcing for performing various tasks online increased significantly in the past few years. The low cost and flexibility of crowdsourcing, in particular, attracted researchers in the field of subjective multimedia evaluations and Quality of Experience (QoE). Since online assessment of multimedia content is challenging, several dedicated frameworks were created to aid in the designing of the tests, including the support of the testing methodologies like ACR, DCR, and PC, setting up the tasks, training sessions, screening of the subjects, and storage of the resulted data. In this paper, we focus on the web-based frameworks for multimedia quality assessments that support commonly used crowdsourcing platforms such as Amazon Mechanical Turk and Microworkers. We provide a detailed overview of the crowdsourcing frameworks and evaluate them to aid researchers in the field of QoE assessment in the selection of frameworks and crowdsourcing platforms that are adequate for their experiments.},
keywords = {},
pubstate = {},
tppubtype = {inproceedings}
}

Calixto, G., Keimel, C., de Costa, L., Merkel, K., Zuffo, M.: "Analysis of coexistence of Ginga and HbbTV in DVB and ISDB-Tb", Consumer Electronics - Berlin (ICCE-Berlin), 2014 IEEE Fourth International Conference on, pp. 83-87, 2014.
[Abstract] [PDF] [BibTeX] [DOI]

In this paper, we examine the possible coexistence of Ginga and HbbTV as interactive TV (iTV) systems in the Brazilian and European broadcasting systems, ISDB-Tb and DVB, respectively. We compare both systems architectures, in particular with respect to their functional modules. Our analysis provides the necessary information to assess the possibilities of a joint framework that includes both Ginga and HbbTV, consequently leading to a potential foundation of a system that supports both Ginga and HbbTV applications.
Download citation as [.bib File]
@inproceedings{Calixto-ICCE-Berlin2014,
title = {Analysis of coexistence of Ginga and HbbTV in DVB and ISDB-Tb},
author = {Gustavo Moreira Calixto and Christian Keimel and Laisa Caroline de Paula Costa and Klaus Merkel and Marcelo Knorich Zuffo},
doi = {10.1109/ICCE-Berlin.2014.7034224},
year = {2014},
date = {2014-01-01},
booktitle = {Consumer Electronics - Berlin (ICCE-Berlin), 2014 IEEE Fourth International Conference on},
pages = {83-87},
abstract = {In this paper, we examine the possible coexistence of Ginga and HbbTV as interactive TV (iTV) systems in the Brazilian and European broadcasting systems, ISDB-Tb and DVB, respectively. We compare both systems architectures, in particular with respect to their functional modules. Our analysis provides the necessary information to assess the possibilities of a joint framework that includes both Ginga and HbbTV, consequently leading to a potential foundation of a system that supports both Ginga and HbbTV applications.},
keywords = {},
pubstate = {},
tppubtype = {inproceedings}
}

Technical Reports

Hoßfeld, T., Hirth, M., Redi, J., Mazza, F., Korshunov, P., Naderi, B., Seufert, M., Gardlo, B., Egger, S., Keimel, C.: "Best Practices and Recommendations for Crowdsourced QoE - Lessons learned from the Qualinet Task Force ''Crowdsourcing''", 2014.
[Abstract] [PDF] [BibTeX]

Crowdsourcing is a popular approach that outsources tasks via the Internet to a large number of users. Commercial crowdsourcing platforms provide a global pool of users employed for perform-ing short and simple online tasks. For quality assessment of multimedia services and applications, crowdsourcing enables new possibilities by moving the subjective test into the crowd resulting in larger diversity of the test subjects, faster turnover of test campaigns, and reduced costs due to low reimbursement costs of the participants. Further, crowdsourcing allows easily addressing additional features like real-life environments. Crowdsourced quality assessment however is not a straight-forward implementation of existing subjective testing methodologies in an Internet-based environment. Additional challenges and differences to lab studies occur, in conceptual, technical, and motivational areas [9, 25, 26]. For example, the test contents need to be transmitted to the user over the Internet; test users may have low resolution screens influencing the user experience; also users may not understand the test or do not execute the test carefully resulting in unreliable data. This white paper summarizes the recommendations and best practices for crowdsourced qual-ity assessment of multimedia applications from the Qualinet Task Force on "Crowdsourcing". The European Network on Quality of Experience in Multimedia Systems and Services Qualinet (COST Action IC 1003, see www.qualinet.eu) established this task force in 2012. Since then it has grown to more then 30 members. The recommendation paper resulted from the experience in designing, implementing, and conducting crowdsourcing experiments as well as the analysis of the crowdsourced user ratings and context data. For understanding the impact of the crowdsourcing environment on QoE assessment and to derive a methodology and setup for crowdsourced QoE assessment, data from traditional lab experiments were compared with results from crowdsourc-ing experiments. Within the crowdsourcing task force, several different application domains and scientific questions were considered, among others: • video and image quality in general, • QoE for HTTP streaming [31, 32] and HTTP adaptive streaming [19, 30], • selfie portrait images perception in a recruitment context [10], • privacy in HDR images and video [39, 20, 36], • compression of HDR images [37] [38], • evaluation of 3D video [38], • image recognizability and aesthetic appeal [12, 13], • multidimensional modeling of web QoE [14], • QoE factors of cloud storage services [21], • enabling eye tracking experiments using web technologies [41]. From a crowdsourcing perspective, the following mechanisms and approaches were investigated which are relevant to understand for crowdsourced quality assessment.
Download citation as [.bib File]
@techreport{Hossfeld-QualinetCrowdsourcing-WHP,
title = {Best Practices and Recommendations for Crowdsourced QoE - Lessons learned from the Qualinet Task Force ''Crowdsourcing''},
author = {Tobias Hoßfeld and Matthias Hirth and Judith Redi and Filippo Mazza and Pavel Korshunov and Babak Naderi and Michael Seufert and Bruno Gardlo and Sebastian Egger and Christian Keimel},
url = {https://hal.archives-ouvertes.fr/hal-01078761},
year = {2014},
date = {2014-01-01},
abstract = {Crowdsourcing is a popular approach that outsources tasks via the Internet to a large number of users. Commercial crowdsourcing platforms provide a global pool of users employed for perform-ing short and simple online tasks. For quality assessment of multimedia services and applications, crowdsourcing enables new possibilities by moving the subjective test into the crowd resulting in larger diversity of the test subjects, faster turnover of test campaigns, and reduced costs due to low reimbursement costs of the participants. Further, crowdsourcing allows easily addressing additional features like real-life environments. Crowdsourced quality assessment however is not a straight-forward implementation of existing subjective testing methodologies in an Internet-based environment. Additional challenges and differences to lab studies occur, in conceptual, technical, and motivational areas [9, 25, 26]. For example, the test contents need to be transmitted to the user over the Internet; test users may have low resolution screens influencing the user experience; also users may not understand the test or do not execute the test carefully resulting in unreliable data. This white paper summarizes the recommendations and best practices for crowdsourced qual-ity assessment of multimedia applications from the Qualinet Task Force on "Crowdsourcing". The European Network on Quality of Experience in Multimedia Systems and Services Qualinet (COST Action IC 1003, see www.qualinet.eu) established this task force in 2012. Since then it has grown to more then 30 members. The recommendation paper resulted from the experience in designing, implementing, and conducting crowdsourcing experiments as well as the analysis of the crowdsourced user ratings and context data. For understanding the impact of the crowdsourcing environment on QoE assessment and to derive a methodology and setup for crowdsourced QoE assessment, data from traditional lab experiments were compared with results from crowdsourc-ing experiments. Within the crowdsourcing task force, several different application domains and scientific questions were considered, among others: • video and image quality in general, • QoE for HTTP streaming [31, 32] and HTTP adaptive streaming [19, 30], • selfie portrait images perception in a recruitment context [10], • privacy in HDR images and video [39, 20, 36], • compression of HDR images [37] [38], • evaluation of 3D video [38], • image recognizability and aesthetic appeal [12, 13], • multidimensional modeling of web QoE [14], • QoE factors of cloud storage services [21], • enabling eye tracking experiments using web technologies [41]. From a crowdsourcing perspective, the following mechanisms and approaches were investigated which are relevant to understand for crowdsourced quality assessment.},
note = {Lessons learned from the Qualinet Task Force ''Crowdsourcing'' COST Action IC1003 European Network on Quality of Experience in Multimedia Systems and Services (QUALINET)},
keywords = {},
pubstate = {},
tppubtype = {techreport}
}

2013

Incollections

Keimel, C.: "Crowdsourcing of Multimedia QoE Subjective Experiments", Hoßfeld, Tobias; Tran-Gia, Phuoc; Vukovic, Maja (Ed.): Dagstuhl Reports - Crowdsourcing: From Theory to Practice and Long-Term Perspectives (Dagstuhl Seminar 13361), 3 (9), pp. 1–33, Schloss Dagstuhl--Leibniz-Zentrum fuer Informatik, Dagstuhl, Germany, 2013, ISSN: 2192-5283.
[Abstract] [PDF] [BibTeX] [DOI]

Download citation as [.bib File]
@incollection{Hossfeld-Dagstuhl2013,
title = {Crowdsourcing of Multimedia QoE Subjective Experiments},
author = {Christian Keimel},
editor = {Tobias Hoßfeld and Phuoc Tran-Gia and Maja Vukovic},
url = {http://drops.dagstuhl.de/opus/volltexte/2013/4354},
doi = {10.4230/DagRep.3.9.1},
issn = {2192-5283},
year = {2013},
date = {2013-01-01},
booktitle = {Dagstuhl Reports - Crowdsourcing: From Theory to Practice and Long-Term Perspectives (Dagstuhl Seminar 13361)},
journal = {Dagstuhl Reports},
volume = {3},
number = {9},
pages = {1--33},
publisher = {Schloss Dagstuhl--Leibniz-Zentrum fuer Informatik},
address = {Dagstuhl, Germany},
keywords = {},
pubstate = {},
tppubtype = {incollection}
}

Inproceedings

Seybold, T., Keimel, C., Knopp, M., Stechele, W.: "Towards an Evaluation of Denoising Algorithms with Respect to Realistic Camera Noise", Multimedia (ISM), 2013 IEEE International Symposium on, pp. 203-210, 2013, ISBN: 978-0-7695-5140-1.
[Abstract] [PDF] [BibTeX] [DOI]

The development and tuning of denoising algorithms is usually based on readily processed test images that are artificially degraded with additive white Gaussian noise (AWGN). While AWGN allows us to easily generate test data in a repeatable manner, it does not reflect the noise characteristics in a real digital camera. Realistic camera noise is signal-dependent and spatially correlated due to the demosaicking step required to obtain full-color images. Hence, the noise characteristic is fundamentally different from AWGN. Using such unrealistic data to test, optimize and compare denoising algorithms may lead to incorrect parameter tuning or sub optimal choices in research on denoising algorithms. In this paper, we therefore propose an approach to evaluate denoising algorithms with respect to realistic camera noise: we describe a new camera noise model that includes the full processing chain of a single sensor camera. We determine the visual quality of noisy and denoised test sequences using a subjective test with 18 participants. We show that the noise characteristics have a significant effect on visual quality. Quality metrics, which are required to compare denoising results, are applied, and we evaluate the performance of 10 full-reference metrics and one no-reference metric with our realistic test data. We conclude that a more realistic noise model should be used in future research to improve the quality estimation of digital images and videos and to improve the research on denoising algorithms.
Download citation as [.bib File]
@inproceedings{Seybold-ISM2013,
title = {Towards an Evaluation of Denoising Algorithms with Respect to Realistic Camera Noise},
author = {Tamara Seybold and Christian Keimel and Marion Knopp and Walter Stechele},
doi = {10.1109/ISM.2013.39},
isbn = {978-0-7695-5140-1},
year = {2013},
date = {2013-12-01},
booktitle = {Multimedia (ISM), 2013 IEEE International Symposium on},
pages = {203-210},
abstract = {The development and tuning of denoising algorithms is usually based on readily processed test images that are artificially degraded with additive white Gaussian noise (AWGN). While AWGN allows us to easily generate test data in a repeatable manner, it does not reflect the noise characteristics in a real digital camera. Realistic camera noise is signal-dependent and spatially correlated due to the demosaicking step required to obtain full-color images. Hence, the noise characteristic is fundamentally different from AWGN. Using such unrealistic data to test, optimize and compare denoising algorithms may lead to incorrect parameter tuning or sub optimal choices in research on denoising algorithms. In this paper, we therefore propose an approach to evaluate denoising algorithms with respect to realistic camera noise: we describe a new camera noise model that includes the full processing chain of a single sensor camera. We determine the visual quality of noisy and denoised test sequences using a subjective test with 18 participants. We show that the noise characteristics have a significant effect on visual quality. Quality metrics, which are required to compare denoising results, are applied, and we evaluate the performance of 10 full-reference metrics and one no-reference metric with our realistic test data. We conclude that a more realistic noise model should be used in future research to improve the quality estimation of digital images and videos and to improve the research on denoising algorithms.},
keywords = {},
pubstate = {},
tppubtype = {inproceedings}
}

Redi, J., Hoßfeld, T., Korshunov, P., Mazza, F., Povoa, I., Keimel, C.: "Crowdsourcing-based Multimedia Subjective Evaluations: A Case Study on Image Recognizability and Aesthetic Appeal", Proceedings of the 2nd ACM International Workshop on Crowdsourcing for Multimedia, pp. 29-34, ACM, Barcelona, Spain, 2013, ISBN: 978-1-4503-2396-3.
[Abstract] [PDF] [BibTeX] [DOI]

Research on Quality of Experience (QoE) heavily relies on subjective evaluations of media. An important aspect of QoE concerns modeling and quantifying the subjective notions of 'beauty' (aesthetic appeal) and 'something well-known' (content recognizability), which are both subject to cultural and social effects. Crowdsourcing, which allows employing people worldwide to perform short and simple tasks via online platforms, can be a great tool for performing subjective studies in a time and cost-effective way. On the other hand, the crowdsourcing environment does not allow for the degree of experimental control which is necessary to guarantee reliable subjective data. To validate the use of crowdsourcing for QoE assessments, in this paper, we evaluate aesthetic appeal and recognizability of images using the Microworkers crowdsourcing platform and compare the outcomes with more conventional evaluations conducted in a controlled lab environment. We find high correlation between crowdsourcing and lab scores for recognizability but not for aesthetic appeal, indicating that crowdsourcing can be used for QoE subjective assessments as long as the workers' tasks are designed with extreme care to avoid misinterpretations.
Download citation as [.bib File]
@inproceedings{Redi-CrowdMM2013,
title = {Crowdsourcing-based Multimedia Subjective Evaluations: A Case Study on Image Recognizability and Aesthetic Appeal},
author = {Judith Alice Redi and Tobias Hoßfeld and Pavel Korshunov and Filippo Mazza and Isabel Povoa and Christian Keimel},
doi = {10.1145/2506364.2506368},
isbn = {978-1-4503-2396-3},
year = {2013},
date = {2013-10-01},
booktitle = {Proceedings of the 2nd ACM International Workshop on Crowdsourcing for Multimedia},
pages = {29-34},
publisher = {ACM},
address = {Barcelona, Spain},
series = {CrowdMM '13},
abstract = {Research on Quality of Experience (QoE) heavily relies on subjective evaluations of media. An important aspect of QoE concerns modeling and quantifying the subjective notions of 'beauty' (aesthetic appeal) and 'something well-known' (content recognizability), which are both subject to cultural and social effects. Crowdsourcing, which allows employing people worldwide to perform short and simple tasks via online platforms, can be a great tool for performing subjective studies in a time and cost-effective way. On the other hand, the crowdsourcing environment does not allow for the degree of experimental control which is necessary to guarantee reliable subjective data. To validate the use of crowdsourcing for QoE assessments, in this paper, we evaluate aesthetic appeal and recognizability of images using the Microworkers crowdsourcing platform and compare the outcomes with more conventional evaluations conducted in a controlled lab environment. We find high correlation between crowdsourcing and lab scores for recognizability but not for aesthetic appeal, indicating that crowdsourcing can be used for QoE subjective assessments as long as the workers' tasks are designed with extreme care to avoid misinterpretations.},
keywords = {},
pubstate = {},
tppubtype = {inproceedings}
}

Horch, C., Keimel, C., Habigt, J., Diepold, K.: "Length-independent refinement of video quality metrics based on multiway data analysis", Image Processing (ICIP), 2013 20th IEEE International Conference on, pp. 44-48, 2013.
[Abstract] [PDF] [BibTeX] [DOI]

In previous publications it has been shown that no-reference video quality metrics based on a data analysis approach rather than on modeling the human visual system lead to very promising results and outperform many well-known full-reference metrics. Furthermore, the results improve when taking the temporal structure of the video sequence into account by using multiway analysis methods. This contribution shows a way of refining these multiway quality metrics in order to make them more suitable for real-life applications and maintaining the performance at the same time. Additionally, our results confirm the validity of H.264/AVC bitstream no-reference quality metrics using multiway PLSR by evaluating this concept on an additional dataset.
Download citation as [.bib File]
@inproceedings{Horch-ICIP2013,
title = {Length-independent refinement of video quality metrics based on multiway data analysis},
author = {Clemens Horch and Christian Keimel and Julian Habigt and Klaus Diepold},
doi = {10.1109/ICIP.2013.6738010},
year = {2013},
date = {2013-09-01},
booktitle = {Image Processing (ICIP), 2013 20th IEEE International Conference on},
pages = {44-48},
abstract = {In previous publications it has been shown that no-reference video quality metrics based on a data analysis approach rather than on modeling the human visual system lead to very promising results and outperform many well-known full-reference metrics. Furthermore, the results improve when taking the temporal structure of the video sequence into account by using multiway analysis methods. This contribution shows a way of refining these multiway quality metrics in order to make them more suitable for real-life applications and maintaining the performance at the same time. Additionally, our results confirm the validity of H.264/AVC bitstream no-reference quality metrics using multiway PLSR by evaluating this concept on an additional dataset.},
keywords = {},
pubstate = {},
tppubtype = {inproceedings}
}

Redl, A., Keimel, C., Diepold, K.: "Saliency based video quality prediction using multi-way data analysis", Quality of Multimedia Experience (QoMEX), 2013 Fifth International Workshop on, pp. 188-193, 2013.
[Abstract] [PDF] [BibTeX] [DOI]

Saliency information allows us to determine which parts of an image or video frame attracts the focus of the observer and thus where distortions will be more obvious. Using this knowledge and saliency thresholds, we therefore combine the saliency information generated by a computational model and the features extracted from the H.264/AVC bitstream, and use the resulting saliency-weighted features in the design of a video quality metric with multi-way data analysis. We used two different multi-way methods, the two dimensional principal component regression (2D-PCR) and multi-way partial least squares regression (PLSR) in the design of a no-reference video quality metric, where the different saliency levels are considered as an additional direction. Our results show that the consideration of the the saliency information leads to more stable models with less parameters in the model and thus the prediction performance increases compared to metrics without saliency information for the same number of parameters.
Download citation as [.bib File]
@inproceedings{Redl-Qomex2013,
title = {Saliency based video quality prediction using multi-way data analysis},
author = {Arne. Redl and Christian Keimel and Klaus Diepold},
doi = {10.1109/QoMEX.2013.6603235},
year = {2013},
date = {2013-07-01},
booktitle = {Quality of Multimedia Experience (QoMEX), 2013 Fifth International Workshop on},
pages = {188-193},
abstract = {Saliency information allows us to determine which parts of an image or video frame attracts the focus of the observer and thus where distortions will be more obvious. Using this knowledge and saliency thresholds, we therefore combine the saliency information generated by a computational model and the features extracted from the H.264/AVC bitstream, and use the resulting saliency-weighted features in the design of a video quality metric with multi-way data analysis. We used two different multi-way methods, the two dimensional principal component regression (2D-PCR) and multi-way partial least squares regression (PLSR) in the design of a no-reference video quality metric, where the different saliency levels are considered as an additional direction. Our results show that the consideration of the the saliency information leads to more stable models with less parameters in the model and thus the prediction performance increases compared to metrics without saliency information for the same number of parameters.},
keywords = {},
pubstate = {},
tppubtype = {inproceedings}
}

Keimel, C., Pangerl, C., Diepold, K.: "Comparison of lossless video codecs for crowd-based quality assessment on tablets", Seventh International Workshop on Video Processing and Quality Metrics for Consumer Electronics - VPQM 2013, pp. 37-41, 2013.
[Abstract] [PDF] [BibTeX]

Video quality evaluation with subjective testing is both time consuming and expensive. A promising new approach to traditional testing is the so-called crowdsourcing, moving the testing effort into the Internet. The advantages of this approach are not only the access to a larger and more diverse pool of test subjects, but also the significant reduction of the financial burden. Extending this approach to tablets, allows us not only to assess video quality in a realistic environment for an ever more important use case, but also provides us with a well-defined hardware platform, eliminating on of the main drawbacks of crowdsourced video quality assessment. One prerequisite, however, is the support of lossless coding on the used tablets. We therefore examine in this contribution the performance of lossless video codecs on the iPad platform. Our results show, that crowdbased video testing is already feasible for CIF-sized videos on tablets, but also that there may be limits for higher resolution videos.
Download citation as [.bib File]
@inproceedings{Keimel-VPQM2013,
title = {Comparison of lossless video codecs for crowd-based quality assessment on tablets},
author = {Christian Keimel and Christopher Pangerl and Klaus Diepold},
url = {http://www.vpqm.org/},
year = {2013},
date = {2013-01-01},
booktitle = {Seventh International Workshop on Video Processing and Quality Metrics for Consumer Electronics - VPQM 2013},
pages = {37-41},
abstract = {Video quality evaluation with subjective testing is both time consuming and expensive. A promising new approach to traditional testing is the so-called crowdsourcing, moving the testing effort into the Internet. The advantages of this approach are not only the access to a larger and more diverse pool of test subjects, but also the significant reduction of the financial burden. Extending this approach to tablets, allows us not only to assess video quality in a realistic environment for an ever more important use case, but also provides us with a well-defined hardware platform, eliminating on of the main drawbacks of crowdsourced video quality assessment. One prerequisite, however, is the support of lossless coding on the used tablets. We therefore examine in this contribution the performance of lossless video codecs on the iPad platform. Our results show, that crowdbased video testing is already feasible for CIF-sized videos on tablets, but also that there may be limits for higher resolution videos.},
keywords = {},
pubstate = {},
tppubtype = {inproceedings}
}

Technical Reports

Brunnström, K., Beker, S., Moor, K., Dooms, A., Egger, S., Garcia, M., Hossfeld, T., Jumisko-Pyykkö, S., Keimel, C., Larabi, C., Lawlor, B., Callet, P., Möller, S., Pereira, F., Pereira, M., Perkis, A., Pibernik, J., Pinheiro, A., Raake, A., Reichl, P., Reiter, U., Schatz, R., Schelkens, P., Skorin-Kapov, L., Strohmeier, D., Timmerer, C., Varela, M., Wechsung, I., You, J., Zgank, A.: "Qualinet White Paper on Definitions of Quality of Experience", 2013.
[Abstract] [PDF] [BibTeX]

This White Paper is a contribution of the European Network on Quality of Experience in Multimedia Systems and Services, Qualinet (COST Action IC 1003, see www.qualinet.eu), to the scientific discussion about the term "Quality of Experience" (QoE) and its underlying concepts. It resulted from the need to agree on a working definition for this term which facilitates the communication of ideas within a multidisciplinary group, where a joint interest around multimedia communication systems exists, however approached from different perspectives. Thus, the concepts and ideas cited in this paper mainly refer to the Quality of Experience of multimedia communication systems, but may be helpful also for other areas where QoE is an issue. The Network of Excellence (NoE) Qualinet aims at extending the notion of network-centric Quality of Service (QoS) in multimedia systems, by relying on the concept of Quality of Experience (QoE). The main scientific objective is the development of methodologies for subjective and objective quality metrics taking into account current and new trends in multimedia communication systems as witnessed by the appearance of new types of content and interactions. A substantial scientific impact on fragmented efforts carried out in this field will be achieved by coordinating the research of European experts under the catalytic COST umbrella. The White Paper has been compiled on the basis of a first open call for ideas which was launched for the February 2012 Qualinet Meeting held in Prague, Czech Republic. The ideas were presented as short statements during that meeting, reflecting the ideas of the persons listed under the headline "Contributors" in the previous section. During the Prague meeting, the ideas have been further discussed and consolidated in the form of a general structure of the present document. An open call for authors was issued at that meeting, to which the persons listed as "Authors" in the previous section have announced their willingness to contribute in the preparation of individual sections. For each section, a coordinating author has been assigned which coordinated the writing of that section, and which is underlined in the author list preceding each section. The individual sections were then integrated and aligned by an editing group (listed as "Editors" in the previous section), and the entire document was iterated with the entire group of authors. Furthermore, the draft text was discussed with the participants of the Dagstuhl Seminar 12181 "Quality of Experience: From User Perception to Instrumental Metrics" which was held in Schloß Dagstuhl, Germany, May 1-4 2012, and a number of changes were proposed, resulting in the present document. As a result of the writing process and the large number of contributors, authors and editors, the document will not reflect the opinion of each individual person at all points. Still, we hope that it is found to be useful for everybody working in the field of Quality of Experience of multimedia communication systems, and most probably also beyond that field.
Download citation as [.bib File]
@techreport{LeCallet-QualinetQoE-WHP,
title = {Qualinet White Paper on Definitions of Quality of Experience},
author = {Kjell Brunnström and Sergio Ariel Beker and Katrien De Moor and Ann Dooms and Sebastien Egger and Marie-Neige Garcia and Tobias Hossfeld and Satu Jumisko-Pyykkö and Christian Keimel and Chaker Larabi and Bob Lawlor and Patrick Le Callet and Sebastian Möller and Fernando Pereira and Manuela Pereira and Andrew Perkis and Jesenka Pibernik and Antonio Pinheiro and Alexander Raake and Peter Reichl and Ulrich Reiter and Raimund Schatz and Peter Schelkens and Lea Skorin-Kapov and Dominik Strohmeier and Christian Timmerer and Martin Varela and Ina Wechsung and Junyong You and Andrej Zgank},
editor = {Patrick Le Callet and Sebastian Möller and Andrew Perkis},
url = {https://hal.archives-ouvertes.fr/hal-00977812},
year = {2013},
date = {2013-01-01},
abstract = {This White Paper is a contribution of the European Network on Quality of Experience in Multimedia Systems and Services, Qualinet (COST Action IC 1003, see www.qualinet.eu), to the scientific discussion about the term "Quality of Experience" (QoE) and its underlying concepts. It resulted from the need to agree on a working definition for this term which facilitates the communication of ideas within a multidisciplinary group, where a joint interest around multimedia communication systems exists, however approached from different perspectives. Thus, the concepts and ideas cited in this paper mainly refer to the Quality of Experience of multimedia communication systems, but may be helpful also for other areas where QoE is an issue. The Network of Excellence (NoE) Qualinet aims at extending the notion of network-centric Quality of Service (QoS) in multimedia systems, by relying on the concept of Quality of Experience (QoE). The main scientific objective is the development of methodologies for subjective and objective quality metrics taking into account current and new trends in multimedia communication systems as witnessed by the appearance of new types of content and interactions. A substantial scientific impact on fragmented efforts carried out in this field will be achieved by coordinating the research of European experts under the catalytic COST umbrella. The White Paper has been compiled on the basis of a first open call for ideas which was launched for the February 2012 Qualinet Meeting held in Prague, Czech Republic. The ideas were presented as short statements during that meeting, reflecting the ideas of the persons listed under the headline "Contributors" in the previous section. During the Prague meeting, the ideas have been further discussed and consolidated in the form of a general structure of the present document. An open call for authors was issued at that meeting, to which the persons listed as "Authors" in the previous section have announced their willingness to contribute in the preparation of individual sections. For each section, a coordinating author has been assigned which coordinated the writing of that section, and which is underlined in the author list preceding each section. The individual sections were then integrated and aligned by an editing group (listed as "Editors" in the previous section), and the entire document was iterated with the entire group of authors. Furthermore, the draft text was discussed with the participants of the Dagstuhl Seminar 12181 "Quality of Experience: From User Perception to Instrumental Metrics" which was held in Schloß Dagstuhl, Germany, May 1-4 2012, and a number of changes were proposed, resulting in the present document. As a result of the writing process and the large number of contributors, authors and editors, the document will not reflect the opinion of each individual person at all points. Still, we hope that it is found to be useful for everybody working in the field of Quality of Experience of multimedia communication systems, and most probably also beyond that field.},
note = {Qualinet White Paper on Definitions of Quality of Experience Output from the fifth Qualinet meeting, Novi Sad, March 12, 2013},
keywords = {},
pubstate = {},
tppubtype = {techreport}
}

2012

Inproceedings

Keimel, C., Habigt, J., Diepold, K.: "Challenges in crowd-based video quality assessment", Quality of Multimedia Experience (QoMEX), 2012 Fourth International Workshop on, pp. 13-18, 2012, ISBN: 978-1-4673-0725-3.
[Abstract] [PDF] [BibTeX] [DOI]

Video quality evaluation with subjective testing is both time consuming and expensive. A promising new approach to traditional testing is the so-called crowdsourcing, moving the testing effort into the Internet. The advantages of this approach are not only the access to a larger and more diverse pool of test subjects, but also the significant reduction of the financial burden. Recent contributions have also shown that crowd-based video quality assessment can deliver results comparable to traditional testing in some cases. In general, however, new problems arise, as no longer every test detail can be controlled, resulting in less reliable results. Therefore we will discuss in this contribution the conceptual, technical, motivational and reliability challenges that need to be addressed, before this promising approach to subjective testing can become a valid alternative to the testing in standardized environments.
Download citation as [.bib File]
@inproceedings{Keimel-QoMEX2012-ChallengesCrowdsourcing,
title = {Challenges in crowd-based video quality assessment},
author = {Christian Keimel and Julian Habigt and Klaus Diepold},
doi = {10.1109/QoMEX.2012.6263866},
isbn = {978-1-4673-0725-3},
year = {2012},
date = {2012-07-01},
booktitle = {Quality of Multimedia Experience (QoMEX), 2012 Fourth International Workshop on},
pages = {13-18},
abstract = {Video quality evaluation with subjective testing is both time consuming and expensive. A promising new approach to traditional testing is the so-called crowdsourcing, moving the testing effort into the Internet. The advantages of this approach are not only the access to a larger and more diverse pool of test subjects, but also the significant reduction of the financial burden. Recent contributions have also shown that crowd-based video quality assessment can deliver results comparable to traditional testing in some cases. In general, however, new problems arise, as no longer every test detail can be controlled, resulting in less reliable results. Therefore we will discuss in this contribution the conceptual, technical, motivational and reliability challenges that need to be addressed, before this promising approach to subjective testing can become a valid alternative to the testing in standardized environments.},
keywords = {},
pubstate = {},
tppubtype = {inproceedings}
}

Keimel, C., Habigt, J., Horch, C., Diepold, K.: "Video quality evaluation in the cloud", Packet Video Workshop (PV), 2012 19th International, pp. 155-160, 2012, ISBN: 978-1-4673-0299-9.
[Abstract] [PDF] [BibTeX] [DOI]

Video quality evaluation with subjective testing is both time consuming and expensive. An interesting new approach to traditional testing is the so-called crowdsourcing, moving the testing effort into the internet. The QualityCrowd framework allows codec independent, crowd-based video quality assessment with a simple web interface, usable with common web browsers. However, due to its codec independent approach, the framework can pose high bandwidth requirements on the coordinating server. We therefore propose in this contribution a cloud-based extension of the QualityCrowd framework in order to perform subjective quality evaluation as a cloud application. Moreover, this allows us to access an even larger pool of potential participants due to the improved connectivity. We compare the results from an online subjective test using this framework with the results from a test in a standardized environment. This comparison shows that QualityCrowd delivers equivalent results within the acceptable inter-lab correlation.
Download citation as [.bib File]
@inproceedings{Keimel-PV2012,
title = {Video quality evaluation in the cloud},
author = {Christian Keimel and Julian Habigt and Clemens Horch and Klaus Diepold},
doi = {10.1109/PV.2012.6229729},
isbn = {978-1-4673-0299-9},
year = {2012},
date = {2012-05-01},
booktitle = {Packet Video Workshop (PV), 2012 19th International},
pages = {155-160},
abstract = {Video quality evaluation with subjective testing is both time consuming and expensive. An interesting new approach to traditional testing is the so-called crowdsourcing, moving the testing effort into the internet. The QualityCrowd framework allows codec independent, crowd-based video quality assessment with a simple web interface, usable with common web browsers. However, due to its codec independent approach, the framework can pose high bandwidth requirements on the coordinating server. We therefore propose in this contribution a cloud-based extension of the QualityCrowd framework in order to perform subjective quality evaluation as a cloud application. Moreover, this allows us to access an even larger pool of potential participants due to the improved connectivity. We compare the results from an online subjective test using this framework with the results from a test in a standardized environment. This comparison shows that QualityCrowd delivers equivalent results within the acceptable inter-lab correlation.},
keywords = {},
pubstate = {},
tppubtype = {inproceedings}
}

Keimel, C., Redl, A., Diepold, K.: "The TUM High Definition Video Data Sets", Fourth International Workshop on Quality of Multimedia Experience (QoMEX 2012), pp. 91-102, 2012, ISBN: 978-1-4673-0725-3.
[Abstract] [PDF] [BibTeX] [DOI]

The research on video quality metrics depends on the results from subjective testing for both the design and development of metrics, but also for their verification. As it is often too cumbersome to conduct subjective tests, freely available data sets that include both mean opinion scores and the distorted videos are becoming ever more important. While many datasets are already widely available, the majority of these data sets focus on smaller resolutions. We therefore present in this contribution the TUM high definition datasets that include videos in both 1080p25 and 1080p50, encoded with different coding technologies and settings, H.264/AVC and Dirac, but also different presentation devices from reference monitors to home-cinema projectors. Additionally a soundtrack is provided for the home-cinema scenario. The datasets are made freely available for download under a creative commons license.
Download citation as [.bib File]
@inproceedings{Keimel-QoMEX2012-DataSets,
title = {The TUM High Definition Video Data Sets},
author = {Christian Keimel and Arne Redl and Klaus Diepold},
doi = {10.1109/QoMEX.2012.6263865},
isbn = {978-1-4673-0725-3},
year = {2012},
date = {2012-01-01},
booktitle = {Fourth International Workshop on Quality of Multimedia Experience (QoMEX 2012)},
pages = {91-102},
abstract = {The research on video quality metrics depends on the results from subjective testing for both the design and development of metrics, but also for their verification. As it is often too cumbersome to conduct subjective tests, freely available data sets that include both mean opinion scores and the distorted videos are becoming ever more important. While many datasets are already widely available, the majority of these data sets focus on smaller resolutions. We therefore present in this contribution the TUM high definition datasets that include videos in both 1080p25 and 1080p50, encoded with different coding technologies and settings, H.264/AVC and Dirac, but also different presentation devices from reference monitors to home-cinema projectors. Additionally a soundtrack is provided for the home-cinema scenario. The datasets are made freely available for download under a creative commons license.},
keywords = {},
pubstate = {},
tppubtype = {inproceedings}
}

Keimel, C., Habigt, J., Horch, C., Diepold, K.: "QualityCrowd -- A framework for crowd-based quality evaluation", Picture Coding Symposium (PCS), 2012, pp. 245-248, 2012, ISBN: 978-1-4577-2048-2.
[Abstract] [PDF] [BibTeX] [DOI]

Video quality assessment with subjective testing is both time consuming and expensive. An interesting new approach to traditional testing is the so-called crowdsourcing, moving the testing effort into the internet. We therefore propose in this contribution the QualityCrowd framework to effortlessly perform subjective quality assessment with crowdsourcing. QualityCrowd allows codec independent quality assessment with a simple web interface, usable with common web browsers. We compared the results from an online subjective test using this framework with the results from a test in a standardized environment. This comparison shows that QualityCrowd delivers equivalent results within the acceptable inter-lab correlation. While we only consider video quality in this contribution, QualityCrowd can also be used for multimodal quality assessment.
Download citation as [.bib File]
@inproceedings{Keimel-PCS2012,
title = {QualityCrowd -- A framework for crowd-based quality evaluation},
author = {Christain Keimel and Julian Habigt and Clemens Horch and Klaus Diepold},
doi = {10.1109/PCS.2012.6213338},
isbn = {978-1-4577-2048-2},
year = {2012},
date = {2012-01-01},
booktitle = {Picture Coding Symposium (PCS), 2012},
pages = {245-248},
abstract = {Video quality assessment with subjective testing is both time consuming and expensive. An interesting new approach to traditional testing is the so-called crowdsourcing, moving the testing effort into the internet. We therefore propose in this contribution the QualityCrowd framework to effortlessly perform subjective quality assessment with crowdsourcing. QualityCrowd allows codec independent quality assessment with a simple web interface, usable with common web browsers. We compared the results from an online subjective test using this framework with the results from a test in a standardized environment. This comparison shows that QualityCrowd delivers equivalent results within the acceptable inter-lab correlation. While we only consider video quality in this contribution, QualityCrowd can also be used for multimodal quality assessment.},
keywords = {},
pubstate = {},
tppubtype = {inproceedings}
}

Keimel, C., Habigt, J., KlausDiepold, .: "Hybrid No-Reference Video Quality Metric Based on Multiway PLSR", European Signal Processing Conference (EUSIPCO),2012, pp. 1244-1248, 2012, ISSN: 2219-5491.
[Abstract] [PDF] [BibTeX]

In real-life applications, no-reference metrics are more useful than full-reference metrics. To design such metrics, we apply data analysis methods to objectively measurable features and to data originating from subjective testing. Unfortunately, the information about temporal variation of quality is often lost due to the temporal pooling over all frames. Instead of using temporal pooling, we have recently designed a H.264/AVC bitstream no-reference video quality metric employing multiway Partial Least Squares Regression (PLSR), which leads to an improved prediction performance. In this contribution we will utilize multiway PLSR to design a hybrid metric that combines both bitstream-based features with pixel-based features. Our results show that the additional inclusion of the pixel-based features improves the quality prediction even further.
Download citation as [.bib File]
@inproceedings{Keimel-EUSIPCO2012,
title = {Hybrid No-Reference Video Quality Metric Based on Multiway PLSR},
author = {Christian Keimel and Julian Habigt and KlausDiepold},
issn = {2219-5491},
year = {2012},
date = {2012-01-01},
booktitle = {European Signal Processing Conference (EUSIPCO),2012},
pages = {1244-1248},
abstract = {In real-life applications, no-reference metrics are more useful than full-reference metrics. To design such metrics, we apply data analysis methods to objectively measurable features and to data originating from subjective testing. Unfortunately, the information about temporal variation of quality is often lost due to the temporal pooling over all frames. Instead of using temporal pooling, we have recently designed a H.264/AVC bitstream no-reference video quality metric employing multiway Partial Least Squares Regression (PLSR), which leads to an improved prediction performance. In this contribution we will utilize multiway PLSR to design a hybrid metric that combines both bitstream-based features with pixel-based features. Our results show that the additional inclusion of the pixel-based features improves the quality prediction even further.},
keywords = {},
pubstate = {},
tppubtype = {inproceedings}
}

Keimel, C., Redl, A., Diepold, K.: "Influence of viewing experience and stabilization phase in subjective video testing", Gaykema, Frans; Burns, Peter D (Ed.): Image Quality and System Performance IX, pp. 829313-1 - 829313-9, 2012.
[Abstract] [PDF] [BibTeX] [DOI]

In this contribution, we will examine two important aspects of subjective video quality assessment and their overall influence on the test results in detail: the participants' viewing experience and the quality range in the stabilization phase. Firstly, we examined if the previous viewing experience of participants in subjective tests influence the results. We performed a number of single- and double-stimulus tests assessing the visual quality of video material compressed with both H.264/AVC and MPEG2 not only at different quality levels and content, but also in different video formats from 576i up to 1080i. During these tests, we collected additional statistical data on the test participants. Overall, we were able to collect data from over 70 different subjects and analyse the influence of the subjects' viewing experience on the results of the tests. Secondly, we examined if the visual quality range presented in the stabilization phase of a subjective test has significant influence on the test results. Due to time constraints, it is sometimes necessary to split a test into multiple sessions representing subsets of the overall quality range. Consequently, we examine the influence of the quality range presented in the stabilization phase on the overall results, depending on the quality subsets included in the stabilization phase.
Download citation as [.bib File]
@inproceedings{Keimel-SPIE-EI2012,
title = {Influence of viewing experience and stabilization phase in subjective video testing},
author = {Christian Keimel and Arne Redl and Klaus Diepold},
editor = {Frans Gaykema and Peter D Burns},
doi = {10.1117/12.907967},
year = {2012},
date = {2012-01-01},
booktitle = {Image Quality and System Performance IX},
journal = {Image Quality and System Performance IX},
volume = {8293},
number = {8293},
pages = {829313-1 - 829313-9},
abstract = {In this contribution, we will examine two important aspects of subjective video quality assessment and their overall influence on the test results in detail: the participants' viewing experience and the quality range in the stabilization phase. Firstly, we examined if the previous viewing experience of participants in subjective tests influence the results. We performed a number of single- and double-stimulus tests assessing the visual quality of video material compressed with both H.264/AVC and MPEG2 not only at different quality levels and content, but also in different video formats from 576i up to 1080i. During these tests, we collected additional statistical data on the test participants. Overall, we were able to collect data from over 70 different subjects and analyse the influence of the subjects' viewing experience on the results of the tests. Secondly, we examined if the visual quality range presented in the stabilization phase of a subjective test has significant influence on the test results. Due to time constraints, it is sometimes necessary to split a test into multiple sessions representing subsets of the overall quality range. Consequently, we examine the influence of the quality range presented in the stabilization phase on the overall results, depending on the quality subsets included in the stabilization phase.},
keywords = {},
pubstate = {},
tppubtype = {inproceedings}
}

Redl, A., Keimel, C., Diepold, K.: "Influence of viewing device and soundtrack in HDTV on subjective video quality", Gaykema, Frans; Burns, Peter D (Ed.): Image Quality and System Performance IX, pp. 829312-1 - 829312-9, SPIE, 2012.
[Abstract] [PDF] [BibTeX] [DOI]

Today in many homes big TV screens and hifi systems are common. But is the perception of subjective video quality under professional test conditions the same as at home? Therefore we examined two things: How large is the influence of the presentation device but also the influence of the soundtrack, both in HDTV (1080p50). Previous work has shown that a difference is noticeable, but there have not been studies with consumer devices, yet. It was also shown that there is an influence of the soundtrack, but only in SDTV or lower resolutions. Therefore we conducted subjective video tests: One test with different devices, a 23-inch-reference monitor, a high quality 56-inch-LCD-TV and an HD-projector, and one test in which we presented a soundtrack on a 7.1-channel hifi system in addition to the HD-projector. The results show two things: First the test subjects had a higher quality of experience with the consumer devices than with the reference monitor, although the video quality of the reference monitor itself was rated better in an additional questionnaire and the mean opinion score (MOS). The second result is that there is no significant difference in the MOS between showing the videos on the projector with or without sound.
Download citation as [.bib File]
@inproceedings{Redl-SPIE-EI2012,
title = {Influence of viewing device and soundtrack in HDTV on subjective video quality},
author = {Arne Redl and Christian Keimel and Klaus Diepold},
editor = {Frans Gaykema and Peter D Burns},
doi = {10.1117/12.907015},
year = {2012},
date = {2012-01-01},
booktitle = {Image Quality and System Performance IX},
journal = {Image Quality and System Performance IX},
volume = {8293},
number = {1},
pages = {829312-1 - 829312-9},
publisher = {SPIE},
abstract = {Today in many homes big TV screens and hifi systems are common. But is the perception of subjective video quality under professional test conditions the same as at home? Therefore we examined two things: How large is the influence of the presentation device but also the influence of the soundtrack, both in HDTV (1080p50). Previous work has shown that a difference is noticeable, but there have not been studies with consumer devices, yet. It was also shown that there is an influence of the soundtrack, but only in SDTV or lower resolutions. Therefore we conducted subjective video tests: One test with different devices, a 23-inch-reference monitor, a high quality 56-inch-LCD-TV and an HD-projector, and one test in which we presented a soundtrack on a 7.1-channel hifi system in addition to the HD-projector. The results show two things: First the test subjects had a higher quality of experience with the consumer devices than with the reference monitor, although the video quality of the reference monitor itself was rated better in an additional questionnaire and the mean opinion score (MOS). The second result is that there is no significant difference in the MOS between showing the videos on the projector with or without sound.},
keywords = {},
pubstate = {},
tppubtype = {inproceedings}
}

2011

Journal Articles

Keimel, C., Rothbucher, M., Shen, H., Diepold, K.: "Video is a Cube", Signal Processing Magazine, IEEE, 28 (6), pp. 41-49, 2011, ISSN: 1053-5888.
[Abstract] [PDF] [BibTeX] [DOI]

Quality of experience (QoE) is becoming increasingly important in signal processing applications. In taking inspiration from chemometrics, we provide an introduction to the design of video quality metrics by using data analysis methods, which are different from traditional approaches. These methods do not necessitate a complete understanding of the human visual system (HVS). We use multidimensional data analysis, an extension of well-established data analysis techniques, allowing us to better exploit higher-dimensional data. In the case of video quality metrics, it enables us to exploit the temporal properties of video more properly; the complete three-dimensional structure of the video cube is taken into account in metrics' design. Starting with the well-known principal component analysis and an introduction to the notation of multiway arrays, we then present their multidimensional extensions, delivering better quality prediction results. Although we focus on video quality, the presented design principles can easily be adapted to other modalities and to even higher dimensional data sets as well.
Download citation as [.bib File]
@article{Keimel-SPM2011,
title = {Video is a Cube},
author = {Christian Keimel and Martin Rothbucher and Hao Shen and Klaus Diepold},
doi = {10.1109/MSP.2011.942468},
issn = {1053-5888},
year = {2011},
date = {2011-01-01},
journal = {Signal Processing Magazine, IEEE},
volume = {28},
number = {6},
pages = {41-49},
abstract = {Quality of experience (QoE) is becoming increasingly important in signal processing applications. In taking inspiration from chemometrics, we provide an introduction to the design of video quality metrics by using data analysis methods, which are different from traditional approaches. These methods do not necessitate a complete understanding of the human visual system (HVS). We use multidimensional data analysis, an extension of well-established data analysis techniques, allowing us to better exploit higher-dimensional data. In the case of video quality metrics, it enables us to exploit the temporal properties of video more properly; the complete three-dimensional structure of the video cube is taken into account in metrics' design. Starting with the well-known principal component analysis and an introduction to the notation of multiway arrays, we then present their multidimensional extensions, delivering better quality prediction results. Although we focus on video quality, the presented design principles can easily be adapted to other modalities and to even higher dimensional data sets as well.},
keywords = {},
pubstate = {},
tppubtype = {article}
}

Inproceedings

Keimel, C., Habigt, J., Klimpke, M., Diepold, K.: "Design of no-reference video quality metrics with multiway partial least squares regression", Quality of Multimedia Experience (QoMEX), 2011 Third International Workshop on, pp. 49 -54, 2011, ISBN: 978-1-4577-1334-7.
[Abstract] [PDF] [BibTeX] [DOI]

No-reference video quality metrics are becoming ever more popular, as they are more useful in real-life applications compared to full-reference metrics. One way to design such metrics is by applying data analysis methods on both objectively measurable features and data from subjective testing. Partial least squares regression (PLSR) is one such method. In order to apply such methods, however, we have to temporally pool over all frames of a video, loosing valuable information about the quality variation over time. Hence, we extend the PLSR into a higher dimensional space with multiway PLSR in this contribution and thus consider video in all its dimensions. We designed a H.264/AVC bitstream no-reference video quality metric in order to verify multiway PLSR against PLSR with respect to the prediction performance. Our results show that the inclusion of the temporal dimension with multiway PLSR improves the quality prediction and its correlation with the actual quality.
Download citation as [.bib File]
@inproceedings{Keimel-QoMEX2011,
title = {Design of no-reference video quality metrics with multiway partial least squares regression},
author = {Christian Keimel and Julian Habigt and Manuel Klimpke and Klaus Diepold},
doi = {10.1109/QoMEX.2011.6065711},
isbn = {978-1-4577-1334-7},
year = {2011},
date = {2011-09-01},
booktitle = {Quality of Multimedia Experience (QoMEX), 2011 Third International Workshop on},
pages = {49 -54},
abstract = {No-reference video quality metrics are becoming ever more popular, as they are more useful in real-life applications compared to full-reference metrics. One way to design such metrics is by applying data analysis methods on both objectively measurable features and data from subjective testing. Partial least squares regression (PLSR) is one such method. In order to apply such methods, however, we have to temporally pool over all frames of a video, loosing valuable information about the quality variation over time. Hence, we extend the PLSR into a higher dimensional space with multiway PLSR in this contribution and thus consider video in all its dimensions. We designed a H.264/AVC bitstream no-reference video quality metric in order to verify multiway PLSR against PLSR with respect to the prediction performance. Our results show that the inclusion of the temporal dimension with multiway PLSR improves the quality prediction and its correlation with the actual quality.},
keywords = {},
pubstate = {},
tppubtype = {inproceedings}
}

Keimel, C., Klimpke, M., Habigt, J., Diepold, K.: "No-reference video quality metric for HDTV based on H.264/AVC bitstream features", Image Processing (ICIP), 2011 18th IEEE International Conference on, pp. 3325-3328, 2011, ISSN: 1522-4880.
[Abstract] [PDF] [BibTeX] [DOI]

No-reference video quality metrics are becoming ever more popular, as they are more useful in real-life applications compared to full-reference metrics. Many proposed metrics extract features related to human perception from the individual video frames. Hence the video sequences have to be decoded first, before the metrics can be applied. In order to avoid decoding just for quality estimation, we therefore present in this contribution a no-reference metric for HDTV that uses features directly extracted from the H.264/AVC bitstream. We combine these features with the results from subjective tests using a data analysis approach with partial least squares regression to gain a prediction model for the visual quality. For verification, we performed a cross validation. Our results show that the proposed no-reference metric outperforms other metrics and delivers a correlation between the quality prediction and the actual quality of 0.93.
Download citation as [.bib File]
@inproceedings{Keimel-ICIP2011,
title = {No-reference video quality metric for HDTV based on H.264/AVC bitstream features},
author = {Christian Keimel and Manuel Klimpke and Julian Habigt and Klaus Diepold},
doi = {10.1109/ICIP.2011.6116383},
issn = {1522-4880},
year = {2011},
date = {2011-09-01},
booktitle = {Image Processing (ICIP), 2011 18th IEEE International Conference on},
pages = {3325-3328},
abstract = {No-reference video quality metrics are becoming ever more popular, as they are more useful in real-life applications compared to full-reference metrics. Many proposed metrics extract features related to human perception from the individual video frames. Hence the video sequences have to be decoded first, before the metrics can be applied. In order to avoid decoding just for quality estimation, we therefore present in this contribution a no-reference metric for HDTV that uses features directly extracted from the H.264/AVC bitstream. We combine these features with the results from subjective tests using a data analysis approach with partial least squares regression to gain a prediction model for the visual quality. For verification, we performed a cross validation. Our results show that the proposed no-reference metric outperforms other metrics and delivers a correlation between the quality prediction and the actual quality of 0.93.},
keywords = {},
pubstate = {},
tppubtype = {inproceedings}
}

Keimel, C., Redl, A., Diepold, K.: "Comparison of HDTV formats in a consumer environment", Farnand, Susan P; Gaykema, Frans (Ed.): pp. 786716-1 - 786716-7, 2011.
[Abstract] [PDF] [BibTeX] [DOI]

High definition television (HDTV) has become quite common in many homes. Still, there are two different formats used currently in commercial broadcasting: one interlaced format, 1080i50/60, and one progressive format, 720p50/60.There have already been quite a few contributions comparing the visual quality of these formats subjectively under common standard conditions. These conditions, however, dont necessarily represent the viewing conditions in the real-life consumer environment. In this contribution we therefore decided to do a comparison under conditions more representative of the consumer environment with respect to display and viewing conditions. Furthermore, we decided to select not specially prepared test sequences, but real-life content and coding conditions. As we were not interested in the influence of the transmission errors, we captured the sequences directly in the play-out centre of a cable network provider in both 1080i50 and 720p50. Also we captured for comparison the same content in digital PAL-SDTV. We conducted extensive subjective tests with overall 25 test subjects and a modified SSIS method. The results show that both HDTV formats outperform SDTV significantly. Although 720p50 is perceived to have a better quality than 1080i50, this difference is not significant in a statistical sense. This supports the validity of previous contributions results, gained in standard conditions, also for the real-life consumer environment.
Download citation as [.bib File]
@inproceedings{Keimel-SPIE-EI2011-HDTV,
title = {Comparison of HDTV formats in a consumer environment},
author = {Christian Keimel and Arne Redl and Klaus Diepold},
editor = {Susan P Farnand and Frans Gaykema},
doi = {10.1117/12.876781},
year = {2011},
date = {2011-01-01},
journal = {Image Quality and System Performance VIII},
volume = {7867},
pages = {786716-1 - 786716-7},
abstract = {High definition television (HDTV) has become quite common in many homes. Still, there are two different formats used currently in commercial broadcasting: one interlaced format, 1080i50/60, and one progressive format, 720p50/60.There have already been quite a few contributions comparing the visual quality of these formats subjectively under common standard conditions. These conditions, however, dont necessarily represent the viewing conditions in the real-life consumer environment. In this contribution we therefore decided to do a comparison under conditions more representative of the consumer environment with respect to display and viewing conditions. Furthermore, we decided to select not specially prepared test sequences, but real-life content and coding conditions. As we were not interested in the influence of the transmission errors, we captured the sequences directly in the play-out centre of a cable network provider in both 1080i50 and 720p50. Also we captured for comparison the same content in digital PAL-SDTV. We conducted extensive subjective tests with overall 25 test subjects and a modified SSIS method. The results show that both HDTV formats outperform SDTV significantly. Although 720p50 is perceived to have a better quality than 1080i50, this difference is not significant in a statistical sense. This supports the validity of previous contributions results, gained in standard conditions, also for the real-life consumer environment.},
keywords = {},
pubstate = {},
tppubtype = {inproceedings}
}

Keimel, C., Rothbucher, M., Diepold, K.: "Extending video quality metrics to the temporal dimension with 2D-PCR", Farnand, Susan P; Gaykema, Frans (Ed.): pp. 786713-1 - 786713-10, SPIE, 2011.
[Abstract] [PDF] [BibTeX] [DOI]

The aim of any video quality metric is to deliver a quality prediction similar to the video quality perceived by human observers. One way to design such a model of human perception is by data analysis. In this contribution we intend to extend this approach to the temporal dimension. Even though video obviously consists of spatial and temporal dimensions, the temporal aspect is often not considered well enough. Instead of including this third dimension in the model itself, the metrics are usually only applied on a frame-by-frame basis and then temporally pooled, commonly by averaging. Therefore we propose to skip the temporal pooling step and use the additional temporal dimension in the model building step of the video quality metric. We propose to use the two dimensional extension of the PCR, the 2D-PCR, in order to obtain an improved model. We conducted extensive subjective tests with different HDTV video sequences at 1920×1080 and 25 frames per seconds. For verification, we performed a cross validation to get a measure for the real-life performance of the acquired model. Finally, we will show that the direct inclusion of the temporal dimension of video into the model building improves the overall prediction accuracy of the visual quality significantly.
Download citation as [.bib File]
@inproceedings{Keimel-SPIE-EI2011-2D-PCR,
title = {Extending video quality metrics to the temporal dimension with 2D-PCR},
author = {Christian Keimel and Martin Rothbucher and Klaus Diepold},
editor = {Susan P Farnand and Frans Gaykema},
doi = {10.1117/12.872406},
year = {2011},
date = {2011-01-01},
journal = {Image Quality and System Performance VIII},
volume = {7867},
number = {1},
pages = {786713-1 - 786713-10},
publisher = {SPIE},
abstract = {The aim of any video quality metric is to deliver a quality prediction similar to the video quality perceived by human observers. One way to design such a model of human perception is by data analysis. In this contribution we intend to extend this approach to the temporal dimension. Even though video obviously consists of spatial and temporal dimensions, the temporal aspect is often not considered well enough. Instead of including this third dimension in the model itself, the metrics are usually only applied on a frame-by-frame basis and then temporally pooled, commonly by averaging. Therefore we propose to skip the temporal pooling step and use the additional temporal dimension in the model building step of the video quality metric. We propose to use the two dimensional extension of the PCR, the 2D-PCR, in order to obtain an improved model. We conducted extensive subjective tests with different HDTV video sequences at 1920×1080 and 25 frames per seconds. For verification, we performed a cross validation to get a measure for the real-life performance of the acquired model. Finally, we will show that the direct inclusion of the temporal dimension of video into the model building improves the overall prediction accuracy of the visual quality significantly.},
keywords = {},
pubstate = {},
tppubtype = {inproceedings}
}

Technical Reports

Horch, C., Keimel, C., Diepold, K.: "QualityCrowd - Crowdsourcing for Subjective Video Quality Tests", Institute for Data Processing, Technische Universität München 2011.
[Abstract] [PDF] [BibTeX]

Despite continuing research on the development of better quality metrics, subjective tests are still indispensable for the assessment of video quality. These tests are both time-consuming and expensive and require installing a suitable laboratory that fulfills the corresponding ITU recommendations. In this thesis the use of crowdsourcing in conjunction with the internetbased performing of such tests shall be examined comparing the results of such a test and the results of conventional laboratory tests. For performing this test the web-based software QualityCrowd was developed, which allows the simple planning and conducting of subjective tests. The software uses Amazon's crowdsourcing platform Mechanical Turk to assign the assessment of the videos to the crowd. Amazon provides the infrastructure for distibuting large numbers of almost any task and paying the workers afterwards. Another aspect is the evaluation of the technical issues that arise from an internet-based video test. In particular, the problems concerning the compression, delivery and playback of the videos in the participants' browsers are discussed. After considering the various possibilities, a decision in favour of lossless compression using H.264/AVC and playback with Adobe's Flash Player is taken. The gathered data show very high correlation with the data from the laboratories they are compared with. Although there are also some significant deviations, the results in general are quite promising and indicate the suitability of the use of crowdsourcing for subjective video tests. Even though the test could not be conducted publicly and the workers be paid, the costs of a test like this one are estimated. It shows that - compared to conventional laboratory tests - a clear cut in costs can be achieved.
Download citation as [.bib File]
@techreport{Horch-TR-QualityCrowd-2011,
title = {QualityCrowd - Crowdsourcing for Subjective Video Quality Tests},
author = {Clemens Horch and Christian Keimel and Klaus Diepold},
url = {https://mediatum.ub.tum.de/node?id=1082600},
year = {2011},
date = {2011-01-01},
institution = {Institute for Data Processing, Technische Universität München},
abstract = {Despite continuing research on the development of better quality metrics, subjective tests are still indispensable for the assessment of video quality. These tests are both time-consuming and expensive and require installing a suitable laboratory that fulfills the corresponding ITU recommendations. In this thesis the use of crowdsourcing in conjunction with the internetbased performing of such tests shall be examined comparing the results of such a test and the results of conventional laboratory tests. For performing this test the web-based software QualityCrowd was developed, which allows the simple planning and conducting of subjective tests. The software uses Amazon's crowdsourcing platform Mechanical Turk to assign the assessment of the videos to the crowd. Amazon provides the infrastructure for distibuting large numbers of almost any task and paying the workers afterwards. Another aspect is the evaluation of the technical issues that arise from an internet-based video test. In particular, the problems concerning the compression, delivery and playback of the videos in the participants' browsers are discussed. After considering the various possibilities, a decision in favour of lossless compression using H.264/AVC and playback with Adobe's Flash Player is taken. The gathered data show very high correlation with the data from the laboratories they are compared with. Although there are also some significant deviations, the results in general are quite promising and indicate the suitability of the use of crowdsourcing for subjective video tests. Even though the test could not be conducted publicly and the workers be paid, the costs of a test like this one are estimated. It shows that - compared to conventional laboratory tests - a clear cut in costs can be achieved.},
keywords = {},
pubstate = {},
tppubtype = {techreport}
}

2010

Inproceedings

Keimel, C., Diepold, K.: "On the use of reference monitors in subjective testing for HDTV", Quality of Multimedia Experience (QoMEX), 2010 Second International Workshop on, pp. 35-40, 2010, ISBN: 978-1-4244-6959-8.
[Abstract] [PDF] [BibTeX] [DOI]

Most international standards recommend the use of reference monitors in subjective testing for visual quality. But do we really need to use reference monitors? In order to find an answer to this question, we conducted extensive subjective tests with reference, color calibrated high quality and uncalibrated standard monitors. We not only used different HDTV sequences, but also two fundamentally different encoders: AVC/H.264 and Dirac. Our results show that using the uncalibrated standard monitor, the test subjects underestimate the visual quality compared to the reference monitor. Between the reference and a less expensive color calibrated high quality monitor, however, we were unable to find a statistically significant difference in most cases. This might be an indication that both can be used equivalently in subjective testing, although further studies will be necessary in order to get a definitive answer.
Download citation as [.bib File]
@inproceedings{Keimel-QOMEX2010,
title = {On the use of reference monitors in subjective testing for HDTV},
author = {Christian Keimel and Klaus Diepold},
doi = {10.1109/QOMEX.2010.5518305},
isbn = {978-1-4244-6959-8},
year = {2010},
date = {2010-01-01},
booktitle = {Quality of Multimedia Experience (QoMEX), 2010 Second International Workshop on},
pages = {35-40},
abstract = {Most international standards recommend the use of reference monitors in subjective testing for visual quality. But do we really need to use reference monitors? In order to find an answer to this question, we conducted extensive subjective tests with reference, color calibrated high quality and uncalibrated standard monitors. We not only used different HDTV sequences, but also two fundamentally different encoders: AVC/H.264 and Dirac. Our results show that using the uncalibrated standard monitor, the test subjects underestimate the visual quality compared to the reference monitor. Between the reference and a less expensive color calibrated high quality monitor, however, we were unable to find a statistically significant difference in most cases. This might be an indication that both can be used equivalently in subjective testing, although further studies will be necessary in order to get a definitive answer.},
keywords = {},
pubstate = {},
tppubtype = {inproceedings}
}

Keimel, C., Diepold, K., Sarkis, M.: "Improving the visual quality of AVC/H.264 by combining it with content adaptive depth map compression", Picture Coding Symposium (PCS), 2010, pp. 494-497, 2010, ISBN: 978-1-4244-7134-8.
[Abstract] [PDF] [BibTeX] [DOI]

The future of video coding for 3DTV lies in the combination of depth maps and corresponding textures. Most current video coding standards, however, are only optimized for visual quality and are not able to efficiently compress depth maps. We present in this work a content adaptive depth map meshing with tritree and entropy encoding for 3D videos. We show that this approach outperforms the intra frame prediction of AVC/H.264 for the coding of depth maps of still images. We also demonstrate by combining AVC/H.264 with our algorithm that we are able to increase the visual quality of the encoded texture on average by 6 dB. This work is currently limited to still images but an extension to intra coding of 3D video is straightforward.
Download citation as [.bib File]
@inproceedings{Keimel-PCS2010,
title = {Improving the visual quality of AVC/H.264 by combining it with content adaptive depth map compression},
author = {Christian Keimel and Klaus Diepold and Michel Sarkis},
doi = {10.1109/PCS.2010.5702545},
isbn = {978-1-4244-7134-8},
year = {2010},
date = {2010-01-01},
booktitle = {Picture Coding Symposium (PCS), 2010},
pages = {494-497},
abstract = {The future of video coding for 3DTV lies in the combination of depth maps and corresponding textures. Most current video coding standards, however, are only optimized for visual quality and are not able to efficiently compress depth maps. We present in this work a content adaptive depth map meshing with tritree and entropy encoding for 3D videos. We show that this approach outperforms the intra frame prediction of AVC/H.264 for the coding of depth maps of still images. We also demonstrate by combining AVC/H.264 with our algorithm that we are able to increase the visual quality of the encoded texture on average by 6 dB. This work is currently limited to still images but an extension to intra coding of 3D video is straightforward.},
keywords = {},
pubstate = {},
tppubtype = {inproceedings}
}

Keimel, C., Habigt, J., Habigt, T., Rothbucher, M., Diepold, K.: "Visual quality of current coding technologies at high definition IPTV bitrates", Multimedia Signal Processing (MMSP), 2010 IEEE International Workshop on, pp. 390-393, 2010, ISBN: 978-1-4244-8111-8.
[Abstract] [PDF] [BibTeX] [DOI]

High definition video over IP based networks (IPTV) has become a mainstay in today's consumer environment. In most applications, encoders conforming to the H.264/AVC standard are used. But even within one standard, often a wide range of coding tools are available that can deliver a vastly different visual quality. Therefore we evaluate in this contribution different coding technologies, using different encoder settings of H.264/AVC, but also a completely different encoder like Dirac. We cover a wide range of different bitrates from ADSL to VDSL and different content, with low and high demand on the encoders. As PSNR is not well suited to describe the perceived visual quality, we conducted extensive subject tests to determine the visual quality. Our results show that for currently common bitrates, the visual quality can be more than doubled, if the same coding technology, but different coding tools are used.
Download citation as [.bib File]
@inproceedings{Keimel-MMSP2010,
title = {Visual quality of current coding technologies at high definition IPTV bitrates},
author = {Christian Keimel and Julian Habigt and Tim Habigt and Martin Rothbucher and Klaus Diepold},
doi = {10.1109/MMSP.2010.5662052},
isbn = {978-1-4244-8111-8},
year = {2010},
date = {2010-01-01},
booktitle = {Multimedia Signal Processing (MMSP), 2010 IEEE International Workshop on},
pages = {390-393},
abstract = {High definition video over IP based networks (IPTV) has become a mainstay in today's consumer environment. In most applications, encoders conforming to the H.264/AVC standard are used. But even within one standard, often a wide range of coding tools are available that can deliver a vastly different visual quality. Therefore we evaluate in this contribution different coding technologies, using different encoder settings of H.264/AVC, but also a completely different encoder like Dirac. We cover a wide range of different bitrates from ADSL to VDSL and different content, with low and high demand on the encoders. As PSNR is not well suited to describe the perceived visual quality, we conducted extensive subject tests to determine the visual quality. Our results show that for currently common bitrates, the visual quality can be more than doubled, if the same coding technology, but different coding tools are used.},
keywords = {},
pubstate = {},
tppubtype = {inproceedings}
}

Keimel, C., Oelbaum, T., Diepold, K.: "Improving the prediction accuracy of video quality metrics", Acoustics Speech and Signal Processing (ICASSP), 2010 IEEE International Conference on, pp. 2442-2445, 2010, ISBN: 978-1-4244-4296-6.
[Abstract] [PDF] [BibTeX] [DOI]

To improve the prediction accuracy of visual quality metrics for video we propose two simple steps: temporal pooling in order to gain a set of parameters from one measured feature and a correction step using videos of known visual quality. We demonstrate this approach on the well known PSNR. Firstly, we achieve a more accurate quality prediction by replacing the mean luma PSNR by alternative PSNR-based parameters. Secondly, we exploit the almost linear relationship between the output of a quality metric and the subjectively perceived visual quality for individual video sequences. We do this by estimating the parameters of this linear relationship with the help of additionally generated videos of known visual quality. Moreover, we show that this is also true for very different coding technologies. Also we used cross validation to verify our results. Combining these two steps, we achieve for a set of four different high definition videos an increase of the Pearson correlation coefficient from 0.69 to 0.88 for PSNR, outperforming other, more sophisticated full-reference video quality metrics.
Download citation as [.bib File]
@inproceedings{Keimel-ICASSP2010,
title = {Improving the prediction accuracy of video quality metrics},
author = {Christian Keimel and Tobias Oelbaum and Klaus Diepold},
doi = {10.1109/ICASSP.2010.5496299},
isbn = {978-1-4244-4296-6},
year = {2010},
date = {2010-01-01},
booktitle = {Acoustics Speech and Signal Processing (ICASSP), 2010 IEEE International Conference on},
pages = {2442-2445},
abstract = {To improve the prediction accuracy of visual quality metrics for video we propose two simple steps: temporal pooling in order to gain a set of parameters from one measured feature and a correction step using videos of known visual quality. We demonstrate this approach on the well known PSNR. Firstly, we achieve a more accurate quality prediction by replacing the mean luma PSNR by alternative PSNR-based parameters. Secondly, we exploit the almost linear relationship between the output of a quality metric and the subjectively perceived visual quality for individual video sequences. We do this by estimating the parameters of this linear relationship with the help of additionally generated videos of known visual quality. Moreover, we show that this is also true for very different coding technologies. Also we used cross validation to verify our results. Combining these two steps, we achieve for a set of four different high definition videos an increase of the Pearson correlation coefficient from 0.69 to 0.88 for PSNR, outperforming other, more sophisticated full-reference video quality metrics.},
keywords = {},
pubstate = {},
tppubtype = {inproceedings}
}

Keimel, C., Oelbaum, T., Diepold, K.: "Improving the prediction accuracy of PSNR by simple temporal pooling", Fifth International Workshop on Video Processing and Quality Metrics for Consumer Electronics - VPQM 2010, 2010.
[Abstract] [PDF] [BibTeX]

PSNR is still one of the most often and universally used visual quality metrics. Although it is not very well suited to describe the human perception of visual quality, its simplicity and familiarity lead to its extensive use in many applications. We propose to improve the predication accuracy of PSNR by simple temporal pooling and thus not only using the mean PSNR, but also to exploit other statistical properties. In order to support this approach, we conducted extensive subjective testing of HDTV video sequences at typical bit rates for consumer and broadcasting applications. Using temporal pooling, we were able to achieve an improvement of nearly 10 % in the predication accuracy of PSNR for visual quality while not increasing the computational complexity significantly. Also this approach may be extendible to other frame-based metrics.
Download citation as [.bib File]
@inproceedings{Keimel-VPQM2010,
title = {Improving the prediction accuracy of PSNR by simple temporal pooling},
author = {Christian Keimel and Tobias Oelbaum and Klaus Diepold},
url = {http://www.vpqm.org/},
year = {2010},
date = {2010-01-01},
booktitle = {Fifth International Workshop on Video Processing and Quality Metrics for Consumer Electronics - VPQM 2010},
abstract = {PSNR is still one of the most often and universally used visual quality metrics. Although it is not very well suited to describe the human perception of visual quality, its simplicity and familiarity lead to its extensive use in many applications. We propose to improve the predication accuracy of PSNR by simple temporal pooling and thus not only using the mean PSNR, but also to exploit other statistical properties. In order to support this approach, we conducted extensive subjective testing of HDTV video sequences at typical bit rates for consumer and broadcasting applications. Using temporal pooling, we were able to achieve an improvement of nearly 10 % in the predication accuracy of PSNR for visual quality while not increasing the computational complexity significantly. Also this approach may be extendible to other frame-based metrics.},
keywords = {},
pubstate = {},
tppubtype = {inproceedings}
}

Technical Reports

Klimpke, M., Keimel, C., Diepold, K.: "Visuelle Qualitätsmetrik basierend auf der multivariaten Datenanalyse von H.264/AVC Bitstream-Features", Institute for Data Processing, Technische Universität München 2010.
[Abstract] [PDF] [BibTeX]

Download citation as [.bib File]
@techreport{Klimpke-TR-BitstreamFeatures-2010,
title = {Visuelle Qualitätsmetrik basierend auf der multivariaten Datenanalyse von H.264/AVC Bitstream-Features},
author = {Manuel Klimpke and Christian Keimel and Klaus Diepold},
url = {https://mediatum.ub.tum.de/node?id=1120198},
year = {2010},
date = {2010-11-01},
institution = {Institute for Data Processing, Technische Universität München},
keywords = {},
pubstate = {},
tppubtype = {techreport}
}

2009

Journal Articles

Oelbaum, T., Keimel, C., Diepold, K.: "Rule-Based No-Reference Video Quality Evaluation Using Additionally Coded Videos", Selected Topics in Signal Processing, IEEE Journal of, 3 (2), pp. 294-303, 2009, ISSN: 1932-4553.
[Abstract] [PDF] [BibTeX] [DOI]

This contribution presents a no-reference video quality metric, which is based on a set of simple rules that assigns a given video to one of four different content classes. The four content classes distinguish between video sequences which are coded with a very low data rate, which are sensitive to blocking effects, which are sensitive to blurring, and a general model for all other types of video sequences. The appropriate class for a given video sequence is selected based on the evaluation of feature values of an additional low-quality version of the given video, which is generated by encoding. The visual quality for a video sequence is estimated using a set of features, which includes measures for the blockiness, the blurriness, the spatial activity, and a set of additional continuity features. The way these features are combined to compute one overall quality value is determined by the feature class, to which the video has been assigned. We also propose an additional correction step for the visual quality value. The proposed metric is verified in a process, which includes visual quality values originating from subjective quality tests in combination with a cross validation approach. The presented metric significantly outperforms peak-signal-to-noise ratio as a visual quality estimator. The Pearson correlation between the estimated visual quality values and the subjective test results takes on values as high as 0.82.
Download citation as [.bib File]
@article{Oelbaum-JSTSP2009,
title = {Rule-Based No-Reference Video Quality Evaluation Using Additionally Coded Videos},
author = {Tobias Oelbaum and Christian Keimel and Klaus Diepold},
doi = {10.1109/JSTSP.2009.2015473},
issn = {1932-4553},
year = {2009},
date = {2009-04-01},
journal = {Selected Topics in Signal Processing, IEEE Journal of},
volume = {3},
number = {2},
pages = {294-303},
abstract = {This contribution presents a no-reference video quality metric, which is based on a set of simple rules that assigns a given video to one of four different content classes. The four content classes distinguish between video sequences which are coded with a very low data rate, which are sensitive to blocking effects, which are sensitive to blurring, and a general model for all other types of video sequences. The appropriate class for a given video sequence is selected based on the evaluation of feature values of an additional low-quality version of the given video, which is generated by encoding. The visual quality for a video sequence is estimated using a set of features, which includes measures for the blockiness, the blurriness, the spatial activity, and a set of additional continuity features. The way these features are combined to compute one overall quality value is determined by the feature class, to which the video has been assigned. We also propose an additional correction step for the visual quality value. The proposed metric is verified in a process, which includes visual quality values originating from subjective quality tests in combination with a cross validation approach. The presented metric significantly outperforms peak-signal-to-noise ratio as a visual quality estimator. The Pearson correlation between the estimated visual quality values and the subjective test results takes on values as high as 0.82.},
keywords = {},
pubstate = {},
tppubtype = {article}
}

Inproceedings

Keimel, C., Oelbaum, T., Diepold, K.: "Improving the verification process of video quality metrics", Quality of Multimedia Experience, 2009. QoMEx 2009. International Workshop on, pp. 121 -126, 2009, ISBN: 978-1-4244-4370-3.
[Abstract] [PDF] [BibTeX] [DOI]

The most important step in the development process of a video quality metric is its verification with regards to the subjective quality experience. Even though guidelines in the form of standards and recommendations are well known, there are still quite often shortcomings in the verification process of many metrics. In this contribution we revisit these rules, point out important details and review contributions to video quality metrics for typical shortcomings. We will highlight in detail five steps that should be followed in order to improve the overall quality of the verification process of video quality metrics: using a large and diverse data base, planing and conducting subjective tests carefully, using different data for calibration and verification of a metric, avoiding unnecessary data fitting steps, and a clear and meaningful presentation of the results. Also we will provide short examples how an improper verification might affect the results of a video quality metric.
Download citation as [.bib File]
@inproceedings{Keimel-QOMEX2009,
title = {Improving the verification process of video quality metrics},
author = {Christian Keimel and Tobias Oelbaum and Klaus Diepold},
doi = {10.1109/QOMEX.2009.5246966},
isbn = {978-1-4244-4370-3},
year = {2009},
date = {2009-07-01},
booktitle = {Quality of Multimedia Experience, 2009. QoMEx 2009. International Workshop on},
pages = {121 -126},
abstract = {The most important step in the development process of a video quality metric is its verification with regards to the subjective quality experience. Even though guidelines in the form of standards and recommendations are well known, there are still quite often shortcomings in the verification process of many metrics. In this contribution we revisit these rules, point out important details and review contributions to video quality metrics for typical shortcomings. We will highlight in detail five steps that should be followed in order to improve the overall quality of the verification process of video quality metrics: using a large and diverse data base, planing and conducting subjective tests carefully, using different data for calibration and verification of a metric, avoiding unnecessary data fitting steps, and a clear and meaningful presentation of the results. Also we will provide short examples how an improper verification might affect the results of a video quality metric.},
keywords = {},
pubstate = {},
tppubtype = {inproceedings}
}

Keimel, C., Oelbaum, T., Diepold, K.: "No-reference video quality evaluation for high-definition video", Acoustics, Speech and Signal Processing, 2009. ICASSP 2009. IEEE International Conference on, pp. 1145-1148, 2009, ISSN: 1520-6149.
[Abstract] [PDF] [BibTeX] [DOI]

A no-reference video quality metric for High-Definition video is introduced. This metric evaluates a set of simple features such as blocking or blurring, and combines those features into one parameter representing visual quality. While only comparably few base feature measurements are used, additional parameters are gained by evaluating changes for these measurements over time and using additional temporal pooling methods. To take into account the different characteristics of different video sequences, the gained quality value is corrected using a low quality version of the received video. The metric is verified using data from accurate subjective tests, and special care was taken to separate data used for calibration and verification. The proposed no-reference quality metric delivers a prediction accuracy of 0.86 when compared to subjective tests, and significantly outperforms PSNR as a quality predictor.
Download citation as [.bib File]
@inproceedings{Keimel-ICASSP2009,
title = {No-reference video quality evaluation for high-definition video},
author = {Christian Keimel and Tobias Oelbaum and Klaus Diepold},
doi = {10.1109/ICASSP.2009.4959791},
issn = {1520-6149},
year = {2009},
date = {2009-04-01},
booktitle = {Acoustics, Speech and Signal Processing, 2009. ICASSP 2009. IEEE International Conference on},
pages = {1145-1148},
abstract = {A no-reference video quality metric for High-Definition video is introduced. This metric evaluates a set of simple features such as blocking or blurring, and combines those features into one parameter representing visual quality. While only comparably few base feature measurements are used, additional parameters are gained by evaluating changes for these measurements over time and using additional temporal pooling methods. To take into account the different characteristics of different video sequences, the gained quality value is corrected using a low quality version of the received video. The metric is verified using data from accurate subjective tests, and special care was taken to separate data used for calibration and verification. The proposed no-reference quality metric delivers a prediction accuracy of 0.86 when compared to subjective tests, and significantly outperforms PSNR as a quality predictor.},
keywords = {},
pubstate = {},
tppubtype = {inproceedings}
}