Visual quality assessment

2013

Inproceedings

Keimel, C., Pangerl, C., Diepold, K.: "Comparison of lossless video codecs for crowd-based quality assessment on tablets", Seventh International Workshop on Video Processing and Quality Metrics for Consumer Electronics - VPQM 2013, pp. 37-41, 2013.
[Abstract] [PDF] [BibTeX]

Video quality evaluation with subjective testing is both time consuming and expensive. A promising new approach to traditional testing is the so-called crowdsourcing, moving the testing effort into the Internet. The advantages of this approach are not only the access to a larger and more diverse pool of test subjects, but also the significant reduction of the financial burden. Extending this approach to tablets, allows us not only to assess video quality in a realistic environment for an ever more important use case, but also provides us with a well-defined hardware platform, eliminating on of the main drawbacks of crowdsourced video quality assessment. One prerequisite, however, is the support of lossless coding on the used tablets. We therefore examine in this contribution the performance of lossless video codecs on the iPad platform. Our results show, that crowdbased video testing is already feasible for CIF-sized videos on tablets, but also that there may be limits for higher resolution videos.
Download citation as [.bib File]
@inproceedings{Keimel-VPQM2013,
title = {Comparison of lossless video codecs for crowd-based quality assessment on tablets},
author = {Christian Keimel and Christopher Pangerl and Klaus Diepold},
url = {http://www.vpqm.org/},
year = {2013},
date = {2013-01-01},
booktitle = {Seventh International Workshop on Video Processing and Quality Metrics for Consumer Electronics - VPQM 2013},
pages = {37-41},
abstract = {Video quality evaluation with subjective testing is both time consuming and expensive. A promising new approach to traditional testing is the so-called crowdsourcing, moving the testing effort into the Internet. The advantages of this approach are not only the access to a larger and more diverse pool of test subjects, but also the significant reduction of the financial burden. Extending this approach to tablets, allows us not only to assess video quality in a realistic environment for an ever more important use case, but also provides us with a well-defined hardware platform, eliminating on of the main drawbacks of crowdsourced video quality assessment. One prerequisite, however, is the support of lossless coding on the used tablets. We therefore examine in this contribution the performance of lossless video codecs on the iPad platform. Our results show, that crowdbased video testing is already feasible for CIF-sized videos on tablets, but also that there may be limits for higher resolution videos.},
keywords = {},
pubstate = {},
tppubtype = {inproceedings}
}

2012

Inproceedings

Keimel, C., Redl, A., Diepold, K.: "The TUM High Definition Video Data Sets", Fourth International Workshop on Quality of Multimedia Experience (QoMEX 2012), pp. 91-102, 2012, ISBN: 978-1-4673-0725-3.
[Abstract] [PDF] [BibTeX] [DOI]

The research on video quality metrics depends on the results from subjective testing for both the design and development of metrics, but also for their verification. As it is often too cumbersome to conduct subjective tests, freely available data sets that include both mean opinion scores and the distorted videos are becoming ever more important. While many datasets are already widely available, the majority of these data sets focus on smaller resolutions. We therefore present in this contribution the TUM high definition datasets that include videos in both 1080p25 and 1080p50, encoded with different coding technologies and settings, H.264/AVC and Dirac, but also different presentation devices from reference monitors to home-cinema projectors. Additionally a soundtrack is provided for the home-cinema scenario. The datasets are made freely available for download under a creative commons license.
Download citation as [.bib File]
@inproceedings{Keimel-QoMEX2012-DataSets,
title = {The TUM High Definition Video Data Sets},
author = {Christian Keimel and Arne Redl and Klaus Diepold},
doi = {10.1109/QoMEX.2012.6263865},
isbn = {978-1-4673-0725-3},
year = {2012},
date = {2012-01-01},
booktitle = {Fourth International Workshop on Quality of Multimedia Experience (QoMEX 2012)},
pages = {91-102},
abstract = {The research on video quality metrics depends on the results from subjective testing for both the design and development of metrics, but also for their verification. As it is often too cumbersome to conduct subjective tests, freely available data sets that include both mean opinion scores and the distorted videos are becoming ever more important. While many datasets are already widely available, the majority of these data sets focus on smaller resolutions. We therefore present in this contribution the TUM high definition datasets that include videos in both 1080p25 and 1080p50, encoded with different coding technologies and settings, H.264/AVC and Dirac, but also different presentation devices from reference monitors to home-cinema projectors. Additionally a soundtrack is provided for the home-cinema scenario. The datasets are made freely available for download under a creative commons license.},
keywords = {},
pubstate = {},
tppubtype = {inproceedings}
}

Keimel, C., Redl, A., Diepold, K.: "Influence of viewing experience and stabilization phase in subjective video testing", Gaykema, Frans; Burns, Peter D (Ed.): Image Quality and System Performance IX, pp. 829313-1 - 829313-9, 2012.
[Abstract] [PDF] [BibTeX] [DOI]

In this contribution, we will examine two important aspects of subjective video quality assessment and their overall influence on the test results in detail: the participants' viewing experience and the quality range in the stabilization phase. Firstly, we examined if the previous viewing experience of participants in subjective tests influence the results. We performed a number of single- and double-stimulus tests assessing the visual quality of video material compressed with both H.264/AVC and MPEG2 not only at different quality levels and content, but also in different video formats from 576i up to 1080i. During these tests, we collected additional statistical data on the test participants. Overall, we were able to collect data from over 70 different subjects and analyse the influence of the subjects' viewing experience on the results of the tests. Secondly, we examined if the visual quality range presented in the stabilization phase of a subjective test has significant influence on the test results. Due to time constraints, it is sometimes necessary to split a test into multiple sessions representing subsets of the overall quality range. Consequently, we examine the influence of the quality range presented in the stabilization phase on the overall results, depending on the quality subsets included in the stabilization phase.
Download citation as [.bib File]
@inproceedings{Keimel-SPIE-EI2012,
title = {Influence of viewing experience and stabilization phase in subjective video testing},
author = {Christian Keimel and Arne Redl and Klaus Diepold},
editor = {Frans Gaykema and Peter D Burns},
doi = {10.1117/12.907967},
year = {2012},
date = {2012-01-01},
booktitle = {Image Quality and System Performance IX},
journal = {Image Quality and System Performance IX},
volume = {8293},
number = {8293},
pages = {829313-1 - 829313-9},
abstract = {In this contribution, we will examine two important aspects of subjective video quality assessment and their overall influence on the test results in detail: the participants' viewing experience and the quality range in the stabilization phase. Firstly, we examined if the previous viewing experience of participants in subjective tests influence the results. We performed a number of single- and double-stimulus tests assessing the visual quality of video material compressed with both H.264/AVC and MPEG2 not only at different quality levels and content, but also in different video formats from 576i up to 1080i. During these tests, we collected additional statistical data on the test participants. Overall, we were able to collect data from over 70 different subjects and analyse the influence of the subjects' viewing experience on the results of the tests. Secondly, we examined if the visual quality range presented in the stabilization phase of a subjective test has significant influence on the test results. Due to time constraints, it is sometimes necessary to split a test into multiple sessions representing subsets of the overall quality range. Consequently, we examine the influence of the quality range presented in the stabilization phase on the overall results, depending on the quality subsets included in the stabilization phase.},
keywords = {},
pubstate = {},
tppubtype = {inproceedings}
}

2011

Inproceedings

Keimel, C., Redl, A., Diepold, K.: "Comparison of HDTV formats in a consumer environment", Farnand, Susan P; Gaykema, Frans (Ed.): pp. 786716-1 - 786716-7, 2011.
[Abstract] [PDF] [BibTeX] [DOI]

High definition television (HDTV) has become quite common in many homes. Still, there are two different formats used currently in commercial broadcasting: one interlaced format, 1080i50/60, and one progressive format, 720p50/60.There have already been quite a few contributions comparing the visual quality of these formats subjectively under common standard conditions. These conditions, however, dont necessarily represent the viewing conditions in the real-life consumer environment. In this contribution we therefore decided to do a comparison under conditions more representative of the consumer environment with respect to display and viewing conditions. Furthermore, we decided to select not specially prepared test sequences, but real-life content and coding conditions. As we were not interested in the influence of the transmission errors, we captured the sequences directly in the play-out centre of a cable network provider in both 1080i50 and 720p50. Also we captured for comparison the same content in digital PAL-SDTV. We conducted extensive subjective tests with overall 25 test subjects and a modified SSIS method. The results show that both HDTV formats outperform SDTV significantly. Although 720p50 is perceived to have a better quality than 1080i50, this difference is not significant in a statistical sense. This supports the validity of previous contributions results, gained in standard conditions, also for the real-life consumer environment.
Download citation as [.bib File]
@inproceedings{Keimel-SPIE-EI2011-HDTV,
title = {Comparison of HDTV formats in a consumer environment},
author = {Christian Keimel and Arne Redl and Klaus Diepold},
editor = {Susan P Farnand and Frans Gaykema},
doi = {10.1117/12.876781},
year = {2011},
date = {2011-01-01},
journal = {Image Quality and System Performance VIII},
volume = {7867},
pages = {786716-1 - 786716-7},
abstract = {High definition television (HDTV) has become quite common in many homes. Still, there are two different formats used currently in commercial broadcasting: one interlaced format, 1080i50/60, and one progressive format, 720p50/60.There have already been quite a few contributions comparing the visual quality of these formats subjectively under common standard conditions. These conditions, however, dont necessarily represent the viewing conditions in the real-life consumer environment. In this contribution we therefore decided to do a comparison under conditions more representative of the consumer environment with respect to display and viewing conditions. Furthermore, we decided to select not specially prepared test sequences, but real-life content and coding conditions. As we were not interested in the influence of the transmission errors, we captured the sequences directly in the play-out centre of a cable network provider in both 1080i50 and 720p50. Also we captured for comparison the same content in digital PAL-SDTV. We conducted extensive subjective tests with overall 25 test subjects and a modified SSIS method. The results show that both HDTV formats outperform SDTV significantly. Although 720p50 is perceived to have a better quality than 1080i50, this difference is not significant in a statistical sense. This supports the validity of previous contributions results, gained in standard conditions, also for the real-life consumer environment.},
keywords = {},
pubstate = {},
tppubtype = {inproceedings}
}

2010

Inproceedings

Keimel, C., Diepold, K.: "On the use of reference monitors in subjective testing for HDTV", Quality of Multimedia Experience (QoMEX), 2010 Second International Workshop on, pp. 35-40, 2010, ISBN: 978-1-4244-6959-8.
[Abstract] [PDF] [BibTeX] [DOI]

Most international standards recommend the use of reference monitors in subjective testing for visual quality. But do we really need to use reference monitors? In order to find an answer to this question, we conducted extensive subjective tests with reference, color calibrated high quality and uncalibrated standard monitors. We not only used different HDTV sequences, but also two fundamentally different encoders: AVC/H.264 and Dirac. Our results show that using the uncalibrated standard monitor, the test subjects underestimate the visual quality compared to the reference monitor. Between the reference and a less expensive color calibrated high quality monitor, however, we were unable to find a statistically significant difference in most cases. This might be an indication that both can be used equivalently in subjective testing, although further studies will be necessary in order to get a definitive answer.
Download citation as [.bib File]
@inproceedings{Keimel-QOMEX2010,
title = {On the use of reference monitors in subjective testing for HDTV},
author = {Christian Keimel and Klaus Diepold},
doi = {10.1109/QOMEX.2010.5518305},
isbn = {978-1-4244-6959-8},
year = {2010},
date = {2010-01-01},
booktitle = {Quality of Multimedia Experience (QoMEX), 2010 Second International Workshop on},
pages = {35-40},
abstract = {Most international standards recommend the use of reference monitors in subjective testing for visual quality. But do we really need to use reference monitors? In order to find an answer to this question, we conducted extensive subjective tests with reference, color calibrated high quality and uncalibrated standard monitors. We not only used different HDTV sequences, but also two fundamentally different encoders: AVC/H.264 and Dirac. Our results show that using the uncalibrated standard monitor, the test subjects underestimate the visual quality compared to the reference monitor. Between the reference and a less expensive color calibrated high quality monitor, however, we were unable to find a statistically significant difference in most cases. This might be an indication that both can be used equivalently in subjective testing, although further studies will be necessary in order to get a definitive answer.},
keywords = {},
pubstate = {},
tppubtype = {inproceedings}
}