Akamai: service providers in need of more sophisticated QoE metrics

Delivering complex content such as sport or high action onto a large-screen TV via streaming requires a minimum of 6Mbps to achieve an ‘excellent’ viewer rating, whilst smartphones require between 2-3Mbps, according to a study by CDN provider Akamai with Eurofins.

For less-complex content, optimal bit rates are lower and 2-4Mbps will still deliver an excellent visual rating on large-screen TVs, according to the study.

The Akamai-Eurofins study looked at the experience of viewing different types of content on different smartphones, set-top boxes, desktop PCs, tablets and smart TVs. Researchers used SSIMplus – a Quality of Experience measure that is said to closely represent the human perception of video quality – to rank the perceived quality of the video, where anything rating above 80 out of 100 is deemed ‘excellent’, or the equivalent of HDTV.

The tests covered three different content genres – talking heads, drama and high action content. Different combinations of player, content and bitrate, were tested under both normalised and challenging network conditions.

The results showed significant variation in the bitrate required to deliver a high-definition image. The two smartphones tested achieved optimal perceptual quality scores with bitrates of less than 3Mbps but had significant perceptual quality variation between them. The smart TVs on test required encode profiles of beyond 6Mbps to achieve a similar score. Desktop PC’s_ on test required bitrates of around 5Mbps but again showed significant variation in their perceptual quality.

The set-top box tested delivered a better result under challenging conditions than the smart TV.

The study also found that smartphones delivered a consistently high-quality video stream even when network conditions were especially challenging and did this noticeably better than most of the other players tested and recommended the use of transport protocols such as UDP rather than TCP to improve stream consistency.

According to the study’s authors, delivering content at lower bitrates to avoid buffering can have a negative impact on perceived picture quality, with viewers who are used to HjDTV now expecting similar levels of quality from OTT services.

The Akamai-Eurofins study found that measures such as rebuffering ratio, video start time, and video start failures do not provide a full picture of video quality and argued that a meaningful measure of video quality for OTT services should now include a PQ score as an objective measure of picture quality in addition to the current KPIs typically used to measure quality.

The study is the second phase of a broader research project from Akamai investigating issues around the quality of IP-delivered video streams. The first phase, via a study undertaken on behalf of Akamai by Sensum into how we as viewers react to video quality, focused on how quality of experience impacts commercial success.