Checks and balances – quality assurance, service providers and the web

QoE and QoS monitoring for pay TV services is increasingly seen as a key feature of the churn reduction toolset, while web-based services are throwing up challenges of their own. Anna Tobin reports.

There was a time when the quality control department of most major broadcasters consisted of one person tucked away in a back room somewhere watching output for any sign of poor picture or sound quality. Now that content providers have to distribute hundreds of hours of programming, some live, some pre-recorded and on-demand in several different formats to dozens of different platforms things have got a little bit more complicated.

Customers may moan when their PC delivers a few seconds of buffering before their catch-up programme arrives at it or when the box office film they have paid for has frozen on their TV, but given the technological feat of delivering all this stuff, it’s amazing that everything arrives without a hitch for the large majority of the time.
Video quality can be helped along by proactive Quality of Experience (QoE) and Quality of Service (QoS) checks and balances that are put in place by content distributors. It is difficult to quantitatively measure QoE, however, and so the focus tends to be on the more objective QoS metrics. The first is based on subjective measurements and analytics and the latter is based on objective real data.

“The subjective side is about how the picture looks, what type of pixilation is happening to the image, but if you are going to measure pixilation issues you have to measure every different picture, which is quite a challenge,” says Simen Frostad, chairman of Bridge Technologies. “The objective side looks at real data, is there packet loss, are there any detailed errors, etc. Whilst it is important to be able to measure and quantify what the users are seeing, you have to start with objective data and if you have this you have 100% reference to what is going on. So you can measure packet loss, find the error and cure it before the customer notices it.”

It is the fear of disappointing the customer that largely drives interest in this technology. Each operator is competing for eyeballs and if those eyeballs don’t like what they see they will jump ship to another provider. Superior service is becoming a key distinguishing factor in maintaining a competitive edge. All it takes is one slip up, poorly timed in the middle of a World Cup football final, and customers will leave in droves.

“In the highly competitive markets in Europe and North America, providing your customers with a high QoE is very important, as there are a multitude of delivery platforms available and each platform is competing for delivery,” says Andrew Sachs, Volicon’s vice-president of product management. In this situation, operators need to be so on top of their output that they can foresee problems even before they’ve played out, says Sachs. “In a hypercompetitive market such as the US, you have to be very proactive. Move over to India, however, and QoE isn’t such a high priority.”

Operators in highly developed multi-channel, multi-platform markets tend to be more focused on quality control than their counterparts in less developed regions where technical hitches are almost a given, because audiences in countries such as India have different expectations of quality of output. But, even in more error-tolerant markets, we will increasingly see more proactive monitoring put in place, says Seth Vermulm, application engineering manager at Sencore. “QoE monitoring is an essential part of the service provider’s toolset. You can’t do without it. If you do you risk customer churn. Without proper monitors you are driving blind and if anything goes wrong there is no easy way to find out where the problem is occurring and no simple way of remedying it quickly,” says Vermulm.

Even in the current economic climate, with delivery platforms looking to make savings, it is seen as vital that they don’t make any drastic cuts to test and measurement systems, which are often seen as an easy target, says Vermulm. “Leave an integrated overall proactive solution out of the picture and customer churn is proven to increase. Those companies that do have this in place actually tend to grow faster than those that don’t.”

Cutting back on monitoring is a false economy, says Eric Le Trehour, marketing director at Witbe. “Customer complaints generate an increase in operational costs with greater demands on people at the customer care centre and people for the truck-rolls,” he says.

Customer complaints generate an increase in operational costs with greater demands on people at the customer care centre and people for the truck-rolls.

Eric Le Trehour, Witbe

You can pick and choose where in your delivery network you want to place monitors to predict potential problems, but unless operators have a fully integrated system they are never fully covered. It’s not an area of their business where they can afford to cut corners.

“A good monitoring coverage from service creation at the headend to consumption in the home is the foundation for correct and efficient decisions,” says Mikael Dahlgren, CEO at Agama Technologies.

Most providers have now added to their networks some kind of error recovery mechanism. John Williams, director, emerging markets, at JDSU communications test and measurement business segment points out a couple of examples: “MediaRoom from Microsoft includes an error recovery approach that re-transmits to the set-top box lost packets, thus repairing, up to certain limits, network performance problems in this key area and Cisco offers a similar error recovery approach based on re-transmission of lost packets.”

Adding to the burden

As demand increases, a growing number of operators are now offering, or imminently plan to offer, IPTV services, as well as a host of catch-up and on-demand choices. And, of course, the more there is on the menu, the more likelihood there is of something going wrong. “Whenever you add an interactive service it increases the complexity of the system. There are more moving parts and more opportunities for error, so testing and monitoring becomes even more crucial,” says Volicon’s Sachs.

In a competitive market, where operators are having to fight to stop content choosing their rival’s pipes for delivery, QoS and QoE becomes a vital part of their armoury and content providers want to see proof of low margins of error.

The digitisation of cable set the quality threshold that is generally applied today, says Frostad of Bridge Technologies. “At the time of cable moving over to digital, there was an FCC recommendation that you shouldn’t have more than one noticeable artefact in the picture in the hour; if you have more than that then it isn’t broadcast quality. We see this as the same with packet loss, but there are some subjectives to this. If you are talking about a huge soccer game, for example, the threshold for putting up with even this is much lower than it would be for, say, a rerun of Cheers,” says Frostad. “We say that in a modern IPTV technology system you shouldn’t have packet loss in any part of your system as it’s a huge indication that something is wrong with the system architecture or operations.” [icitspot id=”16077″ template=”box-story”]

There are some key parameters that any IPTV system is going to be measuring against, including jitter, dropped packets, MPEG and compression. “The thresholds are often very syste-dependent,” says Vermulm at Sencore, “so it’s key to normalise the system and understand what thresholds are tolerable and what needs to be in place to prevent errors. There are no specific industry standard thresholds that need to be met; recommendations will develop but hard numbers are a long way off.”

With no standardization in this area, however, quality is difficult to measure, although you could judge by customer loyalty or the lack of it. “If people show that they are happy to pay for a service, could you say that quality of service and experience is being met?” says Sachs at Volicon. He argues that it is better to take a proactive stance so that any errors are picked up and fixed before the customer has even noticed them and had a chance to call customer service. He explains: “If we are talking about network issues, customer care shouldn’t be telling your network engineers that they have issues, they should know that already. If you have to wait until 35 people complain to customer care before your engineers react, then that is an indication that your internal processes have failed.”

In the event that a problem does filter through to the viewer and customer service calls rise, the QoE system needs to help the customer care operative turn the situation into a positive. “If a customer calls in and says ‘my Box Office is out’, the customer care operative should be able to look at the monitoring system and say ‘yes, we can see the problem and it will be fixed shortly.’ This saves them having to go through the whole tiresome questioning process that irritates customers,” says Kirk George, director of marketing at IneoQuest.

To use the data obtained from QoE and QoS checks most effectively, you need to be able to use them and apply them to all areas of your system from service creation to consumption, says Dahlgren at Agama Technologies. “Headend technicians need information on technical quality of the service creation, both real-time, such as alarms, and historical, as well as aggregated information. In the same way, staff at the NOC need filtered root causes as alarms to escalate the right situations to the right part of the organization, whilst in the customer care centre the focus is instead easy-to-understand problem isolation information for a quick problem resolution.”

To aim for or maintain market leader status, content providers need to work through dozens of different scenarios that their users can play out from their networks and ensure that they have the QoE and QoS systems in place to ensure that each of these scenarios is played out at optimum performance. As the scenarios keep developing it isn’t going to be easy to stay ahead of the game.