Testing times

A slow-down in subscriber uptake in the face of stiff competition, as well as the looming threat from OTT providers, could force IPTV operators to invest more in quality assurance. Stuart Thomson reports.

It is well-known that IPTV networks require a great deal of care and attention, as telecom infrastructure was not designed with video in mind. Yet investment in systems to ensure quality to some extent took a back seat in the rush to get video services up and running – a task that required a great deal of effort and ingenuity in itself.
Now, as telecom operators begin to find it harder to grow their IPTV subscriber bases in the face of a resurgent cable industry and stiff competition from satellite pay-TV operators, as well as threats from digital-terrestrial services and, more recently, over-the-top providers, quality assurance is moving centre stage.

Linear and VOD

The first quality assurance measure operators must undertake is clearly to design the network properly to accommodate both video-on-demand and linear services.
The former has its own quality assurance requirements. The unicast nature of the service means that traffic engineering planning is key, particularly at peak viewing times. If the network can’t cope, problems will arise. The fact that VOD services are often sold on a pay-per-view or subscription basis means that protecting the quality of the service becomes all the more important.

VOD systems require a number of different levels of quality assurance, according to Mikael Dahlgren, managing director of Sweden-based quality assurance specialist Agama Technologies. First, the video quality of assets needs to be verified before they are loaded onto the video service. Second, the servers themselves have to be tested thoroughly to ensure that the video can be delivered when requested. Beyond, that, operators should, he says, monitor each session at the set-top level. “This is the only place in a delivery chain where one can know how a VOD service scales,” he says. “It is also the only place where it makes sense to understand how the viewer experienced the VOD service and measure set-up time as well as other important session-oriented parameters of delivery.”

While IPTV networks share a number of characteristics with cable, IPTV providers do not broadcast content, as cable operators do, with the set-top taking on the role of tuning to the required channel.
IP-based networks also suffer from particular weak spots. According to Dahlgren, difficult areas include the switched/routed and packet-oriented IP transport itself as well as the bridging points to lower bandwidth access networks. The transport network problems can be exacerbated by jitter – the displacement of pulses in a digital signal. “Generally what differs from a transport perspective between a cable service and an IPTV service is just the use of another media – packet-oriented IP instead of the fixed bit-rate modulated signal on cable,” says Dahlgren. “This is  [deployed] in combination with the use of single programme streams and set-top boxes with de-jittering buffers making them more robust.”

However, despite cable network operators’ advantage over telcos of being designed for video distribution from the start, a few technologists believe that IPTV has some intrinsic advantages. “The HFC [cable] plant requires QAM or RF modulation at the edge of the network for digital video,” says Ron Shanks, video solutions owner at technology provider JDSU. “The multiplexing and modulation process, which is very complex and prone to error, is handled at the edge of the network and results in much more complexity at many more locations than with IPTV operators.”

Close to the edge

For Ben Schwarz, an industry consultant and former head of IPTV deployment at Orange, a key recent trend is a move by service providers to monitor closer to the edge of the network, the logical conclusion being to monitor the set-top itself. “Monitoring the headend is now accepted as standard practice. So when people in the industry say that monitoring is moving from the headend closer to the edge, what I see is that more and more operators recognise that monitoring the set-top with an embedded agent is a necessary step,” says Schwarz.

“Telcos have become more content-oriented whereas previously they were more packet-oriented.”
Danny Wilson, Pixelmetrix

John Williams, director of emerging markets at technology provider JDSU, agrees, arguing that operators need to put in place a comprehensive end-to-end service assurance programme. “This includes thorough new service testing of the access plant, and ownership of the in-home distribution network from a quality point of view,” he says “Remote monitoring of consumer premises equipment performance is critical after installation. Service providers have now taken this on-board and are making significant investments to deliver that expected Quality of Experience.”
Williams’ colleague Shanks concurs, putting forward the view that the emergence of the use of the consumer premises equipment as a network monitoring device marks a step change in service assurance methodology. “Most of the problems in the network occur on the last mile and in the home and the only way to cost effectively monitor this portion of the network is to use the CPEs – residential gateways, set-top boxes, ATA, ONT, etc – as service monitoring devices,” he says.

Jean-Michel Planche, CEO of quality assurance specialist Witbe, says that the user experience is the element that technologies should bear in mind, focusing on the availability of services, performance (including channel change times) and integrity (the absence of blockiness and jitter and the presence of a clear picture). Planche points out that IPTV providers were initially focused on getting something up and running in order to compete with existing service providers including cable operators. Quality assurance therefore took a back seat to other considerations. “New kinds of service need new monitoring technology, fully end-to-end and fully user centric,” says Planche. “Classic QoS is too network-centric…but the goal is user satisfaction.”

Agama’s Dahlgren agrees, up to a point. “In the end, I think it is easy to agree that the only thing that matters from a business perspective is the service experience with the end user. Just understanding this parameter would make it hard to improve the quality – one also has to understand the root causes to be able to improve the quality,” he says.
Quality assurance systems need to look at the headend, the network and the home. For Agama, viewer experience should be simulated at various points in the network, because, for example, a problem found part-way down a multicast network will extend to all viewers downstream.

Quality of Experience

The main criticism leveled at the end-user Quality of Experience approach to monitoring is the difficulty of managing it when services reach a large number of subscribers. Schwarz says that the largest deployment he knows of with active probes at the consumer premises is only 250,000. However, he believes that the case for embedded probes is strong.
“At last operators get to know what subscribers are really doing with the service,” says Schwarz. “This enables proactive troubleshooting but also, when a client calls up a help centre they are answered by someone who knows what the problem is and doesn’t say stupid things like ‘have you plugged it into the power outlet?”

[icitspot id=”9354″ template=”box-story”]

New monitoring technologies are coming to the fore that could help operators in the future. At this year’s NAB show in Las Vegas, test and measurement specialist Tektronix showcased a new product for operators that have to make a trade off between bit-rate and packet quality. According to Richard Duvall, video marketing manager, interest in Quality of Experience is being driven primarily by the growth of HD services. Tektronix is primarily focused on the cable market at this stage, but the growth of IP as a transport technology in cable has also contributed to the need for new solutions. While previous monitoring and testing based on the TR-101290 European standard for monitoring allowed operators to set different levels of alarms when things went wrong, there was no way to ascertain if the problem actually was having an impact on the viewer experience. Tektronix has developed a family of network probes that can be deployed across the network up to the point where content is encrypted at the QAM and which examine the video in detail, analysing the extent of impairments including black frames, macro-blocking and compression artefacts. “We have had interest from IPTV service providers in the same soft of thing,” says Duvall.

Solving problems once they arise can be complex in the case of IPTV networks in particular. Andrew Sachs, cable and IPTV senior director of technology at Quality of Experience monitoring specialist Volicon, points out that traditional broadcast technologies including forward error correction do not really work in an IPTV system. While some system providers, including the widely deployed Microsoft Mediaroom as well as Cisco, have developed their own error correction technologies for IPTV, this presents other IPTV service providers with a major problem. Volicon’s own technology provides Quality of Experience monitoring upstream from the set-top. While monitoring at the set-top level is the ideal solution, it is not currently practical, he says. A solution that captures the user experience upstream of the actual set-top, on the other hand, can be deployed economically and will identify the majority of problems. (One ingenious solution to the problem of finding out what is happening at the set-top level has been developed by Norway-based Bridge Networks. Bridge’s MicroVB product is a small piece of monitoring hardware that can be mailed to the subscriber’s premises, self-installed and used to correlate what’s visible in the subscriber’s household with the rest of the network).

Struggle to compete

Concerns about IPTV quality have come to the fore to a greater extent in part because the early phase of strong subscriber growth by IPTV providers has more or less ended. Operators now face a struggle to compete with a reinvigorated cable industry as well as DTH and digital-terrestrial pay and free offers.
“IPTV operators now realise they are competing against [cable operators such as] KDG or Virgin Media,” says Danny Wilson, CEO of quality assurance technology provider Pixelmetrix. “Whereas at the beginning they were worried about things like packet delays, now the concern is do you have a service reaching the viewers that’s comparable to those of the competition.” As a result, they have been forced to undergo a cultural change in order to get the video part of their service offer right. “Telcos have become more content-oriented whereas previously they were more packet-oriented.” A key trend that has been part of the change has been the appointment by telcos of an individual with overall responsibility for end-to-end video delivery, says Wilson.

For Schwarz, quality remains the key factor that consumers look when deciding whether to opt for IPTV or a rival service provider. “The investment in Quality monitoring seems directly proportional to the level of competition of a given market,” he says. “Thus in Europe French operators probably have invested most. Indeed the French market is the only one with two independent public Quality of Experience benchmarks published every month. In many markets, there is little competition between operators on the content because they all get the same deals and despite their future promise, advanced features aren’t yet reasons for choosing an IPTV operator. So if you have choice, QoE will be important in making your mind up.”

Not all technology providers in the space are focused on the edge of the network and set-top monitoring. Test and measurement specialist IneoQuest used its presence at this year’s IPTV World Forum in London to promote a new key performance indicator for video delivery that it says will revolutionise the way service providers manage quality while minimising operating expenditure.

Stewart Newton, managing director of European operations at IneoQuest, says that the concept of ‘Programme Availability’ will have a significant impact on the way that service providers (including IPTV operators) delivery quality. According to IneoQuest, the new measurement tool overcomes the existing fragmented nature of video delivery quality assurance tools and techniques (comprising an MPEG monitoring system from origination to the headend, a network monitoring system from the core network to the edge, and DSL or RF monitoring systems at the subscriber premises) by providing a tool that be used to assess video quality at any point in the network.
Previously, said Newton, monitoring practices only provided partial or fragmented views of end-to-end delivery chains. “Only the MPEG monitoring actually looked at the MPEG transport itself,” says Newton. With the introduction of multichannel line-ups, IPTV services, and more complex video delivery chains involving multiple organisations, problems became compounded. “There are physical divisions between different companies or political divisions within the same organisation,” says Newton.

According to Newton, Programme Availability moves forward from the Media Delivery Index (MDI), a metric created by IneoQuest that subsequently became a standard industry key performance indicator (KPI). “This moves from reactive alarming to a more proactive system for monitoring availability,” he says. Programme Availability is a standard measure that can be applied across the entire network, based on the ‘errored second’, a second in time where something happened that could have an impact on the subscriber. The number of errored seconds per month gives an indication of the nature of any problems across the entire network from ingest to subscriber. “We provide the capability to look to this Programme Availability figure and you can start to set target levels for Programme Availability on a per-channel basis at each point in the network,” says Newton. As well as providing a standard measure of performance over time that will enable service providers to control opex by reducing the need for reactive measures, Programme Availability will also better enable service providers to fulfill service level agreements with third parties.

Not all technology vendors are convinced. Pixelmetrix’s Wilson argues that use of a single measure is unlikely to be a cure-all for the problems actually experienced by viewers. Quality of Experience is what counts in Wilson’s view. “It’s all about delivering the service. It doesn’t matter how the IP packets come in – you have to see that the football game is reaching the viewer,” he says. Practical issues such as making sure the right channel is reaching the right set-top at the right command are to be prioritised.

Cure-all

JDSU’s Shanks is sceptical about the use of a single metric to measure quality. “While certain KPIs may have a more significant impact than others it’s important to measure a wide variety of Quality of Service indicators,” he says.  “KPIs such as those outlined in TR-101290 validate whether the content has been encoded properly or if errors have been induced at the MPEG layer by processes such as encryption, ad-splicing or multiplexing, while KPIs such as packet loss and network jitter are used to measure the performance of the transport network. Stream bandwidth and availability are also key metrics.  By not making all the appropriate measurements the visibility into the service is reduced.” His colleague Williams argues that the most important KPIs are packet loss and jitter. Error recovery mechanisms include packet-level Forward Error Correction and re-transmission, both of which Williams admits have limitations. “PCR jitter is another key metric which judges both encoder and other source quality issues and its performance directly impacts the set-top box decoding functions. It can be used to ID source issues v network performance issues if properly analysed,” he says.

Other vendors also look to a range of standard measures in order to provide a clear overall view of what is happening.  Witbe’s Planche argues the case for that Mean Opinion Score (MOS), which can provide a score for the actual quality experienced by the user. Whatever the arguments over the usefulness of particular metrics, it is clear that the scope of the use of test and measurment systems has widened considerably since the early phase of IPTV deployment. “When we started out supporting the first IPTV deployments back around the turn of the century it was all about technical challenges in getting the service to work,” says Agama’s Dahlgren. “Today, TV service assurance solutions have moved from being the technicians’ standalone tool to end-to-end solutions providing mission critical information to a large part of the operator organisation. It is no longer enough to provide technical KPIs to technicians. Today it is about providing technical and non-technical service quality input for each and every customer to core business processes spanning from top management to customer support, NOC and field technicians.” Most operators, he says, have started by focusing on parts of the network that affect large numbers of viewers – the headend and core – and then moved down towards the last mile, where problems occur but affect a smaller number of people.

Top level metrics certainly have their uses, however. Dahlgren says that Quality of Experience metrics can be particularly useful in facilitating understanding between network-oriented IP technicians and traditional broadcast specialists, who may take very different approaches to quality assurance. “This is an area where we at Agama try to produce top level QoE metrics to simplify the interaction, and minimise discussions on what problems that can be seen in the NOC compared to trouble reports coming in from viewers,” he says. “Correlation of quality reports from set-tops have come in very handy here to enable fault location by cross-section analysis.” QoE scores which can be understood at all levels of an organisation nevertheless need to be complemented by more specialised metrics such as TR-101290 parameters for broadcast engineering staff and correlation between video and network problems for network people.

Prevention of problems

For Ralph Bachofen, vice-president of sales and head of marketing at Triveni Digital, the best thing for IPTV service providers to do is to “start small” by introducing a few analysers at key points in the network. Triveni recently integrated its monitoring technology with Juniper routers, enabling it to monitor video quality more efficiently on the network backbone. This, says Bachofen, enables a degree of proactive troubleshooting by giving information to the router to switch traffic when problems arise.

While approaches to quality assurance differ between technology providers, most would agree that the ideal, in the long term, would be to put in place systems that can anticipate and prevent problems rather than react to complaints to call centres. However, for now this largely remains an aspiration rather than a reality. “Operators will be forced to move from an environment of getting alarms/events and troubleshooting failures and outages in real time to making changes in the network to prevent the failures and outages in the first place,” says JDSU’s Shanks. “This will be impossible without constant long-term service measurement and performance management. It is also incredibly difficult to isolate transient failures in the network without a proactive long term monitoring approach.”

As IPTV and other pay-TV providers seek to expand their footprints they are increasingly becoming indifferent to the types of network over which they distribute their services. They are increasingly looking to deliver their services over competitors’ infrastructure. This has some implications for quality assurance – it is more problematic to guarantee as certain level of quality if you have no control over the network.

The use of third-party networks means that service providers will increasingly seek to operate via service-level agreements. However, says Schwarz, traditional Quality of Service-based service level agreements will only guarantee that the third party (the wholesaler) will provide sufficient bandwidth. “There is no guarantee that the TV viewer is having a good experience. I predict a major shake up as the different stakeholders turn to Quality of Experience-type of metrics to enforce the service level agreements. These can be as simple as channel change times or service availability,” he says.

As the range of business models expands, and competition between IPTV and other service providers intensifies, the need for quality assurance technologies looks set to grow.