The cost of quality assurance

The success of the iPad has helped change consumers’ expectations of what web video should be, but quality assurance costs money. Stuart Thomson reports.

Mariner NetworkVu offers customised views of service performance.

Uptake of  TV Everywhere services and over-the-top subscription video offerings has been rapid. By comparison, the adoption of pay TV services – including, over the past 10-15 years, managed IPTV services – was a much slower and more manageable affair.

Nevertheless, viewers are increasingly demanding that video delivered over the web should be the same or of higher quality than broadcast video. The challenge from the point of view of content and service providers is how to deliver a quality of experience comparable with managed networks, in an environment where returns – from subscriptions or advertising – are marginal or non-existent and where the complexity of delivery is greater.

Unlike with managed networks, OTT and TV Everywhere video is delivered over the open web and mobile networks, and service providers rely on the ability of content delivery networks (CDNs) and third-party service providers (ISPs) to deliver it without buffering and loss of quality. They are also delivering to a very wide – and growing – variety of devices, with an ever-expanding range of operating systems and players.

The explosion in the use of second screens to view video has clearly galvanised the OTT market, according to Fabien Maisl, director of marketing at quality assurance specialist Witbe. “Things have been moving very quickly over the last 12 months. Before that video on the PC or tablet was seen as a nice feature but it was OK if it was best effort. Service providers were trying to do it right but people would accept some issues,” says Maisl. This year, however, maintaining a good level of service on such platforms is seen as an essential part of service providers’ businesses.

“It’s a very serious subject for servce providers and content owners alike. They are putting in measures to make sure they can deliver good quality,” says Maisl.

Witbe provides software-based ‘robots’ that sit on client devices and mimic the actions of  subscribers by browsing menus and logging in to a video-on-demand portal, for example. Maisl says that the company is offering its products to telecom and cable operators and also has customers from the broadcast and content provider business that are delivering additional revenues – notably advertising – from these types of services.

The popularity of tablets and other devices notwithstanding, business models for OTT and multiscreen have still to be worked out. According to Simen Frostad, CEO of quality assurance specialist Bridge Technologies, the lack of significant revenues accruing to multiscreen services plays an important role in determining how much service providers are willing to invest in quality assurance.

“All the feedback we hear is that [service providers] are excited about it but don’t see any money in it,” says Frostad. “That influences how people monitor the services. You have advanced services using completely different technologies going over the public internet with multiple CDNs, WiFi and mobile players, and TCP-IP transmission. At the same time there is no money in it.”  He says that continuous quality assessment is important in this environment, enabling content providers to be sure of what they are sending out before it disappears into the ‘black box’ of the CDN infrastructure prior to delivery to end users. Bridge provides an OTT engine that monitors the origin server for the metadata and the ‘chunks’ of video that make up adaptive bit-rate streams, “so you know what’s going on before you hand over to the CDN”, he says. However, Frostad adds this does not give insight into how the video is playing back at the end of the chain. For this reason, he says, it is important for service providers to use monitoring robots “post-cloud” to validate what is happening in different regions and with different ISPs.

“The ultimate goal would be to see how each individual device is coping with media access,” says Frostad. This can be done via in-app monitoring devices. However, monitoring all devices in all households is prohibitively costly, especially given the limited revenues coming from multiscreen deployments today.

Increasing expectations

While the fact that limited incremental revenue from OTT and multiscreen services clearly inhibits investment in quality assurance, it is also true that viewers’ expectations are increasing. Mikael Dahlgren, CEO of Agama Technologies, points out that with OTT services, much of users’ fury when things go wrong is vented via social networking sites including Twitter, as well as on consumer forums. So bad news can spread very fast among a service’s users. “It’s much quicker movement in the customer base. It’s easy to sign up [for services] and it’s a quick process [to churn],” he says.

For Huw Price-Stephens, head of strategy and business development, EMEA at Mariner Partners, the emergence of the tablet computer has been “a game changer” in setting user expectations at a new and higher level.

“Most people wouldn’t watch long format content on a smartphone but they may do on a tablet, so the tablet is a game changer where the worlds of high-quality IPTV and video delivery onto an alternative screen join together,” he says. With high-speed networks, says Price-Stephens, OTT video can match or even exceed the quality of managed services, meaning that viewers’ expectations are only likely to increase further. If adaptive bit-rate streaming guarantees continuity and eliminates buffering, much of what upsets consumers is eliminated. If the technology had been available 20 years ago, says Price-Stephens, it could have transformed the economics of IPTV by extending the reach of services much further from the DSLAM.

Mariner Partners focuses on analytics and delivering visualisation of data from the sources available rather than on building hardware probes or client devices. Price-Stephens points out that in the multiscreen world it would be impossible to deliver monitoring of the consumer experience on every device in a proprietary way. Information is instead collaged from web services logs, backup and disaster recovery servers and so on. “For Mariner the core value is in what we do with that data,” says Price-Stephens. “Our goal is to take that data, understand it and play back to agents a single screen with a summary of the key information that might have an impact on the customer that they are speaking to.”

Delivery chain

The essential thing is to deliver a comprehensible view of the service that can be used to solve practical problems. “If you provide operators with 20 different screens it becomes self defeating. If you can provide visual cues and non-technical staff can look at a screen and interpret what’s happening rapidly, that works,” says Price-Stephens.

The key difference between IPTV and OTT, he says, is that “in a managed IPTV environment the operator has strict control of the whole delivery chain”, meaning that a view of the network is important. In the case of OTT, the operator lacks the ability to see what’s happening in the cloud between playout and the subscriber, but still knows that end users are getting a good video experience. “Today for OTT services often the service provider isn’t making the same promise of absolute quality to the end consumer because they put in place caveats about their ability to receive services. Things can be outside the service provider’s control unlike in a managed IPTV environment,” he says. Players in devices can provide information about the end user experience to the content provider, while additional information could be provided by the adaptive bitrate server. “We do have a library we can ask people to incorporate in a client device or player technology, but it’s not a lot of data. What we expect is that we won’t have to build our business around getting our technology out there [to the players],” says Price-Stephens. [icitspot id=”75262″ template=”box-story”]

Increasingly, he adds, service providers are looking to an OTT-centric model that will allow them to deliver content to multiple screens as well as to the TV. “Some operators are deploying OTT as a main service in areas that can’t economically be reached by IPTV, where OTT can become the TV service. People still have questions over the robustness of the technology, but why go for IPTV if you are starting in a greenfield site?” says Price-Stephens. “What’s happening is that the value of OTT and the cost of the platform means operators think it’s a lot cheaper to go this way than traditional IPTV.”

Agama’s Dahlgren concurs that IPTV and OTT are beginning to converge, particularly in certain cases involving partnerships between different types of providers. “One of our first customers was working with open city networks. The approach they had wasn’t different from other CDNs. They had lots of different networks contracted, so the end-to-end network was more like an OTT network than a traditional IPTV solution,” he says.

The complexity of the delivery chain, and in particular the range of devices now targeted by content and service providers, has implications for the way that quality assurance measures can be effectively put in place. “The first aspect is quality automation. Each tablet has a different operating system, even within the Android [environment]. There are many different operating systems that you need to validate, so all these app developers face a nightmare making sure they work,” says Witbe’s Maisl. “We are providing test automation platforms for all these devices.”

Maisl says that test scenarios for individual devices have also become more complex. “You need to test more features in less time – the first thing is to make sure apps work across all devices… The second thing is the need for live monitoring of these services all day long, all year long. [Service providers] want to monitor if their CDN providers comply with service-level agreements (SLAs) and identify where they have network issues and invest and renegotiate contracts to provide the best quality.”

According to Maisl, while some providers are looking to ensure that revenue-earning features such as advertising are properly played back at the correct time, the main preoccupation is ensuring playback of content across an ever expanding range of devices as quickly as possible and the provision of dashboards that can be used to enforce contracts with CDN providers. “The main issue is that legacy quality assurance equipment that broadcasters have deployed in the network is not relevant any more for OTT and multiscreen,” says Maisl. “You don’t own the delivery network or the viewing device. There is no point in tracing packet loss. There is no way you can put probes in the network. All this quality assurance infrastructure is almost useless.” While providers need MPEG analysers that check what is actually being played out of their headends, further down the delivery network the only way to understand the quality experience of users is to monitor what the end device is actually playing back. “That’s a new paradigm that operators need to adopt. Most operators have been network-centric up to now, but now the only thing that matters is the quality of what’s being played by the terminal.”

Maisl says that Witbe combines a sampling approach by mimicking what’s happening on end devices at various points with embedded software in apps to deliver real-time metrics including the incidence of rebuffering, network delay and MPEG errors. This will be put into context depending on the type of device and operating system and the networks used by the subscriber.

For Johan Görsjö, director of product management at Agama, it is easy to exaggerate the differences between traditional IPTV platforms and OTT, despite the lack of visibility OTT service providers have into the network in between their own playout and playback on the consumer’s device. “You are trying to solve the same problem fundamentally. It’s not a game changer in monitoring,” says Görsjö. “The general approach is the same but the products are different and the protocols are different so we have different product lines for it. The network is very different from the case with managed services as you no longer have access to the network. The next point [of visibility] is when you come out to the player end. Players are players. You have software and it’s approximately the same in a set-top. So this approach with the embedded [software] concept works equally well in all these scenarios.”

Automation

Andrew Sachs, senior director of marketing at Volicon agrees with Maisl that automation of quality assurance is important if the economics of OTT are to make sense. While a single linear broadcast or IPTV HD channel comprises one stream, an OTT service can comprise 16 different streams to deliver multiple bitrates across multiple streaming formats. Volicon has developed Observer OTT for Flash and encrypted OTT content, which is available in hardware or software versions. “The same system can be integrated in software. There is not much revenue so far and it’s important from a strategy perspective,” he says.

For Sachs, OTT means added complexity for limited additional revenue. Channels are therefore subject to less oversight than broadcast services, even if ad insertion demands can be more complex than for linear TV services. “There are a lot more of those streams but there is not the same level of engineering oversight. And there are ad insertion issues and so on,” he says. “Multiple encoders are being pushed to their limits. You have to make sure they behave properly, even as you are paying less per channel for encoding.” The transmission path is also more complex with multiple interfaces with multiple CDNs.

Sachs believes there is nevertheless an imperative to invest in quality assurance, driven by the sale of subscription services on devices such as the iPad, from which users expect a high-quality experience.

In some cases, such as IPTV providers adding multiscreen to an existing suite of offerings, the operator will have  its own CDN infrastructure. In this case, says Görsjö, service providers will want to know what’s happening, for example, in relation to their edge caches. Where service providers provide web-delivered content over their own networks, there is also a greater danger that their brand could be damaged if the service fails to meet users’ expectations than is the case with a ‘pure’ OTT service over a third party network. For service providers without their own infrastructure, they will likely use multiple CDNs, meaning they have to look after their own SLAs.

Transmission routes

“There are also lots of things the CDN doesn’t control like performance in the player and other things that could affect what’s causing the problem. You must identify who your customers are and what parts [of the chain] you must take care of,” says Dahlgren.

OTT delivery is different from traditional broadcast or IPTV in a number of ways. Danny Wilson, CEO of quality specialist Pixelmetrix, contrasts OTT, based on clients requesting or pulling content from a web server with the ‘push’ nature of traditional TV delivery. Testing OTT services is more complex than DVB, which “you test with a meter”, he says. Pixelmetrix has produced the OTT Media Grinder to simulate the behaviour of client devices. “The platform takes a sample and can simulate up to 200 clients at different places around the network,” says Wilson. The company plans to develop products that look deeper into the network, focusing on the origin servers. “While the OMG simulates real people, the extension will focus more on what’s happening inside the network,” says Wilson.  Multiple problems can arise. CDNs cache content streamed from the origin server. With live content, often the cache will not know when the master copy at the origin server is updated.

Witbe’s solutions can enable providers to analyse infrastructure peformance.

“Video is broken up into chunks. The cache requests chunks according to a manifest list. If the file is a VOD movie there is a fixed start and end point, but for live content there is no fixed end and the index list has to be updated every so often. You have to keep requesting index lists and often the cache caches the same index list and doesn’t go back to the origin server to get the new one, so you can get a 10-minutes segment repeating again and again,” says Wilson. “This might never be noticed by an individual because the picture quality will be perfect and the loop will be seamless.” There is thus a need for monitoring of the complete system instead of just the end point.

Others take a slightly different view. According to Bridge Technologies’ Frostad, most service providers can trust large CDN providers to take care of delivery needs, particularly for free and advertising-supported services. Nevertheless, says Frostad, problems are likely to mount as the number of internet-delivered video services increases. Providing technology that delivers massive amounts of data about packet loss will not solve the content provider’s problems, meaning that there is a need for intelligent application of alarm systems, he says. Currently, he says, major content providers are putting in place systems to check that their output is in a pristine condition before it leaves the building. He says they are likely to continue evaluating client-based monitoring systems for the next few years because of the complexity involved.

Once content is played out, it is in the hands of CDN providers that have put systems in place to monitor what goes through their network. “In the cloud we don’t see a lot of necessity for monitoring tools because the CDNs are good at maintaining data integrity from ingress to egress,” says Frostad.

Ensuring Quality of Service at the end point is more complex, and is generally regarded as a matter for the consumer and his individual ISP. However, demarcation between the CDN provider, that has been contracted and is subject to a service level agreement, and the ISP, is difficult to police. In addition, while CDN providers have put in place their own systems in order to give their clients confidence that service level agreements are being adhered to, there is an obvious conflict of interest. “CDN providers are generally offering software agents built into the device that will provide quality metrics to the content owner which is in some ways similar to what we provide,” says Witbe’s Maisl. “But the thing content owners come to us for is if they can’t rely on a particular CDN provider’s information. You cannot deliver the service and also the quality assurance metrics at the same time. Also [operators can] have several CDN providers. They need a source of information that is more objective and also works across all networks.”

For Kirk George, marketing director at quality assurance specialist IneoQuest, quality assurance systems can also be used by CDN suppliers themselves to support service level agreements with content providers. “We are seeing a shift where CDNs are looking to guarantee their services,” he says.

Adaptive bit-rate streaming

One obvious technology solution that has emerged to transform the experience of watching video over an unmanaged network is adaptive bit-rate streaming. By dynamically varying the video profile according to the available bandwidth, adaptive streaming has changed the framework of internet delivery of content by making it easier to ensure a sort of continuity of experience, if not absolute quality.

However, adaptive bitrate technology is not a cure-all. “There was a general thought in the market that if you set up adaptive bitrate most of the problems would solve themselves,” says Agama’s Dahlgren. “There are ups and downs with technologies. With small rollouts most of the problems do sort themselves out, but as you scale up there can be problems.”

Service quality and the scale of the deployment of an OTT service can be inversely correlated, and OTT is inherently complex. IneoQuest developed VeriStream, a technology to deliver Quality of Service assurance for adaptive bitrate streams, three years ago. “We patented the Media Delivery Index [for IPTV] and our competitors licensed it in their probes, and we came up with VeriStream for multiscreen, which is the only quality metric for adaptive bitrate,” says George. “What we have done from VeriStream is create the Video Experience Index which is the experience [of video] on the device itself.”

Bridge Technologies’ Frostad says that standardisation around MPEG DASH could help deliver quality assurance by reducing complexity. “When you reduce that complexity you will see a lot more usage of OTT even for traditional broadcast. And a huge investment in quality assurance may not be needed,” he says. Standardisation can eliminate a host of problems. Adding subtitling and other features means that content providers need to deliver additional metadata, which adds to the complexity of multiple formats.

The promise of MPEG DASH notwithstanding, adaptive bitrate both helps deliver an acceptable quality of experience and adds to the complexity of delivery of OTT services. As more video migrates to unmanaged distribution over the internet, the need for innovative – and cost-effective – quality assurance technologies is only likely to grow.