Why Network Capacity Isn’t Enough: Caching Strategies for Better Streaming

To keep up with the growing volume of media content, the Verizon Media, now Edgio Media, Platform has invested in expanding our global cache footprint. In 2019 alone, we added more than 25 Tbps of capacity, seven global PoPs, and close to 900 last-mile connections. While effective at improving performance, raw capacity is not enough, nor is it a sustainable business model for meeting the ever-growing global demand for streaming content.

To maximize the availability of our network capacity, we invest equally in technologies, processes, and tools that keep operational and infrastructure costs in check. Our research team continually pushes the boundary of caching technologies, applying and refining processes to give our network operators granular control over how, when, and where content is cached.

Modern Caching Strategies

The goal of any caching strategy is to keep the most popular content in the cache while quickly and efficiently removing the less popular content. Over the years, researchers and software developers have devised countless strategies intended to solve the caching challenge. These range from relatively simple to extremely complex. Some of the more popular strategies include:

• The least recently used (LRU)
• Least frequently used (LFU)
• First in, first out (FIFO)

It would be convenient if there were a single caching strategy to rule all situations. However, such a solution has yet to be developed, and the effectiveness of a particular strategy can vary greatly depending on server and disk sizes, traffic patterns, and other factors. Based on extensive testing, we’ve found that LRU offers the best compromise between hit rate and disk I/O, providing 60% fewer writes than FIFO while maintaining high hit rates. Additionally, for the disk sizes used in our CDN, LRU performs on par with more complex policies like S4LRU (Quadruply-segmented LRU). You can get more detail in this paper we published last year at the Passive and Active Measurement Conference (PAM) held in Puerto Varas, Chile.

Evolving Caching Strategy with Hybrid LRU

Even though LRU works very well for our environment, we’re always looking for ways to drive innovation and improve customer performance. This has led to a new capability we recently added to our platform called Hybrid LRU. It’s called hybrid because it adds a layer of abstraction on top of LRU. If we don’t use the hybrid functionality, the system continues to operate normally, so it’s very easy to understand and activate or deactivate.

What we’re doing with the hybrid approach is tweaking the LRU system to give us more control over specific pieces of content. By control, we mean having the ability to explicitly store some content for a longer or shorter duration based on predefined settings.

This is important due to changes occurring across the video streaming landscape, particularly the rapid growth in live streaming. Our network alone has hosted hundreds of thousands of live events, many of which are delivered to millions of concurrent viewers. Despite the massive popularity of such events, once a live event stream is completed, it’s not likely to be re-streamed at any significant volume. With Hybrid LRU, we can specify a shorter cache period, freeing up valuable cache resources for other media and content.

We are experimenting with locking down certain content and providing a best-effort assurance that it will remain in our cache. This can be particularly useful for live video streams with limited shelf life but may still be in high demand for a few hours following a live event, which becomes a normal piece of video-on-demand content. This functionality can also be used in conditions where a content provider explicitly wants to lock some content for a specific period of time so that it does not hit their origin servers.

Hybrid LRU also allows us to store some content for a longer duration of time. This is useful if the origin is located in a remote part of the world, for example, which can lead to poor QoE when the CDN does not have requested content in its cache. In such cases, a new client request would trigger a cache miss that the origin will need to fill, potentially resulting in rebuffering. Aging this content slower will stay in the cache longer and reduce the number of such origins fills.

Hybrid LRU Usage Parameters

Hybrid LRU consists of two tunable parameters that give us the ability to either delay or speed up the eviction or removal of specific content from our caches:

• Aging Rate
• Time to Live (TTL)

The Time to Live (TTL) parameter reduces the age of an item by a specified amount. It works by giving an extremely low eviction score to an item for the duration set by this variable. This forces an item to stay in the cache for a specified duration since it was last accessed. The default is 0 seconds, which means no special preference.

The charts below show how these tunable parameters work to adjust how long content stays in the cache. Considering these parameters as knobs or dials that can be precisely adjusted to match content demands is useful. The charts show how objects age over time on server caches while waiting to be accessed.

First, let’s look at Aging Rate. Traditional LRU objects age at the same rate over time. But as we turn up the aging rate dial, items age faster over time. Similarly, when we turn the dial in the opposite direction, items age slower than LRU. Turn the dial enough, and the slow-aging items never exceed the “eviction threshold,” as figure one shows. With this control, we can either remove items sooner to free up space or keep items on disk longer as needed to reduce origin pulls or for other reasons.