GigaOM Radar Edge Platform Leader & Outperformer 2024 | 
In the last several years, our network has supported hundreds of thousands of live-streamed sporting events, massive software downloads, billions of hours of streaming video content, and demands from thousands of web applications that all require real-time responsiveness and globally consistent performance. To support this massive growth, since 2015, our global network has grown to 250+ Tbps, requiring thousands of new servers across our 300 PoPs.
To keep up with the challenge of deploying a network footprint that is always growing and changing, our Operations teams developed an IT automation platform using StackStorm. It’s a giant leap forward that has transformed how our teams deploy, change, repair, and decommission server infrastructure, spanning more than 95% of the server population across our global network.
Here’s a look at how we built an automation platform that enabled us to:
- Add more than 12,000 servers to our network in under three years.
- Automate ticket generation in some cases and resolve issues without tickets in others.
- Give our teams tools so they can collaborate in real-time.
- Reduce engineer time spent following a manual series of steps.
Customizing StackStorm into project Crayfish
Recognizing the value and increasing maturity of IT automation platforms, in 2015, we formed the Infrastructure Automation team under the project name Crayfish. We began developing an IT automation solution in earnest — no small task for an organization of our size. The choice of name for the project — which we are still using — was intentional. We’ve long had fish and nautical themes for our tools, but more than that, we chose Crayfish because they serve as filter feeders, helping to keep the environment they live in clean.
Initially, we considered developing our own IT automation framework, but decided early on that StackStorm, with additional enhancements to support our unique requirements and scale, allowed us to move forward most effectively. StackStorm is an open-source event-driven platform that supports an “infrastructure as code” approach to DevOps automation. It excels at running workflows based on events, integrates with Slack, which we use across our operations teams, and has native ChatOps support.
Some configuration changes have been made to StackStorm to enable it to run at the scale we need. Primarily, this involved increasing the number of running service instances and reducing data retention. When we started the project, our network consisted of 8,000 servers. Now we’re at 20,000 servers and counting. The key to our scalability has been the ability to run any number of StackStorm instances as workers rather than trying to make a single StackStorm instance handle everything. As shown in the illustration below, this is made possible by something we’ve developed called Policy Engine, which gates the submission of requests to StackStorm based on our business logic. Policy Engine can optionally queue up requests and automatically run them when the policies determine it is safe. These policies monitor concurrency, capacity, traffic, production status, infrastructure, blacklists, and failure rate.
Submission requests route through a Policy Engine instance to any number of StackStorm workers for virtually unlimited scalability.
Codifying tribal knowledge
The basic approach in any IT automation project is to capture existing processes and workflows, turn them into code that can implement the workflow steps without manual intervention, and store the codified workflows in a centralized repository where they can be called up as needed, either as part of other automation or through ChatOps commands.
Once the project got underway, the Crayfish team captured existing workflows and processes from across the organization. Fortunately, the majority of the most important processes used by different groups, such as the Network Operation Center (NOC), Data Center Operations, or SysOps, were well-documented and could quickly be codified using the Python programming language and added to Crayfish.
For example, if our NOC wanted to implement a reboot workflow, it would need to verify that it is safe to work on the machine based on various policy checks. For example, if a server were in production, the system would take steps to drain connections and update the status. Next, it would perform the actual reboot and the steps necessary to verify that the machine is ready for production before finally returning it to production.
To be sure, there were plenty of edge cases. Among the most challenging were processes that had not made it to documentation. These tribal-knowledge cases required cross-functional collaboration and speaking to the individuals whose expertise ensured they were handled correctly. One side benefit of the automation effort was we were able to capture this knowledge, ensuring that it stays within our organization going forward.
In some cases, we were even able to identify more efficient ways of handling a particular task by connecting with several engineers and technicians who understood what was going on for a specific task or how to fix a particular problem. We found that in some cases, there were different levels of knowledge among various team members. That’s when we would gather everyone’s intelligence and develop a single workflow that everyone agreed would provide the most efficient solution.
Deploying new servers on demand
Crayfish has transformed our IT operations on many levels, such as simplifying server administration, reducing server downtime, and speeding up server deployment. In 2018 alone, ITOps added 22 Tbps capacity to our global network. Adding thousands of servers would not have been possible without the automation of Crayfish. And it was accomplished without significantly increasing staff levels.
Currently, Crayfish provides robust automation for about 20 different server types. In our organization, we define a server type as a group of servers that run more or less the same service. Different applications run on different server types, and each server type has different support needs when in production. Today, Crayfish supports the full server life cycle for about 10 server types and can also do most repairs, revisions, and patching for about 10 additional server types. It also provides varying degrees of support for about 40 or 50 more. Since we focused on the most widely deployed server types, Crayfish supports approximately 96% of our global server population.
The use of Crayfish has led to an overall increase in the performance of the CDN and its ability to accommodate unexpected events. Our network is well equipped to support massive, ever-increasing traffic from major sporting event streams and peak software downloads, for example, by keeping more servers in production and reducing the interruption of removing servers from production during peak traffic times.
Our scale contributes to why Crayfish is essential from a performance perspective. If your company has one data center, you can schedule downtime and stay on top of the needs of that data center. But when you’re talking about a data center in almost every time zone, as shown in the network map below, you have to contend with different profiles, usage, and usage times. Taking all those factors into account case-by-case can be extremely complex, but Crayfish handles it easily.
With Crayfish, we can still schedule tasks such as provisioning or upgrades to occur automatically, but the system is smart enough to accommodate local needs. Let’s say a news event in Latvia caused a large spike in video streaming right when we wanted to reprovision all the servers in Latvia. Because of the integration between Crayfish and our metrics collection system, the system would see the traffic increase and reprovision would stop. It could even put more servers into production if needed.
Catching infrastructure failures before they become service outages
Crayfish and other monitoring systems significantly reduce individual server downtime and repair time. Our systems continuously monitor the network for problems or failures and can automatically create tickets or directly apply fixes. With automated systems, the staff is not required to have “eyes on glass” to spot network errors — computers never get bored or tired and monitor continuously.
There’s very little limitation to the amount of checking that can be done in this fashion. Our metrics system continuously runs hardware checks that don’t require taking machines out of production, while another system looks for bad values and will call Crayfish to check on the machine when problems are found. With this approach, we can catch hardware failures very early on.
For instance, if metrics show that a hard drive is starting to fail, automated systems will kick off a workflow in Crayfish that verifies the error on the impacted machine and collects details about the failure, such as what hard drive failed in what slot, the shield number and so on. It then creates a specially crafted ticket for the Data Center Operations group and a technician will be dispatched to install a replacement.
Historically, any manual workflow that required multiple departments to coordinate or hand off between steps introduced significant delays along the way. With an automated workflow, the time to completion can be minutes instead of days spent in a ticket queue waiting for the request to pop up. This puts more control in the hands of the individual operations teams who know their systems and applications the best.
Giving engineers tools that make them more productive
Our previous approach to NOC work involved issuing a command and waiting for things to happen. If a NOC technician spends 10, 20 or 30 minutes waiting for a server to gracefully shut down, you have to ask why. Is that a good use of time? Crayfish can run many, many executions in parallel. And, as noted earlier, Crayfish was designed to scale horizontally. So, if we need more work, we can add a new node. If we suddenly find ourselves having to do a lot of work all at once, we can scale up a couple of nodes across the entire CDN and then scale it back down.
Also, technicians only have so much memory regarding how many machines they can work on at any one time. Only the most exceptional multitasker can handle more than five or six machines in different states without losing track or making errors. But with Crayfish, there’s no need to jump between various tasks, they can issue a command and move on.
Building on existing smart provisioning systems developed by the Software Infrastructure and Systems Operations teams, other aspects of server life cycles are fully supported within Crayfish and happen largely autonomously, including operating system updates, server provisioning or reprovisioning firmware updates, and security patches. When a new patch is released, Crayfish speeds up validation testing to ensure that the patch is okay to be released.
Because Crayfish is integrated with Slack, it supports normal IT processes and facilitates collaborative work as needs and requirements change. Rather than waiting on a single department to implement a change, teams can now define their manual process as an automated workflow. They can then give the workflow to end users to run on demand. Policies limiting when a workflow can safely run can also be implemented. This also allows an engineer standing in front of a server in a data center to issue a command to change the production status of a server they would like to work on.
Final thoughts
With thousands of customers pushing terabytes of data over our global network of more than 20,000 servers daily, our implementation of StackStorm has become an essential operational tool to keep our services healthy and secure. It’s enabled us to scale our network more quickly to meet customer demand while offering better performance despite dynamic changes and unexpected traffic surges. This operational agility makes us a more reliable service provider that can scale to meet streaming media and web application demands.
While most organizations don’t operate on the same scale as we do, there are still several operational table stakes that can make automation attractive, including its ability to:
- Focus developers on work that generates a higher return
- Standardize routine and mundane tasks.
- Improve collaboration across departments.
- Reduce server downtime.
- Improve security by implementing firmware updates and security patches.
Without question, the move to IT automation offers a very attractive return on investment, but it’s important to ensure you’re making a move for the right reasons. A good approach is to think of automation as a way to give engineers tools that will make them more productive and proficient and allow them to focus on improvements.
The transition to workflow-based automation is an excellent way to socialize and codify tribal knowledge. Instead of everyone having their own approach — we bring the tribe together, come up with a common workflow, and enable everyone to contribute future knowledge gained to a single bottom line.
