EP9 – Holiday Readiness

An Introduction to Beyond the Edge Episode 9 – Holiday Readiness

In this episode of “Beyond the Edge,” Howie Ross hosts a discussion on Holiday Readiness, diving into the essential steps businesses need to take to prepare for the upcoming holiday season rush. Howie, the Senior Director of Product Management for Edgio Applications and Application Security and Performance Platform, draws on his extensive experience in web development and cloud architecture across various industries, including Fintech and e-commerce. Joined by Ellery Womack, Senior Director of Engineering at Edgio, and Tom Gorup, Vice President of Security Operations at Edgio, the conversation covers a range of topics, from security concerns to performance optimization strategies. With Black Friday, Cyber Monday, and the entire holiday season looming, the team emphasizes the importance of early preparation to ensure websites can handle the increased traffic and deliver optimal performance. They discuss the significance of load testing, stress testing, and identifying potential application errors and slowdowns in performance ahead of time. Additionally, the discussion highlights the critical role of security measures, including visibility, threat detection, and bot management, in safeguarding against potential attacks during this crucial period. As businesses gear up for the holiday rush, Howie, Ellery, and Tom provide actionable insights and best practices to help organizations navigate the challenges and capitalize on the opportunities presented by the holiday season.

Howie Ross: Welcome to Beyond the Edge, where we dig into the insurance and outs of the trends affecting modern digital businesses. 

I’m your host, Howie Ross. I’m the Senior Director of Product Management for Edgio Applications and Application Security and Performance Platform where I focus on web acceleration including CDN and edge computing. I’ve been doing web development and cloud architecture for about 20 years and during that time I’ve worked in many industries including Fintech and e-commerce where I worked with brands including Urban Outfitters, Coach, Verizon and M&M’s. 

I’m joined by Ellery Womack. 

Ellery Womack: Hi, I’m Ellery, Senior Director of Engineering at Edgio. I work with a team of architects, engineers and Q&A to help Edgio customers drive the most value from our products. Over the past three years, my team has helped customers drive incremental revenue and SEO benefits by improving their Core Web Vitals with Edgio technology and performance engineering. 

Howie Ross: Thank you Ellery. And we’re also joined today by Tom Gorup. 

Tom Gorup: Thanks for having me. This is going to be a lot of fun. I’m Tom Gorup, Vice President of Security Operations or Security Services at Edgio. So everything security and service related, the combination of those two falls within my team. So that’s your 24×7 security operations. You’re security, architects, and threat intelligence. All that great stuff falls within my purview. 

Howie Ross: Excellent. So today, we’re going to be discussing how to get your website ready for the holiday season. 

I know that it seems far off, but to really take advantage of the insanity that is the holiday buying season, we need to start prepping now because websites experience dramatically increased traffic during this holiday when the sales and promotions are running on the vast majority of B2C businesses in retail, travel, and hospitality. 

So, businesses really need to prepare for this increased load as many businesses depend on the increased revenue from Black Friday, Cyber Monday and the entire holiday season. And not hitting your targets during this critical period can be devastating for businesses. Tom, what should organizations be thinking about and doing right now to ensure a successful holiday season? 

Tom Gorup: That’s a great question. I think preparation is important, right? The further you can get ahead of this, the better off you’ll be, a better planning you can put in place. When I look at the world from a security standpoint, I kind of put it into three different categories or three pillars defined as part of your security posture. It’s visibility, exposures, and threats, and really, it all starts with visibility. 

I think what businesses can do starting now is start to discover and understand those assets that are critical to whatever sales are coming up and to whatever application needs to be fully stable and secure when those holiday events kick off. A great way to do that could be like a tax service management tool, something that scans the Internet and scours it for maybe APIs that you didn’t know existed or open ports that you didn’t know were there. We see a lot of that. So, I think the first thing you can do is start to really build an inventory of those resources and those assets that you need to prep up, test, and secure when the time comes. 

Howie Ross: Yeah, that makes a lot of sense ’cause it’s very difficult to secure what you don’t know is there. Do you feel like there are increased risks during this holiday period? 

Tom Gorup: Yeah, 100%. The holidays are significant revenue. For some businesses, 50-80% of the revenue is propped up through the holiday season. So, it’s critical that their web resources or the conduit of which customers are purchasing from is highly available. Attackers love to take advantage of these scenarios, and it’s not uncommon to see you know DDoS as ransom, or there are sorts of attacks that inhibit the ability for the sales to go on. So we’re starting to get busy right about now in preparation for those events for our customers, and you know, when it’s game day, man, it’s all hands on deck across the board, right? 

Howie Ross: So Ellery, what else in addition to security concerns should businesses be thinking about and preparing for to ensure a successful holiday season? 

Ellery Womack: Thanks Howie. So a lot of our customers are worried about not being able to handle the increased load from Black Friday traffic. And again, ahead of that, they’ve started load testing and stress testing their applications. Often times, when Black Friday rolls around, businesses will send out, you know, marketing campaigns in the form of SMS blasts and emails, which trigger large spikes in load on their applications. So many of our customers have started doing these load tests, and they’re working with us to identify the root cause of application errors and slowdowns in performance. 

Howie Ross: Yeah, that makes a lot of sense. And then, you know, once you get the results of that load test, it’s critical that you make sure that you can sustain the load and, as you mentioned, a little bit more than you prepared for. This was many years ago, but I was working with, I live in Philadelphia, and I was working with a locally headquartered discount retailer and I determined that they were woefully under-provisioned for what we thought was going to be their holiday load. But given that, you know they’re at the time when most of their business was in their brick-and-mortar stores, they didn’t want to add additional resources until it was Black Friday. 

Of course, we’re having issues as a result of the traffic on the website kept crashing. And then at that point they were all too ready to sign up for additional monitoring tools and add additional servers. And you know, that had to be done in, you know, a fire drill type scenario as opposed to a more controlled and prepared scenario. So yeah, it’s critical that we understand the load that our infrastructure can sustain and plan for it appropriately. 

Tom Gorup: It’s unfortunate it often takes an event for somebody to for some businesses to start investing that goes in security as well. I was actually having a conversation earlier today as a security analyst. Well, entry-level is kind of looking, hey, how do I kind of break into the industry? One of the questions he asked was, you know, do you have to convince businesses that they need security, and it’s a mixed result? But I think, more often than not, businesses want to do the right thing. They just don’t know where to start. 

It’s come a long way then, you know, 10 years ago when you’re trying to convince a business that they need security or, you know, need those extra servers. These days, finding more businesses want to do it; they just don’t know where to start, and also, you know, can they afford it? Those are like some of the challenges that they write into. 

Howie Ross: Yeah, that’s a great point. You know, you talked about attack service management and understanding your exposure. So, you know then what’s the next step once we know what we’ve got out there, what are some of the additional steps we should be taking now to protect our properties during this critical period. 

Tom Gorup: Yeah, great question. First, the visibility where we have an inventory and nobody’s going to have 100%, right? You can’t expect 100%. So, I always say don’t stop or get caught up into like oh, I don’t have full visibility yet, I can’t keep going. Take what you have and then start looking at it. 

The next pieces to the puzzle are exposures and threats. Exposures: where are your vulnerabilities, where are the chips in your armor, and then how are you being attacked? I kind of landed at these three pillars when I was thinking about how I secured battle positions in Afghanistan and Iraq and these were the things that I looked at, right, I can’t protect what I can’t see any and make sure I have a good field of fire. I need to have good cover and concealment, and if I have exposures, I need to be aware of them. 

Sometimes, you accept those risks, and sometimes you mitigate them, but also, when that fight happens, when the attack kicks off, knowing how the enemy’s attacking you allows you to adjust your posture. You need these three elements to start making those sorts of decisions, and you’re likely being attacked today so you can use that intel to drive good decisions. 

Those are two things to start incorporating in there, where are your vulnerabilities, how are you being attacked, get that good visibility and then start planning, you know, maybe do some, you know, I already mentioned load testing, what happens if you run into a DDoS attack, what happens next? 

I mean, a lot of team members are going to jump on a phone call, and they’re going to be looking for someone to guide them, do tabletop exercises, run a couple prior to, and I think that’ll go a long way in your responsiveness to the inevitable attack. But you can reduce dwell time, and you can reduce the amount of time that site is offline, and you know make it a big win at the end of the day if you’re if you’re prepared well enough. 

Howie Ross: Yeah, great point. I would say that almost as bad as a site that’s offline as a result of a DDoS attack or a security incident is a site that is, you know, underperforming and not taking advantage of this increased traffic, and the promotions that are being run during this holiday period. So, Ellery, what else can we do to ensure that our sites are performing optimally? 

Ellery Womack: It’s a great question. One of the things that our customers are looking at is A/B testing. So, they’re trying out new experiences for various types of pages to see what gets the best user engagement, how can we optimize conversion rate, average order value, page views per session and other important business KPIs. And so, people are testing out new page templates, different types of promotions, product discounts, etcetera all across their websites and they want to do this in a way that’s highly performing as well. 

Using an edge experimentation product like the one in Edgio 7 has become a popular use case for them. But there are a number of just intentional performance improvement things that all of our customers can do. So, we’re constantly looking for opportunities to help customers with caching on our network. Cached content is going to be more secure. 

We want to make sure that response times for various APIs are very fast. And I think customers identify that with our observability and analytics tools is important. People might not know this, but a website that takes 4.2 seconds to load is going to convert at half the rate than one that loads in under 2 1/2 seconds. These things are really important to optimize and we have a team that can help customers with this. A/B testing and performance improvements are at the forefront of that, and that’ll even take some load off your origin as well. You’re going to need fewer servers on Black Friday if your servers have to do less work to complete the same operations. 

Howie Ross: Yeah, great points. We’ve actually had customers come to us, you know, months from now, right, a month before Black Friday and say my website is too slow. I’m not passing, Google’s core web vitals. I’m not going to get as much organic traffic. My conversion rates are going to be lower. What can you do? And we can actually help them but the tools that we have at our disposal at that point are somewhat limited. We can help you do you know some prefetching and do some acceleration in certain use cases. But really to take advantage of that increased traffic and get those conversion rates. You want to start that optimization well in advance so that you can you know get those high cache hit rates and get that site performing as optimally as possible. 

Ellery Womack: Yeah, that’s absolutely right. And you know, if you want to work backward from that, if Black Friday and the holiday season starts in November, most of our customers are looking to do a code freeze or code chill sometime in early October. We’ve been able to help customers become 30% faster in as little as a month. So, if you’re looking for performance improvements, I would definitely look to start that no later than June or July. 

Howie Ross: Yeah, that’s a great point because you know by the time we reach the fall, we should be in that code phase or as we call it a code chill, cause of course you know if there’s a critical bug, or a security issue. We’re going to have a thaw and patch that. But we should effectively be penciled down at that point. And so, you know this ties into our team, you know our most important asset, our people. So, what aspects of preparation should we be doing with our with members to make sure that they’re ready for this increased traffic? 

Ellery Womack: I guess I can get that one started. Some of our customers are working with us for dry runs of incidents and making sure we’re ready for, you know, the incident response process and we’re documenting that both in our runbooks and customer runbooks. To Tom’s point earlier, you know you need to be ready for these things and be prepared, but you don’t know how prepared you are until you’ve actually done the dry run. 

So can we trigger an alert from our observability tool saying the website is down and we’re having a major issue and make sure that this is escalated properly, the right people are responding in the right time frames. We can assemble the right people on a call or in a war room to debug or diagnose the issue in an effective way, making sure that you’re prepared for having the right people on call as well. Do you have a calendar that shows when people who satisfy critical roles in the instant response process are available? So you constantly have coverage, and I would highly recommend having at least two people available. As we like to say, two is one and one is none. 

Tom Gorup: That’s good. I like that. Yeah, taking that further, the preparedness is important, running tabletop exercises, making sure you know your process, who you’re going to call. You know, when does the CEO get to find out that you’re under attack, right? These things need to be hashed out, otherwise it’s going to be chaos when the time comes, but also just general user awareness from a security standpoint, holiday season and is a great opportunity for attackers to jump in, send phishing emails, find other ways to squeeze into your infrastructure. So it’s important to raise awareness around holiday season. 

There’s a lot of emails going back and forth. There’s a lot of Black Friday Deals and and and links being shared. There’s also a lot of maliciousness within that information. So it’s good to make sure that your your users from your HR department to your engineers are aware of the risks that are out there. We also have an attack trends report coming out here at Edgio. And one thing that we took a look at was all the CVE’s (Common Vulnerabilities and Exposures), all the common vulnerabilities that were discovered or actually more specifically that were talked about in the first quarter of this year. 

We broke it down a little bit further to look at the common weaknesses. So what was the vulnerability that existed within the product that resulted in that vulnerability and what we saw there was top, we’ll call it so top three was remote code execution, denial service and privilege escalation vulnerabilities. When you’re thinking about what sort of training or awareness can you bring to your engineers, those would be 3 big ones that can we do some extra security training around what remote code execution might look like in our app. And how can we put preventive measures in place at the code level, right. And what would a denial service attack look like in our app? Start doing some of that threat modelling but also training your engineers on how to circumvent it at the code level. 

Ellery Womack: So remote code execution in my understanding is about executing code and remote servers. What can people do to make sure that a malicious actor doesn’t inject code into a website and start doing some type of a data skimming attack like a Magecart? 

Tom Gorup: Client-Side Protection (CSP) is a great one that you can leverage a lot of input sanitization is another piece on that remote code execution side is we do not want anything to be running on the server side. 

What kind of input sanitization can we do? Edgio itself can make sure the API is properly sanitized before it gets sent across the line. You can put that schema within the tool itself and do that sanitization. There are a lot of controls that you can put in place to protect yourself. 

Howie Ross: Quick question, Ellery. So let’s say that you know we have been hard at work for, for months now and we have our, you know, our analytics and our observability in place to be able to detect our traffic and any attacks that are against us. We’ve optimized for performance. We’ve run our tabletop exercises and it’s now the fall and we are you know the holiday season is upon us and we’re probably not doing as many code deployments or at least we hope not. So what can we be doing? 

Ellery Womack: You know now in the fall typically what we end up helping customers with is fine tuning a lot of monitoring and dashboards and analytics. You’ve honed in on all the important things but what else can we go monitor for. It’s a great time when you don’t have the pressure of delivering new features and new capabilities to customers that we can actually focus a lot more on monitoring and observability. 

Making sure they’re prepared for that and making sure that we give the right tools to people who are going to be doing eyes on glass monitoring, especially on days like Black Friday and Cyber Monday. We typically have eyes on glass and people who are checking in on the health of the web application and APIs on a very regular basis. Making sure they have all the insights they need to glean; you know meaningful insights is really important. 

Howie Ross: Great point. And Tom, is there anything that we can be doing you know in the fall as the holiday season is upon us to from a security perspective? 

Tom Gorup: It’s not uncommon for us like our security architects, we work with our customers to fine tuning the app, but also fine tuning the WAF, the Web Application Firewall as well and also the bot management. 

Bot management‘s a big one that that could be something you can be taking a look at as well. You want to make sure you have good SEO ratings, you don’t want to break good bots, but you also want to maybe protect those pages that are specific for the event. Maybe you don’t want that new splash page to be released just yet. Make sure that you have the controls in place using bot management, using laugh to make sure that those are properly protected ahead of time. We do a lot of tuning on the WAF leading up to events, making sure that we’re blocking the right traffic and allowing, excuse me, yeah, blocking the right traffic and also allowing the right to good traffic to make its way through. 

Rate limiting is another one. Taking the information that you’ve learned from your load testing and applying that into your security tools to make sure that you can protect on the front end before your server tips over on the other side of that. Those are some things that we do ahead of time. It’s just making sure everything’s buttoned up tightened and in place because we want a successful holiday event. 

Howie Ross: You mentioned bot management and I just want to kind of double check on that because you know so we from a security perspective we often thinks of bots as malicious and trying to do nefarious things on our site. They’re trying to take over people’s accounts and things of that nature. But you know bots are critical to this holiday season traffic increase because bots are what you know search engines and you know other services use to understand what our website offers. You call it bot management as opposed to mitigation. We believe that it is as important to be tuned for the good bots as it is to be tuned to prevent the bad bots 100%. 

Tom Gorup: What we noticed is building out this, our tech trends report also is there’s a lot of web administrators blocking good bots for specific pages that they don’t want indexed by Google, Yandex, what have you. So it is bot management. You can block good bots without having an negative impact to your SEO, but be purposeful around that. And then we also want to block bad bots because we don’t want, you know, maybe we don’t want our prices to be scraped or seats to be reserved. There’s a lot of value that comes from bot management. 

Howie Ross: OK. So you know, so we’ve made it into the fall. And let’s say that now it’s game day and it’s Black Friday. What can we do be doing now other than, kicking our feet up and eating those Thanksgiving leftovers? 

Tom Gorup: I don’t think anybody’s kicking their feet up just yet, at least in the security industry. That’s just when we’re raring getting ready to go. You want to skip that big lunch or you can get a, you know a turkey coma right at the console and that’s not a good place to go to sleep. It’s all hands-on deck. We often run bridges with customers where everybody’s on that bridge talking about you know, and I’m sure Ellery can dig into some of that the observability around the site performance. 

Then we’re also constantly inspecting traffic. Are we seeing attacks and swatting down even the smallest of attacks to make sure that we can ensure space. Security isn’t just protecting the app from the attacks, but it’s also ensuring that we maximize availability here, right? Why waste web requests or CPU cycles on what do we see as known attacks when we can shut them down? A lot of scrutiny, watching the console and just keeping up in pace with the engineering team, the network teams to make sure that we’re all in lock sync. Usually it’s run through a, hey, we’re going to open it up at this time and run 346 hours afterwards, make sure we have good coverage of that event. 

Ellery Womack: Then once your shift ends, you can do some retail therapy with all the discounts. 

Howie Ross: And then Ellery, in addition to that retail therapy, what are the engineering and business teams doing at this time? 

Ellery Womack: Well, one thing we’re constantly at is the analytics. People are checking to make sure are we hitting our targets. A lot of our customers have already done sort of dry runs of events for Black Friday and Cyber Monday where they’ve done promotions that help them forecast the spikes in demand and load and browsing patterns of their users. We’re validating that to make sure that we’re getting the right number of users, we’re getting the right number of add to cart events and people, you know, browsing products and generating product impressions and so on. We’re making sure that things are within our expectations. We want to make sure that performance is on par. Checking, you know, live performance data with a run tool which is real time user monitoring is very helpful. 

You want to make sure that the website is behaving the way it should. Not only from like a server perspective that the static assets and the content is being served, but also that the user’s browser is rendering everything appropriately is another key aspect of that. We want to make sure that the site stays running smooth. We have live insights on everything and beyond just the performance of the website, but the business KPIs and other metrics that are meaningful to the business stakeholders are also being met makes a lot of sense. 

I hope this has provided an overview of the things that organizations should be thinking about and really executing starting right now to make sure that they are prepared for this critical holiday season. We touched on the importance of security and performance testing and remediation having proper observability in regards to our traffic analytics and other and other insights. Make sure we all have all of our security protocols and tools in place and that most importantly our teams are prepared and are going to be available during this. When we know a lot of people travel and you want to try to take some time away. But given how important this time of year is for businesses, we really hope that this has given you all some food for thought and some actions that you can take immediately. 

I just want to thank Ellery and Tom once again for joining us here and thank you all for listening to this episode of Beyond the Edge. See you next time.