Preventing Outages in 2024

Outages have affected some of the most prominent names in the tech industry, underscoring the critical need for robust IT resilience. From AWS’s trio of outages in December 2021 to the major disruption in October 2021 that brought down Facebook, Instagram, WhatsApp, and related services, these incidents highlight the widespread impact outages can have. Even seemingly minor outages, such as Amazon’s search function being unavailable to 20% of global users for two days in December 2022, can disrupt key functionalities and erode user trust. Most recently, the Microsoft CrowdStrike outage in July 2024 further illustrated the vulnerability of even the most advanced IT infrastructures. In this blog learn about preventing outages in 2024.

When significant incidents like these occur, the stakes are high, affecting not only revenue and the bottom line but also a company’s reputation and brand. This is why vigilance and proactive strategies are essential. Although preventing every outage is impossible, the right measures can significantly mitigate their impact. This article explores six critical lessons learned from recent failures and offers practical advice to help organizations enhance their IT resilience and avoid becoming the next headline.

1. Monitor What Matters

Understanding that not everything is within our control is crucial. IT teams often focus on the elements they can directly influence, such as containers, VMs, hardware, and code. While this is important, it’s equally vital to monitor the entire system, including components beyond immediate control. Issues can arise in third-party services like CDNs, managed DNS, and backbone ISPs, which can impact users and the business. Developing a comprehensive Internet Performance Monitoring (IPM) strategy that includes monitoring output and performance is essential. This approach ensures that even external factors affecting user experience are under surveillance, enabling prompt detection and resolution of issues.

2. Map Your Internet Stack

A common misconception is that unchanged components will continue to function flawlessly. However, the internet’s infrastructure, including DNS, BGP, TCP configurations, SSL, and networks, is complex and interconnected. Over-reliance on cloud services can obscure the underlying network’s visibility, making problem detection challenging. Continuous monitoring of these critical elements and having a well-prepared response plan are crucial. Teams must practice their responses regularly to maintain muscle memory, ensuring quick and efficient resolution when issues arise.

3. Intelligently Automate

Automation has revolutionized IT operations, enhancing efficiency and reducing errors. However, it’s essential to apply the same rigor to automation as to production systems. Design flaws in automation scripts, like those seen in the Facebook outage of October 2021, can lead to significant disruptions. Thorough testing and design consideration for potential failures are necessary to ensure robust automation. Integrating comprehensive testing into the automation design and implementation processes helps prevent surprises and minimizes risks.

4. Trust and Verify

Relying on multiple vendors and teams for critical operations necessitates a “trust and verify” approach. Changes made by one team or vendor can inadvertently impact others, spreading issues across the system. Understanding the dependencies within your Internet Stack is vital. Regularly verifying the plans and changes implemented by vendors ensures that your operations remain unaffected by external changes. This proactive approach helps identify and mitigate potential risks before they escalate into full-blown outages.

5. Implement an Internet Performance Monitoring Plan

A well-defined Internet Performance Monitoring (IPM) plan is crucial for maintaining system reliability. Establishing performance baselines before changes allows for accurate comparisons and trend analysis. This approach helps detect issues like increased latency, dropped connections, or slower DNS lookups early. Monitoring both internal and external environments ensures comprehensive visibility into system performance from the user’s perspective. This holistic approach to monitoring provides a 360-degree view, helping identify and address performance issues promptly.

6. Practice, Practice, Practice

The most critical lesson is the importance of regular practice. Ensuring teams are prepared for failures involves more than just having a plan. Regularly practicing crisis response, designing robust playbooks, and planning for vendor outages are essential steps. Turning practice sessions into engaging, game-like scenarios can help teams remain sharp and responsive during actual outages. This proactive preparation minimizes response times and reduces the mean time to repair (MTTR), ensuring swift recovery from disruptions.

Conclusion

Preventing outages in 2024 requires a multifaceted approach that includes monitoring, mapping, automation, verification, and continuous practice. By learning from past failures and implementing these strategies, organizations can enhance their IT infrastructure’s resilience and reliability, ensuring smooth operations and uninterrupted user experiences.

The recent outages among major tech giants highlight the critical importance of robust IT resilience. Events like AWS’s outages, Facebook’s October 2021 disruption, Amazon’s search functionality issue, and the recent Microsoft CrowdStrike outage in July 2024 demonstrate that no company is immune to these incidents. However, by implementing proactive strategies, organizations can significantly mitigate their impact.

At Protected Harbor, we understand what’s at stake during significant outages, from revenue loss to reputational damage. Our Managed Services Program offers a comprehensive solution to achieve and maintain Internet resilience. With 24/7/365 support, our seasoned experts provide training, onboarding assistance, and best-practice processes tailored to your needs. We can extend or complement your team, providing regular KPI updates and optimization opportunities, ensuring world-class expertise and an extra layer of protection.

Find out more and ensure your organization’s resilience with Protected Harbor at: https://www.protectedharbor.com/it-audit