Preventing Outages in 2024

Outages have affected some of the most prominent names in the tech industry, underscoring the critical need for robust IT resilience. From AWS’s trio of outages in December 2021 to the major disruption in October 2021 that brought down Facebook, Instagram, WhatsApp, and related services, these incidents highlight the widespread impact outages can have. Even seemingly minor outages, such as Amazon’s search function being unavailable to 20% of global users for two days in December 2022, can disrupt key functionalities and erode user trust. Most recently, the Microsoft CrowdStrike outage in July 2024 further illustrated the vulnerability of even the most advanced IT infrastructures. In this blog learn about preventing outages in 2024.

When significant incidents like these occur, the stakes are high, affecting not only revenue and the bottom line but also a company’s reputation and brand. This is why vigilance and proactive strategies are essential. Although preventing every outage is impossible, the right measures can significantly mitigate their impact. This article explores six critical lessons learned from recent failures and offers practical advice to help organizations enhance their IT resilience and avoid becoming the next headline.

1. Monitor What Matters

Understanding that not everything is within our control is crucial. IT teams often focus on the elements they can directly influence, such as containers, VMs, hardware, and code. While this is important, it’s equally vital to monitor the entire system, including components beyond immediate control. Issues can arise in third-party services like CDNs, managed DNS, and backbone ISPs, which can impact users and the business. Developing a comprehensive Internet Performance Monitoring (IPM) strategy that includes monitoring output and performance is essential. This approach ensures that even external factors affecting user experience are under surveillance, enabling prompt detection and resolution of issues.

2. Map Your Internet Stack

A common misconception is that unchanged components will continue to function flawlessly. However, the internet’s infrastructure, including DNS, BGP, TCP configurations, SSL, and networks, is complex and interconnected. Over-reliance on cloud services can obscure the underlying network’s visibility, making problem detection challenging. Continuous monitoring of these critical elements and having a well-prepared response plan are crucial. Teams must practice their responses regularly to maintain muscle memory, ensuring quick and efficient resolution when issues arise.

3. Intelligently Automate

Automation has revolutionized IT operations, enhancing efficiency and reducing errors. However, it’s essential to apply the same rigor to automation as to production systems. Design flaws in automation scripts, like those seen in the Facebook outage of October 2021, can lead to significant disruptions. Thorough testing and design consideration for potential failures are necessary to ensure robust automation. Integrating comprehensive testing into the automation design and implementation processes helps prevent surprises and minimizes risks.

4. Trust and Verify

Relying on multiple vendors and teams for critical operations necessitates a “trust and verify” approach. Changes made by one team or vendor can inadvertently impact others, spreading issues across the system. Understanding the dependencies within your Internet Stack is vital. Regularly verifying the plans and changes implemented by vendors ensures that your operations remain unaffected by external changes. This proactive approach helps identify and mitigate potential risks before they escalate into full-blown outages.

5. Implement an Internet Performance Monitoring Plan

A well-defined Internet Performance Monitoring (IPM) plan is crucial for maintaining system reliability. Establishing performance baselines before changes allows for accurate comparisons and trend analysis. This approach helps detect issues like increased latency, dropped connections, or slower DNS lookups early. Monitoring both internal and external environments ensures comprehensive visibility into system performance from the user’s perspective. This holistic approach to monitoring provides a 360-degree view, helping identify and address performance issues promptly.

6. Practice, Practice, Practice

The most critical lesson is the importance of regular practice. Ensuring teams are prepared for failures involves more than just having a plan. Regularly practicing crisis response, designing robust playbooks, and planning for vendor outages are essential steps. Turning practice sessions into engaging, game-like scenarios can help teams remain sharp and responsive during actual outages. This proactive preparation minimizes response times and reduces the mean time to repair (MTTR), ensuring swift recovery from disruptions.

Conclusion

Preventing outages in 2024 requires a multifaceted approach that includes monitoring, mapping, automation, verification, and continuous practice. By learning from past failures and implementing these strategies, organizations can enhance their IT infrastructure’s resilience and reliability, ensuring smooth operations and uninterrupted user experiences.

The recent outages among major tech giants highlight the critical importance of robust IT resilience. Events like AWS’s outages, Facebook’s October 2021 disruption, Amazon’s search functionality issue, and the recent Microsoft CrowdStrike outage in July 2024 demonstrate that no company is immune to these incidents. However, by implementing proactive strategies, organizations can significantly mitigate their impact.

At Protected Harbor, we understand what’s at stake during significant outages, from revenue loss to reputational damage. Our Managed Services Program offers a comprehensive solution to achieve and maintain Internet resilience. With 24/7/365 support, our seasoned experts provide training, onboarding assistance, and best-practice processes tailored to your needs. We can extend or complement your team, providing regular KPI updates and optimization opportunities, ensuring world-class expertise and an extra layer of protection.

Find out more and ensure your organization’s resilience with Protected Harbor at: https://www.protectedharbor.com/it-audit

CrowdStrike vs. Delta: Who’s to Blame for the Global Tech Outage?

A heated legal battle has erupted between cybersecurity giant CrowdStrike and Delta Air Lines over a recent global technology outage that caused major disruptions worldwide. The outage, which many initially attributed solely to a flawed software update from CrowdStrike, left Delta struggling to recover, resulting in the cancellation of about 5,000 flights, roughly 37% of its schedule, over four days. Crowdstrike vs. Delta: Who’s to blame for the global tech outage?

Delta Points Fingers, CrowdStrike Pushes Back

Delta’s chief executive, Ed Bastian, estimated that the outage cost the airline $500 million, covering expenses like compensation and hotel stays for affected passengers. Delta has since hired Boies Schiller Flexner, a prominent law firm, to pursue legal claims against CrowdStrike.

In a letter to Delta, CrowdStrike’s lawyers from Quinn Emanuel Urquhart & Sullivan pushed back against the airline’s claims. They emphasized that while the software update did cause disruptions, many other businesses, including several airlines, managed to recover within a day or two. Delta, on the other hand, faced prolonged issues, with about 75% of its remaining flights delayed.

Breakdown in Communication

CrowdStrike apologized for the inconvenience caused and highlighted their efforts to assist Delta’s information security team during the outage. They noted that their CEO had offered on-site help to mitigate the damage, but Delta did not respond to or accept the offer. CrowdStrike’s letter also questioned why Delta’s recovery lagged behind other airlines and suggested that any liability should be limited to under $10 million.

Investigation and Expert Opinions

The U.S. Department of Transportation has launched an investigation into the incident, with Secretary Pete Buttigieg pointing out that Delta might have been particularly vulnerable due to its reliance on affected software and its overloaded crew scheduling system.

Other major carriers like American and United Airlines managed to rebound more quickly. Aviation experts suggest that Delta’s strategy of leaning heavily on cancellations rather than delays, coupled with the intense activity at its main hub in Atlanta, contributed to its extended recovery time.

Learning from the Past

The situation echoes Southwest Airlines’ ordeal in 2022 when severe winter storms caused massive disruptions. Southwest struggled due to insufficient equipment and an overwhelmed crew scheduling system, ultimately canceling nearly 17,000 flights over ten days.

Conclusion

As the investigation unfolds and legal actions progress, it remains clear that proactive measures and robust IT infrastructure are crucial for managing such crises. At Protected Harbor, we pride ourselves on delivering unmatched uptime and proactive monitoring to prevent and swiftly address any issues. Our commitment to excellence ensures that our clients enjoy seamless operations, well above industry standards.

For more insights on tech outages and proactive IT solutions, check out our previous blog on the Microsoft CrowdStrike outage.

How a Software Update Crashed Computers Globally

And why the CrowdStrike outage is proving difficult to resolve.

On Friday 19 July, the world experienced a rare and massive global IT outage. These events, while infrequent, can cause significant disruption. They often originate from errors in centralized systems, such as cloud services or server farms. However, this particular outage was unique and has proven to be difficult and time-consuming to resolve. The culprit? A faulty software update was pushed directly to PCs by CrowdStrike, a leading cybersecurity firm serving over half of the Fortune 500 companies.

Windows Global IT Outage: The Beginning

The outage began with a Windows global IT outage stemming from faulty code distributed by CrowdStrike. This update caused affected machines to enter an endless reboot loop, rendering them offline and virtually unusable. The severity of the problem was compounded by the inability to issue a fix remotely.

Immediate Impacts of the IT Outage

The immediate aftermath saw a widespread Microsoft server down scenario. Systems across various industries were disrupted, highlighting the dependency on stable cybersecurity measures. With computers stuck in an endless cycle of reboots, normal business operations ground to a halt, creating a ripple effect that was felt globally.

The Challenges of a Remote Fix

Why the Global IT Outage is Harder to Fix

One of the most significant challenges in this global IT outage is the inability to resolve the issue remotely. The faulty code rendered remote fixes ineffective, necessitating manual intervention. This meant that each affected machine had to be individually accessed to remove the problematic update.

Manual vs. Automated Fixes

Unless experts can devise a method to fix the machines remotely, the process will be painstakingly slow. CrowdStrike is exploring ways to automate the repair process, which would significantly expedite resolution. However, the complexity of the situation means that even an automated solution is not guaranteed to be straightforward.

Broader Implications of the Outage

Understanding the Broader Impact

The Windows global IT outage has exposed vulnerabilities in how updates are managed and deployed. This incident serves as a stark reminder of the potential risks associated with centralized update systems. Businesses worldwide are now reevaluating their dependence on single-point updates to avoid similar disruptions in the future.

Preventing Future IT Outages

Moving forward, organizations could implement more rigorous testing protocols and fail-safes to prevent such widespread disruptions. Additionally, there may be a shift towards more decentralized update mechanisms to minimize the risk of a single point of failure.

Conclusion

The global IT outage caused by a faulty CrowdStrike update serves as a critical lesson for the tech industry. The incident underscores the need for more resilient and fail-safe update mechanisms to ensure that such disruptions do not occur again. As organizations worldwide continue to grapple with the consequences, the focus will undoubtedly shift towards preventing future occurrences through improved practices and technologies.

FAQs

What caused the global IT outage?

The outage was caused by a faulty CrowdStrike software update, which led to affected computers to enter an endless reboot loop.

How widespread was the outage?

The outage was global, affecting businesses and systems across various industries worldwide.

Why is it difficult to fix the outage?

The affected machines cannot be remotely fixed due to the nature of the faulty code. Each computer needs to be manually accessed to remove the problematic update.

Is there a way to automate the fix?

CrowdStrike is exploring automated solutions, but the complexity of the issue means that a straightforward automated fix may not be feasible.

What are the broader implications of the outage?

The incident highlights the vulnerabilities in centralized update systems and may lead to more rigorous testing protocols and decentralized update mechanisms.

How can future IT outages be prevented?

Implementing more robust testing procedures and decentralized update systems can help prevent similar outages in the future.

Microsoft Windows Outage: CrowdStrike Falcon Sensor Update

Like millions of others, I tried to go on vacation, only to have two flights get delayed because of IT issues.  As an engineer who enjoys problem-solving and as CEO of the company nothing amps me up more than a worldwide IT issue, and what frustrates me the most is the lack of clear information.

From the announcements on their website and on social media, CloudStrike issued an update and that update was defective, causing a Microsoft outage. The computers that downloaded the update go into a debug loop; attempt to boot, error, attempt repair, restore system files, boot, repeat.

The update affects only Windows systems, Linux and Macs are unaffected.

The wide-spread impact and Windows server down focus; is because Microsoft outsourced part of their security to Cloudstrike, allowing CloudStrike to directly patch the Windows Operating System.

Microsoft and CrowdStrike Responses

Microsoft reported continuous improvements and ongoing mitigation actions, directing users to its admin center and status page for more details. Meanwhile, CrowdStrike acknowledged that recent crashes on Windows systems were linked to issues with the Falcon sensor.

The company stated that symptoms included the Microsoft server down and the hosts experiencing a blue screen error related to the Falcon Sensor and assured that their engineering teams were actively working on a resolution to this IT outage.

There is a deeper problem here, one that will impact us worldwide until we address it.  The technology world is becoming too intertwined with too little testing or accountability leading to a decrease in durability, stability, and an increase in outages.

Global Impact on Microsoft Windows Users 

Windows users worldwide, including those in the US, Europe, and India, experienced the Windows server outage or Windows server downtime, rendering their systems unusable. Users reported their PCs randomly restarting and entering the blue screen error mode, interrupting their workday. Social media posts showed screens stuck on the recovery page with messages indicating Windows didn’t load correctly and offering options to restart the PC.

If Microsoft had not outsourced certain modules to CloudStrike, then this Windows server outage wouldn’t have occurred. Too many vendors build their products based on assembling a hodgepodge of tools, leading to outages when one tool fails.

The global IT outage caused by CrowdStrike’s Falcon Sensor has highlighted the vulnerability of interconnected systems, especially during Windows server downtime.

I see it in the MSP industry all the time; most (if not all) of our competitors use outsourced support tools, outsourced ticket systems, outsourced hosting, outsourced technology stack, and even outsourced staff. If everything is outsourced, then how do you maintain quality?

We are very different, which is why component outages like what is occurring today do not impact us. The tools we use are all running on servers we built, those servers are running in clusters we own, which are running in dedicated data centers we control. We plan for failures to occur, which to clients translates into unbelievable uptime, and that translates into unbelievable net promotor scores.

The net promotor score is an industry client “happiness” score; for the MSP industry, the average score is 32-38, but at Protected Harbor, our score is over 90.

Because we own our own stack, because all our staff are employees with no outsourcing, and because 85%+ of our staff are engineers, we can deliver amazing support and uptime, which translates into customer happiness.

If you are not a customer of ours and your systems are affected by this Windows server outage in the US, wait. Microsoft downtime will likely resolve soon when an update is issued, however, a manual update process might be required. If your local systems are not impacted yet, turn them off right now and wait for a couple of hours for Windows server outage in the US updates. For our clients, go to work; everything is functioning perfectly. If your local systems or home system are impacted, contact support, and we will get you running.

What went wrong and why?

On July 19, 2024, CrowdStrike experienced a significant incident due to a problematic Rapid Response Content update, which led to a Windows crash, widely recognized as the Windows Blue Screen of Death (BSOD). The issue originated from an IPC Template Instance that passed the Content Validator despite containing faulty content data. This bug triggered an out-of-bounds memory read, Windows outage cause operating systems to crash. The problematic update was part of Channel File 291, and while previous instances performed as expected, this particular update resulted in widespread disruptions.

The incident highlighted the need for enhanced testing and deployment strategies to prevent such occurrences. CrowdStrike plans to implement staggered deployment strategies, improved monitoring, and additional validation checks to ensure content integrity. They also aim to provide customers with greater control over content updates and detailed release notes. This incident underscores the critical need for robust content validation processes to prevent similar issues from causing outages, such as the one experienced with Microsoft.

Preventing Outages with High Availability (HA)

High Availability (HA) is a fundamental part of data management, ensuring that critical data remains accessible and operational despite unforeseen challenges. It’s a comprehensive approach that employs various strategies and technologies to prevent outages, minimize downtime, and maintain continuous data accessibility. The following are five areas that comprise a powerful HA deployment.

Redundancy and Replication:  Redundancy and replication involve maintaining multiple copies of data across geographically distributed locations or redundant hardware components. For instance, in a private cloud environment, data may be replicated across multiple availability data centers. This redundancy ensures that if one copy of the data becomes unavailable due to hardware failures, natural disasters, or other issues, another copy can seamlessly take its place, preventing downtime and ensuring data availability. For example: On Premise Vs private cloud (AWS) offers services like Amazon S3 (Simple Storage Service) and Amazon RDS (Relational Database Service) that automatically replicate data across multiple availability zones within a region, providing high availability and durability.

Fault Tolerance:  Fault tolerance is the ability of a system to continue operating and serving data even in the presence of hardware failures, software errors, or network issues. One common example of fault tolerance is automatic failover in database systems. For instance, in a master-slave database replication setup, if the master node fails, operations are automatically redirected to one of the slave nodes, ensuring uninterrupted access to data. This ensures that critical services remain available even in the event of hardware failures or other disruptions.

Automated Monitoring and Alerting:  Automated monitoring and alerting systems continuously monitor the health and performance of data storage systems, databases, and other critical components. These systems use metrics such as CPU utilization, disk space, and network latency to detect anomalies or potential issues. For example, monitoring tools like PRTG and Grafana can be configured to track key performance indicators (KPIs) and send alerts via email, SMS, or other channels when thresholds are exceeded or abnormalities are detected. This proactive approach allows IT staff to identify and address potential issues before they escalate into outages, minimizing downtime and ensuring data availability.

For example, we write custom monitoring scripts, for our clients, that alert us to database processing pressure and long-running queries and errors.  Good monitoring is critical for production database performance and end-user usability.

Load Balancing:  Load balancing distributes incoming requests for data across multiple servers or nodes to ensure optimal performance and availability. For example, a web application deployed across multiple servers may use a load balancer to distribute incoming traffic among the servers evenly. If one server becomes overloaded or unavailable, the load balancer redirects traffic to the remaining servers, ensuring that the application remains accessible and responsive. Load balancing is crucial in preventing overload situations that could lead to downtime or degraded performance.

Data Backup and Recovery:  Data backup and recovery mechanisms protect against data loss caused by accidental deletion, corruption, or other unforeseen events. Regular backups are taken of critical data and stored securely, allowing organizations to restore data quickly in the event of a failure or data loss incident.

Continuous Software Updates and Patching:  Keeping software systems up to date with the latest security patches and updates is essential for maintaining Data High Availability. For example, database vendors regularly release patches to address security vulnerabilities and software bugs. Automated patch management systems can streamline the process of applying updates across distributed systems, ensuring that critical security patches are applied promptly. By keeping software systems up-to-date, organizations can mitigate the risk of security breaches and ensure the stability and reliability of their data infrastructure.

Disaster Recovery Planning:  Disaster recovery planning involves developing comprehensive plans and procedures for recovering data and IT systems in the event of a catastrophic failure or natural disaster. For example, organizations may implement multi-site disaster recovery strategies, where critical data and applications are replicated across geographically dispersed data centers. These plans typically outline roles and responsibilities, communication protocols, backup and recovery procedures, and alternative infrastructure arrangements to minimize downtime and data loss in emergencies.

We develop database disaster automatic failure procedures and processes for clients and work with programmers or IT departments to help them understand the importance of HA and how to change their code to optimize their use of High Availability.

An Essential Tool

Data High Availability is essential for preventing outages and ensuring continuous data accessibility in modern IT environments. By employing the strategies we outlined, you can mitigate the risk of downtime, maintain business continuity, and ensure the availability and reliability of critical data and services.

High Availability is available on all modern database platforms and requires a thoughtful approach. We’d be happy to show you how we can help your organization and make your applications and systems fly without disruption. Call us today.

How to Prevent Crashes and Outages?

Today’s workforce relies heavily on computers for day-to-day tasks. If a computer crashes, we tend to get more than just a little agitated.

Fear of being unable to work and get our jobs done for the day races through our minds while anger takes its place in the forefront of trying to fix whatever went wrong, throwing all logic out the window.

When a system abruptly ceases to work, it crashes. The scope of a system failure can vary significantly from one that affects all subsystems to one that is just limited to a particular device or just the kernel itself.

System hang-ups are a related occurrence in which the operating system is nominally loaded. Still, the system stops responding to input from any user/device and ceases producing output. Another way to define such a system is as frozen.

This blog will explain how to prevent crashes and outages in 6 easy steps.

What is a System Crash and an Outage?

A system crash is a term used to describe a situation in which a computer system fails, usually due to an error or a bug in the software. An outage may also be caused by an application program, system software, driver, hardware malfunction, power outage, or another factor.

“A system freeze,” “system hang,” or “the blue screen of death” are the other terms for a system crash.

An outage is a general term for an unexpected interruption to a service or network. Outages can be planned (for example, during maintenance) or unplanned (a fault occurs). Outages can last for minutes, hours, days, or even weeks.

Main Reasons for Crashes and Outages

System outages can be caused by various factors, from hardware failures to software glitches. In many cases, outages are the result of a combination of factors. The following are some of the most common causes of system outages:

• Hardware failures: A defective component can cause an entire system to fail. Servers, hard drives, and other components can fail, leading to an outage.
• Software glitches: Software glitches can also cause system outages. A coding error or a bug in the software can disrupt the system’s regular operation.
• Power outages: A power outage can cause the entire system to fail. The system may be damaged permanently if the power is not restored quickly.
• Natural disasters: Natural disasters such as hurricanes, tornadoes, and earthquakes can damage or destroy critical components of the system.

System crashes can be caused by various things, from software defects to hardware failures. Sometimes, the crash may even be caused by something as simple as a power outage or due to a more severe issue, such as a virus or malware infection.

• Overheating: When a computer’s CPU or graphics card gets too hot, it can cause the system to freeze or crash. This is often the result of inadequate cooling or dust and dirt buildup inside the computer.
• Bad drivers: If a driver is outdated, corrupt, or incompatible with the operating system, it can cause the system to crash. In some cases, this can even lead to data loss or permanent damage to the computer.

Preventions Against Crashes and Outages

Nobody wants their computer to crash, but it will happen eventually. Here are a few ways to help prevent them and keep your computer running smoothly.

1.    Keep Your Software Up to Date by Installing Updates

One of the best ways to prevent crashes and outages is by updating your software. This means installing updates as soon as they become available. You should also keep your operating system and programs up to date. These updates can fix bugs and security vulnerabilities, so installing them as soon as they are released is essential.

2.    Avoid Clicking on Links or Downloading Files from Unknown Sources

It’s essential to be proactive in preventing crashes and outages. One way to do this is to avoid clicking on links or downloading files from unknown sources, as these can often contain malware that can harm your computer or network. Additionally, you should routinely back up your data to recover it if something goes wrong.

3. Make Sure You Have Good Antivirus and Anti-Malware Programs

One of the most important things you can do to prevent crashes is to ensure that your antivirus and anti-malware programs are up to date. These programs can help protect your computer from malware infections, which can cause crashes.

4.    Close Programs You’re Not Using

One of the best ways to prevent crashes and outages is to close any programs you’re not using. When too many programs are open, your computer’s performance can suffer, leading to crashes and outages.

5.    Delete Unwanted Files

Another way to improve your computer’s performance is to regularly delete files you no longer need. This will free up space on your hard drive, allowing your computer to run more efficiently.

6.    Try a Trusted Disk Clean-Up to Free Up Some Space

This will help your computer run faster and smoother. You can even defragment your hard drive occasionally to keep it organized and running smoothly.

Remember to install updates for your operating system and software as soon as they are available. Keeping your computer clean and organized will help prevent crashes and outages.

Final Words

Don’t forget that you are the one running the computer, not the other way around. Therefore, it is your top priority to maintain the computers for improved performance and to continually check for any disruptions that could result in computer failures.

Try to pay attention to the little warnings your system sends you so you can save not just your computer but also yourself from a mental spiral.

Now that you know what causes crashes and outages, you can stay on top of them by following a few simple rules. Regularly monitoring your system resources, updating your software, keeping your system up to date, and having a good antivirus are the best ways to keep your computer running smoothly and keep both crashes and outages at bay.

Taking care of your data can help you to protect it from crashes and outages. You can get expert help from Protected Harbor to manage and maintain your systems and data. Protected Harbor provides an added layer of security that helps to ensure the uninterrupted flow of business-critical data. Additionally, our expert team monitors and detects any threats or updates to your system in order to ensure a smooth, efficient operation that saves it from crashing.

We help you to avoid the most common causes of data loss and system outages. These include network issues due to malicious activity, viruses, and system overload; natural disasters; power outages; and accidental deletion or corruption of data. You’re less likely to experience a system outage or lose critical data if you have a backup, plus 99.99% uptime is our guarantee.

Sign up now and get a free consultation to learn more about how Protected Harbor can keep your company’s data secure and your business up and running.