AWS global outage; disrupts services and aftermath

Facebook, Alexa, Reddit, Netflix, and more apps were affected by the AWS outage.

If you faced problems logging in to Amazon.com for shopping ahead of Christmas, you’re not alone. On Tuesday, December 7, large parts of the internet and apps reported disrupted services based on the AWS platform. Netflix, Alexa, Disney+, Reddit, and IMDB are some of the services reported downtime.

UPDATE: 19:35 EST/16:35 PST, The official Amazon Web Services dashboard published the following affirmation. ” With the network device problems resolved, we are now operating towards the recovery of any impaired services. We will roll out additional updates for impaired services within the connected entry in the Service Health Dashboard.

AWS down

Users began reporting issues around 10:45 AM ET on Tuesday about the outage and took to Twitter and other social media platforms to discuss. More than 24,000 people reported cases with Amazon, which included Prime Video and other services, on DownDetector.com. The website collects outage reports from multiple sources, including user-submitted errors.

3 hours later we have a detailed message with the important information like the real affected services. #awsDown pic.twitter.com/t3ooQohWn1

— José Antonio Ruiz (@joseant_ruiz) December 7, 2021

The AWS global outage recovery problems came from the US-EAST-1 AWS region in Virginia, so users elsewhere may not have noticed as many issues, and even if you were affected, you might have seen a slightly slower loading time while the network redirected your requests.

Peter DeSantis, AWS’ vice president of infrastructure, led a 600-person internal call about the then-ongoing outage. Some said it was likely an internal issue, and others pointed to more nefarious possibilities.
“We have mitigated the underlying issues that caused network devices in the US-EAST-1 Region to be impaired,” AWS said on its status page.

What caused the outage?

Engineers at Amazon Web Services (AWS), the enormous cloud computing provider in the US, are still unsure of AWS global outage causes on December 7. AWS does not list any issues on the status page currently. Previous outages have also not been reflected on the status page or even brought down the site entirely, so it is not unusual.
There is, however, a 500 Server error on the specific page for the us-east-1 AWS Management Console Home, instead of information about the Northern Virginia region.

A 500 server internal error means their server is trying to show the requested web page (the technical answer is delivered rather than the web page). But it can’t show the webpage because something within the server failed – for example, the storage failed, so the file is unavailable.

“Possible causes are internal routing problems within Amazon, a defective Amazon-wide update, an Amazon-wide misconfiguration. A defective API (application programming interface) or network device issue might also be a cause of the amazon console down,” said Richard Luna, CEO, Protected Harbor.

Amazon global outage comes just a few months after Meta Platforms, Inc. (FB) went offline due to network problems, affecting some of its most popular apps, including WhatsApp, Instagram, and Facebook Messenger.
The research firm Gartner Inc. estimates that major cloud platforms suffer significant outages once per quarter per year. Many people felt the AWS service disruption; however, since AWS controls about 90% of the cloud infrastructure market and many people continue to work and study from home during the pandemic, the outage was widely felt. Gartner vice president Sid Nag told The Wall Street Journal that these guys have become almost too big to fail. Our day-to-day lives rely heavily on cloud computing services.

Hasn’t This Happened Before?

Yes, AWS downtime is not a new occurrence. The last major AWS global outage happened in November 2020. Numerous other disruptive and lengthy cloud service interruptions have involved various providers. In June, the behind-the-scenes content distributor Fastly experienced a failure that briefly took down dozens of major internet sites, including CNN, The New York Times, and Britain’s government home page. Another cloud service interruption that month affected provider Akamai during peak business hours in Asia.

In the October outage, Facebook — now known as Meta Platforms — blamed a “faulty configuration change” for an hours-long worldwide AWS downtime that took down Instagram and WhatsApp in addition to its titular platform.

Credible solutions

On Tuesday, the world received a reminder of just how much we rely on Amazon Web Services and AWS global outage recovery. A simple outage for a brief period disrupted the operations and services of millions of people. Amazon is in the monopoly and would never partner with another provider. So the simplest solution is to opt for a service provider who puts customers first.

Amazon, as big it is, is still just one location and provides a single server location to the clients. At its core, it is one batch of servers. Protected Harbor solves this problem by spreading the customers across multiple server locations, preventing a site-wide misconfiguration. We protect our clients by using various services; we expect one service to fail- that gives us time to resolve and repair the situation quickly.

We differentiate from other providers by being proactive and planning for failures like this. We do it all the time- partner with other providers to deliver unmatched services to the customers because their satisfaction comes first.

Key Takeaways:

• An hours-long AWS outage crippled popular websites and disrupted smart devices, as well as creating delivery delays at Amazon warehouses.
• Companies like Facebook, Netflix, Reddit, IMDB, Disney+, and more were affected by the outage.
• Amazon stated that it “identified the root cause” but yet to reveal what precisely the root cause was?
• AWS controls almost 90% of the cloud services market, and the outages are not uncommon.
• Now is the time to choose the provider which satisfies you and your business needs.

Go complete risk-free

Protected Harbor is the underdog player in the market that exceeds the customer’s expectations. With its Datacenter and Managed IT services, it has stood the test of customers, and “Beyond expectations” is quoted by all customers. Best in segment cloud services with optimum IT support, safety, and security, it’s a no-brainer why organizations choose to stay with us. This way to the crème de la crème.

Facebook Down Globally: A Case of the Mondays for Facebook, Instagram, and WhatsApp as they go dark midday Monday

Some of the biggest social media sites on the planet, including Facebook, went down globally starting at noon EDT and are still not up in some regions. That’s right, no Instagram #motivationmondays or “Ugh, is it Friday Yet?” Facebook posts from your first semester freshman year college roommate. As the sky was falling for millennials (myself included) and your favorite newly-political aunt, the teams at Facebook were scrambling to keep their sites (including Instagram and WhatsApp, of which both are Facebook-owned) operating.

Facebook Chief Technology Officer Mike Schoepfer took to Twitter to address the situation:

“*Sincere* apologies to everyone impacted by outages of Facebook-powered services right now. We are experiencing networking issues and teams are working as fast as possible to debug and restore as fast as possible”

Facebook outages of this magnitude are rare, to have Facebook down globally for this amount of time is something that hasn’t happened in years. To put in perspective just how impactful the Facebook outage is, the term “Facebook down” was Googled more than 5,000,000 times today alone.
The cause of the outage is speculated to be tied to a recently aired “60 Minutes” segment where whistleblower and former Product Manager at Facebook, Frances Haugen claimed that Facebook knows the platform is used to spread hate and that they have tried hiding evidence of it, of course, Facebook denies this claim.

“The interview followed weeks of reporting about and criticism of Facebook after Haugen released thousands of pages of internal documents to regulators and the Wall Street Journal. Haugen is set to testify before a Senate subcommittee on Tuesday.” According to CNN

Jake Williams, CTO of cybersecurity firm BreachQuest mentioned to the Associated Press that this was an “operational issue” caused by human error.

Regardless of the reasoning, I’m sure this will be an issue that will be discussed for quite some time in the technology space as the outage was global and not regional. Facebook shares opened at $335.50 and closed at $326.32, a drop of 4.89%.

Nonetheless, as I’m sure many were beside themselves that they couldn’t post a nice “Los Angeles” filtered photo of their lunch on Instagram to show their followers, we can only hope, for Facebook’s sake, they can have it fixed by the time we want to show off our dinner.

It has been confirmed that per a Facebook blog that the outage was due to a botched configuration change. Facebook posted the following:

“Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication. This disruption to network traffic had a cascading effect on the way our data centers communicate, bringing our services to a halt.”

Information about the depth of the outage continues to grow, it’s reported that Facebook’s internal chat was also down limiting communications within the company, it even went so far as the employee’s keycards began to fail which made them unable to enter certain buildings.

The Krebs on Security blog explains the problem as follows:

“…sometime this morning Facebook took away the map telling the world’s computers how to find its various online properties. As a result, when one types Facebook.com into a web browser, the browser has no idea where to find Facebook.com, and so returns an error page.”

The Facebook campus was only the beginning, due to the sites interconnectivity it stretched to sites that were utilizing Facebook’s authentication process as well, these effects resonated across the board, from those who rely on Facebook/WhatsApp for primary communication purposes, to small businesses unable to get in touch with their customer base, and even the large number of folks in countries where Facebook is their internet.

We will continue to update as information becomes available.

Protected Harbor’s New PBX Phone System

Protected Harbor is proud to introduce a state of the art phone system 3CX.  It is available to all our current clients and anyone in the market looking to upgrade their current set up.  Protected Harbor partnered with 3CX, and provides system configuration and support.

The benefits of this new phone system are plentiful, starting with the end-user being in complete control.  This PBX can be installed on-premise and virtualized on a Linux or Windows platform.   It is easily set up via iOS and Android apps for remote work via QR code.  Staff can easily be added, and voicemail can be set up in minutes.  The user management is simple and easy which will save countless hours of work.

The biggest benefit perhaps, of our system is that it has a softphone so you can make and receive business calls from your PC, tablet or smartphone.  There is no need to be tied to a hard physical phone in your office.  It’s perfect for businesses that have employees working from home during COVID-19.

Moreover, many of the phone features that other vendors classify as add-ons, such as video conferencing, iOS and Android mobile apps or call center features, are included with Protected Harbor’s PBX. So, there are no hidden costs for the features you need.

Switching to this configuration makes complete sense when you compare pricing to that of other PBX vendors such as 8×8 and Avaya. Our system provides a complete unified communications solution that is easy to manage, flexible and affordable. Whether you are a small business or a large enterprise, you can save thousands and avoid all the hassle of purchasing additional extensions and add-ons as your business grows.

Think less about maintaining your PBX, and more about your business – call or email us today to find out more about Protected Harbor’s new phone system! www.protectedharbor.com

Disadvantages of AWS, Azure, and Other Big Brand Hosting

When it comes to hosting for a business, you don’t want to use just anyone. There are many critical factors to consider including security, stability, uptime, scalability, and more. Because of this, many businesses gravitate towards big, established brands for hosting and management such as Amazon’s AWS or Microsoft’s Azure.

Companies like these can likely provide well beyond your technical needs. That’s not to say they’re all the exact same. Azure caters to Microsoft products, allowing large companies to move their Windows-based infrastructure online more easily. Meanwhile, AWS boasts their general flexibility and universal capabilities.

Each brand has its unique strengths. When it comes to weaknesses, however, there are some overlapping issues that basically any large-scale hosting company deals with.

Overwhelming Options

Right from the start, many businesses are overwhelmed with the variety of packages and services offered by large hosting companies. AWS, for example, greets you with an entire library of services and products to choose from. Simply trying to find basic website hosting proves to be difficult and confusing.

Unclear Pricing Structures

Equally confusing are the pricing structures. Many companies try to sign you up on free trials or temporary discount pricing, only for you to discover the true inflated price months down the road.

These companies also tend to work off a pay-per-use model. In other words, the more data you process and store, the more your hosting costs. While it sounds nice in theory, as you only pay for what you use, it can make it very difficult to predict your monthly costs as prices fluctuate.

It also leaves you severely exposed to DDoS attacks.

DDoS attacks infect a large number of devices with malware and then use them to unleash a coordinated flood of traffic on an unsuspecting network. In addition to slowing down and (likely) crashing your systems, it results in a massage spike of data use.

The average size of a DDoS attack is 2.5 gigabits per second. If you’re being charged per data used, you’ll be left with a very large hosting bill following a DDoS attack.

Advanced Knowledge

Once you’ve figured out what your business needs, the real difficultly begins. Within any given service, there are countless add-ons, tools, settings, and more. While this provides a lot of flexibility and customization, it requires a lot of work and understanding. The deeper your needs go, the deeper your understanding needs to be.

Each platform is different, which means you either need to hire someone who is experienced on a particular platform, or you’ll need to invest in training a current employee. The question is, do you want someone learning a new platform as they’re managing your IT needs?

Support Problems

Platforms like AWS and Azure do offer their own technical support, should you require it. In fact, they often provide a certain amount of free support when you sign up. However, those hours can quickly be eaten up during the onboarding. After that, you’ll pay for support.

Things can get very expensive very quickly.

A better solution is generally to find a third party to help manage and maintain your hosting needs. This can provide more affordable support, but it also adds complexity to your hosting management and expenses.

A Simpler, Yet Powerful Hosting Solution

At Expedient Technology Services, we provide straightforward, yet diverse hosting solutions for businesses of all sizes. Whether you’re a start-up or an enterprise, we have the capabilities to meet your specific needs.

We operate under flat rate costs, so you know exactly what you’ll pay every month. We can even bundle in support hours so that you get professional assistance when you need it. As your company grows, we can easily scale our services with you.

While our initial costs may seem higher, they’re generally cheaper in the long run. Best of all, they’re much less stressful to understand and manage. After all, ETS exists to provide Stress-Free IT. For hosting, computer services, and technical support in Dayton, Ohio, and beyond, contact ETS today.

What Does Downtime Cost the Average Business?

How did Twitter get hacked?

On July 15^th many Twitter accounts were compromised.  How did this happen to a company like Twitter?

‘This was the worst social media hack ever happened in history’

The security involvement of the hack are also wide-reaching, not just for Twitter but for other social platforms.

Early suggestions are the hackers managed to access administration privileges, which allowed them to bypass the passwords of any account they wanted.

Twitter appeared to confirm this in a tweet saying: “We detected what we believe to be a co-ordinated social-engineering attack by people who successfully targeted some of our employees with access to internal systems and tools.”

As we generate more content online we are creating a larger digital footprint.  These attackers simply contacted Twitter and asked for the names of key personal, the head of the customer service, their CIO, etc.  Once the attackers knew the identity of key individuals they then researched their web pages, Facebook links, LinkedIn profiles, etc.

The attackers were able to gain enough information from those pages to be able to correctly answer Twitter’s support questions and gain access to those accounts.

Once the attackers had access to an Admin account they could reset end-user accounts and then login as those users.  It was that easy.

Some questions that should be asked; What would have helped prevent this disaster?  Is your system(s) vulnerable to a similar attack?   How can your system(s) be protected?

2FA or Two Factor Authentication would have stopped this attack.  With 2FA the mobile device is registered to the account and the login is not possible until a code on the mobile device is entered.

At Protected Harbor we support 2FA for all systems, allowing our customers to be safe, secure, and protected, as in Protected Harbor.

TAKING YOUR PRIVACY BACK FROM GOOGLE

With every new technology comes benefits and concerns.

Now Google can present coupons for stores you have purchased from while you are driving past them, and tell you what you need to buy in your smart refrigerator, or based on your driving pattern can tell you when you will next visit a location.

As someone who works IT, I do not like my refrigerator telling Google what I am eating and I don’t like Google knowing all the details of my life.

Google does not make it easy to control your data because that is how they earn revenue. Google doesn’t charge for Android, the majority operating system for cell phones because they earn money by monetizing your behavior and selling what you are interested in. Do a Google search for “Cruises from Boston” for example and see what ads appear in Google and are emailed to you.

Google tracks your location, what you search for, your buying habits and most activity on your cell phone.

For example, when I looked up a friend’s location history, I could go back to 2015 and see where they had been and for how long.

It is possible to take control over much of your Google data by visiting; https://myactivity.google.com/more-activity?authuser=1/ . This page presents a menu of data collection items, each one can be adjusted or disabled.

WITH EVERY NEW TECHNOLOGY COMES BENEFITS & CONCERNS
Google tracks your location, what you search for, your buying habits and most activity on your cell phone.