August 2007

We have been advised by Paypal that due to technical issues Paypal payments are currently delivered with a short delay.

We apologize for any inconveniences this is causing.

UPDATE: This issue has been resolved and all orders are delivered normally again.

We have been informed that users may receive error “checkcode (kontrollkood) is invalid” when making payment using Hansabank or Ühisbank in Estonia. We hope to get the issue resolved as soon as possible but until then, please choose English as browsing language at www.skype.com and continue making payment in English. This way you are able to place successful payment until the issue is resolved. We will also update once the issue is resolved.

We apologize for any inconveniences.

Yesterday, we posted an explanation of what happened to Skype on August 16. Some reactions to the explanation, however, have reminded us of one of the basic tenets of communication: It’s not what you say. It’s what they hear. We’d therefore like to clear a few misunderstandings that emerged in yesterday’s reactions to our explanation of what transpired last week.

1. Are we blaming Microsoft for what happened?

We don’t blame anyone but ourselves. The Microsoft Update patches were merely a catalyst — a trigger — for a series of events that led to the disruption of Skype, not the root cause of it. And Microsoft has been very helpful and supportive throughout.

The high number of post-update reboots affected Skype’s network resources. This caused a flood of log-in requests, which, combined with the lack of peer-to-peer network resources at the time, prompted a chain reaction that had a critical impact. The self-healing mechanisms of the P2P network upon which Skype’s software runs have worked well in the past. Simply put, every single time Skype has needed to recover from reboots that naturally accompany a routine Windows Update, there hasn’t been a problem.

Unfortunately, this time, for the first time, Skype was unable to rise to the challenge and the reasons for this were exceptional. In this instance, the day’s Skype traffic patterns, combined with the large number of reboots, revealed a previously unseen fault in the P2P network resource allocation algorithm Skype used. Consequently, the P2P network’s self-healing function didn’t work quickly enough. Skype’s peer-to-peer core was not properly tuned to cope with the load and core size changes that occurred on August 16. The reboots resulting from software patching merely served as a catalyst. This combination of factors created a situation where the self-healing needed outside intervention and assistance by our engineers.

2. What was different about this set of Microsoft update patches?

In short – there was nothing different about this set of Microsoft patches. During a joint call soon after problems were detected, Skype and Microsoft engineers went through the list of patches that had been pushed out. We ruled each one out as a possible cause for Skype’s problems. We also walked through the standard Windows Update process to understand it better and to ensure that nothing in the process had changed from the past (and nothing had). The Microsoft team was fantastic to work with, and after going through the potential causes, it appeared clearer than ever to us that our software’s P2P network management algorithm was not tuned to take into account a combination of high load and supernode rebooting.

3. How come previous Microsoft update patches didn’t cause disruption?

That’s because the update patches were not the cause of the disruption. In previous instances where a large number of supernodes in the P2P network were rebooted, other factors of a “perfect storm” had not been present. That is, there had not been such a combination of high usage load during supernode rebooting. As a result, P2P network resources were allocated efficiently and self-healing worked fast enough to overcome the challenge.

4. Has the bug been fixed? Should Skype users worry about future Microsoft Update patches and reboots?

Yes, the bug has been squashed. The parameters of the P2P network have been tuned to be smarter about how similar situations should be handled. Once we found the algorithmic fix to ensure continued operation in the face of high numbers of client reboots, the efforts focused squarely on stabilising the P2P core. The fix means that we’ve tuned Skype’s P2P core so that it can cope with simultaneous P2P network load and core size changes similar to those that occurred on August 16. We’d like to reassure our users across the globe that we’ve done everything we need to do to make sure this doesn’t happen again. We’ve already introduced a number of improvements to our software to ensure our users will not be similarly affected – in the unlikely possibility of this combination of events recurring.

(Posted at 5:35 AM GMT, updated first two paragraphs at 11:45 AM GMT)

On Thursday, 16th August 2007, the Skype peer-to-peer network became unstable and suffered a critical disruption. The disruption was triggered by a massive restart of our users’ computers across the globe within a very short timeframe as they re-booted after receiving a routine set of patches through Windows Update.

The high number of restarts affected Skype’s network resources. This caused a flood of log-in requests, which, combined with the lack of peer-to-peer network resources, prompted a chain reaction that had a critical impact.

Normally Skype’s peer-to-peer network has an inbuilt ability to self-heal, however, this event revealed a previously unseen software bug within the network resource allocation algorithm which prevented the self-healing function from working quickly. Regrettably, as a result of this disruption, Skype was unavailable to the majority of its users for approximately two days.

The issue has now been identified explicitly within Skype. We can confirm categorically that no malicious activities were attributed or that our users’ security was not, at any point, at risk.

This disruption was unprecedented in terms of its impact and scope. We would like to point out that very few technologies or communications networks today are guaranteed to operate without interruptions.

We are very proud that over the four years of its operation, Skype has provided a technically resilient communications tool to millions of people worldwide. Skype has now identified and already introduced a number of improvements to its software to ensure that our users will not be similarly affected in the unlikely possibility of this combination of events recurring.

The Skype community of users has been incredibly supportive and we are very grateful for all their good wishes.

Hello friends,

Take a deep breath. Skype is back to normal.

On Monday, we’ll provide a more detailed explanation of what happened. Until then, we’d like to apologize and thank you. Precisely in that order.

We know how difficult and frustrating the past two days have been. And still, your good wishes kept flowing in. Thank you for the amazing patience, trust and support!

(Posted at 11 am GMT)

Hello again,

We are pleased to announce that the situation continues to improve. The sign-on problems have been resolved. Skype presence and chat may still take a few more hours to be fully operational. We know what our faithful users have been going through and we thank you for your patience and kind support.

If you are one of the minority who may still be experiencing problems, please be patient. You do not need to adjust or restart your computer. Skype will start working for you very soon.

We will issue a further update when we know that Skype is functioning normally, or if there is further material news.

We’ve commandeered extra supplies of pizza and coffee, and we can promise that the Skype people aren’t going anywhere until they’re happy that everything is back to normal.

(Posted at 00.02 AM GMT)

(Update at 1:44 AM GMT: Please ignore the “Problems with Skype login” text in the Heartbeat status box above, next to “Sign-in and registration”, as well as “P2P network core”. The text should read “All working normally”.)

Hello all,

As Europe has woken up to a new day and Asia is entering the evening hours, here’s the latest on the sign-on problem.

We’re on the road to recovery. Skype is stabilizing, but this process may continue throughout the day.

An encouraging number of users can now use Skype once again. We know we’re not out of the woods yet, but we are in better shape now than we were yesterday.

Finally, we’d like to dispel a couple of theories that we are still hearing. Neither Wednesday’s planned maintenance of our web-based payment services nor any form of attack was related to the current sign-on issues in any way.

We’ll update you again as soon as we can. Thanks for hanging tight.

Hey,

even though it is too early to call out anything definite yet we are now seeing signs of improvement in our sign-on performance. We continue to monitor the situation through Europe waking up this morning and have more for you in approximately 4 hours.

My client is now occasionally bringing in stacks of unread multichats. Hope yours too.

Hello,

just wanted to drop a line that everyone at Skype working on resolving the sign-on issue is still up. We’re fixing issues in our networking software and monitoring the clients getting online with increased success.

It’s been a loooong day (and night) for many and we are gradually moving to new shifts of fresh brains to help out those getting to well deserved few hours of sleep. Be sure that the Skype family as a whole will not retire before you can get back to all the Skype conversations you want, need or just feel like having. Sorry for keeping you split that long.

(Updated at 5:00GMT)

Hello again,

Everyone at Skype continues to work hard at resolving the current software issue. We are making good progress. We feel that we are on the right track to bring back services to normal.

We thank you for your continued support and are thinking of you every step of the way.

(Updated at 2:15am GMT)

Hello everyone,

Apologies for the delay, but we can now update you on the Skype sign-on issue. As we continue to work hard at resolving the problem, we wanted to dispel some of the concerns that you may have. The Skype system has not crashed or been victim of a cyber attack. We love our customers too much to let that happen. This problem occurred because of a deficiency in an algorithm within Skype networking software. This controls the interaction between the user’s own Skype client and the rest of the Skype network.

Rest assured that everyone at Skype is working around the clock — from Tallinn to Luxembourg to San Jose — to resume normal service as quickly as possible.

(Updated at 10pm GMT)

We’d like to thank everyone who has taken the time to send us their thoughts, concerns and good wishes. It means the world to those working so hard to resolve this thing.

UPDATED 14:02 GMT: Some of you may be having problems logging into Skype. Our engineering team has determined that it’s a software issue. We expect this to be resolved within 12 to 24 hours. Meanwhile, you can simply leave your Skype client running and as soon as the issue is resolved, you will be logged in. We apologize for the inconvenience.

Additionally, downloads of Skype have been temporarily disabled. We will make downloads available again as quickly as possible.

Our web paid services might be unavailable due to the planned maintenance on 15.07.2007 07:00 - 07:30 GMT. During that time you might have problems entering the web paid services, Payment In Client and My Account In Client. This means that you might encounter problems while logging in to our website and making payments. If you have any problems entering the these services during that time, please try again after half an hour.

We apologize for any inconvenience that might cause to you.

UPDATE: The planned maintenance was completed at 15.08.2007 7:10 GMT and now all systems work normally again.

Our payment service provider is having a maintenance, due to that some credit card payments might be processed with delay. We will let you know when all payments will be processed normally again.

We apologize for any inconvenience that this issue might cause to you.

UPDATE: this issue has been resolved and all payments are working normally again