The DDoS that didn't break the camel's VAC*

For a little over a week, OVH has been in the public eye, with two different sets of eyes, that is. On one hand, the watchful eye of hundreds of thousands of comprised IP cameras used to cause the largest DDoS attack ever recorded - which was withstood well by OVH. On the other hand, journalists from all over the world with their own cameras, eager to learn more about this off-the-chart attack, its intensity (peaks of up to 1 Tbps) and its modus operandi (over 145,000 connected objects hacked to send numerous requests simultaneously). Torn between a need for transparency towards its customers and the risks attached to sharing the news about such an attack, OVH hadn't said much until now. Some inaccuracies found in a few articles here and there prompted us to speak up in order to set the record straight… and reassure everyone!*VAC: a combination of technologies developed by OVH to mitigate DDoS attacks

Was the goal of the DDoS attacks to "hack " or "kidnap data", as was allegedly reported in some articles?

Absolutely not! There are 3 types of computer attacks: the ones aimed at making a service unavailable, the ones aimed at stealing and/or leaking files, and those aimed at altering data. The DDoS attacks that occurred at the end of September fall into the first category—their goal was to make a service unavailable. In this case, the target of the attack wasn't OVH but rather a few websites belonging to customers hosted by OVH. The way a DDoS attack (Distributed Denial of Service attack) operates is easy enough to understand. The key is to overload the bandwidth of a server (the "pipe" leading all the way to the machine) or to completely monopolise its resources by sending a multitude of simultaneous connections originating from various points of the Internet (hence "Distributed"). So this has nothing to do with the exfiltration or destruction of data.

OVH being attacked by "zombie machines"... now that's really scary isn't it?

Yes, absolutely. Just like all the images used to illustrate this topic: a gloved hand typing on a keyboard, a hooded gangster standing next to a rack of servers.... Now let's play it down a little: launching a DDoS attack requires the use of several machines connected to the Internet. In order to create this network of machines capable of working together to carry out an attack—what we call a botnet—hackers rely on vulnerabilities. They then stealthily take control of the machines so they can strike at the appropriate time.
Until then, those "zombies" were for the most part PCs belonging to ordinary people. These machines had been infected by using booby-trapped emails or programs downloaded from pornographic or torrent websites. In a more marginal way, servers were then put to work inside botnets, which called for a little ingenuity depending on whether it was simply about taking advantage of an administrator who showed negligence by not securing his/her machine properly or about exploiting a security vulnerability spotted in the operating system or inside one of the installed applications. Then the Internet of Things arrived: from smartphones to cameras, to home thermostats, TVs and even cars, a whole bunch of devices have been connected to the Internet… without the manufacturers unduly worrying about their high degree of vulnerability.

This brings about multiple issues, starting with the respect for user privacy. Already back in 2014, the website was launched online by a Russian hacker in order to raise awareness with customers and manufacturers regarding the low level of security found in IP cameras. It is very easy - and perfectly legal—to access the stream of inadequately secured or unsecured IP cameras around the world…

Even though this initiative was passed on through the media, it didn't cause the anticipated shockwave. A year earlier when security experts started to inform the public about the threat of computer attacks launched by comprised connected devices, a lot of people were amused by this prophecy of refrigerators and connected thermostats being the source of a DDoS attack. In reality, the threat became real sooner than expected!
And for good reason: not only are connected devices always available unlike PCs and easier to compromise than servers (these are monitored, so finding malware becomes easier), but there is also more of them. A compromised camera will not only become a slave to the botnet but it is also a contaminating agent since it will in turn scan the net in search of other poorly secured devices to infect. This is how some very large botnets of connected objects have been formed, similar to those that attacked OVH. Several malware programs at the heart of those infections were identified during the recent attacks: Mirai (whose source code has just been made public and its author has proudly claimed responsibility for infecting 380,000 devices) was one, but also Bashlite (its source code was leaked at the end of 2015, which led to the appearance of variations on the initial code).

Now here's the last hurdle, but certainly not the least: the devices used for those attacks are more often than not connected to the Internet through home networks, which are in turn connected to the Internet through ISPs. There is a certain number of ISPs who allocate dynamic temporary IP addresses to their users. This means that even if we identify an IP as being one of the sources of an attack, it is possible that this IP will be attributed to a legitimate network user a few days later.

Are cameras the only connected devices to blame? How many are there? What network do they come from?

We spoke extensively about cameras because there's plenty of them around due to their decreasing cost but we've also seen infected DVRs used for home video surveillance as well as NAS, routers and Raspberry Pis. What all these connected devices have in common is the existence of security vulnerabilities caused by a flawed software design or gross negligence on the part of their manufacturers, which will often use the same factory password for all their devices. Similarly, it's not uncommon to see installers who won't take the time to modify those passwords when deploying the equipment. We can also assume the existence of backdoors: one of the software programs involved had a backdoor and was commercialized as unbranded to several manufacturers, which is why we can't rule out anything intentional. And finally, these devices all have the necessary computing power as well as a bandwidth large enough to send request streams. Therefore, one of the last DDoS attacks recorded by OVH came from unsecured NTUs (network termination unit) distributed to subscribers by some ISPs in Southern Europe.
In short, if zombie cameras scare you, be ready for even more chills. While our internal investigation (which is still ongoing) has identified close to 145,000 infected connected devices as the source of the recent attacks, network service provider Level3 has recently assessed their number at more than a million. So we're only at the beginning of the problem, not to mention the fact that Internet connection rates are constantly growing, notably due to the ever-increasing availability of VDSL, SDSL, and fiber optics. And so we've recently learned that just 60 hacked cameras in Finland (a very well connected country), were able to cause an attack with the resulting impact as one originating in Taiwan that had over 1000 cameras at it command.

"The largest DDoS attack to date". And who's been hit, by the way?

A DDoS attack never lasts for very long - rarely more than 2 minutes - but it will often reoccur every 10 to 15 minutes for several hours, days or weeks. As mentioned earlier, the goal of the attackers is to make a handful of websites unavailable. However, the means being used can also cause the network to get saturated, which in turn causes part of the infrastructure of the hosting provider to become inaccessible to internet users. This causes timeouts: the packets are lining up to enter a tunnel that's become too narrow and if they can't go through, they expire and get dropped. But such a critical situation didn't happen and that is why we can say OVH was able to withstand the attacks.
However, this had an impact on some of our customers, which we would like to go over in more details. During the attacks, internet users from the Southern European countries have experienced traffic slowdowns as they were trying to access servers hosted by OVH because the DDoS coming from those regions were massive. There were congestions on our interconnections with a local ISP in Spain so we sped up investments in our Madrid point of presence, and our exchange capacities will be multiplied by 10 very soon.

Furthermore, if the VAC (a combination of technologies developed by OVH to mitigate DDoS attacks) is protecting all OVH customers by default, it only activates when an attack is detected. It then filters legitimate traffic so that the targeted server remains available. Customers with specific security needs have access to an option allowing for a permanent activation of the VAC. Their entire traffic is constantly being filtered, even when they're not under attack. During the week of September 19, we disabled this option in order to free some bandwidth inside the VAC so we could receive an attack without saturating our protective equipment. In the following days, we reactivated permanent mitigation for the impacted customers. Before the end of 2016, the first generation of anti-DDoS protection, we released back in 2013, will be replaced by a new technology developed in-house and based on FGPA (programmable integrated circuits) and codes that have been in development for the past 18 months. This will allow us to offer up a VAC capable of withstanding DDoS attacks with peaks up to 5 Tbps without slowing down our network.

And finally, the strength of the latest attacks seems enormous at first glance but what needs to be understood is that very few hosting providers have the capacity to even receive such peaks, for the simple reason that their backbone would get saturated even before the peaks could reach it. We have direct interconnections with practically all the major players of the Internet in Europe and in the US, as well as substantial excess bandwidth. Our Internet connection capacity is currently over 7 Tbps and this explain why we were able to receive an attack of 1 Tbps - whereas the previous record was set at 600 Gbps - without saturating our connections (except for the Spanish ISP previously mentioned).

What is the situation at OVH? The press is talking about "fully mobilized" teams on maximum alertness.

Everyone is indeed on high alert but if we're mobilizing our teams these days, it's more likely in anticipation of our 4th OVH Summit that will take place on October 11 in Paris. Even though the intensity of the attacks took us a little by surprise, we were prepared… and we're convinced that there will be more. If a period of calm can follow the "surrendering" of the hacker responsible for the Mirai malware - who claims to have closed the backdoor he/she was using to infect cameras, there's every reason to believe that variations will appear, all of them as ingenious as the other. In short, the current situation isn't exceptional except for the media excitement surrounding this latest attack. We continue to innovate, to manage the services we provide to our customers... and to fend off attacks, which are part of a phenomenon that can't be avoided in the hosting business.

Every day, around 1,200 OVH clients are being protected by our anti-DDoS (VAC) system without even realizing it. Once the attack is over, we're the ones informing them about what happened. We have a team dedicated to this project, including R&D engineers, and we're devoting substantial resources to this endeavour. In other words, a DDoS attack is no big deal! All companies have at least one competitor bent on trying to take them down. This is why an anti-DDoS protection isn't an option at OVH but rather something we provide to all our customers by default.

Who are the bad guys? What exactly do they want?

Some people got ahead of themselves by assuming that the OVH's size and ambitions made it a prefect target for the attackers. The reality is a little more complicated than that. Let's go back to the beginning.

Behind the botnets used during the recent DDoS attacks lay the supervillains. They have a very cunning way of coding the malware used to infect the connected devices by exploiting their vulnerabilities and they operate their networks of zombie devices from their Command & Control (C2), the absolute master of the now slave devices. These botnets are offered to the highest bidder by the supervillains for a price that's proportional to their striking power and the damage they can inflict. And just like the money they generate, these botnets never stop working. Because there's a good number of villains who are willing to pay a very hefty price to get this type of service to harm a competitor or even their wife's lover by bringing down their company website.

In a nutshell, saying that OVH is a prime target is a little too easy. The truth is, since many servers and applications are being hosted by OVH, a certain number of those will obviously be a target of choice for all sorts of attackers. This is something we've been taking into account for a long time now, since a DDoS attack doesn't necessarily target customers from sectors deemed as very competitive or where this type of practice is common (gaming, for example). The flipside of having so many customers is that we have the means to defend ourselves.
And finally, there's no need to bury our heads in the sand: the Internet is not all about puppies and unicorns. Some people could be tempted to justify the use of DDoS attacks for what they deem as a fair fight. We believe it's a mistake. And we will continue to invest as much as we can into protective measures that are always more effective so that DDoS attacks never become a means of censorship (a risk very well explained by American security expert Brian Krebs).

Why then, did you make the attack public on Twitter?

When Octave Klaba, our CTO, chose to make the attack public on Twitter on September 23, his goal was to draw the attention of security experts and competent authorities regarding the scope of the botnet (the network of cameras controlled by the hackers) that caused the DDoS, as well as the intensity of the peak loads that OVH had to absorb.

Just like all the other major players on the web, OVH has the capacity to withstand attacks of that magnitude—which is something they all have to face on a regular basis without necessarily talking about it. But this isn't the case with smaller operators, notably because of the costs involved in implementing security measures and the excess network capacity permanently needed to face these attacks.

Withstanding DDoS attacks aimed at our customers is our job. However, disabling botnets that are becoming bigger and bigger isn't our mission. We neither have the right, nor probably the means to do so (except when it comes to neutralizing the C2s that we could be hosting), and it would involve working alongside many other players. For example, what can we do if equipment manufacturers don't fix their software's vulnerabilities and resellers don't have the courage to warn their customers that their product is infected?

Why did you then refuse to talk to the media about it?

The reason why we didn't follow up on requests from the press—and there were a lot—is because communication centering around DDoS attacks is a very sensitive matter and, for the most part, counter-productive. We must indeed tread very carefully when trying to explain how an attack occurred and what we did to counter it without giving away valuable information to the attackers, inspiring others to do the same, or destroying evidence that could have been used by legal authorities over the course of a possible investigation. Disclosing information after withstanding an attack often results in motivating assailants to intensify their efforts. It can also be disquieting for our customers who start to wonder if OVH isn't more at risk for DDoS attacks than other companies. This is why we will continue to be transparent during the next attacks while still being discrete.
If we didn't refrain from pointing out approximations and inaccuracies on this subject, note that we have quoted some very good papers in this article and we highly recommend that you read them if you're interested in this topic, including the ones mentioned below.

Previous article

Black Friday in Europe