Meltdown, Spectre bug impacting x86-64 CPU - OVH fully mobilised

Like all IT industry players, OVH has been informed of security vulnerabilities on x86-64 processors, which were exposed by security researchers. These vulnerabilities make it possible to carry out, on a large-scale, a type of attack (side-channel attacks) that up until now has been infrequent due to the complexity of implementation.

[Update January 9 at 10 p.m. CET]

OVH continues to secure its infrastructures. Regarding the restart of services, concerned customers have been alerted by email.
The two documents published by OVH on January 7, 2018 are intended to informin real time the status of each of our services regarding Meltdown and Spectrum
security vulnerabilities and availability of patches for each operating system. These documents have been updated regularly over the past 48 hours to provide you with the most comprehensive information possible.

So that you can better understand the technical details, we published a post, written by one of our security architects, about the different variants of Spectrum and Meltdown.

Attention, reading reserved for an informed public!
For those who might have missed it, one of our security engineers shared a tool on GitHub to test the vulnerability of your Linux distribution against the 3 variants of Spectrum and Meltdown.

[Updated January 7 at 2:00 p. m CET.]

OVH teams have been on duty all week, including this weekend.
We have already protected some of our services, through the deployment of patches available to date.
For other services the security is in progress or to come.
All the operations associated with this action plan will appear on the site travaux.ovh.net.

Overall, the situation is under control.

However, we are aware of the difficulty that some users may have in measuring the current situation, and understand what they need to do on their side to secure their services against Metldown and Spectrum vulnerabilities.

This complexity can be explained by the variety of software and hardware configurations, and by the fact that your services are partially or totally managed by OVH, which means that there is in some occasions no actions to be carried out on the user side, while user participation will be necessary in some cases.
Moreover, once the patches are already available, they can only protect one or two of the 3 attack vectors.

Therefore, we worked on a page that list all OVH service aggregating two important information’s:

-what OVH is doing to secure the service;

-what does the user has to do to make the service secure (when required).

We will update this page on a regular basis.
We will keep you informed of the latest developments in terms of security operations performed and, when you have actions to take, we will inform you what they are and especially when it is recommended to take this specific action.

To go a little further in the support we can provide you, we have assembled a second status page online that explains what to do, if necessary, depending on the operating system of your servers to apply the necessary patch to the kernel. This page is also being updated on a daily basis.

In addition, in order to regularly update the information available to you, our objective is now to make this mass of information as digestible as possible, so that each of you, whatever your situation is, can easily find what you need.

Thank you for your understanding.

Statement published January 4

These vulnerabilities, made public on the night of 3rd-4th January by Google's Project Zero, are now known as Meltdown and Spectre and include 3 distinct attack vectors:

  • CVE-2017-5715 (branch target injection - Spectre)

  • CVE-2017-5753 (bounds check bypass - Spectre)

  • CVE-2017-5754 (rogue data cache load - Meltdown)

To date, OVH has not received any information demonstrating that the concerned vulnerabilities have been exploited outside of a research laboratory setting. Their operation today requires technically complex processes, but there is no doubt that simpler processes will emerge.

Intel, the world leader in microprocessors, has confirmed the existence of these vulnerabilities on its CPU. With its partners, and operating system editors, the company is working on solutions to reduce the exposure of its chips to these types of attack, via patches to be implemented at several levels.

  • Operating system and Virtual Machine Manager

  • Microcode processor (via BIOS/UEFI)

At OVH, a dedicated team of security experts is fully mobilised. We are in close contact with the main players involved in the implementation of patches, i.e. the main editors of free operating systems (including GNU/Linux distributions), proprietary operating systems (Microsoft or VMware), and motherboard manufacturers.

Most of these players were warned of vulnerabilities discovered several weeks ago, and were already working on patches while under an embargo. Patches are therefore beginning to be announced and released by various editors and communities. Our teams are progressively testing patches so that once stability has been assured, updates can be released as quickly as possible. Since the patches make major modifications to the kernel design, the risks of instability are not negligible.

It is still too early to accurately identify the impact of the available patches and the future performance of servers. Furthermore, our internal tests indicate that the potential impact on performance varies greatly depending on workload and running services. In addition, it is likely that the effectiveness of the first available updates will improve over time, mitigating the effects induced by this design change at the kernel level.

Given the unusually short development time of these patches, making tests on all market configurations is unlikely, there is a good chance that more updates will follow, correcting any bugs that may appear. We already know that official patches will only be available, except in exceptional cases, for the versions of kernels and operating systems that are currently maintained (depending on the editor).

At the same time, we are studying in detail the possible exploitation of vulnerabilities exposed, so that we can assess risks more accurately and share our recommendations with our customers.

In parallel, we are studying various scenarios for deploying official patches as quickly as possible, while minimising the impact on the availability of your services. Some infrastructures may have to be rebooted in order to update the kernels and firmware affected by vulnerabilities.

We will inform you as soon as possible of the action plan and related operations. When the time comes, we will provide you with more details about what you need to do next to apply the patches on your physical or virtual machines.