KernelCare: Patching the Kernel On-the-Fly

As for today, we do not have to turn off and reboot a server with Linux operating system for months. But sooner or later, we’ll have to do it anyway. For example, to install kernel updates. As a rule, this procedure is a real pain for sysadmins. First of all, they have to choose time of the minimal user activity. They should also send out warning emails to all users in advance. Besides, there’s always a risk of an emergency when the server downtime can last for too long.

There are also special programming solutions, with the help of which we can install patches and kernel updates without rebooting. As an example, we should name Ksplice – a product by Oracle Company, which is distributed under GPL v.2 license. It supports the following Linux distributives: Oracle, Linux, RHEL, Ubuntu (the desktop versions only) and Fedora. On the whole, Ksplice copes with allotted tasks but it has one drawback. He can work with not nearly all existing security patches.

At the beginning of 2014 RedHat Enterpise Linux developers introduced their solution – Kpatch. The product is distributed under GPL2 license. You can find its source code at GitHub. Unfortunately, it is still “raw” and can not be recommended for use. We can say the same about kGraft –a solution by SUSE developers.

Our colleagues from CloudLinux Company have quite recently introduced their tool. With its help we can install all security patches and updates on-the-fly, without actually rebooting the server at all. It is KernelCare.

CloudLinux developers cautiously monitor all the information about kernel vulnerability. As soon as they find a weak point in any of supported kernels, they create a patch that will eliminate them. Patches are located on distribution servers. Each patch is adjusted to the kernel of a certain distributive. Installed on the client server, KernelCare Agent periodically contacts distribution servers, downloads and installs all new patches. All of that is executed in the background and there’s no need to reboot the server.

The first article announcing KernelCare appeared on the Internet early this year. Starting from June, the product is distributed by paid subscription. A test license is valid for 15 days.

As for today, KernelCare supports the following Linux distributives (it operates with 64-bit OS only):

• RedHat 6.x;
  
• CentOS 6.x;
  
• CloudLinux 6.x.
  

Starting from April, it also supports OpenVZ kernels. Debian and Ubuntu support are scheduled to be released very soon.

Testing KernelCare

/The testing has been carried out on CentOS 6 OS/

Let’s install a test version of KernelCare using the following command:

# rpm -i http://patches.kernelcare.com/kernelcare-latest.el6.x86_64.rpm

Right after the installation KernelCare will automatically download and apply the necessary updates. After that it will display a message as follows:

Downloading updates
Patch Level 9 applied
Kernel is safe

We can look at the applied patches utilizing the following command:

# /usr/bin/kcarectl --info
kpatch-state: patch is applied
kpatch-for: Linux version 2.6.32-358.23.2.el6.x86_64 ([email protected]) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-3) (GCC) ) #1 SMP Wed Oct 16 18:37:12 UTC 2013
kpatch-build-time: Mon May 12 23:50:58 2014
kpatch-description: 9

It will provide detailed information on everything KernelCare is doing. We can also do it with the help of dmesg command:

# dmesg|grep 'kcare'
kcare: registered device with node 10:57
kcare: allocated 278112 bytes for patch at ffffc900005c4000
kcare: verifying patch...
kcare: verified successfully
kcare: allocating memory in module space...
kcare: allocated 278112 bytes at ffffffffa0207000
kcare: 865 relocations to fixup...
kcare: fixed 865 relocations
kcare: jumping to ffffffffa020d9a0
kcare: registered device with node 10:57
kcare: allocated 278112 bytes for patch at ffffc900005c4000
kcare: verifying patch...
kcare: verified successfully
kcare: allocating memory in module space...
kcare: allocated 278112 bytes at ffffffffa0207000
kcare: 865 relocations to fixup...
kcare: fixed 865 relocations

KernelCare runs a check for the availability of new patches every 4 hours. It downloads and applies all patches automatically. We can disable the automatic update. We should open /etc/sysconfig/kcare configuration file for that purpose. It contains the only parameter – AUTO_UPDATE. We should change its value from True to False:

AUTO_UPDATE = False

When the automatic update is disabled, we can download and apply a new patch using the following command:

/usr/bin/kcarectl --update
Updates already downloaded
Patch Level 9 applied
Kernel is safe

We can roll back all applied changes by running the following command:

kcaretl --unload
Updates already downloaded
KernelCare protection disabled, kernel might not be safe

Summary

KernelCare is a really good and useful tool. Among its evident merits are the following:

• it’s easy to install and adjust;
  
• its speed of downloading and applying patches;
  
• it does not affect the system performance;
  
• we can always roll back the applied changes.
  

CloudLinux developers are improving the product. Therefore, there’s every good reason to believe that in the short run it will become popular among sysadmins.