Enabling live kernel patching on Ubuntu Server 16.04 LTS

A couple of years ago, a set of changes was merged into the kernel core (version 4.0) that enabled kernel patching without a reboot. Although a similar functionality has been available for a while in commercial Linux distributions as a paid service (Oracle Ksplice, RHEL Live Patching and SUSE Live Patching), it’s now possible to configure it for up to 3 machines with Ubuntu for free. Live kernel patching in Ubuntu is known as Canonical Livepatch Service.

Let’s check how it works.

Is there anything to patch?

First, let’s see what is our current kernel version:

root@bernard:~# uname -a
Linux bernard 4.8.0-41-generic #44~16.04.1-Ubuntu SMP Fri Mar 3 17:11:16 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

What is the latest available kernel version? Let’s run apt-get update and then:

root@bernard:~# apt-cache madison linux-generic-hwe-16.04
linux-generic-hwe-16.04 | 4.8.0.46.18 | http://it.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
linux-generic-hwe-16.04 | 4.8.0.46.18 | http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages

root@bernard:~# ls /boot/vmlinuz*
/boot/vmlinuz-4.4.0-66-generic /boot/vmlinuz-4.4.0-70-generic /boot/vmlinuz-4.4.0-72-generic /boot/vmlinuz-4.8.0-41-generic /boot/vmlinuz-4.8.0-45-generic /boot/vmlinuz-4.8.0-46-generic

So it looks like a newer kernel is available and it has already been installed by unattended upgrades. But to enable it, we would have to do a server reboot. Let’s install Canonical Livepatch Service instead.

Enabling kernel livepatching

First, we need to generate a service token – we can do it here: https://auth.livepatch.canonical.com/. Then we’ll use the snap utility to install the service on the server.

root@bernard:~# snap list
No snaps are installed yet. Try "snap install hello-world".

root@bernard:~# snap install canonical-livepatch
canonical-livepatch 7 from 'canonical' installed

root@bernard:~# snap list
Name                 Version  Rev   Developer  Notes
canonical-livepatch  7        22    canonical  -
core                 16-2     1577  canonical  -

Next, we’ll enable the service using the generated token:

root@bernard:~# canonical-livepatch enable [TOKEN]
2017/04/05 23:15:17 Error executing enable?auth-token=[TOKEN].
This machine ID is already enabled with a different key or is non-unique. 
Either "sudo canonical-livepatch disable" on the other machine, or regenerate a 
unique /etc/machine-id on this machine with 
"sudo rm /etc/machine-id /var/lib/dbus/machine-id && sudo systemd-machine-id-setup" : 
{"error": "Conflicting machine-id"}

Well, since bernard is a VPS server, it has a pregenerated machine-id which has been used. Fortunately, regenerating the id fixes the problem.

root@bernard:~# cat /etc/machine-id
cbaaaaaaaaaaaaaaaaaaaaaaaaaaaabd

root@bernard:~# rm /etc/machine-id /var/lib/dbus/machine-id && systemd-machine-id-setup
Initializing machine ID from random generator.

root@bernard:~# cat /etc/machine-id
e8aaaaaaaaaaaaaaaaaaaaaaaaaaaac4

root@bernard:~# canonical-livepatch enable [TOKEN]
Successfully enabled device. Using machine-token: [MACHINE_TOKEN]

root@bernard:~# canonical-livepatch status --verbose
client-version: "7.21"
machine-id: e8aaaaaaaaaaaaaaaaaaaaaaaaaaaac4
machine-token: [MACHINE_TOKEN]
architecture: x86_64
cpu-model: Intel(R) Xeon(R) CPU E5-2650L v3 @ 1.80GHz
last-check: 2017-04-05T23:18:47.066474268+02:00
boot-time: 2017-03-20T23:40:15+01:00
uptime: 382h38m56s
status:
- kernel: 4.8.0-41.44~16.04.1-generic
  running: true
  livepatch:
    checkState: checked
    patchState: nothing-to-apply
    version: ""
    fixes: ""

root@bernard:~# uname -a
Linux bernard 4.8.0-41-generic #44~16.04.1-Ubuntu SMP Fri Mar 3 17:11:16 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Well, unfortunately it seems it changed nothing. It might be because I’m using Hardware Enablement (HWE) with a different kernel.

After a little bit of Googling I’ve found that indeed Kernel Livepatching works only for Ubuntu 16.04 LTS with “GA kernel” (without HWE), which means kernel 4.4. Here’s the relevant link to the FAQ. So right now it’s either a newer kernel without livepatching (4.8) or an older kernel with livepatching (4.4). The HWE kernel might get livepatching this year according to this article:

Q: What about other releases of Ubuntu?
A: The Canonical Livepatch Service is provided for Ubuntu 16.04 LTS’s Linux 4.4 kernel. Older releases of Ubuntu will not work, because they’re missing the Linux kernel support. Interim releases of Ubuntu (e.g. Ubuntu 16.10) are targeted at developers and early adopters, rather than Long Term Support users or systems that require maximum uptime. We will consider providing livepatches for the HWE kernels in 2017.

So for know, I’ve decided to downgrade to kernel 4.4, as there’s nothing in HWE that I would need at the moment.

root@bernard:~# uname -a
Linux bernard 4.4.0-72-generic #93-Ubuntu SMP Fri Mar 31 14:07:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

root@bernard:~# ls /boot/vmlinuz*
/boot/vmlinuz-4.4.0-66-generic /boot/vmlinuz-4.4.0-70-generic /boot/vmlinuz-4.4.0-72-generic

root@bernard:~# canonical-livepatch status --verbose
client-version: "7.21"
machine-id: e8aaaaaaaaaaaaaaaaaaaaaaaaaaaac4
machine-token: [MACHINE_TOKEN]
architecture: x86_64
cpu-model: Intel(R) Xeon(R) CPU E5-2650L v3 @ 1.80GHz
last-check: 2017-04-06T00:13:09.240517748+02:00
boot-time: 2017-04-06T00:09:04+02:00
uptime: 4m15s
status:
- kernel: 4.4.0-72.93-generic
  running: true
  livepatch:
    checkState: checked
    patchState: nothing-to-apply
    version: ""
    fixes: ""

Comparison to existing solutions

It’s very well outlined in Canonical Livepatch Data Sheet PDF:

How does this service compare to Oracle Ksplice, RHEL Live Patching and SUSE Live Patching?

While the concepts are largely the same, the technical implementations and the commercial terms are very different. Oracle Ksplice uses it’s own technology which is not in upstream Linux. RHEL and SUSE currently use their own homegrown kpatch/kgraft implementations, respectively. Canonical Livepatching uses the upstream Linux Kernel Live Patching technology.

– Ksplice is free, but unsupported, for Ubuntu Desktops, and only available for Oracle Linux and RHEL servers with an Oracle Linux Premier Support license ($2299/node/year).

– It’s a little unclear how to subscribe to RHEL Kernel Live Patching, but it appears that you need to first be a RHEL customer, and then enroll in the SIG (Special Interests Group) through your TAM (Technical Account Manager), which requires Red Hat Enterprise Linux Server Premium Subscription at $1299/node/year.

– SUSE Live Patching is available as an add-on to SUSE Linux Enterprise Server 12 Priority Support subscription at $1,499/node/year, but does come with a free music video.

– Canonical Livepatching is available for every Ubuntu Advantage customer, starting at our entry level UA Essential for $150/node/year, and available for free to community users of Ubuntu.