Introduction

The Linux® kernel is one of the widely used operating systems that is well known for its availability. The robustness that makes it carrier grade comes from thousands of test runs against the Linux kernel by various individuals and companies for all kernel releases. Most of the issues are discovered and fixed during the release cycle and some get discovered later by the users or by special interest groups, such as security analysts who test the kernels extensively. Some of these issues can even compromise the system by either denial-of-service attack (DoS) or by gaining superuser privileges if the systems are not patched with the security fixes when available.

Updates to the kernel require scheduled downtime, and a lengthy and well-planned schedule that affects the least number of users, and a backup plan in case of system unavailability after the update. The live patching infrastructure that is introduced in the Linux kernel helps updating the running kernel with security fixes by live patching it, without any disturbance to the mission-critical workloads running on the system or a need to restart the applications. All the running processes in the system start seeing the fixed version of the affected kernel functions immediately without any delay after the kernel is live patched.

Overview

The live patching infrastructure is now available on little-endian mode in the IBM Power Systems™ server (ppc64le). Live patching is achieved by redirecting the call to a kernel function to its patched version with the fix and it depends on the Ftrace tracing framework that is available in the Linux kernel to profile kernel functions during their invocation. Every redirection requires a call back to be registered with the Ftrace framework to redirect any call made to a function to its patched version. See the following figure for more details.

Figure 1. Function call redirection to patch through Ftrace framework

Before describing how redirection works, this section provides high-level information about the Ftrace framework. Function profiling is enabled only if the compiler (GCC) supports the -pg flag. When the Linux kernel is compiled with the -pg flag, the compiler inserts a branch instruction to a special compiler-generated function _mcount in the prologue sequence of every kernel function. See the following figure for more details.

Figure 2. Objdump of cmdline_proc_show() function compiled with -pg flag

Note: Not every kernel function is traceable due to their implications.

When enabled, Ftrace replaces call to the _mcount function with the ftrace_caller function. The ftrace_caller function records profiling information about the function during its invocation, whereas call to _mcount is replaced with no-op instruction, having zero overhead when the Ftrace framework is disabled.

Live patching, like Ftrace on Power Systems, depends on compiler support. In case of live patching, compiler support for the -mprofile-kernel flag is required. This flag inserts the branch to the _mcount function immediately after the function local entry setup and before stack creation in the prologue sequence of the function. Compare the instructions of Figure 3 with Figure 2 for _mcount location. The reason is that during function redirection, all the registers and arguments that are available for the original function must be available to the patched function too.

Figure 3. Objdump of cmdline_proc_show function compiled with -pg and -mprofile-kernel flags

Prerequisites

To enable the live patching infrastructure, you need to build kernel with the following configuration enabled:

CONFIG_MPROFILE_KERNEL=y
CONFIG_HAVE_DYNAMIC_FTRACE_WITH_REGS=y
CONFIG_DYNAMIC_FTRACE_WITH_REGS=y
CONFIG_HAVE_LIVEPATCH=y
CONFIG_LIVEPATCH=y

Building a live kernel patching module

This section describes the steps to build a simple live patching kernel module and how to use it to live patch the kernel:

Build a module with the modified version of the function that you want to live patch, along with other functions that are required to register the patch with the live patching infrastructure. Authoring a live patch module is similar to writing any other kernel module. For example, consider a sample patch.

--- a/fs/proc/cmdline.c
+++ b/fs/proc/cmdline.c
@@ -6,8 +6,7 @@
static int cmdline_proc_show(struct seq_file *m, void *v)
{
- seq_puts(m, saved_command_line);
- seq_putc(m, '\n');
+ seq_puts(m, "Livepatched kernel command line\n");
return 0;
}

Here, the kernel command line is live patched to display the string, &Livepatched kernel command line&, when read through /proc/cmdline, instead of the booted kernel command line.

1. Include the standard header files and the live patch header file that exports the patch registering functions in the live patching kernel module file.

    #include <linux/module.h>
    #include <linux/kernel.h>
    #include <linux/livepatch.h>
    #include <linux/seq_file.h>
   

   Show more

2. Copy the original function and make changes to the boot command line string that has to be displayed, and prefix the fixed version of the function name with livepatch_. The patched function is included in the module as a complete function. Live patching works by redirection and hence the complete function must be a part of the module.

    static int livepatch_cmdline_proc_show(struct seq_file *m, void *v)
    {
    seq_printf(m, "Livepatched kernel command line\n");
    return 0;
    }
   

   Show more

   Every function that needs to be live patched must be a part of the klp_func array, where the original function name and the patched function name are provided. Thus, when the original function is called, it is redirected to the patched function, also known as new_func.

    static struct klp_func funcs[] = {
    {
    .old_name = "cmdline_proc_show",
    .new_func = livepatch_cmdline_proc_show,
    }, { }
    };
    static struct klp_object objs[] = {
    {
    .funcs = funcs,
    }, { }
    };
    static struct klp_patch patch = {
    .mod = THIS_MODULE,
    .objs = objs,
    };
   

   Show more

3. Include the following structures listed here for registering the functions that must be live patched.

    static int livepatch_init(void)
    {
    int ret;
    ret = klp_register_patch(&patch);
    if (ret)
    return ret;
    ret = klp_enable_patch(&patch);
    if (ret) {
    WARN_ON(klp_unregister_patch(&patch));
    return ret;
    }
    return 0;
    }
    static void livepatch_exit(void)
    {
    WARN_ON(klp_unregister_patch(&patch));
    }
    module_init(livepatch_init);
    module_exit(livepatch_exit);
    MODULE_LICENSE("GPL");
    MODULE_INFO(livepatch, "Y");
   

   Show more

   The module must have both the init and exit functions, which are run when the module is loaded and unloaded respectively.

   When the module is being loaded, as a part of the initialization process, functions to be patched are registered and the following fail-safe checks are performed:

   • Is live patch supported in the current running kernel

   • Is the function to be patched valid and traceable

     A function that is non-traceable by the Ftrace framework cannot be live patched. Live patching depends on Ftrace for redirection to a patched version of a function. As the next step in the registration process, the callback for the original function cmdline_proc_show is registered with the Ftrace framework. After the registration is successful, the patched version of the function is enabled through the klp_enable_patch() application binary interface (ABI). The klp_unregister_patch() ABI unregisters the Ftrace handler for the patched version of the function and restores the call to the original function instead of the live patched version of the function. The most important line is the last line where the module is flagged as a live patching module, which differentiates a regular kernel module and a live patching module.

4. Compile the module, which is similar to compiling an out-of-tree module.

    $ cat Makefile
    obj-m = my-first-livepatch.o
    KERNEL_VER = $(shell uname -r)
    all:
    make -C /lib/modules/$(KERNEL_VER)/build M=$(PWD) modules
    clean:
    make -C /lib/modules/$(KERNEL_VER)/build M=$(PWD) clean
   

   Show more

5. After compiling the module, you can live patch the kernel using the following commands:

    $ sudo insmod ./my-first-livepatch.ko
    $ cat /proc/cmdline
    Livepatched kernel command line
   

   Show more

Enabling or disabling the patched version of the kernel function

One advantage that the live patching infrastructure provides is rolling back to the original functionality (consider the previous example, where disabling the live patch would roll back to display the original kernel command line string).

After the module is inserted, a directory with the module name is created in the /sys/kernel/livepatch/, where the patch can be enabled or disabled. To restore the original functionality, the redirection to the fixed version of the function can be disabled by running the following command:

$ sudo echo 0 > /sys/kernel/livepatch/my_first_livepatch/enable
$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinux-4.15.0-22-generic
root=UUID=7d64960d-34b6-4ae4-bd61-d532b53e2d7f
ro crashkernel=auto net.ifnames=0 biosdevname=0 rd.lvm.lv=rhel00/root
rd.lvm.lv=rhel00/swap

Re-enable the live patched version by running the following command:

$ sudo echo 1 > /sys/kernel/livepatch/my_first_livepatch/enable
$ cat /proc/cmdline
Livepatched kernel command line

Removing the live patch module

You can remove the live patch module like other kernel modules. Ensure that the module is disabled by passing the following command before unloading the module:

b sudo echo 0 > /sys/kernel/livepatch/my_first_livepatch/enable'

After the patch is disabled by the kernel live patching infrastructure, it writes &0& into the /sys/kernel/livepatch/my_first_livepatch/transition file indicating that it is safe to remove the live patching module. The module can be unloaded after the transition file reads &0& using the rmmod command.