In 1991, two unrelated events took place, each of which heralded two very different freedoms: the end of the Cold War and the birth of Linux.
The possibility of patching the kernel real-time came in 2008, when Linux was still a teenager. Now that the Linux kernel is about to hit 30, real-time patching is ripe, too, and poised to belittle its reputation as an optional add-on that is just "nice to have in the box."
There are two reasons for this. First is the dominance of Linux as the platform of choice for economical and versatile web hosting: more than halfall known websites are currently running on Linux. Second, the recognition that real-time patching is more than a convenience; it is also an effective and low-cost way to improve the security of your Linux system.
Ksplice, the first real-time Linux kernel patching solution, effortlessly gained lucrative status after launch, only strengthening its leadership when tech giant Oracle bought it in 2011. Today it is the most famous of five organizations offering automated security patching services for Linux. In alphabetical order, these are:
- Canonical Livepatch Service for Ubuntu.
- KernelCare for Ubuntu, Red Hat Enterprise Linux, Oracle Linux, Amazon Linux and more.
- Kpatch for Red Hat Enterprise Linux.
- Ksplice for Oracle Linux.
- SUSE Linux Enterprise Live Patching for SUSE Linux Enterprise.
Each company promotes its services using the same boilerplate keywords, the same standard phrases, repeated endlessly. I will give you the most popular ones and how you should understand them.
- Rebootless : Refreshes the Linux kernel without rebooting.
- Automated: Checks and installs kernel security updates without an observer.
- Easy install: Installation and setup is very simple or unnecessary.
- Fully supported: you don't need to write patches yourself. (Look at the manual for writing patches on Kpatch, to understand why this is important)
Ignore these blindingly obvious selling points and you will find other features to consider, other factors you can use to compare one service to another. This article explores some of them, lightly seasoned with affirmation and edification. But first, let's recall what live patching is.
What is real-time Linux kernel patching?
The Linux kernel contains over 20 million lines of code written mostly (presumably) by human programmers. As with all software, it has bugs. In this age of increased focus on cybersecurity, bugs imply vulnerabilities . To solve this problem, Linux vendors try to release updates to their kernels as soon as possible. Linux sysadmins are also trying to react quickly by installing these updates, making scheduling more difficult. This is because there is no pattern for detecting vulnerabilities. Here are some reasons why:
- There are many different versions of the Linux kernel in use at any given time. Some will need fixing; some don't.
- , .
- Linux , .
- Linux . , , — .
There is one more thing that is rarely mentioned: it is the instinctive disgust of the system administrator for stopping the machine, causing physical discomfort from having to say: "The server is down for maintenance."
Why stop the server? Because the kernel update won't take effect until the reboot. If you don't reboot - if you put it on the back burner or forget it - you will find yourself in a quandary, because a server without the latest patches is insecure . We will talk about this mantra in more detail later, but its essence is this:
When vulnerabilities become public, the same happens with their exploits.
Or, in other words, you have a vulnerability that you didn't know about before, but now you do. And everyone else too.
The savior in this scenario is live patching . This is a way to update not the entire Linux kernel, but only its problematic part. And this is done without stopping processes or interrupting users.
However, there are limitations. Realtime kernel patching cannot fix all kinds of bugs. The overwhelming complexity of the kernel code and the technical challenges of changing code on the fly means this technique is only used to troubleshoot critical issues, usually security-related.
Despite this, a system with real-time patching facilities is safer than a system without it. Here are a few questions to ask when looking for a real-time patching solution:
1. What's inside the patch (and who is behind it)?
When you sign up for a live patching service, you pay more than just the software itself. This becomes clear when you start digging through the Linux kernel source and realize how a poorly written patch can harm an entire Linux system.
I mentioned earlier that the kernel is a huge body of work that seems to expand in both activity and volume. Note the increase in the number of lines of code in the master branch of the Linux kernel repository. (Counts as of January 1 of each year. Only C / C ++ files and headers, Python and Perl files are counted.)
With so much code, writing patches for the Linux kernel is no easy task. A deep understanding of kernel architecture and data structures is required. Experience with Linux community coding conventions and quality standards. And it takes a special talent to invent solutions that do not affect functionality, performance and stability.
There are two opposite methods that patch vendors use to develop and deliver kernel patches. I call them incremental and monolithic .
Incremental patches, like a ball of chewing gum, are stacked on top of each other, each modifying the behavior of the previous one. As a customer, you must keep track of which patches you install and which you don’t, and the order in which they are installed. Failure to pay attention to these issues can lead to unpredictable changes in the behavior of your system. This patching method allows vendors to quickly release small targeted patches. But over time, the system can become unstable to the point that only a full kernel update can restore it.
A monolithic patch is a patch that includes all previous patches. Basically, there is only one master patch that replaces the active one rather than being added to it. This approach makes the platform more stable, significantly increasing server uptime, which is often measured in thousands of days.
2. How long to wait until the next patch?
There is inevitably a delay between reporting a vulnerability and fixing it. It takes time to analyze kernel errors and time to assess their impact. It takes ingenuity and skill to find a solution, and tons of tests to make sure it works. It is then subject to mandatory review and approval as part of the Linux development process.
Avoidable delays occur when the Linux vendor replaces the community severity rating with its own. This means that a vendor can fix multiple vulnerabilities with fewer patches, but because of this, the customer waits on average longer to fix certain issues.
Linux administrators who understand the reasons for the erratic nature of patch releases do not benefit much more than those who do not. None of them find comfort in the knowledge that while they wait for patches, their systems are even more vulnerable to an exploit than they were before it was discovered.
Here's why: Members of the cybersecurity research community love to announce vulnerabilities along with a visual use case. They usually take the form of a proof of concept, a detailed technical description of how to reproduce a bug or use an exploit. These descriptions are in good faith to help kernel developers find and patch the hole. But they also save hackers a lot of time and effort by giving them a shortcut to a literal recipe for disaster in their race to exploit vulnerability as a weapon.
3. How serious should the error be?
Almost all newly discovered vulnerabilities are assigned a CVE identifier. Later, upon closer examination by security researchers, a vulnerability is rated as severity, that is, a measure of its impact.
An important rating scheme is a general system Vulnerability Assessment (CVSS - Common Vulnerability Scoring System) . It represents a vulnerability as a set of numbers, each of which is a score for a characteristic, for example:
- How easy is it to reproduce (use).
- How difficult it is to fix it.
- The scale of the impact on the availability of servers and services.
- Importance or confidentiality of open data.
The complete set includes many more similar assessments. The algorithm combines these into a baseline score , one neat number representing the severity of the vulnerability, ranging from 0 (low) to 10 (high). CVSS of the second version divides the ranges of these numbers into the keywords LOW, MEDIUM and HIGH. The new version of CVSS 3 adds two more: NONE and CRITICAL.
Thus, the number of vulnerabilities varies not only by months and years, but also by severity.
Vulnerability rating schemes such as CVSS enable Linux vendors to assess how to respond to a specific kernel vulnerability. For example, they might want to write patches for vulnerabilities with a cumulative score of 7 or higher. This means HIGH in CVSS v2, HIGH or CRITICAL in CVSS v3. A vendor claiming to target critical vulnerabilities can only be referring to vulnerabilities with severity ratings 9 and 10 when using CVSS v3.
4. Can I roll back a patch?
Installing patches automatically without any supervision is a dreadful idea for many sysadmins. They know that even thoroughly tested fixes can change the behavior of a system, affecting its performance or functionality in a subtle and not entirely obvious way. When this happens, when the server behaves strangely after installing the patch, the quickest way to deal with it is to remove the patch.
Some realtime patching services can remove patches. This makes it easier to determine if a recent update is causing a change in system behavior.
5. Can I host my own patch server?
The patching software agent checks in real time for available patches on the remote Patch servers. It does this at regular, configurable intervals, usually with the ability to perform custom checks. If a patch is available, the agent software will download and install it. But if the patching agent is unable to communicate with the patch server where the vendor's patching service stores patches, live patching will not happen.
To solve this problem, you need to create your own patch server. Such a local server broadcasts patches to all computers in your company under your firewall. Copy Safe downloads the patch and then distributes it through the firewall after checking the integrity of the patch files. You can manage and audit this process at your convenience by making your company's board a little more relaxed.
There are other benefits too:
- You have better control over which patches your servers receive and when.
- You can block a large number of servers for known patch levels.
- It is easier to separate server clusters for development, testing, staging, and production.
Dynamic patching requires a local agent and a remote patch server. Controlling both is critical for enterprise deployments.
Conclusion
Before waking up, here are three additional questions to ask a potential patching solution provider:
- Can I cancel automatic renewal? There are times when you don't want to automatically update kernels.
- What platforms does it work on? While your server environment may preclude any flexibility in choosing a real-time patching solution, it is best if your subscription to the service does not limit which Linux option you need to use.
- ? , Linux. . , .
The fact that they are less prominent in patching vendor product description lists does not make the features described in this article any less important. Any of these can influence your decision to contact a specific supplier. Each of these can directly affect the effectiveness and relevance of the solution for your environment. Since server subscription fees range from a few tens per month to thousands of US dollars per year, it is helpful to be careful when choosing a Linux kernel patching solution.
Read more:
Install and Configure AlienVault SIEM (OSSIM)
10 best practices for securing Docker images. Part 1
10 best practices for securing Docker images. Part 2