As we all know, patching is an important and necessary part of the Support work we do. In terms of Linux, patching of Linux servers is a vital part of ensuring that there are no security holes that could be exploited by outside attackers.
Recently we needed to perform a complete Security Patching Cycle across a whole (large) farm of servers. Essentially, this meant applying the latest Security patches required to guarantee a secure Linux system.
Where to start on doing this can be daunting. The natural path is to go to the Oracle security pages and read through all the available patches, errata, etc. - which looks very daunting.
There is an easier way though - which is to use the yum package : 'yum-plugin-security' - a package that we'll see a lot more of during this discussion.
Although this document references Oracle Linux 6, the approach should also work fine with Oracle Linux 7.
Because this topic is quite large (and detailed), it has been broken up into a number of separate discussions to be published over the coming weeks:
Linux Patching #1 - Introduction and Overview (this one!)
Linux Patching #2 - Setting up local repositories and Apache directories
Linux Patching #3 - Nightly refresh of Repositories
Linux Patching #4 - Scripts to enable and disable non-patching & patching repositories
Linux Patching #5 - Using yum utilities to help in the patching process
Linux Patching #6 - Using RPM tools
Linux Patching #7 - Cataloging the Linux System before and after patching
Linux Patching #8 - Applying the security-minimal patch
Linux Patching #9 - Rolling back all the changes
Linux Patching #10 - Summary of steps for patching an environment
General Repository Considerations
But before that, there is much groundwork to be done before we can actually apply Linux Security patches.
Here is a general list of items we need to consider when patching Linux :
1. Create a new set of local Repositories that can be used for Linux patching work.
This is a critical piece of work. In any secure system, the Linux servers to be patched would not have direct access to the Internet - even via a Proxy. This should be totally forbidden.
Instead, these servers should access a local server which contains the 'yum' repositories required to service and patch those servers.
In this work, we will end up needing a set of repositories that represent :
The base repository that was used to install the original Linux System - typically from the install .iso.
This repository could be vital if we ever found ourselves in a situation where we'd need to back out any applied patches and we need to downgrade back to an original package version (that otherwise may not be available).
The general working repositories, repo-synced from the public repositories when the initial system was first built. It was this repository that was used when additional Puppet and Ansible runs were performed to create the 'Gold Image' Virtual Machine image that all Virtual Machines in the farm are based on.
The patching repositories that are the very latest versions of the Linux public repositories.
2. Set up a nightly refresh mechanism that not only refreshes these repositories but re-creates the Repos and security information for any future patching work.
- These repositories not only are repo-synced nightly, but the Security errata associated with these repositories is also re-applied every night. This results in the very latest Security patches being available for immediate application as required.
3. Create a new 'patching.repo' file that can be distributed to VMs to be patched
- Each Virtual Machine to be patched will need a new .repo file to be added to it's
/etc/yum.repos.d directory so they can access the new local patching repositories.
4. Execute the patching process using 'system management software'. In this case, I'll be using RunDeck.
It's makes good sense to use a some sort of system management software to help manage the patching process and collate the results in a neat and ordered manner.
What software you decide to use is pretty much depends on your own preferences etc., but I have found RunDeck (RunDeck Open Source Project and Downloads ) to be an excellent tool for this sort of work and can totally recommend it.
If you must run the patching process from the command line, ensure that a complete time stamped record is kept of the entire process. This is not only invaluable as a reference, but also for any security auditing that may be carried out across the system.
5. Prove that the applied patches can be rolled back if required.
It's one thing to apply a large set on patches - but we'll also need to prove that we can roll them out again in case disaster strikes.
We need to prove that :
- The Linux System is safely returned to it's pre-patched state
- The Linux System is still stable
This is a critical piece of work and can not be under-estimated.
For example, we'd never want to be in a position where, for some reason, a Production patch has gone bad and there were issues undoing the patch (or even worse, undoing isn't possible!).
Before we Start
Before starting, ensure that the following packages have been installed on the machine that will be hosting the local yum repositories :
- yum install yum-utils
- yum install createrepo
- yum install yum-plugin-security
Also, the yum patching package will need to be installed across this and all VMs that will be patched :
- yum install yum-plugin-security
Another useful resource is the 'yum cheatsheet' - it's well worth looking at.
Other background reading and important sites to look at :
Oracle Linux Security Errata
Oracle Linux 6 Security Guide
Oracle Critical Patch Updates
Stay tuned for the next post in our series - Linux Patching #2: Setting up local repositories and Apache directories.