Hi All,

So now, after all the preparation we are ready to actually apply the security patches!

There are a number of options we have for patching - based on how much we would like to patch (the scope).

From the Oracle Linux 6 Security Guide (page 41), here are the options :

To update all packages for which security-related errata are available to the latest versions of the packages, even if those packages include bug fixes or new features but not security errata, enter:

# yum --security update

To update all packages to the latest versions that contain security errata, ignoring any newer packages that do not contain security errata, enter:

# yum --security update-minimal

In the work we're doing here, we'll be using the 'update-minimal' option.

This is because it's worth keeping the 'change surface' on the servers being patched as small as possible. The larger the number of packages updated (plus those packages dependent on them) the greater the chance that the patch work may introduce some 'side effects' on the servers - which we really want to avoid. The important aspect to this work is to make sure that the Security related issues are dealt with - and 'update-minimal' allows us to do this.

In this article two scripts have been provided :

  1. Script to execute the 'update-minimal' security patching.

  2. An extra script that 'force install' a number of packages that the 'update-minimal' command does not.

The general flow of execution for both (the process the scripts follow internally) is :

  • Disable the general repositories used by the server
  • Enable the specific patching repositories
  • Execute the patching commands
  • Disable the specific patching repositories
  • Enable the general repositories

Please Note :

When I first ran the 'update-minimal' script, the majority of packages were updated but a number of these were not :

No package matched to upgrade: coreutils-0:8.4-46.0.1.el6
No package matched to upgrade: coreutils-libs-0:8.4-46.0.1.el6
No package matched to upgrade: curl-0:7.19.7-53.el6_9
No package matched to upgrade: dhclient-12:4.1.1-53.P1.0.1.el6_9.4
No package matched to upgrade: dhcp-common-12:4.1.1-53.P1.0.1.el6_9.4
No package matched to upgrade: gnutls-0:2.12.23-21.el6
No package matched to upgrade: jasper-libs-0:1.900.1-21.el6_9
No package matched to upgrade: kernel-uek-0:4.1.12-124.23.2.el6uek
No package matched to upgrade: kernel-uek-firmware-0:4.1.12-124.23.2.el6uek
No package matched to upgrade: krb5-libs-0:1.10.3-42z1.el6_7
No package matched to upgrade: libXfont-0:1.4.5-5.el6_7
No package matched to upgrade: libcurl-0:7.19.7-53.el6_9
No package matched to upgrade: libtirpc-0:0.2.1-13.el6_9
No package matched to upgrade: microcode_ctl-1:1.17-25.4.0.2.el6_9
No package matched to upgrade: nspr-0:4.11.0-0.1.el6_7
No package matched to upgrade: nss-softokn-0:3.14.3-23.el6_7
No package matched to upgrade: nss-util-0:3.28.4-1.el6_9
No package matched to upgrade: openssl-0:1.0.1e-48.el6_8.4
No package matched to upgrade: pam-0:1.1.1-20.el6_7.1
No package matched to upgrade: rpcbind-0:0.2.0-13.el6_9
No package matched to upgrade: sos-0:3.2-28.0.1.el6_7.2

On investigation, it was discovered that these particular packages were reliant on a previous version being installed on the server that was somewhat higher than the actual version installed.

To get around this, a custom script was written that explicitly installs the above packages (see below).

So, it's vital that you review the results from the 'update-minimal' patching to ensure that no further patching work is required.

Here are the two scripts - for both 'update-minimal' and 'update-extras'.
Please note that again 'figlet' has been used to create easy to read banners.

Firstly, the 'update-minimal' script :

update-minimal.sh

#!/bin/bash

GREEN="\e[92m"
RED="\e[31m"
STOP="\e[0m"

# Create a banner
printf "${GREEN}"
/usr/bin/figlet -w 180 "=============="
/usr/bin/figlet -w 180 "Start Patch Run"
/usr/bin/figlet -w 180 $(/bin/hostname)
/usr/bin/figlet -w 180 "Start Date / Time     : "
/usr/bin/figlet -w 180 $(/bin/date)
/usr/bin/figlet -w 180 "=============="
printf "${STOP}"

# First disable standard repos
printf "\n\n==========================================================\n"
printf "Disabling Standard Repos ...\n"
printf "==========================================================\n\n"

/rxr/depot/root/patching/disable-vm-repos.sh

# And enable the patching repos
printf "\n\n==========================================================\n"
printf "Enabling Patch Repos ...\n"
printf "==========================================================\n\n"

/rxr/depot/root/patching/enable-vm-patching-repos.sh

/usr/bin/yum repolist

# NOW DO THE PATCHING
/usr/bin/yum -y --security update-minimal

# Now disable patching repos
printf "\n\n==========================================================\n"
printf "Disabling Patching Repos ...\n"
printf "==========================================================\n\n"

/rxr/depot/root/patching/disable-vm-patching-repos.sh

# And enable the standard repos
printf "\n\n==========================================================\n"
printf "Enabling Standard Repos ...\n"
printf "==========================================================\n\n"

/rxr/depot/root/patching/enable-vm-repos.sh

# Do a final Repolist to confirm
printf "\n\n==========================================================\n"
printf "==========================================================\n\n"

/usr/bin/yum repolist

printf "${GREEN}"

/usr/bin/figlet -w 180 "=============="
/usr/bin/figlet -w 180 "Finish Patch Run"
/usr/bin/figlet -w 180 $(/bin/hostname)
/usr/bin/figlet -w 180 "Finish Date / Time     : "
/usr/bin/figlet -w 180 $(/bin/date)
/usr/bin/figlet -w 180 "=============="
printf "${STOP}"

This second script applies the 'extra packages' missed by the first patching run.
Of course, your list of 'extra packages' will likely be different than what follows below :

update-extras.sh

#!/bin/bash

GREEN="\e[92m"
RED="\e[31m"
STOP="\e[0m"

# Create a banner
printf "${GREEN}"
/usr/bin/figlet -w 180 "=============="
/usr/bin/figlet -w 180 "Start Patch-Extras Run"
/usr/bin/figlet -w 180 $(/bin/hostname)
/usr/bin/figlet -w 180 "Start Date / Time     : "
/usr/bin/figlet -w 180 $(/bin/date)
/usr/bin/figlet -w 180 "=============="
printf "${STOP}"

# First disable standard repos
printf "\n\n==========================================================\n"
printf "Disabling Standard Repos ...\n"
printf "==========================================================\n\n"

/rxr/depot/root/patching/disable-vm-repos.sh

# And enable the patching repos
printf "\n\n==========================================================\n"
printf "Enabling Patch Repos ...\n"
printf "==========================================================\n\n"

/rxr/depot/root/patching/enable-vm-patching-repos.sh

/usr/bin/yum repolist

# Patch packages not covered by the minimal update

yum update -y coreutils
yum update -y coreutils
yum update -y curl
yum update -y dhclient
yum update -y dhcp-common
yum update -y gnutls
yum update -y jasper-libs
yum update -y kernel-uek
yum update -y kernel-uek-firmware
yum update -y krb5-libs
yum update -y libXfont
yum update -y libcurl
yum update -y libtirpc
yum update -y microcode_ctl
yum update -y nss-softokn
yum update -y openssl
yum update -y pam
yum update -y rpcbind
yum update -y  sos

# Now disable patching repos
printf "\n\n==========================================================\n"
printf "Disabling Patching Repos ...\n"
printf "==========================================================\n\n"

/rxr/depot/root/patching/disable-vm-patching-repos.sh

# And enable the standard repos
printf "\n\n==========================================================\n"
printf "Enabling Standard Repos ...\n"
printf "==========================================================\n\n"

/rxr/depot/root/patching/enable-vm-repos.sh

# Do a final Repolist to confirm
printf "\n\n==========================================================\n"
printf "==========================================================\n\n"

/usr/bin/yum repolist

printf "${GREEN}"

/usr/bin/figlet -w 180 "=============="
/usr/bin/figlet -w 180 "Finish Patch Run"
/usr/bin/figlet -w 180 $(/bin/hostname)
/usr/bin/figlet -w 180 "Finish Date / Time     : "
/usr/bin/figlet -w 180 $(/bin/date)
/usr/bin/figlet -w 180 "=============="
printf "${STOP}"

After running the above scripts, the Linux system will now have all the relevant security patches applied.

At this point, it's a good idea to review what has been applied, especially in relation to any updated kernals.

It's important to ensure that if a new kernel is scheduled to be activated after a reboot then that is exactly what happens.

Check the current kernel level with : uname -a

Linux server-to-be-patched 3.8.13-68.3.4.el6uek.x86_64 #2 SMP Tue Jul 14 15:03:36 PDT 2015 x86_64 x86_64 x86_64 GNU/Linux

OK - so we are at : __3.8.13-68.3.4.el6uek.x86_64.__

Looking through the 'extras' run, we can see that the kernel was indeed patched to a higher level :

================================================================================
 Package          Arch   Version                Repository                 Size
================================================================================
Installing:
 kernel-uek-firmware
                  noarch 3.8.13-118.29.1.el6uek patching_ol6_UEKR3_latest 2.2 M

Transaction Summary
================================================================================
Install       1 Package(s)

Total download size: 2.2 M
Installed size: 2.8 M
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction

  Installing : kernel-uek-firmware-3.8.13-118.29.1.el6uek.noarch            1/1 

  Verifying  : kernel-uek-firmware-3.8.13-118.29.1.el6uek.noarch            1/1 

Installed:
  kernel-uek-firmware.noarch 0:3.8.13-118.29.1.el6uek                           

Complete!

Perform a reboot of the server and check the new kernel level with : uname -a

If the kernel level has not been updated then we'll need to do this manually - using the 'grubby' command. Here's how :

To list all the available kernels for booting, use the command : grubby --info=ALL (the caps are important!)

[root@server-to-be-patched]# grubby --info=ALL
boot=/dev/xvdb
index=0
kernel=/vmlinuz-3.8.13-118.29.1.el6uek.x86_64
args="ro rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_linux/lv_swap rd_NO_MD rd_LVM_LV=vg_linux/lv_root SYSFONT=latarcyrheb-sun16 crashkernel=auto  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet"
root=/dev/mapper/vg_linux-lv_root
initrd=/boot/initramfs-3.8.13-118.29.1.el6uek.x86_64.img
index=1
kernel=/vmlinuz-3.8.13-68.3.4.el6uek.x86_64
args="ro rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_linux/lv_swap rd_NO_MD rd_LVM_LV=vg_linux/lv_root SYSFONT=latarcyrheb-sun16 crashkernel=auto  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet"
root=/dev/mapper/vg_linux-lv_root
initrd=/boot/initramfs-3.8.13-68.3.4.el6uek.x86_64.img
index=2
kernel=/vmlinuz-2.6.32-573.el6.x86_64
args="ro rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_linux/lv_swap rd_NO_MD rd_LVM_LV=vg_linux/lv_root SYSFONT=latarcyrheb-sun16 crashkernel=auto  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet"
root=/dev/mapper/vg_linux-lv_root
initrd=/boot/initramfs-2.6.32-573.el6.x86_64.img

To make this a bit more readable - and since we only need to know about available vmlinuz images, we can slightly modify the command :

[root@server-to-be-patched network-scripts]# grubby --info=ALL | grep vmlinuz
kernel=/vmlinuz-3.8.13-118.29.1.el6uek.x86_64
kernel=/vmlinuz-3.8.13-68.3.4.el6uek.x86_64
kernel=/vmlinuz-2.6.32-573.el6.x86_64

Just to make sure - let's check what the default kernel is :

[12:44 PM root@server-to-be-patched]# grubby --default-kernel
/boot/vmlinuz-3.8.13-68.3.4.el6uek.x86_64

OK - so we know that there is a later kernel available and that this system is still using the older version.

Let's set the new version :

[root@server-to-be-patched]# grubby --set-default /boot/vmlinuz-3.8.13-118.29.1.el6uek.x86_64

[root@server-to-be-patched]# grubby --default-kernel
/boot/vmlinuz-3.8.13-118.29.1.el6uek.x86_64

Perfect! Now reboot the system and recheck the kernel level with : uname -a

It should be at the correct level now - and the Linux system has been successfully patched.

In the next article, we're going to examine how to roll back patches that have been applied to a system.