Hi All,

This article is the last in this series and discusses how to roll back the patches that have been made on a Linux system.

Rolling back / undoing patching is really important - if an applied patch ends up breaking an major system (like Production) then we need to be sure that there is a way to put the system back into it's original state.

So what's the best way of rolling back patches?

We touched on this earlier in an earlier article, but undoubtedly the best way is to utilize the transaction capabilities of 'yum' to manage the whole process. Not only is it easier, but it's totally reliable.

One important point here is that it's vital that the correct repositories be available during the rollback process.

That is, not only the patching repositories, but also repositories for :

  • ol6_base (the base repository used when the Linux system was originally created)
  • ol6_latest (the generally available repository typically used by the server farm)
  • all the patching repositories

The scripts for enabling and disabling these repositories are :

[04:46 PM root@server-to-be-patched /rxr/depot/root/patching]# cat enable-extra-undo-repos.sh
#!/bin/bash
 
/usr/bin/yum-config-manager -q --enable patching_ol6_base,local_ol6_latest

 
[04:46 PM root@server-to-be-patched /rxr/depot/root/patching]# cat disable-extra-undo-repos.sh
#!/bin/bash
 
/usr/bin/yum-config-manager -q --disable patching_ol6_base,local_ol6_latest

The general sequence for rolling back patches looks like this :

  • disable-vm-repos.sh (disable the standard OEL repositories)
  • enable-vm-patching-repos.sh (enable the new patching repositories)
  • enable-extra-undo-repos.sh (enable the ol6_base and ol6_latest repositories)
  • perform undo of patching work
  • disable-extra-undo-repos.sh (disable the ol6_base and ol6_latest repositories)
  • disable-vm-patching-repos.sh (disable the new patching repositories)
  • enable-vm-repos.sh (enable the standard OEL repositories)

Let's now take a closer look at how yum transactions make the process not as painful as it could be.

The general command is : yum history undo <id> - where <id> can be found from the command : yum history list [all]

So, for example :

yum history list

Loaded plugins: security, ulninfo, versionlock
ID     | Login user               | Date and time    | Action(s)      | Altered
-------------------------------------------------------------------------------
    77 |  <rubiconred>            | 2019-02-03 22:20 | Downgrade      |    9 EE
    76 | root <root>              | 2019-02-03 22:10 | Downgrade      |   37 EE
    75 |  <rubiconred>            | 2019-02-03 22:07 | Install        |    1
    74 | root <root>              | 2019-02-03 21:35 | Downgrade      |    2
    73 | root <root>              | 2019-02-03 21:35 | Downgrade      |    2
    72 | root <root>              | 2019-02-03 21:35 | Downgrade      |    2
    71 | root <root>              | 2019-02-03 21:35 | Downgrade      |    1
    70 | root <root>              | 2019-02-03 21:34 | D, E           |    6
    69 | root <root>              | 2019-02-03 21:34 | D, E           |    4
    68 | root <root>              | 2019-02-03 21:34 | Downgrade      |    1
    67 | root <root>              | 2019-02-03 21:34 | Downgrade      |    1
    66 | root <root>              | 2019-02-03 21:34 | Downgrade      |    3
    65 | root <root>              | 2019-02-03 21:34 | Downgrade      |    1
    64 | root <root>              | 2019-02-03 21:34 | Downgrade      |    1
    63 | root <root>              | 2019-02-03 21:34 | Downgrade      |    1
    62 | root <root>              | 2019-02-03 21:34 | Downgrade      |    1
    61 | root <root>              | 2019-02-03 21:34 | Downgrade      |    1
    60 |  <rubiconred>            | 2019-01-31 22:11 | Update         |    1
    59 | root <root>              | 2019-01-31 22:09 | Update         |    1
    58 | root <root>              | 2019-01-31 22:09 | Update         |    1
history list

The default for 'yum history list' is to display the last 20 transactions.
Using the command 'yum history list all' will display the full transaction history.

If you need to have more information on a specific transaction then use the :
'yum history info <id>' command :

yum history info 73

[root@server-to-be-patched patching]# yum history info 73
Loaded plugins: security, ulninfo, versionlock
Transaction ID : 73
Begin time     : Sun Feb  3 21:35:05 2019
Begin rpmdb    : 730:6c787475e71134897b342d8d2406c8361ccc279e
End time       :                          (0 seconds)
End rpmdb      : 730:c397bf70ed88af1ece8bfec5d19a8da312dc4763
User           : root <root>
Return-Code    : Success
Command Line   : -y history undo 47
Transaction performed with:
    Installed     rpm-4.8.0-47.el6.x86_64      @anaconda-OracleLinuxServer-201507280245.x86_64/6.7
    Installed     yum-3.2.29-69.0.1.el6.noarch @anaconda-OracleLinuxServer-201507280245.x86_64/6.7
Packages Altered:
    Downgrade  curl-7.19.7-46.el6.x86_64          @patching_ol6_base
    Downgraded      7.19.7-53.0.1.el6_9.x86_64    @patching_ol6_latest
    Downgrade  libcurl-7.19.7-46.el6.x86_64       @patching_ol6_base
    Downgraded         7.19.7-53.0.1.el6_9.x86_64 @patching_ol6_latest
history info

There is actually quite a lot of information here - even down to the command line command used to execute transaction 73.
Very cool stuff.

Now, lets say we'd like to undo this undo this transaction - this is how we'd do it ... .

In this example, I don't use the '-y' option since I don't actually want to undo the transaction!

yum history undo 73

[root@server-to-be-patched]# yum history undo 73
Loaded plugins: security, ulninfo, versionlock
Undoing transaction 73, from Sun Feb  3 21:35:05 2019
    Downgrade  curl-7.19.7-46.el6.x86_64          @patching_ol6_base
    Downgraded      7.19.7-53.0.1.el6_9.x86_64    @patching_ol6_latest
    Downgrade  libcurl-7.19.7-46.el6.x86_64       @patching_ol6_base
    Downgraded         7.19.7-53.0.1.el6_9.x86_64 @patching_ol6_latest
Resolving Dependencies
--> Running transaction check
---> Package curl.x86_64 0:7.19.7-46.el6 will be updated
---> Package curl.x86_64 0:7.19.7-53.0.1.el6_9 will be an update
--> Processing Dependency: nss >= 3.36.0-8 for package: curl-7.19.7-53.0.1.el6_9.x86_64
---> Package libcurl.x86_64 0:7.19.7-46.el6 will be updated
---> Package libcurl.x86_64 0:7.19.7-53.0.1.el6_9 will be an update
--> Running transaction check
---> Package nss.x86_64 0:3.18.0-5.3.0.1.el6_6 will be updated
--> Processing Dependency: nss = 3.18.0-5.3.0.1.el6_6 for package: nss-sysinit-3.18.0-5.3.0.1.el6_6.x86_64
--> Processing Dependency: nss(x86-64) = 3.18.0-5.3.0.1.el6_6 for package: nss-tools-3.18.0-5.3.0.1.el6_6.x86_64
---> Package nss.x86_64 0:3.36.0-9.0.1.el6_10 will be an update
--> Processing Dependency: nss-util >= 3.36.0 for package: nss-3.36.0-9.0.1.el6_10.x86_64
--> Processing Dependency: nspr >= 4.19.0 for package: nss-3.36.0-9.0.1.el6_10.x86_64
--> Processing Dependency: libnssutil3.so(NSSUTIL_3.33)(64bit) for package: nss-3.36.0-9.0.1.el6_10.x86_64
--> Processing Dependency: libnssutil3.so(NSSUTIL_3.31)(64bit) for package: nss-3.36.0-9.0.1.el6_10.x86_64
--> Processing Dependency: libnssutil3.so(NSSUTIL_3.24)(64bit) for package: nss-3.36.0-9.0.1.el6_10.x86_64
--> Processing Dependency: libnssutil3.so(NSSUTIL_3.21)(64bit) for package: nss-3.36.0-9.0.1.el6_10.x86_64
--> Running transaction check
---> Package nspr.x86_64 0:4.10.8-1.el6_6 will be updated
---> Package nspr.x86_64 0:4.19.0-1.el6 will be an update
---> Package nss-sysinit.x86_64 0:3.18.0-5.3.0.1.el6_6 will be updated
---> Package nss-sysinit.x86_64 0:3.36.0-9.0.1.el6_10 will be an update
---> Package nss-tools.x86_64 0:3.18.0-5.3.0.1.el6_6 will be updated
---> Package nss-tools.x86_64 0:3.36.0-9.0.1.el6_10 will be an update
---> Package nss-util.x86_64 0:3.18.0-1.el6_6 will be updated
---> Package nss-util.x86_64 0:3.36.0-1.el6 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

==============================================================================================================================================================================
Package                                Arch                              Version                                        Repository                                      Size
==============================================================================================================================================================================
Updating:
curl                                   x86_64                            7.19.7-53.0.1.el6_9                            patching_ol6_latest                            197 k
libcurl                                x86_64                            7.19.7-53.0.1.el6_9                            patching_ol6_latest                            169 k
Updating for dependencies:
nspr                                   x86_64                            4.19.0-1.el6                                   patching_ol6_latest                            114 k
nss                                    x86_64                            3.36.0-9.0.1.el6_10                            patching_ol6_latest                            865 k
nss-sysinit                            x86_64                            3.36.0-9.0.1.el6_10                            patching_ol6_latest                             52 k
nss-tools                              x86_64                            3.36.0-9.0.1.el6_10                            patching_ol6_latest                            460 k
nss-util                               x86_64                            3.36.0-1.el6                                   patching_ol6_latest                             72 k

Transaction Summary
==============================================================================================================================================================================
Upgrade       7 Package(s)

Total download size: 1.9 M
Is this ok [y/N]:

These example pretty much cover how to roll back the 'security-minimal' patch, but there is one more aspect to consider - that being the rolling back of
the 'extra patches' made outside of the 'security-minimal' patch.

The way to approach this is to :

  • Perform a 'yum history list' and note all the transaction numbers from the last one applied back to the transaction for the 'security-minimal' patch.

  • Using this information, start from the latest transaction (largest ID number) and one by one roll back each patch previously applied.

  • In this way the rolling back of the patches is a complete reversal of the order in which they were applied.

At the end of the process, you will need to use 'grubby' to reset the original kernel level since the roll back process will not do this automatically. To save the reader from jumping to a previous article, the steps for using 'grubby' are repeated below :

To list all the available kernels for booting, use the command :
grubby --info=ALL - (the caps are important!)

grubby --info=ALL

[root@server-to-be-patched]# grubby --info=ALL
boot=/dev/xvdb
index=0
kernel=/vmlinuz-3.8.13-118.29.1.el6uek.x86_64
args="ro rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_linux/lv_swap rd_NO_MD rd_LVM_LV=vg_linux/lv_root SYSFONT=latarcyrheb-sun16 crashkernel=auto  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet"
root=/dev/mapper/vg_linux-lv_root
initrd=/boot/initramfs-3.8.13-118.29.1.el6uek.x86_64.img
index=1
kernel=/vmlinuz-3.8.13-68.3.4.el6uek.x86_64
args="ro rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_linux/lv_swap rd_NO_MD rd_LVM_LV=vg_linux/lv_root SYSFONT=latarcyrheb-sun16 crashkernel=auto  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet"
root=/dev/mapper/vg_linux-lv_root
initrd=/boot/initramfs-3.8.13-68.3.4.el6uek.x86_64.img
index=2
kernel=/vmlinuz-2.6.32-573.el6.x86_64
args="ro rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg_linux/lv_swap rd_NO_MD rd_LVM_LV=vg_linux/lv_root SYSFONT=latarcyrheb-sun16 crashkernel=auto  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet"
root=/dev/mapper/vg_linux-lv_root
initrd=/boot/initramfs-2.6.32-573.el6.x86_64.img

To make this a bit more readable - and since we only need to know about available vmlinuz images, we can slightly modify the command :
grubby --info=ALL | grep vmlinuz

[root@server-to-be-patched network-scripts]# grubby --info=ALL | grep vmlinuz
kernel=/vmlinuz-3.8.13-118.29.1.el6uek.x86_64
kernel=/vmlinuz-3.8.13-68.3.4.el6uek.x86_64
kernel=/vmlinuz-2.6.32-573.el6.x86_64

Just to make sure - let's check what the default kernel is :

grubby --default-kernel

[12:44 PM root@server-to-be-patched]# grubby --default-kernel
/boot/vmlinuz-3.8.13-68.3.4.el6uek.x86_64

OK - so we know that there is a later kernel available and that this system is still using the older version.

Let's set the new version :

grubby --set-default /boot/vmlinuz-3.8.13-118.29.1.el6uek.x86_64


[root@server-to-be-patched]# grubby --set-default /boot/vmlinuz-3.8.13-118.29.1.el6uek.x86_64

[root@server-to-be-patched]# grubby --default-kernel
/boot/vmlinuz-3.8.13-118.29.1.el6uek.x86_64

Rolling back patches - with the help of yum transactions - is actually quite straightforward.

Just roll back the transactions in the reverse order they were applied and it should all work well.

In the final article of this series, we'll quickly review all the considerations and steps for Linux Security patching.