[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Savannah-cvs] [476] Bob's Guide to System Upgrades with Debian
|
From: |
bob |
|
Subject: |
[Savannah-cvs] [476] Bob's Guide to System Upgrades with Debian |
|
Date: |
Sat, 25 Jun 2022 18:36:48 -0400 (EDT) |
Revision: 476
http://svn.savannah.gnu.org/viewvc/?view=rev&root=administration&revision=476
Author: rwp
Date: 2022-06-25 18:36:47 -0400 (Sat, 25 Jun 2022)
Log Message:
-----------
Bob's Guide to System Upgrades with Debian
New guide.
Added Paths:
-----------
trunk/sviki/BobsGuideToSystemUpgrades.mdwn
Added: trunk/sviki/BobsGuideToSystemUpgrades.mdwn
===================================================================
--- trunk/sviki/BobsGuideToSystemUpgrades.mdwn (rev 0)
+++ trunk/sviki/BobsGuideToSystemUpgrades.mdwn 2022-06-25 22:36:47 UTC (rev
476)
@@ -0,0 +1,959 @@
+Bob's Guide to System Upgrades with Debian
+==========================================
+
+There are many guides to upgrading Debian and Debian derived systems.
+This one is mine.
+
+As with painting a house the quality of the result is mostly on the
+quality of the preparation for painting. Preparing a system for
+upgrade is the same. It is the preparation that most influences the
+quality of the result.
+
+At this time I am working through an upgrade of Trisquel 9 to Triquel
+10 and am using that as the working example for the documentation
+here.
+
+Release Notes
+-------------
+
+The first thing to do is to read the release notes for the new
+system. If there are special handling instructions then those must be
+followed. Historically while most upgrades have been generic there
+have been upgrades that required stepping the Linux kernel and udev
+forward in a particular ordering. And in the changing of init systems
+there have been requirements about the ordering of particular package
+upgrades. Always read the release notes and pay attention to any
+needed special handling.
+
+Install etckeeper and git
+-------------------------
+
+The etckeeper package is a utility to keep changes of /etc files in a
+version control system such as git. Using git is most popular but
+others can also be used. Then all changes to /etc can be seen in the
+version history. This means that one can "purge" packages without
+fear of the loss of any particular file. Since the changes will all
+be in version control. One can also rm any of those /etc files too.
+Since the chagnes will all be in version control. It's a great safety
+net.
+
+The git version control system is most popular. However git wants to
+know something about the user making commits. In this case the user
+is the root user. Therefore let's set up the root user for git first
+before even installing either git or etckeeper. Write this file to
+the root .gitconfig file.
+
+ cat >/root/.gitconfig <<EOF
+ [user]
+ name = root
+ email = root@localhost
+ EOF
+
+Alternatively there is now a git command to make git configuration
+modifications.
+
+ git config --global user.name root
+ git config --global user.email root@localhost
+
+Since root is actually a pseudo-user I prefer to use generic
+information fot the generic user. This is the default information
+that is logged in the commit log when no other overriding author
+information is provided. Admins that wish to override this may use
+the `GIT_AUTHOR_NAME` and `GIT_AUTHOR_EMAIL` and related env
+variables. Since that is then set up ensure that git is installed and
+then install etckeeper.
+
+ apt-get install git
+ apt-get install etckeeper
+
+At that point upkeep is automated. One can forget that it is there
+for the most part. All commits are then automated. To examine the
+commit log use normal git commands.
+
+ cd /etc
+ git log
+
+This is very useful. I strongly recommend it. It creates a safety
+net such that then one can do cleanup and other modifications without
+fear of losing track of those files and changes.
+
+Clean Old/New conffiles
+-----------------------
+
+The dpkg package manager tracks files in /etc as "conffiles". These
+are tracked such that if dpkg believes the file has NOT been locally
+modified that it automatically upgrade the file on package upgrade.
+However if dpkg believes the file has been modified locally then it
+will present a question as to how to handle it. Does the admin want
+to keep the current file? If so then the new file will be installed
+with a ".dpkg-new" suffix. Does the admin want to upgrade to the new
+package maintainer's file? If so then the new file will be installed
+and the old file will be moved to a ".dpkg-old" suffix. This makes it
+easy in either case for the admin to locate and further take action
+about these files.
+
+Additionally some packages in the maintainer scripts will use the name
+convention ".dpkg-dist" and add custom management. An example package
+that does this is the bind9 package. The use of ".dpkg-dist" is
+similar to ".dpkg-new" but outside the control of dpkg since it is
+custom handling added by the package maintainer.
+
+Some people did not like the handling provided by dpkg. They wanted
+more features in the upgrade. Also some packages wanted to do their
+own handling of /etc files which by Debian Policy needed to have local
+modifications preserved. The "ucf" utility was created for this
+purpose. See the "man ucf" page for documentation. Enough to say
+that ucf is like dpkg but different. Here we only need to know that
+it uses ".ucf-new" and ".ucf-old" suffixes instead of ones using the
+".dpkg-*" string. The cleanup of these is the same.
+
+ find /etc -name '*.dpkg-*'
+ find /etc -name '*.ucf-*'
+ ...I usually type in the simple form above...
+
+ find /etc \( -name '*.dpkg-*' -o -name '*.ucf-*' \) -print
+ ...but one can combine the find into one command...
+
+That will print the list of these files that exist. Review the list.
+Handle them all. Which means that if features should be merged then
+merge them. Or if files should be removed then simply remove them.
+
+Most of the ".dpkg-old" and ".ucf-old" files should simply be removed
+as part of the cleanup task. Those are older versions of the
+configuration files that were not used and are not being used. The
+upcoming upgrade will create more of these files. We want to clean
+the system of these files from previous upgrades so that we don't
+confuse what has happened a decade ago with something that has just
+now happened as part of the current upgrade.
+
+ find /etc \( -name '*.dpkg-old' -o -name '*.ucf-old' \) -print
+
+If that list is okay then remove them.
+
+ find /etc \( -name '*.dpkg-old' -o -name '*.ucf-old' \) -delete
+
+Look at new version of the file that are left.
+
+ find /etc \( -name '*.dpkg-*' -o -name '*.ucf-*' \) -print
+
+New versions of these files need attention. These are new package
+maintainer's versions of the configuration files that upon upgrade the
+admin said to retain the old version of the file. Therefore the
+incoming new file was placed with a ".dpkg-new" or ".dpkg-dist" name.
+If one were installing a new pristine system then these are the new
+versions of the file that would have been installed. Since we want to
+leave the upgraded system looking as closely to a newly installed
+system as possible then we need to install these versions of these
+files. This usually means merging local changes into the new version
+of the file and then replacing the current file with the merged file.
+
+Example. In a previous version of the sudo package the sudoers file
+did not include the `secure_path` option. Then the binary executable
+changed to require it and the package maintainer's version started
+including it. At that upgrade admins were presented with the option
+to keep their previous sudoers file or switch to the new sudoers file.
+I daresay that many admins said to keep their existing file. And were
+snagged by now missing the secure_path option! The result was that
+when they ran sudo that they no longer had /usr/sbin:/sbin in PATH.
+The fix for this was to merge in their local changes into the new
+version of the sudoers configuration file.
+
+That example is a good reason why nice packages with configuration
+files provide a way to add local configuration in a different file
+that is not tracked. A file specifically for the local admin to
+modify. That way it will not need to be merged in any future package
+upgrade. In this case sudo includes all files in `/etc/sudoers.d/*`
+making that the best place for local configuration. I suggest using a
+name with "local" in the name so that we can easily tell that it is a
+local downstream configuration. `/etc/sudoers.d/sudoers-local` for
+example. Then that file will never need merging. The nicer packages
+all make this type of configuration available. Use it.
+
+The /etc directory should be cleaned of all '*.dpkg-*' files before
+upgrading. And always checked for required merges after upgrading.
+
+Clean "rc" Packages
+-------------------
+
+Debian's dpkg package manager has an interesting and useful feature
+that removing a package leaves the /etc configuration files behind.
+This allows installing the package again and immediately having the
+same configuration files as before. This means that if testing two
+incompatible programs installing one may push out the other one and
+allows re-installing it again and immediately being back to the same
+state as before. Those /etc configuration files that are managed by
+the dpkg package manager have the special name "conffile". This is
+useful. But in order for the package manager to continue to manage
+those conffiles even after the package has been removed it means that
+removing the package does not actually remove EVERYTHING about the
+package. The package remains installed but in the removed-config
+state or "rc" state where the package has been removed but conffiles
+remain installed.
+
+You can list packages on the system that are in this "rc" state.
+
+ dpkg -l | grep ^rc
+
+Another way to list these packages using the grep-status command.
+Install the `dctrl-tools` package to get this useful utility.
+
+ grep-status -sPackage -n -FStatus 'deinstall ok config-files'
+
+Especially if a system has been operating for a long time and upgraded
+there may be quite a few packages in the rc state. These create
+obstacles for the upgrade process because in addition to the current
+packages and current package dependencies these older packages and
+older package dependencies also must all be considered when performing
+an upgrade.
+
+These packages need to be purged in order to fully remove them from
+the system. Debian's dpkg differentiates the actions "remove" and
+"purge" for this. With dpkg "remove" will remove the non-conffile
+files, the files not in /etc, but will leave the conffiles in /etc on
+the system and will leave the package otherwise listed as installed in
+order to manage those conffiles. The "purge" action will remove those
+conffiles and remove the package from the listing. And of course that
+also opens up any packages in the Depends relation for removal too.
+
+ grep-status -sPackage -n -FStatus 'deinstall ok config-files'
+
+Review the list of removed-config packages and if okay then purge them
+all using this command.
+
+ dpkg --purge $(grep-status -sPackage -n -FStatus 'deinstall ok
config-files')
+
+After these have been purged then other packages may now become
+candidates for automremoval. If a package was installed as a
+dependency of another package then it will be an automatically
+installed package. Automatically installed packages that no longer
+have a dependency keeping them installed are now candidates for
+autoremoval. And when we autoremove we want to use the --purge option
+so that packages are purged and we avoid creating more packages in the
+"rc" state and needing to clean those out again.
+
+ apt-get autoremove --purge
+
+Example.
+
+ root@frontend1:~# apt-get autoremove --purge
+ Reading package lists... Done
+ Building dependency tree
+ Reading state information... Done
+ The following packages will be REMOVED:
+ fonts-dejavu* fonts-dejavu-extra* g++-5* gcc-6-base* imagemagick-6.q16*
+ libapache2-mod-php7.0* libassuan-dev* libconfig-file-perl*
libgpg-error-dev*
+ libpython3.5-minimal* libpython3.5-stdlib* libstdc++-5-dev* libxtables11*
+ linux-image-unsigned-4.4.0-197-generic* linux-modules-4.4.0-197-generic*
+ lua-lpeg* php7.0* php7.0-cli* php7.0-common* php7.0-fpm* php7.0-gd*
+ php7.0-json* php7.0-mysql* php7.0-opcache* php7.0-readline*
python-chardet*
+ python-debian* python-debianbts* python-httplib2* python-pkg-resources*
+ python-pycurl* python-pysimplesoap* python-reportbug* python-setuptools*
+ python-six* python3.5* python3.5-minimal*
+ 0 upgraded, 0 newly installed, 37 to remove and 0 not upgraded.
+ After this operation, 156 MB disk space will be freed.
+ Do you want to continue? [Y/n]
+ (Reading database ... 72633 files and directories currently installed.)
+ Removing fonts-dejavu (2.37-1) ...
+ Removing fonts-dejavu-extra (2.37-1) ...
+ Removing g++-5 (5.5.0-12ubuntu1) ...
+ Removing gcc-6-base:amd64 (6.5.0-2ubuntu1~18.04) ...
+ Removing imagemagick-6.q16 (8:6.9.7.4+dfsg-16ubuntu6.13) ...
+ Removing php7.0 (7.0.33-0ubuntu0.16.04.16) ...
+ Removing libapache2-mod-php7.0 (7.0.33-0ubuntu0.16.04.16) ...
+ apache2_invoke php7.0 prerm: No action required
+ Removing libassuan-dev (2.5.1-2) ...
+ Removing libconfig-file-perl (1.50-3) ...
+ Removing libgpg-error-dev (1.27-6) ...
+ Removing python3.5 (3.5.2-2ubuntu0~16.04.13) ...
+ Removing libpython3.5-stdlib:amd64 (3.5.2-2ubuntu0~16.04.13) ...
+ Removing python3.5-minimal (3.5.2-2ubuntu0~16.04.13) ...
+ Unlinking and removing bytecode for runtime python3.5
+ Removing libpython3.5-minimal:amd64 (3.5.2-2ubuntu0~16.04.13) ...
+ Removing libstdc++-5-dev:amd64 (5.5.0-12ubuntu1) ...
+ Removing libxtables11:amd64 (1.6.0-2ubuntu3) ...
+ Removing linux-image-unsigned-4.4.0-197-generic
(4.4.0-197.229+8.0trisquel3) ...
+ /etc/kernel-img.conf:5: W: ignoring unknown parameter do_boot_enable
+ /etc/kernel/postrm.d/initramfs-tools:
+ update-initramfs: Deleting /boot/initrd.img-4.4.0-197-generic
+ Removing linux-modules-4.4.0-197-generic (4.4.0-197.229+8.0trisquel3) ...
+ Removing lua-lpeg:amd64 (1.0.0-2ubuntu0.18.04.1) ...
+ Removing php7.0-fpm (7.0.33-0ubuntu0.16.04.16) ...
+ apache2_invoke php7.0-fpm prerm: No action required
+ Removing php7.0-cli (7.0.33-0ubuntu0.16.04.16) ...
+ Removing php7.0-mysql (7.0.33-0ubuntu0.16.04.16) ...
+ Removing php7.0-json (7.0.33-0ubuntu0.16.04.16) ...
+ Removing php7.0-gd (7.0.33-0ubuntu0.16.04.16) ...
+ Removing php7.0-opcache (7.0.33-0ubuntu0.16.04.16) ...
+ Removing php7.0-readline (7.0.33-0ubuntu0.16.04.16) ...
+ Removing python-reportbug (6.6.6ubuntu1) ...
+ Removing python-debian (0.1.32) ...
+ Removing python-chardet (3.0.4-1+ubuntu16.04.1+certbot+2) ...
+ Removing python-debianbts (2.7.2) ...
+ Removing python-pysimplesoap (1.16-2) ...
+ Removing python-httplib2 (0.9.2+dfsg-1ubuntu0.3) ...
+ Removing python-setuptools (39.0.1-2) ...
+ Removing python-pkg-resources (39.0.1-2) ...
+ Removing python-pycurl (7.43.0.1-0.2) ...
+ Removing python-six (1.11.0-2) ...
+ Removing php7.0-common (7.0.33-0ubuntu0.16.04.16) ...
+ Processing triggers for install-info (6.5.0.dfsg.1-2) ...
+ Processing triggers for libc-bin (2.27-3ubuntu1.6) ...
+ Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
+ Processing triggers for hicolor-icon-theme (0.17-2) ...
+ Processing triggers for fontconfig (2.12.6-0ubuntu2) ...
+ Processing triggers for mime-support (3.60ubuntu1) ...
+ (Reading database ... 69300 files and directories currently installed.)
+ Purging configuration files for php7.0-mysql (7.0.33-0ubuntu0.16.04.16) ...
+ Purging configuration files for php7.0-readline (7.0.33-0ubuntu0.16.04.16)
...
+ Purging configuration files for linux-modules-4.4.0-197-generic
(4.4.0-197.229+8.0trisquel3) ...
+ dpkg: warning: while removing linux-modules-4.4.0-197-generic, directory
'/lib/modules/4.4.0-197-generic' not empty so not removed
+ Purging configuration files for php7.0-opcache (7.0.33-0ubuntu0.16.04.16)
...
+ Purging configuration files for libapache2-mod-php7.0
(7.0.33-0ubuntu0.16.04.16) ...
+ apache2_invoke postrm: Purging state for php7.0
+ dpkg: warning: while removing libapache2-mod-php7.0, directory
'/etc/php/7.0/apache2/conf.d' not empty so not removed
+ Purging configuration files for php7.0-gd (7.0.33-0ubuntu0.16.04.16) ...
+ Purging configuration files for php7.0-cli (7.0.33-0ubuntu0.16.04.16) ...
+ dpkg: warning: while removing php7.0-cli, directory
'/etc/php/7.0/cli/conf.d' not empty so not removed
+ Purging configuration files for php7.0-fpm (7.0.33-0ubuntu0.16.04.16) ...
+ apache2_invoke php7.0-fpm postrm: No action required
+ dpkg: warning: while removing php7.0-fpm, directory
'/etc/php/7.0/fpm/conf.d' not empty so not removed
+ Purging configuration files for python3.5-minimal
(3.5.2-2ubuntu0~16.04.13) ...
+ Purging configuration files for php7.0-json (7.0.33-0ubuntu0.16.04.16) ...
+ Purging configuration files for linux-image-unsigned-4.4.0-197-generic
(4.4.0-197.229+8.0trisquel3) ...
+ /etc/kernel-img.conf:5: W: ignoring unknown parameter do_boot_enable
+ Purging configuration files for php7.0-common (7.0.33-0ubuntu0.16.04.16)
...
+ dpkg: warning: while removing php7.0-common, directory '/etc/php/7.0' not
empty so not removed
+ Purging configuration files for libpython3.5-minimal:amd64
(3.5.2-2ubuntu0~16.04.13) ...
+ Processing triggers for systemd (237-3ubuntu10.43) ...
+ Processing triggers for ureadahead (0.100.0-21) ...
+
+In addition to autoremove I find using "deborphan" to be another good
+tool for more cleaning. Install deborphan. Run it. Use it to do
+more cleaning. The deborphan tool is an old tool. It's utility has
+mostly been replaced by autoremove. But it is still useful and still
+almost always finds more packages that are good candidates for purging.
+
+ deborphan
+ dpkg --purge $(deborphan)
+
+Repeat the above sequence until there are no more packages listed from
+deborphan. After purging some things listed by deborphan it will then
+open up additional packages as candidates. Keep repeating until the
+list is reduced to zero packages being listed as orphans. In some
+cases you may need to preserve one of the listed packages if it is an
+orphan but just the same one that you want to preserve.
+
+Clean Packages Without An Installation Candidate
+------------------------------------------------
+
+Some packages have gone away over time. They just don't exist
+anymore. They will never be upgraded again. These create obstacles
+to the system since if they don't ever get upgraded the shared
+libraries that they use are also stuck. These packages need to be
+removed in order to allow the upgrade to occur without entanglements
+with ancient libraries and other dependencies.
+
+Some packages have encoded version numbers. The emacs packages are
+examples of ones with an included version number such as emacs21 and
+so on from there. These will never get upgraded and their presence
+can cause problems. For example emacs shares some of the files in a
+common package to be shared across versions. But really old versions
+then cause failures due to the use of newer features. These packages
+need to be removed in order to allow the upgrade to occur without
+entanglements with ancient libraries and other dependencies.
+
+The easiest way to list these is with `apt-show-versions`. Install
+that utility and use it to list packages without an installation
+source. Most packages will show as "uptodate" and so filtering those
+out will show all of the rest of the packages that are interesting.
+Specifically listing out packages without an install source are the
+ones we want to know about.
+
+ apt-show-versions | grep -v -e uptodate
+ apt-show-versions | awk '/No available version in archive/{print$1}'
+
+That will also list a few packages that we want to keep for a bit
+longer. Older Linux kernels that are installed for example. Those
+don't get upgraded but are useful to provide a safety net for
+rebooting back to if needed. Therefore some intelligence and
+filtering needs to be mentally applied to this list. Also any locally
+created packages will be listed if they have been installed as a
+package but do not have an associated repository. Review. Purge
+packages that these identify and that you agree should be cleaned up
+from the system.
+
+Example.
+
+ root@frontend1:~# apt-show-versions | grep -v -e uptodate
+ emacs24-bin-common:amd64 24.5+1-6ubuntu1.1 installed: No available version
in archive
+ emacs24-common:all 24.5+1-6ubuntu1.1 installed: No available version in
archive
+ emacs24-common-non-dfsg:all 24.4+1-2+8.0trisquel2 installed: No available
version in archive
+ emacs24-el:all 24.5+1-6ubuntu1.1 installed: No available version in archive
+ emacs24-nox:amd64 24.5+1-6ubuntu1.1 installed: No available version in
archive
+ gcc-4.9-base:amd64 4.9.3-13ubuntu2 installed: No available version in
archive
+ initscripts:amd64 2.88dsf-59.3ubuntu2 installed: No available version in
archive
+ insserv:amd64 1.14.0-5ubuntu3 installed: No available version in archive
+ libcgmanager0:amd64 0.39-2ubuntu5 installed: No available version in
archive
+ libreadline6:amd64 6.3-8ubuntu2 installed: No available version in archive
+ libreadline6-dev:amd64 6.3-8ubuntu2 installed: No available version in
archive
+ libtasn1-6:amd64 4.13-2+8.0trisquel1 newer than version in archive
+ linux-image-unsigned-4.4.0-210-generic:amd64 4.4.0-210.242+8.0trisquel4
installed: No available version in archive
+ linux-modules-4.4.0-210-generic:amd64 4.4.0-210.242+8.0trisquel4
installed: No available version in archive
+ mountall:amd64 2.54ubuntu1 installed: No available version in archive
+ python-gpgme:amd64 0.3-1.1 installed: No available version in archive
+ sysv-rc:all 2.88dsf-59.3ubuntu2 installed: No available version in archive
+
+In this case all of those packages are okay candidates for removal.
+If a new system were installed then none of those would be installed.
+But this makes a good example of a problem that needs manually
+handling. Because look at libtasn1-6! It is listed as "newer than
+version in archive". WAT?! How can that happen?
+
+This is an odd Trisquel problem.
+
+ root@frontend1:~# apt-show-versions | grep libtasn1-6
+ libtasn1-6:amd64 4.13-2+8.0trisquel1 newer than version in archive
+
+ root@frontend1:~# apt-cache policy libtasn1-6
+ libtasn1-6:
+ Installed: 4.13-2+8.0trisquel1
+ Candidate: 4.13-2+8.0trisquel1
+ Version table:
+ *** 4.13-2+8.0trisquel1 100
+ 100 /var/lib/dpkg/status
+ 4.13-2 500
+ 500 http://mirror.fsf.org/trisquel etiona/main amd64 Packages
+
+It looks like it went forward and then backward in the upstream
+Trisquel repository? This is an unusual Trisquel specfic case. Let's
+manually walk it back to something that matches what is in the repository.
+
+ root@frontend1:~# apt-get install libtasn1-6=4.13-2
+ Reading package lists... Done
+ Building dependency tree
+ Reading state information... Done
+ The following packages will be DOWNGRADED:
+ libtasn1-6
+ 0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 0 not
upgraded.
+ Need to get 36.2 kB of archives.
+ After this operation, 0 B of additional disk space will be used.
+ Do you want to continue? [Y/n]
+ Get:1 http://mirror.fsf.org/trisquel etiona/main amd64 libtasn1-6 amd64
4.13-2 [36.2 kB]
+ Fetched 36.2 kB in 0s (165 kB/s)
+ dpkg: warning: downgrading libtasn1-6:amd64 from 4.13-2+8.0trisquel1 to
4.13-2
+ (Reading database ... 60111 files and directories currently installed.)
+ Preparing to unpack .../libtasn1-6_4.13-2_amd64.deb ...
+ Unpacking libtasn1-6:amd64 (4.13-2) over (4.13-2+8.0trisquel1) ...
+ Setting up libtasn1-6:amd64 (4.13-2) ...
+ Processing triggers for libc-bin (2.27-3ubuntu1.6) ...
+
+Now that problem is handled we return to the issue of removing things
+that are now no longer in the repository.
+
+ root@frontend1:~# apt-show-versions | awk -F: '!/uptodate/{print$1}'
+ emacs24-bin-common
+ emacs24-common
+ emacs24-common-non-dfsg
+ emacs24-el
+ emacs24-nox
+ gcc-4.9-base
+ initscripts
+ insserv
+ libcgmanager0
+ libreadline6
+ libreadline6-dev
+ linux-image-unsigned-4.4.0-210-generic
+ linux-modules-4.4.0-210-generic
+ mountall
+ python-gpgme
+ sysv-rc
+
+ root@frontend1:~# apt-get purge $(apt-show-versions | awk -F:
'!/uptodate/{print$1}')
+ Reading package lists... Done
+ Building dependency tree
+ Reading state information... Done
+ The following packages will be REMOVED:
+ ...
+
+ root@frontend1:~# apt-show-versions | awk -F: '!/uptodate/{print$1}'
+ root@frontend1:~#
+
+And now it is clean for those types of files.
+
+Packages with "obsolete conffiles"
+----------------------------------
+
+This is one of the more confusing aspects of packages and managing
+packages. This is due to buggy maintenance of a package leaving
+behind "obsolete conffiles".
+
+An obsolete file is one that was owned by a previous version of the
+package, removed from a later version of the package, but left behind
+when upgraded. These files are marked by dpkg as obsolete files in
+the package file listing. They are obsolete because if you were to
+install a pristine system with the new version of the package that
+conffile would not be installed. But when doing an upgrade if a
+previous version of the package contained the conffile and if the
+package does not handle the removal of it properly then this conffile
+is left behind as lint.
+
+Here is some mailing list discussion for reference.
+
+* http://lists.debian.org/debian-mentors/2011/07/msg00284.html
+* http://lists.debian.org/debian-mentors/2013/05/msg00115.html
+
+These commands will list out these files.
+
+ dpkg-query -W -f='${Conffiles}\n' | grep obsolete
+ dpkg-query -W -f='${Package} ${Conffiles}\n' | less +/obsolete
+
+That latter one is most useful because it lists out both the package
+as well as the file that is marked as obsolete. Browse that list.
+Unfortunately as Paul noted in that message from 2013(!) there are
+still quite a few packges with these bugs in them. And here it is
+years and years later and we are still fighting with the exact same
+bugs and problems!
+
+Browsing this list will show the files remaining behind that were
+included with a previous version of a package but are now no longer
+included. These fall into a couple of different cases that are useful
+to treat differently.
+
+Moved /etc files. These files are harder to deal with because they
+were removed from one package and added to a different package.
+Simply removing them solves the problem of the obsolete conffile but
+creates a problem of a missing conffile for the other package that now
+includes it. Therefore one cannot simply remove it. Instead one must
+temporarily move it out of the way to a save location, then reinstall
+the package claiming it is obsolete, then move the file back into place.
+
+I must talk about moved files first. Because otherwise we don't know
+if the file was moved and we remove it incorrectly. But in order for
+a file to be in both it will need to be listed as both obsolete for
+one package and not obsolete for another package.
+
+ apt-show-versions /etc/apt/apt.conf.d/20apt-show-versions
e02489f6adb594c2609841ba130345fe
+ /etc/cron.daily/apt-show-versions ceccc6292ad7c2e762bd04f0bee3bf98
+ /etc/bash_completion.d/apt-show-versions db61f00b9796596527fb9510bacc681c
obsolete
+
+ root@frontend1:~# dpkg -S /etc/bash_completion.d/apt-show-versions
+ apt-show-versions: /etc/bash_completion.d/apt-show-versions
+
+That file only belongs to the apt-show-versions package. Therefore it
+is not moved. It has only been removed. This is the simpler case.
+Just remove it and reinstall the package.
+
+ root@frontend1:~# rm -f /etc/bash_completion.d/apt-show-versions
+ root@frontend1:~# apt-get install --reinstall apt-show-versions
+
+If the file is owned by multiple packages however then we have to not
+remove it but instead move it to a saved location, reinstall the
+package marking it as obsolete, then move the file back to continue to
+be part of the other package.
+
+Init files. With the migration to systemd (love it or hate it) many
+package maintainers have summarily removed the `/etc/init.d/*` files
+from the package without handling the removal properly. This leaves
+those init scripts behind. This also includes init scripts from
+"upstart" which is used in Ubuntu. If you are also drinking the
+Kool-Aid of systemd then you will not be using any of these init
+scripts again. In which case removing these files is always the
+correct answer. A pristine installation would not include them.
+However if you are counting on that init script then move it out of
+the way, reinstall the package, and move the init script back. It is
+something the local sysadmin would know.
+
+Here is a simple example with an upstart init script.
+
+ xinetd
+ ...
+ /etc/init/xinetd.conf 8b8bc68acf27ae09755d71b09d3ddab4 obsolete
+
+Mitigation.
+
+ root@frontend1:~# rm -f /etc/init/xinetd.conf
+ root@frontend1:~# apt-get install --reinstall xinetd
+ Reading package lists... Done
+ Building dependency tree
+ Reading state information... Done
+ 0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 0 not
upgraded.
+ Need to get 108 kB of archives.
+ After this operation, 0 B of additional disk space will be used.
+ Get:1 http://mirror.fsf.org/trisquel etiona/main amd64 xinetd amd64
1:2.3.15.3-1 [108 kB]
+ Fetched 108 kB in 1s (106 kB/s)
+ [master 00aad4c] saving uncommitted changes in /etc prior to apt run
+ 2 files changed, 54 deletions(-)
+ delete mode 100644 init/xinetd.conf
+ (Reading database ... 72637 files and directories currently installed.)
+ Preparing to unpack .../xinetd_1%3a2.3.15.3-1_amd64.deb ...
+ Unpacking xinetd (1:2.3.15.3-1) over (1:2.3.15.3-1) ...
+ Setting up xinetd (1:2.3.15.3-1) ...
+ Processing triggers for systemd (237-3ubuntu10.43) ...
+ Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
+ Processing triggers for ureadahead (0.100.0-21) ...
+
+Random /etc files. These are all unique one-off cases that need to be
+looked at individually. There are a lot of files and very likely they
+are not being used. But it is *possible* that you are using one of
+those files, perhaps even customized one of those files. If so then
+removing the file would remove part of your needed customization.
+Review the file. Understand if you need it. Remove it if it is not a
+file that you care about. Here is a simple example.
+
+ base-files
+ /etc/update-motd.d/50-motd-news 1008a5c52c15ce6f484da4ff9c36d294 obsolete
+
+Mitigation.
+
+ root@frontend1:~# rm -f /etc/update-motd.d/50-motd-news
+ root@frontend1:~# apt-get install --reinstall base-files
+ Reading package lists... Done
+ Building dependency tree
+ Reading state information... Done
+ 0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 0 not
upgraded.
+ Need to get 58.4 kB of archives.
+ After this operation, 0 B of additional disk space will be used.
+ Get:1 http://mirror.fsf.org/trisquel etiona-updates/main amd64 base-files
amd64 1:10.1ubuntu2.11+9.0trisquel3 [58.4 kB]
+ Fetched 58.4 kB in 0s (2,398 kB/s)
+ [master 8f9d49a] saving uncommitted changes in /etc prior to apt run
+ 2 files changed, 147 deletions(-)
+ delete mode 100755 update-motd.d/50-motd-news
+ (Reading database ... 72636 files and directories currently installed.)
+ Preparing to unpack
.../base-files_1%3a10.1ubuntu2.11+9.0trisquel3_amd64.deb ...
+ Unpacking base-files (1:10.1ubuntu2.11+9.0trisquel3) over
(1:10.1ubuntu2.11+9.0trisquel3) ...
+ Setting up base-files (1:10.1ubuntu2.11+9.0trisquel3) ...
+ Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
+ Processing triggers for install-info (6.5.0.dfsg.1-2) ...
+
+Then look for another file. Remove it. Reinstall. Repeat until all
+of the obsolete conffiles have been worked through. This manual
+handling is rather tedious. There may be quite a few files. I know
+of no better way than to simply grind through it. Fortunately
+progress is made quickly and we will very soon finish the list and be
+done with this step.
+
+Finding out how many of these files need to be fixed.
+
+ root@frontend1:~# dpkg-query -W -f='${Package} ${Conffiles}\n' | grep -c
obsolete
+ 26
+
+Ugh! Well let's grind through those and get done with them then.
+
+Transition and Dummy Packages
+-----------------------------
+
+Look for packages named as transitional or dummy packages. These are
+cases where the names of the package has been changed. Sometimes
+packages have been reorganized.
+
+ dpkg -l | grep -e transition -e dummy
+
+Review the list. Remove the transitional and dummy packages. But be
+aware that the grep might match on other packages that should not be
+removed. Also in some cases an OS release is made which is buggy and
+even though these packages are transitional packages other packages
+may depend upon them. In which case they cannot be removed without
+also removing the other packages that we want installed.
+
+ ii libncurses5-dev:amd64 6.1+20181013-2+deb10u2 amd64 transitional package
for libncurses-dev
+
+ apt-get purge libncurses5-dev
+
+Init Files Without LSB Headers
+------------------------------
+
+This part is of historical interest on systemd systems. On
+non-systemd systems however this is still an important step.
+
+Look for files in `/etc/init.d/` that are not part of any package but
+may not have LSB headers. If not then the upgrade to insserv will
+fail. Find those and enumerate them first.
+
+ dpkg -S $(find /etc/init.d -type f -perm /a+x) 2>&1 >/dev/null | awk '/not
found/{print$2}'
+
+Each of those files are probably some locally installed file that is
+not part of any package. Ensure that they contain LSB headers so that
+the "insserv" program can work with them without an error.
+
+Completion of Cleaning
+----------------------
+
+All of the above steps are important preparation for an upgrade.
+Whew! If you made it here then you have made it through a long
+preparation and cleaning process! Congratulations!
+
+Upgrade!
+--------
+
+This process basically follows as outlined in the release notes. Read
+them again. Then proceed!
+
+However a generic upgrade process can be outlined and most of the time
+this is the process we use.
+
+Release the Hold on Held Packages
+---------------------------------
+
+List any held packages.
+
+ apt-mark showhold
+
+That will list any packages that are explicitly held to prevent
+upgrades. Examine that list. Can they be released? At the least be
+aware of these. At the most unhold them and let them upgrade.
+
+For an example the mercurial package is currently held due to
+incompatible changes in a security patch upgrade. The issues of the
+security issues did not apply to our case therefore not taking the
+upgrade was not a problem. But the mitigation for the issue was
+completely showstopping breaking. We still do not know what we are
+going to do about this long term.
+
+ apt-mark unhold foopackage
+
+Check for /usr/sbin/policy-rc.d
+-------------------------------
+
+This file controls starting of services by packages. Notably it does
+NOT control starting of packages at boot time by init services. It's
+a specific restriction targeting use of packages and handling of
+upgrades in a chroot container. But this file has been found
+accidentally existing in systems outside of containers. Since it does
+not control boot time init actions it was apparently not noticed when
+the system was provisioned and delivered.
+
+ ls -ld /usr/sbin/policy-rc.d
+
+Remove Stale Preferences Configuration
+--------------------------------------
+
+Remove and/or update any stale preferences files that may have been
+installed and configured.
+
+ ls -l /etc/apt/preferences /etc/apt/preferences.d/*
+
+Silence apt-listchanges
+-----------------------
+
+If the apt-listchanges package is installed then it is very noisy
+during a full system release upgrade. And what is one going to do?
+If one were installing a pristine system then we would get the new
+versions of the packages. This is regardless of whether
+apt-listchanges would warn about possible problems with packages. If
+it is installation time of a pristine system then these will get
+installed regardless. Therefore if apt-listchanges is installed it is
+better to purge it for the upgrade. If you want to install it back
+again after the upgrade then that at least avoids the problems it
+causes during the upgrade.
+
+ apt-get purge apt-listchanges
+
+Free Apt Cache Space
+--------------------
+
+If the system hasn't been getting cleaned regularly then there may be
+gigabytes and gigabytes of old deb package files hanging around in
+`/var/cache/apt/archives` directory. Since we are upgrading we will
+be leaving those old versions behind never to be used again. Remove
+them to free up space which can then be used for new packages.
+
+ apt-get clean
+
+Non-Interactive Upgrades
+------------------------
+
+The way I handle upgrades is a little different. I prefer to do as
+much of the upgrade non-interactively as possible. It's really
+deferring the interactive actions. Then interactively handling the
+parts that needed interaction after the non-interactive part of the
+upgrade has completed.
+
+ export DEBIAN_FRONTEND=noninteractive
+
+These variables control the action of dpkg. It selects the
+non-interactive frontend. This is the part that asks the user whether
+they want to keep the old file or install the new file or show
+differences.
+
+ export DEBCONF_ADMIN_EMAIL=""
+
+When noninteractive is selected dpkg will then decide to email the
+admin some information instead of presenting it interactively on the
+screen. Since I am upgrading many systems this would all be repeat
+information every time it was done. After I have seen the information
+once I don't need to see it every time after then.
+
+ export UCF_FORCE_CONFFNEW=1
+
+The ucf program similar to dpkg manages conffiles and will ask if the
+new file should be used or the old file. This pre-selects using the
+new conffile. We will do the same for dpkg but dpkg does not use
+environment variables for this but command line options instead.
+
+ export UCF_FORCE_CONFFMISS=1
+
+If the conffile has been removed do not treat this as a local
+modification to be preserved but install the package maintainer's
+version of the file the same as if it were a pristine installation.
+(Treating removal as a local modification was successfully argued by a
+pedantic Debian user. It wasn't always that way.)
+
+Update /etc/apt/sources.list files
+----------------------------------
+
+Review the apt sources.list files. Sometimes configuration files will
+be found in `/etc/apt/sources.list.d` that were not known about.
+These most often contain 3rd party repositories.
+
+ ls -l /etc/apt/sources.list.d /etc/apt/sources.list
+
+What to do about 3rd party repositories? It is best to never use 3rd
+party repositories. And if you do then it is best to never upgrade
+the system. Instead create a new system, install new stuff there, and
+then throw away the old system. Because 3rd parties most often do not
+handle upgrades well. They expect pristine installation only.
+
+My advice if 3rd party software has been installed is to purge it
+before the upgrade. Then if it is still required install it again
+after the upgrade. That's safest. This was probably already found
+during the `apt-show-versions` part of the cleaning. But maybe not.
+Remove the 3rd party sources.list entries and then look at what is
+left. Purge off those packages.
+
+ apt-show-versions | grep -v -e uptodate
+
+If everything is ready for the upgrade then change the release name in
+the sources.list file to the new release. Update to download the new
+Packages index files.
+
+ sed --in-place 's/etiona/nabia/g' /etc/apt/sources.list
+ apt-get update
+
+Download All Packages Before Upgrade
+------------------------------------
+
+Download all of the packages first to the local machine to remove the
+dependency upon the network needing to be alive during the
+installation. If network-manager is in use it has a notorious
+reputation for dropping the network during the upgrade and not
+bringing it back online again. Unfortunately network-manager is now a
+default and pulled in by gnome during some upgrades. In any case it
+is much safer to download every package to the cache before starting
+the upgrade. The -d option downloads but does not install.
+
+ apt-get upgrade -d -q -y
+ apt-get upgrade --with-new-pkgs -d -q -y
+ apt-get dist-upgrade -d -q -y
+
+Upgrade. This time with feeling!
+---------------------------------
+
+All of the above has been the preparation. Now we are finally ready
+to perform the actual upgrade. This is the final place where it is
+critically important to have read the release notes and to be aware of
+any specific upgrade needs. Most often needed for udev and kernel
+related upgrades.
+
+I prefer to upgrade as non-interactively as possible. Batchmode.
+Then react to things needing interactive action afterward. These are
+the commands. I will repeat the env variables again to reinforce
+their association though the rationale was explained above.
+
+ export DEBIAN_FRONTEND=noninteractive DEBCONF_ADMIN_EMAIL=""
UCF_FORCE_CONFFNEW=1 UCF_FORCE_CONFFMISS=1
+ apt-get -o DPkg::Options::=--force-confnew -o
DPkg::Options::=--force-confmiss upgrade -q -y < /dev/null
+ apt-get -o DPkg::Options::=--force-confnew -o
DPkg::Options::=--force-confmiss upgrade --with-new-pkgs -q -y < /dev/null
+ apt-get -o DPkg::Options::=--force-confnew -o
DPkg::Options::=--force-confmiss dist-upgrade -q
+
+We pass in the dpkg options to force confnew so that if a conffile has
+been removed, which is almost certainly an accident, then it will be
+installed using the package maintainer's version. This is always what
+we want. (It's only the other way by default because of a pedantic
+user who successfully argued that removing a required conffile
+constituted a deliberate local admin change and must by Debian Policy
+be preserved. I disagree strongly.)
+
+We pass in the dpkg options to select the new conffile to always
+replace the old conffile. The old conffile will be renamed to a name
+with ".dpkg-old" or ".ucf-old" suffix. After the upgrade we will look
+for those files and perform any merges needed.
+
+The upgrade first uses "upgrade" which is a one-for-one package
+upgrade. No new package *names* are installed. No old package names
+are removed. But if packages can be upgraded in place then they are
+upgraded in place. This handles the majority of packages using an
+algorithm that cannot be in error.
+
+Then use `upgrade --with-new-pkgs` which allows new package names to
+be installed due to being pulled in by package dependencies. This
+handles most of the rest of the upgrade topology. No packages are
+allowed to be removed making this another safe upgrade algorithm.
+
+Then use `dist-upgrade` to perform the final bits of the upgrade that
+require removing packages. This is not a safe action and normally
+requires careful review. But what's a person to do? We are most of
+the way there. Therefore we can only do our best and barge forward.
+I removed the -y yes option from the recipe above to require the admin
+to review and confirm the action. But honestly in my own scripts I
+keep the -y yes option there and barge forward.
+
+Post Upgrade Clean-Up
+---------------------
+
+Since we have instructed dpkg to install the new conffiles and save
+the locally modifed ones to ".dpkg-old" everywhere the first thing we
+need to do is to merge in changes. And specifically to the most
+critical services. Critical services such as ssh which we are
+undoubtedly using for the upgrade.
+
+ ls -l /etc/ssh/sshd_config*
+ emacs /etc/ssh/sshd_config*
+ service ssh restart || systemctl restart ssh.service
+
+Examine the old file. Merge in any local changes from there to the
+new sshd_config file. Restart ssh. Verify that login using it is
+still possible.
+
+ find /etc \( -name '*.dpkg-*' -o -name '*.ucf-*' \) -print
+
+Review and merge each of those files. Some of those services will be
+non-functional until that step is done.
+
+The `/etc/skel` files are not upgraded automatically. Since we do not
+customize `/etc/profile` and simply use the base-files copy directly
+we can simply copy it into place. Review the other files there for
+upgrades too.
+
+ diff -u /usr/share/base-files/profile /etc/profile
+ cp /usr/share/base-files/profile /etc/profile
+
+File Bug Reports
+----------------
+
+Repeat all of the same pre-upgrade cleaning all over again. We have
+upgraded almost every package on the system. There will be new
+conffiles that need to be merged. There will be "*.dpkg-*" files to
+be handled. There will be packages no in the rc state. There will be
+new transition packages installed needing removal. Go back to the
+start of the cleaning and repeat all of the cleaning steps again.
+
+Along the way we will find many packages that did not handle the
+upgrade correctly. They will have obsolete conffiles. They will have
+other problems. These are all upgrade bugs. File an appropriate bug
+report for the problems found.
+
+Reboot
+------
+
+This is the final step. Hold your breath. Reboot. The system needs
+to boot in order to run the newly installed kernel. After rebooting
+run the system regression test suite. Any failures? Fix failures.
+
+Savannah external test suite.
+
+* https://git.savannah.nongnu.org/cgit/administration/savannah-tests.git
+
+Congratulations!
+----------------
+
+If you have made it to here then you have completed the full upgrade
+process from one end to the other.
| [Prev in Thread] |
Current Thread |
[Next in Thread] |
- [Savannah-cvs] [476] Bob's Guide to System Upgrades with Debian,
bob <=