[lmi] Working around a problem with tiny partitions

[Greg Chicares]

[For documentation only: this problem has been solved.]

The main drive on a corporate server is partitioned thus (there's
clearly nothing proprietary here):

  ├─VolGroup00-lv_root    253:0    0    9G  0 lvm  /
  ├─VolGroup00-lv_swap    253:1    0   16G  0 lvm  [SWAP]
  ├─VolGroup00-lv_usr     253:2    0    8G  0 lvm  /usr
  ├─VolGroup00-lv_var_log 253:3    0    2G  0 lvm  /var/log
  ├─VolGroup00-lv_var     253:4    0    8G  0 lvm  /var
  ├─VolGroup00-lv_tmp     253:5    0    4G  0 lvm  /var/tmp
  ├─VolGroup00-lv_opt     253:6    0    4G  0 lvm  /opt
  └─VolGroup00-lv_home    253:7    0    2G  0 lvm  /home

but our chroots need more than ten GiB each at inception, and grow over
time, as measured on my own machine:

# du -sxh /srv/chroot/*
33G     /srv/chroot/bullseye0
13G     /srv/chroot/centos7lmi
14G     /srv/chroot/lmi_bookworm_4
21G     /srv/chroot/lmi_bullseye_3

so we've been mounting /srv/chroot on a comparatively generous 80 GiB
drive. Still, we've repeatedly run out of space, because repositories
(and submodules) in /srv/cache_for_lmi were large enough...

#du -sxh /srv/cache_for_lmi
1.7G    /srv/cache_for_lmi

...to have filled up the available remainder of the / partition.

I considered these options to avoid this problem:

(0) [not worth trying] Ask for a less fragmented partition scheme.

(1) [now implemented] Mount the 80 GiB partition as /srv rather than
as /srv/chroot. Then everything just works. (The /srv directory on
this server was empty except for what I had put there, and it doesn't
seem that a default redhat system uses it for anything.) Commands in
detail (recorded here in case they're ever needed again):

# mkdir -p /mnt/sdc1/chroot
# mount /dev/sdc1 /mnt/sdc1
# mv /mnt/sdc1/* /mnt/sdc1/chroot

[Just ignore the harmless warning that the new subdirectory cannot
be moved into itself--it's not worth writing a complex, error-prone
"find...xargs" pipeline just to avoid that.]

[Verify that the new subdirectory actually contains the chroots, and
that everything (including dotfiles) got moved, and that no vestige
remains in the original location.]

# mkdir /mnt/sdc1/cache_for_lmi
# mv /srv/cache_for_lmi/* /mnt/sdc1/cache_for_lmi

[That 'mv' command, unlike the previous one, moves across filesystems
and therefore takes a moment.]

# umount /mnt/sdc1
# rmdir /mnt/sdc1

# vim /etc/fstab

[alter fstab:
- LABEL=lmi /srv/chroot xfs defaults 0 0
+ LABEL=lmi /srv xfs defaults 0 0
and save]

# mount -a

[now, sdc1 was mounted both as /srv and as /srv/chroot ]

# umount /srv/chroot

[That seemed to clean up the old mount. A web search finds advice to
stop and restart some systemd thing, but I ignored it.]

(2) [hideous] Solve the problem by adding a level of indirection,
inserting a common artificial subdirectory into the mounts, e.g.:

  // on the host:
  /dev/sdc1 /artificial
  // in all chroots:
  /artificial/srv/cache_for_lmi /srv/cache_for_lmi
  /artificial/srv/chroot        /srv/chroot

but that would force many changes:
 - to schroot's default directories
 - in all schroots
 - on my personal machine too
 - throughout a complex system of lmi_setup* scripts

(3) [unattractive] Split the 80 GiB drive into two separate partitions,
one for each of the subdirectories we want to use. This would work, but
it's not really in the best of taste to imitate the underlying problem.
And it wouldn't take much less effort than the preferred solution (1).