| 1 |
|
|---|
| 2 | Testbox Imaging (Backup / Restore)
|
|---|
| 3 | ==================================
|
|---|
| 4 |
|
|---|
| 5 |
|
|---|
| 6 | Introduction
|
|---|
| 7 | ------------
|
|---|
| 8 |
|
|---|
| 9 | This document is explores deloying a very simple drive imaging solution to help
|
|---|
| 10 | avoid needing to manually reinstall testboxes when a disk goes bust or the OS
|
|---|
| 11 | install seems to be corrupted.
|
|---|
| 12 |
|
|---|
| 13 |
|
|---|
| 14 | Definitions / Glossary
|
|---|
| 15 | ======================
|
|---|
| 16 |
|
|---|
| 17 | See AutomaticTestingRevamp.txt.
|
|---|
| 18 |
|
|---|
| 19 |
|
|---|
| 20 | Objectives
|
|---|
| 21 | ==========
|
|---|
| 22 |
|
|---|
| 23 | - Off site, no admin interaction (no need for ILOM or similar).
|
|---|
| 24 | - OS independent.
|
|---|
| 25 | - Space and bandwidth efficient.
|
|---|
| 26 | - As automatic as possible.
|
|---|
| 27 | - Logging.
|
|---|
| 28 |
|
|---|
| 29 |
|
|---|
| 30 | Overview of the Solution
|
|---|
| 31 | ========================
|
|---|
| 32 |
|
|---|
| 33 | Here is a brief summary:
|
|---|
| 34 |
|
|---|
| 35 | - Always boot testboxes via PXE using PXELINUX.
|
|---|
| 36 | - Default configuration is local boot (hard disk / SSD)
|
|---|
| 37 | - Restore/backup action triggered by machine specific PXE config.
|
|---|
| 38 | - Boots special debian maintenance install off NFS.
|
|---|
| 39 | - A maintenance service (systemd style) does the work.
|
|---|
| 40 | - The service reads action from TFTP location and performs it.
|
|---|
| 41 | - When done the service removes the TFTP machine specific config
|
|---|
| 42 | and reboots the system.
|
|---|
| 43 |
|
|---|
| 44 | Maintenance actions are:
|
|---|
| 45 | - backup
|
|---|
| 46 | - redo-backup
|
|---|
| 47 | - restore
|
|---|
| 48 |
|
|---|
| 49 | Possible modifier that indicates a subset of disk on testboxes with other OSes
|
|---|
| 50 | installed. Support for partition level backup/restore is not explored here.
|
|---|
| 51 |
|
|---|
| 52 |
|
|---|
| 53 | Storage Server
|
|---|
| 54 | ==============
|
|---|
| 55 |
|
|---|
| 56 | The storage server will have three areas used here. Using NFS for all three
|
|---|
| 57 | avoids extra work getting CIFS sharing right too (NFS is already a pain).
|
|---|
| 58 |
|
|---|
| 59 | 1. /exports/testbox-tftp - TFTP config area. Read-write.
|
|---|
| 60 | 2. /exports/testbox-backup - Images and logs. Read-write.
|
|---|
| 61 | 3. /exports/testbox-nfsroot - Custom debian. Read-only, no root squash.
|
|---|
| 62 |
|
|---|
| 63 |
|
|---|
| 64 | TFTP (/exports/testbox-tftp)
|
|---|
| 65 | ============================
|
|---|
| 66 |
|
|---|
| 67 | The testbox-tftp share needs to be writable, root squashing is okay.
|
|---|
| 68 |
|
|---|
| 69 | We need files from both PXELINUX and SYSLINUX to make this work now. On a
|
|---|
| 70 | debian system, the ``pxelinux`` and ``syslinux`` packages needs to be
|
|---|
| 71 | installed. We actually do this further down when setting up the nfsroot, so
|
|---|
| 72 | it's possible to get them from there by postponing this step a little. On
|
|---|
| 73 | debian 8.6.0 the PXELINUX files are found in ``/usr/lib/PXELINUX`` and the
|
|---|
| 74 | SYSLINUX ones in ``/usr/lib/syslinux``.
|
|---|
| 75 |
|
|---|
| 76 | The initial PXE image as well as associated modules comes in three variants,
|
|---|
| 77 | BIOS, 32-bit EFI and 64-bit EFI. We'll only need the BIOS one for now.
|
|---|
| 78 | Perform the following copy operations::
|
|---|
| 79 |
|
|---|
| 80 | cp /usr/lib/PXELINUX/pxelinux.0 testbox-tftp/
|
|---|
| 81 | cp /usr/lib/syslinux/modules/*/ldlinux.* testbox-tftp/
|
|---|
| 82 | cp /usr/lib/syslinux/modules/bios testbox-tftp/
|
|---|
| 83 | cp /usr/lib/syslinux/modules/efi32 testbox-tftp/
|
|---|
| 84 | cp /usr/lib/syslinux/modules/efi64 testbox-tftp/
|
|---|
| 85 |
|
|---|
| 86 |
|
|---|
| 87 | For simplicitly, all the testboxes boot using good old fashioned BIOS, no EFI.
|
|---|
| 88 | However, it doesn't really hurt to be prepared.
|
|---|
| 89 |
|
|---|
| 90 | The PXELINUX related files goes in the root of the testbox-tftp share. (As
|
|---|
| 91 | mentioned further down, these can be installed on a debian system by running
|
|---|
| 92 | ``apt-get install pxelinux syslinux``.) We need the ``*pxelinux.0`` files
|
|---|
| 93 | typically found in ``/usr/lib/PXELINUX/`` on debian systems (recent ones
|
|---|
| 94 | anyway). It is possible we may need one ore more fo the modules [1]_ that
|
|---|
| 95 | ships with PXELINUX/SYSLINUX, so do copy ``/usr/lib/syslinux/modules`` to
|
|---|
| 96 | ``testbox-tftp/modules`` as well.
|
|---|
| 97 |
|
|---|
| 98 |
|
|---|
| 99 | The directory layout related to the configuration files is dictated by the
|
|---|
| 100 | PXELINUX configuration file searching algorithm [2]_. Create a subdirectory
|
|---|
| 101 | ``pxelinux.cfg/`` under ``testbox-tftp`` and create the world readable file
|
|---|
| 102 | ``default`` with the following content::
|
|---|
| 103 |
|
|---|
| 104 | PATH bios
|
|---|
| 105 | DEFAULT local-boot
|
|---|
| 106 | LABEL local-boot
|
|---|
| 107 | LOCALBOOT
|
|---|
| 108 |
|
|---|
| 109 | This will make the default behavior to boot the local disk system.
|
|---|
| 110 |
|
|---|
| 111 |
|
|---|
| 112 |
|
|---|
| 113 | Images and logs (/exports/testbox-backup)
|
|---|
| 114 | =========================================
|
|---|
| 115 |
|
|---|
| 116 | The testbox-backup share needs to be writable, root squashing is okay.
|
|---|
| 117 |
|
|---|
| 118 | We use a directory layout based on dotted decimal IP addresses here, so for a
|
|---|
| 119 | server with the IP 10.40.41.42 all its file will be under ``10.40.41.42/``.
|
|---|
| 120 |
|
|---|
| 121 |
|
|---|
| 122 | Files found under ``10.40.41.42/``:
|
|---|
| 123 |
|
|---|
| 124 | ``<hostname>``
|
|---|
| 125 | The name of the testbox (empty file). Help finding a testbox by name.
|
|---|
| 126 |
|
|---|
| 127 | ``testbox-info.txt``
|
|---|
| 128 | Information about the testbox. Starting off with the name, decimal IP,
|
|---|
| 129 | PXELINUX style hexadecimal IP, and more.
|
|---|
| 130 |
|
|---|
| 131 | ``maintenance.log``
|
|---|
| 132 | Maintenance log file recording what the maintenance service does.
|
|---|
| 133 |
|
|---|
| 134 | ``disk-devices.lst``
|
|---|
| 135 | Optional list of disk devices to consider backuping up or restoring. This is
|
|---|
| 136 | intended for testboxes with additional disks that are used for other purposes
|
|---|
| 137 | and should touched.
|
|---|
| 138 |
|
|---|
| 139 | ``sda.raw.gz``
|
|---|
| 140 | The gzipped raw copy of the sda device of the testbox.
|
|---|
| 141 |
|
|---|
| 142 | ``sd[bcdefgh].raw.gz``
|
|---|
| 143 | The gzipped raw copy sdb, sdc, sde, sdf, sdg, sdh, etc if any of them exists
|
|---|
| 144 | and are disks/SSDs.
|
|---|
| 145 |
|
|---|
| 146 |
|
|---|
| 147 | Note! If it turns out we can be certain to get a valid host name, we might just
|
|---|
| 148 | switch to use the hostname as the directory name instead of the IP.
|
|---|
| 149 |
|
|---|
| 150 |
|
|---|
| 151 | Debian NFS root (/exports/testbox-nfsroot)
|
|---|
| 152 | ==========================================
|
|---|
| 153 |
|
|---|
| 154 | The testbox-nfsroot share should be read-only and must **not** have root
|
|---|
| 155 | squashing enabled.
|
|---|
| 156 |
|
|---|
| 157 | There are several ways of creating a debian nfsroot, but since we've got a
|
|---|
| 158 | tool like VirtualBox around we've just installed it in a VM, prepared it,
|
|---|
| 159 | and copied it onto the NFS server share.
|
|---|
| 160 |
|
|---|
| 161 | As of writing debian 8.6.0 is current, so a minimal 64-bit install of it was
|
|---|
| 162 | done in a VM. After installation the following modifications was done:
|
|---|
| 163 |
|
|---|
| 164 | - ``apt-get install pxelinux syslinux initramfs-tools gddrescue joe`` and
|
|---|
| 165 | optionally ``apt-get install smbclient cifs-utils``.
|
|---|
| 166 |
|
|---|
| 167 |
|
|---|
| 168 | - ``/etc/default/grub`` was modified to set ``GRUB_CMDLINE_LINUX_DEFAULT`` to
|
|---|
| 169 | ``""`` instead of ``"quiet"``. This allows us to see messages during boot
|
|---|
| 170 | and perhaps spot why something doesn't work on a testbox. Regenerate the
|
|---|
| 171 | grub configuration file by running ``update-grub`` afterwards.
|
|---|
| 172 |
|
|---|
| 173 | - Create the directory ``/etc/systemd/system/[email protected]`` and create
|
|---|
| 174 | the file noclear.conf in it with the following content::
|
|---|
| 175 |
|
|---|
| 176 | [Service]
|
|---|
| 177 | TTYVTDisallocate=no
|
|---|
| 178 |
|
|---|
| 179 | This stops getty from clearing VT1 and let us see the tail of the boot up
|
|---|
| 180 | messages, which includes messages from the testbox-maintenance service.
|
|---|
| 181 |
|
|---|
| 182 | - Copy the ``testbox-maintenance.sh`` file found in the same directory as this
|
|---|
| 183 | document to ``/root/scripts/`` (need to create the dir) and make it
|
|---|
| 184 | executable.
|
|---|
| 185 |
|
|---|
| 186 | - Create the systemd service file for the maintenance service as
|
|---|
| 187 | ``/etc/systemd/system/textbox-maintenance.service`` with the content::
|
|---|
| 188 |
|
|---|
| 189 | [Unit]
|
|---|
| 190 | Description=Testbox Maintenance
|
|---|
| 191 | After=network.target
|
|---|
| 192 | [email protected]
|
|---|
| 193 |
|
|---|
| 194 | [Service]
|
|---|
| 195 | Type=oneshot
|
|---|
| 196 | RemainAfterExit=True
|
|---|
| 197 | ExecStart=/root/scripts/testbox-maintenance.sh
|
|---|
| 198 | ExecStartPre=/bin/echo -e \033%G
|
|---|
| 199 | ExecReload=/bin/kill -HUP $MAINPID
|
|---|
| 200 | WorkingDirectory=/tmp
|
|---|
| 201 | Environment=TERM=xterm
|
|---|
| 202 | StandardOutput=journal+console
|
|---|
| 203 |
|
|---|
| 204 | [Install]
|
|---|
| 205 | WantedBy=multi-user.target
|
|---|
| 206 |
|
|---|
| 207 | - Mount the testbox-nfsroot under ``/mnt/`` with write privileges. (The write
|
|---|
| 208 | privileges are temporary - don't forget to remove them later on.)::
|
|---|
| 209 |
|
|---|
| 210 | mount -t nfs myserver.com:/exports/testbox-nfsroot
|
|---|
| 211 |
|
|---|
| 212 | - Copy the debian root and dev file system onto nfsroot. If you have ssh
|
|---|
| 213 | access to the NFS server, the quickest way to do it is to use ``tar``::
|
|---|
| 214 |
|
|---|
| 215 | tar -cz --one-file-system -f /mnt/testbox-maintenance-nfsroot.tar.gz . dev/
|
|---|
| 216 |
|
|---|
| 217 | An alternative is ``cp -ax . /mnt/. && cp -ax dev/. /mnt/dev/.`` but this
|
|---|
| 218 | is quite a bit slower, obviously.
|
|---|
| 219 |
|
|---|
| 220 | - chroot into the nfsroot: ``chroot /mnt/``
|
|---|
| 221 |
|
|---|
| 222 | - Enable our service: ``systemctl enable testbox-maintenance.service``
|
|---|
| 223 |
|
|---|
| 224 | - Edit ``/etc/initramfs-tools/initramfs.conf`` and change the ``MODULES``
|
|---|
| 225 | value from ``most`` to ``netboot``.
|
|---|
| 226 |
|
|---|
| 227 | - Append ``aufs`` to ``/etc/initramfs-tools/modules``. The advanced
|
|---|
| 228 | multi-layered unification filesystem (aufs) enables us to use a
|
|---|
| 229 | read-only NFS root. [3]_ [4]_ [5]_
|
|---|
| 230 |
|
|---|
| 231 | - Create ``/etc/initramfs-tools/scripts/init-bottom/00_aufs_init`` as
|
|---|
| 232 | an executable file with the following content::
|
|---|
| 233 |
|
|---|
| 234 | # Don't run during update-initramfs:
|
|---|
| 235 | case "$1" in
|
|---|
| 236 | prereqs)
|
|---|
| 237 | exit 0;
|
|---|
| 238 | ;;
|
|---|
| 239 | esac
|
|---|
| 240 |
|
|---|
| 241 | modprobe aufs
|
|---|
| 242 | mkdir -p /ro /rw /aufs
|
|---|
| 243 | mount -t tmpfs tmpfs /rw -o noatime,mode=0755
|
|---|
| 244 | mount --move $rootmnt /ro
|
|---|
| 245 | mount -t aufs aufs /aufs -o noatime,dirs=/rw:/ro=ro
|
|---|
| 246 | mkdir -p /aufs/rw /aufs/ro
|
|---|
| 247 | mount --move /ro /aufs/ro
|
|---|
| 248 | mount --move /rw /aufs/rw
|
|---|
| 249 | mount --move /aufs /root
|
|---|
| 250 | exit 0
|
|---|
| 251 |
|
|---|
| 252 | - Update the init ramdisk: ``update-initramfs -u -k all``
|
|---|
| 253 |
|
|---|
| 254 | - Either copy the initrd.img and corresponding kernel to the
|
|---|
| 255 | ``testbox-tftp/maintenance-boot/`` directory, or create
|
|---|
| 256 | ``testbox-tftp/maintenance-boot`` as a symbolic link to
|
|---|
| 257 | ``testbox-nfsroot/boot/``.
|
|---|
| 258 |
|
|---|
| 259 | - Recreate ``/etc/fstab`` with::
|
|---|
| 260 |
|
|---|
| 261 | proc /proc proc defaults 0 0
|
|---|
| 262 | /dev/nfs / nfs defaults 1 1
|
|---|
| 263 | # Mount tftp and backup too.
|
|---|
| 264 |
|
|---|
| 265 | - xxx
|
|---|
| 266 |
|
|---|
| 267 |
|
|---|
| 268 | - Testing the setup from a VM is kind of useful:
|
|---|
| 269 | - Create a VM using the 64-bit debian profile. Let's call it "pxe-vm".
|
|---|
| 270 | - Mount the TFTP share somewhere, like M: or /mnt/testbox-tftp.
|
|---|
| 271 | - Reconfigure the NAT DHCP and TFTP bits::
|
|---|
| 272 |
|
|---|
| 273 | VBoxManage setextradata pxe-vm VBoxInternal/PDM/DriverTransformations/pxe/AboveDriver NAT
|
|---|
| 274 | VBoxManage setextradata pxe-vm VBoxInternal/PDM/DriverTransformations/pxe/Action mergeconfig
|
|---|
| 275 | VBoxManage setextradata pxe-vm VBoxInternal/PDM/DriverTransformations/pxe/Config/TFTPPrefix M:/
|
|---|
| 276 | VBoxManage setextradata pxe-vm VBoxInternal/PDM/DriverTransformations/pxe/Config/BootFile pxelinux.0
|
|---|
| 277 |
|
|---|
| 278 | - Create the file ``testbox-tftp/pxelinux.cfg/0A00020F`` containing::
|
|---|
| 279 |
|
|---|
| 280 | PATH bios
|
|---|
| 281 | DEFAULT maintenance
|
|---|
| 282 | LABEL maintenance
|
|---|
| 283 | MENU LABEL Maintenance (NFS)
|
|---|
| 284 | KERNEL maintenance-boot/vmlinuz-3.16.0-4-amd64
|
|---|
| 285 | APPEND initrd=maintenance-boot/initrd.img-3.16.0-4-amd64 ro ip=dhcp aufs=tmpfs \
|
|---|
| 286 | boot=nfs root=/dev/nfs nfsroot=10.42.1.1:/exports/testbox-nfsroot
|
|---|
| 287 | LABEL local-boot
|
|---|
| 288 | LOCALBOOT
|
|---|
| 289 |
|
|---|
| 290 |
|
|---|
| 291 | -----
|
|---|
| 292 |
|
|---|
| 293 | .. [1] See http://www.syslinux.org/wiki/index.php?title=Category:Modules
|
|---|
| 294 | .. [2] See http://www.syslinux.org/wiki/index.php?title=PXELINUX#Configuration
|
|---|
| 295 | .. [3] See https://en.wikipedia.org/wiki/Aufs
|
|---|
| 296 | .. [4] See http://shitwefoundout.com/wiki/Diskless_ubuntu
|
|---|
| 297 | .. [5] See http://debianaddict.com/2012/06/19/diskless-debian-linux-booting-via-dhcppxenfstftp/
|
|---|
| 298 |
|
|---|
| 299 |
|
|---|
| 300 | -----
|
|---|
| 301 |
|
|---|
| 302 | :Status: $Id: TestBoxImaging.txt 64498 2016-11-01 04:39:14Z vboxsync $
|
|---|
| 303 | :Copyright: Copyright (C) 2010-2016 Oracle Corporation.
|
|---|
| 304 |
|
|---|
| 305 |
|
|---|
