First Install Stage:
Booting the Kernel
Starting Up the System
Getting Ready to Install
Partitioning and Formatting Disks
Installing the Basic System
Final Touches before Reboot
Second Install Stage:
Starting up the System
Configuring the System
Installing Packages
Finalising the System
Fixing Up the First Stage:
Adding Preseeding
Saving Existing Data
Copying away and restoring local user data becomes impractical (time and space wise) when it becomes 100s or even 1000s of GBytes per machine, and involves over 20 such large machines and nearly 150 normal machines with small amounts of data. Such re-install may become neccessary, in short time and at short notice, such as when tidying up after an system/site has become r00ted, as happened once to our then about 100 machine site in the Debian 2.2 (potato) days.
Debians installation process goes through two basic stages, separated by an reboot. The first runs with the install medias root floppy image (unpacked into RAM disk) as /, the second with the system being installed (called the target) as /.
(Note: It is possible to run just the actual installation of packages, the program /usr/sbin/debootstrap, like this. See the Details in the Debian Installation Guide. But that is not an complete automatic (re-)install, because no partitioning/formatting/mounting is done that way).
For Debian/i386 (which is what we use here) the PC BIOS expects to find an floppy disk, or an floppy image on an El-Torito compatible CD, or an kernel file fetched by PXE. From these it will simply read the first disk sector, check if the last 2 bytes are 0x55 and 0xAA, and if so execute the code in the sector from byte 0 on.
Debians boot.img floppy image (which is what I dissected, in its 1440k variant) contains an VFAT (not just FAT12!) filesystem, and an Syslinux boot loader. Syslinux is an FAT-aware program, which is controlled by an config file (/syslinux.cfg) which in this case calls for loading an kernel file (/linux, 764502 Bytes) and an compressed initrd image file (/initrd.gz, 658413 Bytes). The kernel will then unpack itself, then unpack the initrd, and then place the later into an RAM disk (/dev/rd*), and mount that as /.
This /initrd.gz expands to 1882112 Bytes and contains an ext2 file system, containing an *very* minimal Linux system, of which 534k are libc.so.6 and 724k are modules (649k of these are drivers) and 320k empty space. (No, I don't know why the initrd is such a funny size, as it is not exactly filled anyway.) The /sbin/init of this system is a sh script (run by busybox sh), which only loads various driver modules (for USB keyboards and various storage media). It then prompts the user for disk change to the root floppy (or CD or USB stick).
At this prompt one can also demand an shell, which I have used while this discovery process. Of course one only gets an extremely primitive system that give minimalism an enterely new meaning. There ist not even ls available, you list files with echo *.
Note, that the installation kernel is compiled to use devfs, unlike the production kernels installed and used on the target system. All scripts therefore have devfs style paths in them. Names such as /dev/fd0 become /dev/floppy/0, and both /dev/hda* and /dev/sda* become /dev/discs/disc0/* (I guess this is why they used devfs :-)). But there are also separate /dev/ide/host0/bus0/target0/lun0/* and /dev/scsi/host1/bus0/target0/lun0/* available.
After the user has inserted an root.img floppy and hit enter, the script reads it into an separate second RAM disk, and then uses /sbin/pivot_root (which is part of /bin/busybox) to change / to that, and then runs the second /sbin/init on then, so simulating the kernels boot behavior, as if the original floppy had already contained this larger initrd image.
It it were not for loading modules, one could reconfigure Syslinux to omit the first initrd alltogether. So I assume that the boot.img initrd was only added as a way of reducing the amount of modules on the root initrd, and possibly also for more flexible media choice (and the modules for that) on systems which can not boot from CD or PXE or USB.
This /initrd.gz expands to 5283840 Bytes and contains also an ext2 file system, containing an less minimal Linux system, of which 667k are an new libc.so.6 (the old one will be gone after pivoting root!) and 669k further modules (463k of these network drivers) and 1737k empty space.
This file system contains also an small /sbin/init script to copy to an larger ramdisk (this script is deleted and then bypassed by the boot.imgs /sbin/init script, as it already generates an large ramdisk for loading into). So work continues in the end with /bin/busybox init being called directly.
/bin/busybox init reads an minimal /etc/inittab which runs on the first virtual console exactly two programs. The first one only once, the second one repeated so long until it reboots the system (into the second install stage):
::sysinit:/sbin/debian-installer-startup
::respawn:/sbin/debian-installer
There is also an convenience shell on the 2nd console, which I have often used while debugging my installer modifications and subsequent crashes. Thanks for it! This shell runs directly, without getty or login, and so without entering any password. There is after all no surviving user data to protect during installation, and the BIOS password should prevent users from deliberately booting from such an password-less floppy.
Both /sbin/debian-installer-startup and /sbin/debian-installer are shell scripts, like nearly all of the installer. This makes the installer compact on disk space and easy to modify in place (and also to analyse). There is no need for the old-style (up until Debian 3.0 (woody)) "dbootstrap" installer, with its boot-floppies package full of C code. Nor of modifying that and recompiling it. Or rather replacing it with our own install script (as we did here for our dphys2 autoinstaller (for Debian 3.0 (woody)).
Both of these scripts are pure callers of init.d style script parts, without any own actions (apart from setting one variable, $MENU). This makes the installer easily expandible, by just inserting more numbered scripts. They each run one of these script parts sets:
/lib/debian-installer-startup.d/S[0-9][0-9]*
/lib/debian-installer.d/S[0-9][0-9]*
/lib/debian-installer-startup.d/S[0-9][0-9]* just sets up an really basic Linux system, ending in S55kbd-chooser, which switches the keyboard to Unicode and loads an keyboard layout, if this was demanded from the kernel command line.
/usr/bin/debconf is the program that draws all the dialog boxes that the user will see. It forks an child process which then runs /usr/bin/main-menu. It expects its child (and thats children) to send dialog requests via their stdout, and delivers answers to them via their stdin. This is basically an display/menu server, its child being the client, and ASCII over stdout/stdin providing the IPC mechanism! The actual dialog layouts are not sent by the client, but rather taken from template files (which I did not bother searching for, so no filenames here).
Because of all this you can not use stdout to output anything to the screen. So you must use something like echo > /dev/tty if you want to, as I did to display an fatal error message, before aborting the re-install, if failling an sanity test, if the system being reinstalled was an previous install by our system. One does not part-re-install onto an system of unknown origin, that could easily result in an garbled system!
/usr/bin/main-menu controls the installation process, it does not actually install anything itself. That is left to individual scripts called by it. These are in the form of /var/lib/dpkg/info/*.postinst scripts, which are called by doing udpkg --configure <package>.postinst for each one! BTW, udpkg stands for micro dpkg, an small installer-only clone of dpkg. These *.postinst scripts are the result of all the files for the entire install process being packaged as an set of *.udeb (micro Debian) packages, which have then been installed (unpacked) into the installer image, but not yet configured. Faillure of such an script (exit != 0) leads to /usr/bin/main-menu falling back to displaying an list of all install steps, for manual control.
Which of these *.postinst scripts are called, and in what row, is
something I have not managed to figure out entirely. Using strings
/usr/bin/main-menu does not show an built in list, but access to
/var/lib/dpkg/status. That lists for some of the *.udeb
packages an installer-menu-item: [0-9][0-9] line (yes, that starts
with an "i", not an "I" like all other fields), which sorted by the
[0-9][0-9] numbers gives the correct amount, selection and row of
operations. This amounts to an second init.d style mechanism, but
more inetd like. But removing such lines fails to produce the
expected effect of omitting that script, so there must be something else
active somewhere, which I have missed. [Since writing this I have been
informed that
The scripts languagechooser.postinst ([0-9][0-9] =
10), countrychooser.postinst (11) and
kbd-chooser.postinst (12) question the user the obvious
stuff and set it.
(My many "thanks" to the braindead IBM PC architecture, where keyboards,
despite having an full 8bit microcontroller inside them since over 25
years, not only still don't know what keys are on them (let allone where),
but also not even know which layout table is needed for them. This requires
not only all operating systems to know all existing keyboard layouts, but
also requires every user to select which keyboard they have on every system
they install, not to mention changing it each time they swap keyboard. This
brokenness was copied without improvement to USB HID keyboards, and so has
also infested the Mac world, Apple simply includes on its MacOSX install
disks an file mapping keyboard serial numbers to layout type, which hides
the brokennes so long you use an genuine Apple keyboard. End of rant.)
Then ethdetect.postinst (15) and netcfg.postinst
(18) get network access ready, before
chose-mirror.postinst (23) selects where the user wants
to fetch stuff from. Then the actual show can begin!
download-installer.postinst (also 23, but the name sorts
it later in the alphabet) runs /usr/bin/anna net-retriever. That
anna stands for "Anna is Not Nearly Apt", and it sets up (not
gets!!!) the net-retriever package. Note that this package is
already present and unpacked on the root.img, as the
download-installer package has an Depends: on it. So
anna is really not much more than an script finder/caller, which
in the end runs /usr/lib/debian-installer/retriever/net-retriever.
Then net-retriever at long last goes fetching stuff - not yet the
actual real install, but at least the rest of the installation system,
which would not fit on an single root floppy image! For this it uses an
list of *.udeb packages, which it retreives from
dists/sarge/main/debian-installer/binary-i386/Packages of the
used Debian mirror. So what gets installed here ist determined by the
Debian mirror, and not the install floppys. The actual packages are then
fetched, from pool/main/?/*/*.udeb. After this the install system
has grown to about 15M, about 4-5 times its original 3.5M size.
As all the retrieved stuff consists of *.udeb packages which have
now been unpacked, the collection of *.postinst files has grown by
unpacking them, and /var/lib/dpkg/status lists far more packages
with installer-menu-item: lines. And so the installer knows what
additional stuff it now has to run, and in what row. If any step from now
on fails, then the menu displayed will also now be longer.
After further hardware detection by hw-detect-full.postinst
(35) the system is ready for partitioning and formatting and
mounting and generating /etc/fstab, all of which are done by
partman.postinst (42). That just calls
/bin/partman which together with its large support infrastructure
is part of one of the the *.udeb packages downloaded.
The choice of 23 for download-installer.postinst and
42 for partman.postinst, as two of the most important
parts looks very deliberate. But why does the actual installer then get
65 instead of 69 :-).
I went through this in quite fine detail, as preventing re-formatting
was the whole reason for digging through the system. I had hoped to be
able to fake an user entering the current partitioning, and then skip the
formatting. But this turned out to be too much work, so in the end I simply
opted for preventing the call of /bin/partman and running my own
script to not only clear out the old system (this needed doing anyway), but
also duplicate the wanted work (mounting, /etc/fstab).
/bin/partman is just a shell script, which runs multiple sets of
init.d style script part sets. Yes, that is, after
/usr/lib/debian-installer.d/S[0-9][0-9]* and
installer-menu-item: [0-9][0-9] the third nested level of script
part sets. Together with the actual scripts the calling stuff and
/etc/inittab we are now at about 7th to 9th indirection level!
First to be run are /lib/partman/init.d/[0-9][0-9]*, which set up
resources for partman. Central to these is 30parted which starts
the actual partition table editor. One would have expected an shell script
based installer to dump the partition table to ASCII, modify that using
sed and its friends, and then write it back to binary. But this is
not the case. What debian-installer actually does, is have an single
program /bin/parted_server which is forked, and does all
read/edit/writing of the table, with the shell stuff as client(s) poking
at it.
IPC is done here by three FIFOs, set up by 30parted before
forking. Yes, we are now in an set of shell scripts with an stdout/stdin
IPC relationship to an display/menu server and an second three-FIFO IPC
relationship to an editing server. That is a 3-tier client-server system,
partially written in shell! Access to the FIFOs from the scripts is done by
functions in /lib/partman/definitions.sh, which
open/read-and-write/close file handles 6 and 7 for this. If your head is
spinning by now, that is understandable. If it breaks you get to keep the
pieces. If you ever wanted to know why the "starting up partitioner"
message stays on screen so long, now you know it: lots of scripts being
run.
The last step /lib/partman/init.d/99initial_auto then displays the
first partitioning dialog, before falling back to /bin/partman for
reading and acting on user input. That then uses an function
ask_user (also in /lib/partman/definitions.sh), which
uses plugins (that is the official variable name used for their paths!) in
/lib/partman/choose_partition/[0-9][0-9]*/choices and
.../do_option to navigate the users partitioning and MD-ing and
LVM-ing and formatting actions.
Multiple of these then use an fourth(!) level of script part sets to handle
different filesystem types (ext2/ext3/reiser/xfs/jfs), such as
/lib/partman/fstab.d/*. No, I am not going into further details of
this. Don't panic.
Once the user is through with selections, two more script sets are called,
first /lib/partman/commit.d/[0-9][0-9]* and then
/lib/partman/finish.d/[0-9][0-9]*. This is the reason why there
are so many confirmation dialogs that appear after the end of partitioning.
After all this preparation the actual work can begin. This job is handled
by base-installer.postinst (65), surprisingly directly
from that script, no call to an /bin/<something> program.
This one of course also uses an set of routines called by list. But this
time all the routines are not numbered files, but are implemented as shell
functions, within the base-installer.postinst script itself! Yet
annother mechanism. Their names are stored blank separated into an variable
$WAYPOINTS and then run by an loop. This is definitely the most
strange part of the installer. It looks like someone else wrote it.
The main routines are install_base_system () and
install_kernel ().
install_base_system () first runs an standard script set
/usr/lib/base-installer.d/* which just copies the network config
to the harddisk. Then it calls /usr/sbin/run-debootstrap (an
binary!) which then calls the script /usr/sbin/debootstrap. I have
not bothered to find out for what this binary is needed. [Since writing
this I have been informed that /usr/sbin/run-debootstrap displays
and updates the progress bar, driven by the parsable output of
/usr/sbin/debootstrap.]
/usr/sbin/debootstrap is the actual software install script, which
is also mentioned in the Debian Installation Guide for people doing direct
installs from an running Linux system onto an new (empty, but already
formated) partition (see the Note: in the Booting the
Kernel section for details). This is also the first installer component
that has the same name and function as its woody counterpart! It also has
in /usr/lib/debootstrap/scripts all the support files for woody
and sid, not just the sarge ones!
The /usr/lib/debootstrap/scripts/sarge script contains in its
function work_out_debs () the list of packages that will be
installed, in the variables $required (58 packages) and
$base (annother 61 packages). Surprisingly this list is not
fetched from the Debian mirror, like the installer extensions. Most likely
this is an holdover from reusing the old woody installer. Despite being
more than half of the first stage install time, due to downloading and
unpacking packages, and despite being the central action the users wants,
this part is actually quite simple, and so only has an small description
here.
install_kernel does the obvious thing, after pick_kernel
() has chosen which one is wanted, from the list provided by
apt-cache search kernel-image.
After that the installer still does a bit of tidying up work. This is
driven by prebaseconfig.postinst (78) which simply calls
all of the /usr/lib/prebaseconfig.d/[0-9][0-9]* script parts,
mainly to store information needed by base-config to continue
installation after reboot.
The last of these scripts is 99reboot which then calls
/lib/debian-installer/exit to umount and swapoff and then reboot
into the second install stage.
Booting is now from the targets harddisk. The BIOS runs the first sectors
code (on harddisks this is the MBR code), which will call GRUB or LILO
(whichever got installed) which will fetch in the production kernel. This
is different from the install kernel, in that it is compiled for various
processor variants, and does not use devfs. This will mount the desired
/ read-only and run /sbin/init (now the full size one,
not busybox init).
To continue the installation /sbin/init has been given an
specially prepared /etc/inittab file, taken from
/usr/share/prebaseconfig/inittab of the install system. The proper
/etc/inittab has been installed to /etc/inittab.real for
later activating, when the 2nd install stage is through.
/etc/inittab runs the standard /etc/init.d/rcS and
/etc/init.d/rc 2 scripts, as far as they have already been
installed by actions up to now. Then it runs on the first virtual console
/usr/sbin/base-config, which continues with the rest of the
actual installation.
This is why it is possible to complete installing without having to do an
root login. Only after replacing /etc/inittab does the first
virtual console get an entry with /sbin/getty in it, and then asks
for an user name and then runs /bin/login which asks for the
password.
There are also two convenience shells (with /sbin/getty on the 2nd
and 3rd consoles. These already have full getty and login
on them, so they require proper login, as root).
First the exception, it hardcoded imports the installers debconf database
which was saved by /usr/lib/prebaseconfig.d/93save-install-log
into the target systems debconf database. Why this can not be done by an
called script is not obvious. [Since writing this I have been informed
that /usr/lib/base-config/menu/* may be influenced by debconf
settings, so they need to be merged in before starting it.]
Then it runs all the scripts in /usr/lib/base-config/menu/*, with
all the *.mnu files defining what scripts are to be run, and in
what row (given by the Order: [0-9][0-9] lines in the .mnu Files).
It also determines whether to run the scripts inside debconf (using the
stdout/stdin controls) or outside (re-opening stdout/stdin to point to
/dev/tty) of it (Debconf: true|false lines).
The script debconf-seed ([0-9][0-9] = 5) then
loads all the preseeded values for the 2nd (and also the 1st!) stage from
an temporary file (/var/log/debian-installer/debconf-seed). This
script is always present and run, even if no preseeding was used.
keyboard (20), timezone (30) and
passwd (40) then set up the obvious stuff.
tasksel also shows an list of "tasks" (large-ish groups of
packages commonly used by somewhat standard groups of users) which the user
can select to install. Despite that, it also installs an default set of
packages, even if the users leaves all tasks not selected! The result seems
to be installation of the same stuff which would have been installed, if
the task "Development" had been selected. These default packages are
responsible for the size of an default Debian 3.1 (sarge) installation
becoming about 300M, where Debian 3.0 (woody) was only 110M, because it
default installed no tasks.
Despite this step being more than half of the entire install time, due to
downloading and unpacking of many packages, and being part of the central
action, this part is also actually quite simple, and so has an small
description here.
As last the script finish (120) is called. It will get
the system ready for user to work on it. This includes a few steps:
Run any preseeded command for further user specific installations (we run
this script to install all of our desired
configurations and packages). Preseeding is surprisingly not an separate
script.
Ask if to activate an display manager, if one was installed. But this uses
an hardcoded list (xdm gdm kdm) of known ones, and so fails for any other
one (such as the wdm which we use).
As the last step /etc/inittab.real is put into place as
/etc/inittab and /sbin/init is reloaded, so that as the
next thing the user will get an login prompt. Then as the very last step
the screen is cleared, to give an tidy looking system (and wiping out any
error messages from preseed scripts, frustrating admins who are testing an
new one :-)).
Happy login and linuxing on your fresh Debian system!
Of course by this time we had designed an entire automatic site
setup/deployment system expecting to use this mechanism, so I wanted it
now. Thankfully the entire installer ist put together from *.udeb
packages, and the missing preseeding support also comes in that format. The
only thing I needed were the two packages initrd-preseed and
preseed-common, to be had from any Debian sid mirror. Simply
unpacking/uncompressing the root.img initrd, chroot-ing to it and
installing the two packages using the installers udpkg tool, and
then recompressing/repacking the initrd was all that was needed. Full marks
to DEbian for designing the entire system so modular, as many shell
scripts, and managing the modules as packages.
As side result of using these 2 sid packages, we not only got preseeding,
but also got the newer and better initrd preseeding, which can set
everything, including early stuff that can not be preseeded by any official
sarge preseeder included in other install media. This got rid of trouble
from the early stuff we wanted to set not fitting onto the kernel command
line, which is limited to setting 8 options and 8 variables.
As result of installing these, a few new things appeared:
The file /lib/debian-installer-startup.d/S35initrd-preseed gets
added, for processing by
::sysinit:/sbin/debian-installer-startup (see Starting Up the System section). It reads in an
/preseed.cfg file, which I added to the initrd image at the same
time as installing the two *.udeb files. It processes that using
the function preseed_location () in the also added
/lib/preseed/preseed.sh library. Gone are all questions after the
prompt for the root disk (and with PXE boot, pure kernel and root image,
even that one is gone too).
Also added is /usr/lib/prebaseconfig.d/05preseed for processing by
prebaseconfig.postinst (see Final Touches
section), which is responsible for executing whatever command has been set
with the preseed/late_command line of the preseeding file. This
allows user/site specific commands to be run just before reboot.
We here add installation of our custom kernels, which are required for
having up to date drivers (Debian standard kernels suck in this respect,
see the trouble with server hardware, but also
the trouble with newer graphics cards and old XFree). And security hole
bugfixes are also often slow. As an side effect of this, the 2nd stage
reboot already activates our kernels, saving an further reboot later.
This was just about possible in an planned upgrade for the sarge rollout,
done over multiple weeks. But what in the case of an fast re-install after
an security breach? So we quickly had the desire to not re-partition and
re-formatt on re-installs of systems. Once again patching up the installer
was wanted.
After analysing the operation of partman (see above) I decided to disable
it. Simply removing the installer-menu-item: 42 line from
/var/lib/dpkg/status, so that /usr/bin/main-menu drops
executing it (see Getting Ready to Install section),
failed to work. So now the partman.postinst script simply gets
overwritten with an exit command.
Unfortunately partman.postinst is part of the *.udeb
stuff that is only downloaded at install time, not part of the
root.img floppy. So it can not be modified beforehand. The answer
was to modify download-installer.postinst to do the modifications
to partman.postinst, after doing all its downloading stuff.
After this the actions left by partman have to be duplicated, but leaving
our data intact. This turned out to need: mounting, deleting the old
system, restoring the old /etc/fstab. All these replacement
actions were added directly to download-installer.postinst, saving
us from having to have that add them to partman.postinst just for
them to be then run from there.
This page is by Neil Franklin, last modification 2006.11.17
Partitioning and Formatting Disks
Partman is a complex beast, and one of the two largest actions, so it
warrants its own section in this text. Its job is to partition and format
and mount the targets storage space, and generate the fitting
/etc/fstab which will allow later re-mounting of it.
Installing the Basic System
At long last we are actually ready to install something onto the target
systems harddisk!
Final Touches before Reboot
After partitioning and installing we have an near-complete runnable system.
So it now just requires
grub-installer.postinst (73) or
lilo-installer.postinst (75) or
nobootloader.postinst (77) to ensure that the kernel is
bootable.
Second Install Stage:
Starting up the System
Now the target system is installed so far enough, that it is capable of
standing on its own feet and working on itself. So the install image system
is not needed any more. Therefore the computer is rebooted into the second
stage, where it will then continue to install and configure all the rest.
Configuring the System
/usr/sbin/base-config is responsible for the entire rest of the
installation. Like /usr/bin/main-menu it calls various other
scripts to do the actual work (with one execption). It also uses debconf
as its display/menu server.
Installing packages
Thes the main work is then done by 2 scripts: apt-setup
(70) sets up /etc/apt/sources.list. And then finally
pkgsel (80) gets to install the rest of the system. It
mainly just runs tasksel -ris --new-install, which installs all
packages which are marked with required, important and
standard priority. So what gets installed here ist once again
determined by the Debian mirror chosen, by the
dists/sarge/main/binary-i386/Packages file.
Finalising the System
After installing all the packages the installation is nearly finished.
The script mta (100) is called, to set up the mailer
(exim). Despite being part of the base installation it does not use
Debconf, but rather does its own prompting. This was already bad in woody,
and is absolutly broken in sarge. Particularly as this is an basic
installation package, and just asking for an single number to select which
config file to use.
Fixing Up the First Stage:
Adding Preseeding
According to the Debian Installation
Guide it is possible to run the debian-installer installer
automatically (= without manually filling out all of the debconf dialog
boxes), by providing the data that would be entered to debconf in form of
an preseed
file. What the Guide does not say, is that this does not work in the
floppy version of debian-installer, because the needed code is missing.
It needed an bug Report (#299059)
to get this piece of info. Fixing this on the install floppies is a
post-sarge feature.
Saving Existing Data
When upgrading our over 150 machines from woody to sarge it became clear
that the logistics of copying all local temporary, non backuped, may be
lost at any time (that is the official designation!) user data off of
them was a lot of work. And in the case of some machines this local stuff
had become official data which must be preserved (and is backuped). Running
an forced backup and then restore of 20 such machines would take too long.
On the rest of our machines it had also often become an issue of user
frustration to get them to save and restore it.
Home |
Projects |
dphys3 simple automated installer |
The Debian 3.1 (sarge) debian-installer (d-i) Installation Process