http://www.phys.ethz.ch/~franklin/Projects/dphys-kernel-packages/Logfile author Neil Franklin, last modification see last entry near bottom copyright ETH Zuerich Physics Departement use under either modified/non-advertising BSD or GPL license background is dphys-admin and dphys-config projects, read their Logfiles at: http://www.phys.ethz.ch/~franklin/Projects/dphys-admin/Logfile http://www.phys.ethz.ch/~franklin/Projects/dphys-config/Logfile as part of that wrote this tool to generate host optimised custom kernels here only log of the events pertaining to making this tool 2005.07.01 Fri heeb discussion about making own kernels, existing, own, make-kpkg, ... have each host compile for its processor/SMP/RAM/etc, are fast enough gets rid of central compiling orgy and large number of packages ideal with dphys-config kernel source and .config, local optimised compiling .config large amount of site stuff, and .hostpost a few settings dphys-config be able to handle large files (such as kernel), only fetch once to get rid of devfs drop compile option, lilo conv, inittab conv, fstab conv but preferably like with dphys-admin|config|etc usable for woody and sarge 2005.07.07 Thu looking at what files are needed, not just source and .config, also patches existing package build is on arizona in /scratch/dphys-build/dphys-kernelgen analyse the stuff there, how existing build process works, what we need README what is to be done, at stand of 2 years ago includes symlinking -dphys-install into each modules .tar.gz make clean, debclean, tidy are only various rm stuffs make kernels (includes make deb) does ./makekernels and ./makedebs ./makekernels does: tar xzf, make mrproper, cp config, Makefile extra vers make oldconfig, make dep, make -j 4 bzImage, install in binary/... make -j 4 modules, make modules_install for all modules do: tar xzf, ./dphys-install (inserted into module) ./makedebs does: ./makepackage -g, cp binary/..., cat debian/postinst add Conflicts: and Provides:, ./makepackage -tp make kernelconfig does ./makeconfig ./makeconfig does: tar xzf, make mrproper, cp config, Makefile extra vers make menuconfig, cp config back out ./src-kernel is the kernel used, what we need to now distribute ./src-modules are the patches to add, and ./src-modules/scripts do this ./types/new/clean is base config, to be modified for each kernel type ./types/* are all the modified/adapted configs, for each kernel type but don't seem to be derived from ./types/new/clean, too large diff sorting also does not solve the problem, seems to simply be way older designing new system, possibly better full config, changes for newer kernels and host-type specific edit scripts which do not change wenk discussion, regards central compiling as more reliable than on hosts as some overloaded hosts may have too little RAM and even too little swap 2005.07.08 Fri designing new system, for different variants, use one complete base config and then modify for the other variants, so base can be pure make oldconfig compare todays variants, with base = p1-1gb, ignoring added # lines p3-1gb: -CONFIG_M586TSC +CONFIG_MPENTIUMIII -CONFIG_X86_USE_STRING_486 -CONFIG_X86_USE_STRING_486 -CONFIG_X86_PPRO_FENCE +CONFIG_X86_GOOD_APIC +CONFIG_X86_PGE +CONFIG_X86_USE_PPRO_CHECKSUM +CONFIG_X86_F00F_WORKS_OK p4-1gb: same but +CONFIG_MPENTIUM4 and CONFIG_X86_L1_CACHE_SHIFT 5->7 k7-1gb: same but +CONFIG_MK7 and CONFIG_X86_L1_CACHE_SHIFT 5->6 +CONFIG_X86_USE_3DNOW *-* -> *-smp-*: +CONFIG_SMP +CONFIG_NR_CPUS=32 +CONFIG_HAVE_DEC_LOCK +CONFIG_X86_IO_APIC +CONFIG_X86_LOCAL_APIC *-1gb -> *-4gb: -CONFIG_NOHIGHMEM +CONFIG_HIGHMEM4G +CONFIG_HIGHMEM *-4gb -> *-64gb: -CONFIG_HIGHMEM4G +CONFIG_HIGHMEM64G +CONFIG_X86_PAE so modifying one p1-1gb base to make all others seems to be no problem decided to go for script running on one host, producing all packages there started reworking begun script to fit into this szenario, looping stuff done all sed .config patching up, done Makefile patching up 2005.07.14 Thu rename to dphys-kernel-packages looked at what Debians make-kpkg does, whether it is usefull for this project has already made postinst that can handle lilo and grub do an test compile, with config-p1-1gb, simple make-kpkg, no parameters just compiles the kernel and modules, no packaging running with buildpackage produces Debian-like kernel-image*.deb and kernel-source*.deb, kernel_headers*.deb and kernel_doc*.deb but no kernel-image*.orig.tar.gz proper source package for apt-get source and also no .dsc file for makesourcepackage to use for -ui seems to be an older system, pre apt-get source, do we still want this? but is less work, no making own postinst, is worth it, so use this extra modules, running with modules or modules_image makes packages for these need to make an debian/rules (not debian.rules as man page has it) for example alsa convert existing *-install script to an rules Makefile 2005.07.15 Fri modules tested script, fails to install into proper place, no $(DESTDIR) set target kdist in docs is also wrong, should be kdist_image, docs from 1997 :-( no docs, go and look at debians openafs modules, how their rules file is openafs too complicated, get alsa modules, far simpler requires apparently an full debianisation of modules, rules+control+... with all this in place we actually get an .deb, without contents, just info but install target (and build) is being called, and no error messages make fails to assign variable, if it gets set inside recipy, must be global 2005.08.05 Fri rerun make-kpkg, refreshed what it does, corrected recordings above test run, compiles kernel and makes korrect packages compiles alsa but makes package with only debian/* stuff, no actual stuff nothing from upstream installed into debian/tmp/ but ist installed into debian/, so tmp missing somewhere tried with $(DESTDIR) including tmp, now only debian/* in tmp most likely went and cleared other stuff, but examples have no tmp anyway heeb thinks it is collision of woody tools and sarge style code so try compiling on an sarge system, moved over to dumbledore, same result dh_* think stuff goes in tmp instead of , try DH_COMPAT=3, works 2005.08.11 Thu found some make-kpkg docs, read them, reminded that --initrd needed also will need --revision= to avoid the ugly 10.00.Custom name we will also want --append-to-version to get different /lib/modules//* run make-kpkg clear first, also look at kernel-pkg.conf for settings but also no docs on actually debianising modules further with dphys-kernel-packages script, running make-kpkg, managing modules debianising modules (control|copyright|changelog|rules) 2005.08.12 Fri further with dphys-kernel-packages debianising (rules) test shows version missmatch (with/without extraversion), add all options now complains about existing version, later aborts with Permission denied man says that --append-to-version needs repeat, and setting VERSION_H_OK basically works, but still permission denied in ./configure MODULE_LOC empty because wrong var name, now just debian/rules missing 755 gets to dh_testroot, complains not root, despite --rootcmd fakeroot use direct fakeroot, not option, works, use it so in all places tidy up code, add CONF_ options for setting debianising details fix missing append in module names, so diff kernels diff moduel names varying debianisation for diff modules, seems only debian/rules needs change control and packaging stay same, upstream config/build/install can vary insert in debian/rules an [configure|build|install|clean]-module section 2005.08.18 Thu fixed bug with including debian/rules fragments, now works fully made man page, makesourcepackage -g, Makefile, config example, readme install, faq, index.html.en and .de symlink, .tar, debianisation control 2005.08.19 Fri fetched all other 4 external modules and produced debian/rules fragments 3w-9xxx had Makefile modified for old system, failed as no ${KERNEL} originally no Makefile, modified Makefile.oth, now script sed Makefile.oth Nvidia compiling(!) fails to deinstall(!) existing kernels module as non root switch to different make target, so that only compile and later install Nvidia has _ in directory name, lands in package name, not valid improve package name generation, from .tar.* filename, eliminate _ diferentiate names for unpacked dir, rules fragment base and package name also eliminate upper case alphanumerics, Nvidia manages to have them also added CONF_TARGETDIR and -t for in case .deb Files not wanted in source dir test compile for p3-1gb, install on aragog, fails because woody vs sarge fileutils have been replaced by coreutils, make-kpkg has outdated dependency compile again on clearwater (fastest sarge available), aborts openafs C error 2005.08.25 Thu senguen has installed fileutils package on aragog, kernel installed, works fetched newest kernel and openafs 1.2 archives, test with them, works compiling an .bz2 kernel fails, missing 2nd rev in bz2 vs gz test added SYS_TARGETS so that easy extension from just kernel_image restored aragog test system to stable state, removed first test, and other mess tidy up package naming scheme, seems kernel-image-(-) is fixed for modules use --(-), for dpkg -l 2005.08.26 Fri aragog installed new kernel and modules, works use supermount kernel for further tests, patched kernel new source archive sorts before old one, so wrong one is taken improved heuristics for source archive selection senguen what other modules wanted, only shfs, modularise that has existing debian/* files, so failled to make proper ones overwrite control and rules, extend changelog, leave copyright compiles now, but faills install, $(DESTDIR) missing despite in code and attempted directory to be made lacks -supermount docs added all fragment examples made so far 2005.09.01 Thu shfs package debian/config also lacks -supermount, same all other modules was kernel source Makefile, lacking EXTRAVERSION = -supermount in it $(DESTDIR) missing, is already in Makefile ROOT= not set, rules config fails is $(DESTDIR) not set, is set for other modules install, here used in config compiles now, but -supermount is still missing in module package names it is correctly in Depends: and /lib/modules/*/, changed mod name computing test run final generated stuff, p3-1gb aragog in use so run on norris, works test compile all 9 architectures, 9*7 .debs to sarge pkg server makesourcepackage -l, FAQ/INSTALL/index.html.en, makesourcepackage -p -ui, tar packaged version 20050901.143906-1, changelog: "initial release" 2005.09.02 Fri made dphys-kernel-cputype, for dphys-admin ``, derived from senguens hack determined what CPUs we actually have, in hosts/* found we have p3-smp-4gb p4-4gb and yesterday no compile for these, added made man page for dphys-kernel-cputype, mentioned it in d-k-p man page Makefile added all stuff for these 2 files repackaged -l, -p, woody+sarge -ui added exit 0 to both scripts, and to Logfile of all projects, as bugfix packaged version 20050902.142708-1, changelog: "added cputype detector helper script" 2005.09.08 Thu last weeks problems with not found packages seem to be gone aragog kernel-image-* visible but not installed dphys-kernel-packages visible and installed, mplayer-[codecs|plugin] also replaced CONF_MAINTAINER and CONF_EMAIL w CONF_MAINT_NAME and CONF_MAINT_EMAIL compatible naming with altered more readable makesourcepackage 2005.09.09 Fri add support for ICP Vortex driver, only requires CONFIG_SCSI_GDTH=y is already the case in heeb configs I used, so just test if it works tried on system, failled, wants to mount rd/0, use no root=/dev/rd/0 but was trying to use it as boot kernel, so no harddisk, needs rd and for that requires missing initrd and devfs, make kernel with those compiling without modules directory shows bug, test for none or empty compiled fails to mount, so try compiling with make-kpkg --initrd also set fuer senguen Liste auf welchen Hosts welche Kernel getestet werden koennen packaged version 20050909.135032-1, changelog: "bugfix no or empty modules dir abbort, CONF_ compatible with msp" 2005.09.15 Thu bugfix modules dir test -f -> -d senguen now compiled custom kernel with GDTH (and SCSI and BLK_DEV_SD), works SYS_TARGETS 2 vars, 1 as today for kernel, new also for modules packaged version 20050915.101204-1, changelog: "Bugfix test -f -> -d" 2005.09.22 Thu packaged version 20050922.112922-1, changelog: "bugfix read proper config file name, not outdated one" 2005.09.29 Thu dphys-kernel-cputype fails to recognize Sempron, added case with k7 as output 2005.09.30 Fri senguen tests, module compile fails, with name --*.deb, not -- is because no kernel-*.deb was produced, because problem with new nv driver add an test if kernel-image -deb was produced and FATAL abort if not so problem with make-kpkg packages, they ask questions while in dist-upgrade even worse these require "n" as answer, unlike all other stuff that wants "y" running apt-get dist-upgrade with nice yes n ist our of question fixing this where problem happens, in postinst ist impossible script is too complex (and reason for using make-kpkg) and no place to fix it while generating, nor replace it (loses entire reason for using make-kpkg) will need running separate kernel-image upgrade with ! line in dphys-admin replacement for missing "need-reboot" functionality, will also need extern fix because also a problem with postinst, update existing dphys-reboot each time packaged version 20050930.175657-1, changelog: "cputype recognize Sempron, packaging abort after broken kernel" 2005.10.06 Thu dphys-kernel-cputype repair so that it recognizes properly /proc/cpuinfo fails to show mult proc while running install non-smp kernel /usr/bin/free fails to show full memory while running install 1gb kernel for both of above use lshw, which always reports all, even unused stuff lshw outputs running comment to /dev/tty, no option to prevent, grrr lshw delivers mem size MB or GB output, no option for raw bytes, fix updated output unknown to unknown_cpu_model packaged version 20051006.172445-1, changelog: "bugfixes packages test valid kernel .deb, cputype kernel independant" 2005.10.07 Fri dphys-kernel-packages CONF_NAMETAG to allow setting extraversion-like prefix before compile type, to prevent user need of unpacking source packaged version 20051007.112516-1, changelog: "added name tags, after version and before variant" reduced size of short text in modules debian/control, variant visib in dpkg -l dphys-kernel-packages needs to set /root/need-reboot can not use dphys-reboot hack, because often only 1 kernel type needs reboot forking/fixing make-kpkg postinst is too much work (64k perl postinst) losing make-kpkg even more work, postinst install grub, reason for so large so per kernel variant autogenerated kernel-reboot-version-tag-variant package also in each of these dependency on kernel and all modules so only reboot if all packages installed, and also docs what was compiled so start building before modules, while modules add depends, then finalise packaged version 20051007.175455-1, changelog: "added variant specific kernel-reboot packages" 2005.10.13 Thu senguen dphys-kernel-packages forgot to delete debug output from changes, fixed senguen dphys-kernel-cputype fails to find memory size on one machine lshw broken there, no size: line, because a p1 machine? has capacity: instead of size:, size: is on most machines and some have both with size: right, so use size:, fallback to capacity: lshw also outputs running status direct to /dev/tty, not redirectable and also takes long, because scanning entire machine, despite -class alternative exists called hwinfo, also slow and output and memory is given as a line with multiple individual RAM parts, no sum packaged version 20051013.103743-1, changelog: "bugfix removed debug leftovers, cputype workaround lshw bug" senguen lshw has -disable to save time on unneccessary tests, so used them and 2>/dev/null can get rid of the animated action output, so used it packaged version 20051013.181334-1, changelog: "speed up lshw with -disable, remove animated output of lshw" 2005.10.14 Fri bugfix editor lossage, stray charakter inserted some time while yesterdays edit packaged version 20051014.155454-1, changelog: "bugfix editor lossage (stray charakter inserted)" 2005.10.20 Thu dphys-kernel-cputype fails to recognize Duron, added case with k7 as output packaged version 20051020.095858-1, changelog: "cputype recognize Duron" also fixed SMP detection, the -disable dmi lost the cpu count info used packaged version 20051020.131543-1, changelog: "fixed SMP detection" 2005.10.27 Thu man page and help updates analog to dphys3kernel 2005.12.22 Thu changed CMD_* stuff to same new system as in dphys3 2006.01.12 Thu heeb decided that 2.4.x troubles are mounting, and 2.6.x dropping time to start of looking into changing kernel versions 2006.01.13 Fr started looking into wghat needs to be done for changing to kernel 2.6 fetched 2.6.15 .bz2 from www.kernel.org to experiment with it surfed a bit experiences changing 2.4->2.6, needs totally new .config now only make *config make make *install like with other programs make dep only error message, make bzImage and make modules still work requires udev at least vers 071, sarge has only 056, sid 072 to 079, rest OK sarge has outdated 2.6.8 kernel images, they most likely worked with 056 kernel config produced und compiled, test on hagrid, works except modules forgot copying them, later complains already loaded, compiled in stuff? checked which drivers should be modules, are, likely hotplug fetched them new official named p1 instead of k7, full install, complains, but works complains only piix and Buslogic, musst both be compiled in, for booting test .config extend with all stuff we have in todays production 2.4 2006.01.19 Thu got 2.4 config from senguen, new dir for KERNEL2.6, new 2.6.15.1 from him revise config for 2.6, all we had in 2.4, partially correct it 2006.01.20 Fr further revising config for 2.6, missing 2.4er stuff, compiled, works on hagrid also checked with pomfrey lsmod listing, lacks vmware and supermount, exter config change from generic p1 to pIII, diffed .config, has new fields same for p4, k7, -smp, 4gb, 64gb, also vary, in particular -smp a lot so dphys-kernel-packages requires fixing up config processor dependant dphys-kernel-packages added code to identify if 2.4 or 2.6 kernel being built testing debug code found bug in -D option, shift 1 must be after settings 2006.01.26 Thu difference non-smp -> smp for k7 is identical as for p1, most likely constant so can like in 2.4 do separate cpu and smp and memory sections tidied up config fixing code, so that adding 2.4/2.6 does not produce an mess produced new 2.6 code, added DEBUG_DUMP_CONFIG like in dphys3kernel compiled p1-1gb and k7-1gb, work on hagrid compile k7-4g with modules (NVIDIA and vmware) for menhit fails in NVIDIA conftest.sh for rivafb, test defective, fails make-kpkg paths also faills vmware, because .o -> .ko name, dirs different, rules don't fit compiled an 2.4 to compile vmware dir layout, to make common rules file all upgraded drives tried zo make multiple versions, move old to .old dir 2006.01.27 Fr test run with vmware 2.6-only rules and newest nvida driver vmware 2.6-only works, but newest downloaded nvidia driver still fails made vmware combined version, look into how to fix nvidias broken test fixed rivafb and nvidiafb tests, now fails to find patchlevel seems to be something seriously broken with make-kpkg build environment 2006.02.16 Thu further attempts to fix nvidias broken external modules stuff replace make with one without target name, as noticed in new versions README now no test complaints, but complains of no nvidia.o file next test without any conftest.sh modifications, same result duplicated make call without recompile, debian/rules is running ok is nvidia Makefile which is simply doing nothing, not compiling anything all: install install: package-install package-install: module-install rmmod-sanity-check module-install: suser-sanity-check module module: $(TGT) # TGT = stprof or TGT = mdl mdl: gcc-sanity-check rivafb-sanity-check nvidiafb-sanity-check make "CC=$(CC)" $(KBUILD_PARAMS) modules KBUILD_PARAMS contains -C $(KERNEL_SOURCES) SUBDIRS=$(PWD) then runs linux Makefile with SUBDIRS= to module there KBUILD_EXTMOD ?= $(SUBDIRS) in end should produce nvidia.ko (after finding what in makefile???) was using auto-updated 2.6.15.4, so fall back to 2.6.15.1, to compare doing so fetched .gz archive, dphys-kernel-packages fails on not .bz2, fix added set -e while doing this fix same result as 2.6.15.4, but repeating make by hand old conftest.sh bugs 2006.02.17 Fri unpack nvidia software newly, retry make, still fails seems that first make (by kernel compile) modifies something in linux re-make-ing after killing marker then fails with can not determine type despite the type test (as left after first make) should work, broken heeb discussion about trouble here, do we need this driver decided that we could produce an own kernel patch from nvidia source but nvidia is only test module, card in 2 of 150 machines (same 1 ati) becuase traditional matrox fails to work newer machines, look replacement so for 2.6 upgrade irrelevant, after that test both, working becomes policy drop further investigations here, get on with 2.6 upgrade stuff packaged version 20060217.133822-1, changelog: "added kernel 2.6 support" 2006.04.06 Thu extend to support separate K8 kernel packages dphys-kernel-cputype simple enable the commented out lines dphys-kernel-packages add config modes, look what is needed for them 2.6 works, but 2.4 leaves as k7, use changes analog to 2.6 fixed missing test for if modules directory defined (syntax error if not) 2.4 and 2.6 both compile, give to senguen for test updated compile list to include k8, remove unused false k7 combinations packaged version 20060406.144629-1, changelog: "added k8 support, make full kernel image+source+headers" 2006.04.27 Thu added support in dphys-kernel-cputype for Celeron, allways p4, like Xeon also packaged version 20060427.123553-1, changelog: "added Celeron support" 2006.06.01 Thu senguen reports that 2 hosts failled on dphys-kernel-cputype memory size are ones with missing size:, fallback onto capacity: capacity: seems to be max pluggable, not what is actually there but also are hosts with multiple *-memory sections, because of flash use always the first one, hope this is always main memory packaged version 20060601.153003-1, changelog: "bugfix when lshw muliple *-memory sections (main and flash)" 2006.08.11 Fri added all "packaged version ????????.??????-?, changelog:" lines 2006.10.19 Thu rework for separate upstream and Debian maintainer, only .tar.gz release reworked and updated documentation, FAQ, INSTALL, README, index.html.en remove FAQ and INSTALL and index.html.en references to my old packages index.html.en added existance of dphys-kernel-cputype and its man page remove old debian/ so that senguen can make proper debianisation debian/changelog is all info duplicated in this Logfile, deleted it debian/control is just wget depends and docu text, add to other docs debian/copyright is same as in README, deleted it debian/dphys-config.docs is nothing, deleted it debian/rules has nothing specific, deleted it updated man pages, reworked text set new systematic PATH=, removed all absolute paths in commands uses ./${PNAME} config file, can collide with install as ./${PNAME} put in an test if recursive call to self, else endless recursion checking for cut -f test that can fail if separator missing while doing so added test if kernel is 2.4 or 2.6 also test if demanded variant is malformed (no - in it) adding "${var}" quoting, need better way, make automated quoting check also apply this to dphys-config and dphys-swapfile 2006.10.20 Fri adding "${var}" quoting, and echo "some test" and ${CMD_..} "some text" also VAR="" and [] tests with "${}" instead of x${} lshw "${SPEEDUP}" with SPEEDUP= multiple options fails, without "" same make-kpkg "${SYS_USEINITRD}" and "${SYS_K_TARGETS}", without "" dphys-kernel-cputype simpler GB->MB conversion arithmetic no shell functions, so no keyword function to remove no echo -e, so no need to replace with printf test against recursive call has to be 2 separate ifs, grep inside [ ] fails may be just quoting, but simpler to just have 2 separate ifs make-kpkg now runs, compiles, but then no .deb generated tried adding debug options, now silently dies after .config diff 2006.10.26 Thu is the return value of diff that kills set -e, secure it with || true, now runs still message that no kernel generated, despite compile done .deb file is actually there, is test for file that fails is due to adding "", giving VARKERNELDEB="${VARWORKDIR}"/kernel-image-*.deb can't find anything fitting kernel-image-*.deb like this put in an `echo ...`, like the `ls ...` in all other 4 "${VAR}"/name-*.ext in all of those the ls is needed, to get one filename per line in this last/5th case echo is good enough, as only 1 file there while at making fixes, also defanged quoting bugs in the 2 for statements one variable of config statements, like preproc list in dphys-config, do same other just an ${MODULESDIR}/*.tar.*, simple "${MODULESDIR}"/*.tar.* is OK put in test if non-base-install commands are here, else abort with clear error test for fakeroot make-kpkg make dh_* also add remarks to README FAQ INSTALL index.html.en and the man page improved also man page docs on how to use dphys-kernel-cputype dphys-kernel-cputype man page added remarks about needing lshw also in FAQ INSTALL README dphys-kernel-cputype.1 deleted old test data, 2 .config files for 2.4 and 2.6, host lists examples newer module .rules, add .config files, additional to 2.4 also 2.6 because of this moved from rules.examples to 2.[46].example also added the 2 2.[46].example direcrories to index.html.en 2006.10.27 Fri after all bugfixes and changes new release updated FAQ, INSTALL, README, index.html.en copy to ../dphys-kernel-packages-20061027, made tar file of that remade ../dphys-kernel-packages.tar.gz archive for current state in Debian etch kernel-image-*.deb -> linux-image-*.deb, allow both names repackaged with same release number ------ project status doing: todo: possible extensions: