The stage4 target, specifically the STAGE4_TARBALL target now depends on
the portage/ directory, the appliances/<appliance>/ directory and the
stage3 tarball. It will not be rebuild if these are up-to-date. Therefore
other targets that depend on stage4 (e.g. RAW_IMAGE) won't force stage4 to
be rebuilt if it is up to date.
If you want to force a stage4 rebuild, for example if the appliance pulls
from a live VCS, then the recommended way to do this is:
$ touch appliances/<appliance>
$ make APPLIANCE=<appliance> stage4
Instead of "image" so that if the raw image is built and up-to-date then it
need not be rebuilt to satisfy the other image targets. This excludes the
stage4 "image" which of course is necessary to build the $(RAW_IMAGE)
target.
The raw image making was kind of split up, with different targets to create
the image, partition, copy files, install grub, etc. This change makes it
all one target. After "make image" is done, or, e.g., "make images/base.img"
then the resulting file will be built and contain all the necessary files.
This make the $(RAW_IMAGE) target more uniform with the other image/
targets. It will also make the other image/ targets easier to satify.
The new (2015-02) stage3 tarballs use package.* directories instead of
files, causing the files to fail to be copied over. This change instead
copies them to the directories, creating them if they don't already exist.
This change converts the appliances from openrc to systemd. In addition,
systemd needs to at least be be installed on the build machine as we now
use systemd-nspawn instead of chroot/bind mounts do to the fact that
systemd-nspawn is much more robust and lese prone to errors. In addition:
* All stage4 tarballs and images are now stored in the images/ directory by
default.
* A package dir is now required and is set up in the packages/ directory by
default.
* The portage directory is not unpacked in the chroot every time anymore
and is instead bind-mounted from a directory by systemd-nspawn. This is
the portage/ directory by default.
* In addition there is a distfiles/ directory (by default) which is
bind-mounted in the chroot. It does what you'd probably expect.
* The Makefile learned new target, eclean, which will run eclean-dist and
eclean-pkg to clean out obsolete distfiles and binary packages.
* The REMOVE_PORTAGE_TREE option is removed as portage is no longer
installed.
* The unused/untested PRUNE_CRITICAL functionality has been removed. I
have no idea if it worked and if it didn't I didn't want to fix it.
* As appliances are always using systemd, UDEV is no longer optional.
* Timezones and hostnames are now handled by systemd (systemd-firstboot).
* Stage4 tarballs are now directly importable as systemd containers (and
probably docker too though I haven't tested that yet).
* A number of packages (for example dhcpd, acpid) have been removed from the
appliances when systemd provides equivalent functionality.
* We no longer override etc-update.conf. There's no reason.
* A number of Makefile targets, checkpoints, temp files, etc have been
removed as they are no longer needed.
The reason being that when you are distributing an image you are likely
distributing an general appliance, not a specific host so it makes sense
that the image name be the name of the appliance and not, e.g., ln33. This
can always be overriden on the command line or in a profile.
Instead of having everything in the root directory, split them in seperate
directories (appliances, scripts, & configs). This makes things a little
tidier.
Also added a now Makefile target, appliance-list, that prints a list of the
available appliances. The split directory change made this easier.
The current method no longer works because the text file now contains
non-stage3 files, etc. Eventually this needs to be cleaned up a little
better (using a helper script), but for now we add some extra greps.
emerge will only install the same kernel.
This (and other) logic really needs to be moved out of Makefile and into a
shell script where it can be better maintained and not look so ugly.
don't have the kernel sources and subsequent builds will fail
* kernel: Remove any old kernel and modules before installing new
* Don't link /bin/sh to dash if DASH != YES
* Add appliance name and creation date comments to package list files
appliance with the list of installed packages on that appliance. PKGLIST
is off by default.
* Works with baselayout-2 stage3s, including the fix for the current broken
stage3 tarballs.
* The new stage3 tarballs are much more limited as far as what dev nodes
are included. In addition to being broken stage3s themselves, it also
broke the UDEV=NO option. So I have included my own device node tarball
which is installed when UDEV=NO.
* Put double quotes around hostname in /etc/conf.d/hostname.
The first major change here is the support of baselayout-2/openrc. So far
Gentoo has not released a stage3 tarball with baselayout-2, but I've tested
this enough that I feel it works even when doing an upgrade. The
baselayout-1 stuff has been converted, however it is recommended to not use
this until baselayout-2 is available in Gentoo stage3s as I don't want to
support converting, or you can do this:
stage4 tarball support. I'm not sure why I didn't support this before.
Now not only can stage4 tarballs be build, instead of or in addition to
virtual appliance images, but stage4 tarballs can also be used in lieu of a
stage3. There is a new "stage4" make target, and also when building the
Makefile will first look for stage4/<appliance>-stage4.tar.bz2 and if that
exists use it instead of the Gentoo stage3. If you specify stage4 as a
target and also already have a stage4, then the Makefile will unpack the
stage4 into the chroot, perform updates, and then re-create the stage4
based on the updates (atomically). In addition there is a new Makefile
variable, SOFTWARE. The default value is "1" meaning it will call the
software target (thereby installing/updatein software), but if you set
SOFTWARE=0, then the software phase will not be run. This can allow, for
example, to build a VM image from a stage4 without performing software
updates, in effect a straight stage4 to image for quick image building.
Since most appliances are pretty much based on the "base" appliance, one
could imply copy the base stage4 to the new appliance stage4 and start from
there, instead of having to start from a vanilla stage3, for example:
# cp stage4/base-stage4.tar.bz2 stage4/kde-stage4.tar.bz2
# make APPLIANCE=kde stage4
Will take the already existing base stage4 and simply update it to kde,
which will be easier/faster than starting from a stage3.
There are still some tweaks that need to be done for the stage4 support.
Specifically Makefile targets need to be written with the assumption that
they may be called more than once (e.g. updating a stage4 or going from
stage4 to image with SOFTWARE=1). For the most part things work according
to that assumption, but there are a few things that need to be tweeked,
such as the inittab settings if one is building a "headed" image but it's
based off a headless stage4. New appliance authors should also write their
"preinstall" and "postinstall" targets with the assmption that it may be
called more than once.
* Added UDEV Makefile variable, if set to NO, will disable UDEV and use
static /dev (defaults to YES)
* Run eselct python to get around "which python am i?" situation
* export "inroot" variable
If set to YES, then the kernel sources will not be installed and a kernel
is not built, nor is grub installed/configured. This is can be useful when
building appliances for hypervisors such as Xen and Qemu/KVM which can run
from external kernel/initramfs configurations. Default is NO
So, basically I re-architeched things a bit:
The appliance/Makefile.inc fiels are now appliance/Makefile (again). The
main Makefile will call "make -C appliance preinstall" and "postinstall"
(and in future "clean"). So I got rid of the ugly make variables/include
thing. Some of the main Makefile's variables are exported to the
sub-makes. Appliances don't really need $(APPLIANCE) anymore as the
appliance directory is their CWD.
Added some new targets and smarter targets. I can do more with this, but
it's a big improvment from last time. Still learning a lot of Makefile
magic (been reading other people's Makefiles).
Verified that "make -j3" works (at least on the base appliance) but will
kill your hard drive :D
Introduced "profiles" Which are files with variables you want to override.
The file will be "include"ed by the main Makefile. For example, I have a
file, "local.cfg" that looks like this:
--- 8< -----------------------------
CHROOT = /var/scratch/marduk/vabuild
HEADLESS = YES
PRUNE_CRITICAL = NO
VIRTIO = YES
TIMEZONE = EST5EDT
DISK_SIZE = 60.0G
SWAP_SIZE = 48
PKGDIR = /var/scratch/packages
NBD_DEV = /dev/nbd8
all: qcow
--- 8< ------------------------------
Then, e.g. i can run "make PROFILE=local APPLIANCE=kde". If you don't
specify a PROFILE variable, then it will default to the empty string, which
means the main Makefile will attempt to include .cfg
So, for example i have:
$ ln -s local.cfg .cfg
$ make APPLIANCE=kde
Don't set PROFILE inside your .cfg file (why would you?). Also, if the
[pro]file does not exist, the include fails silently.
I will put this info in the wiki eventually...
Albert Hopkins