Build system fixes

This fixes #418 (closed), but it also does much more:

• remove the obsolete -a, -e and -w options

• add a new generic -t / -T build target option which automatically selects the proper type of package for the host platform

• add a new -k option to bypass the cleaning stage (this reduces the amount of unnecessary recompiles, especially while testing changes, but it's not perfect yet, see below)

• make tar_em_up.sh properly catch errors during the build and provide useful diagnostics; this finally lets us detect and diagnose error conditions during a build in a reliable fashion, especially in CI builds

• fix a lot of nagging little bugs in the build scripts and Makefiles which could make a build fail for mysterious (and stupid) reasons

• make INCREMENTAL (Gem-less) builds work on Windows

• add a toplevel Makefile so that you can now just run make, make incremental, make clean, make realclean, etc. and be done with it (see the comments at the beginning of the Makefile for details)

The latter alone should be a heaven-sent for new users who don't have to deal with the arcane options of the build script any more, but can now just run familiar make commands.

Other than that, the overhauled tar_em_up.sh should be 100% backward-compatible (except for the obsolete options that I removed).

Remaining Issues

The -k option is not perfect yet, since the Makefiles under packages and externals want to recompile a lot of stuff anyway, even if preserving all artifacts from a previous build. I'm not sure why that is, we'll have to take another deep look into the hall of Makefile mirrors to see what is going on there, but that's for another time. ;-) In any case, the amount of unconditional recompilation seems tolerable now (except maybe on Windows, but that's probably because Windows always takes 90% of the compile time and produces 90% of the mysterious issues anyway).

How To Test

After a fresh checkout, the following sequence of commands should result in three valid builds, without any failures (replace make with make incremental to save a lot of compile time):

make
make
make clean
make
make realclean
make

I tested this as good as I could, although I eventually had to give up on Windows, because in a VM even an incremental build takes an eternity there. But I did check that incremental builds work on Windows, and the CI should tell us about the full build. I also ran my existing Arch PKGBUILD on this branch and it works without any hitches. I'll try the debuild later tonight. Again, CI will tell us about a direct Debian build using tar_em_up.sh -B. (I don't expect any further issues there, because I'm also using -B in the Arch build, which does everything but the Debian packaging itself, and that part hasn't changed at all.)

mentioned in issue #418 (closed)

Note that the test witness above should really be executed in a pristine working copy. Otherwise, when using an existing wc, you may have build artifacts lying around from "bad ol' tar_em_up", some of which may even be invisible to git status, which may break the initial build -- use make realclean before the first make in that case.

Sounds great.

There's some problem with the CI atm. Once I get that sorted out I'll merge this.

In the meantime-- is it possible to change "clean" and "realclean" targets to a choice of words that doesn't imply the first one lied to me? :)

They're customary (GNU and automake standard I believe), ingrained in many developers' muscle memories, so I wouldn't want to change them. But if it's really a big issue for you, then we might call the second one distclean if that seems more honest. ;-)

But stay tuned, I still need to fix one minor issue in the make clean target and another one in debuild, that shouldn't take long.

added 1 commit

• 13b06b4f - Fix in make clean target.

Compare with previous version

added 1 commit

• dfb3a0f8 - Fix in make clean target.

Compare with previous version

Concerning the naming, I really think that make clean should always be available, and it should be the least brutal clean option. Anything else would just confuse people. This is just an approximation anyway, we can't do any better right now without putting in way more effort.

added 1 commit

• bd3eeb8e - Update debuild

Compare with previous version

added 1 commit

• 361f65b8 - Update debuild

Compare with previous version

Ok, should be good now. I'm still trying a debuild on Ubuntu, this will take a little while. I already tested a straight Debian build (tar_em_up.sh -b a.k.a. make incremental) and that works fine on Ubuntu, too. Looking good!

debuild is still broken. :( Anyway, that's surely a minor issue and won't affect the CI, so I can fix that in a separate PR. I've already wasted too much time on this anyway (need to finish my LAC paper), I call it a day for now.

added 1 commit

• ce6e3b76 - debuild burps on non-existing extensions subdir in make distclean.

Compare with previous version

I think that the debuild error is a symptom of something going awry in tar_em_up.sh -B, so I'm putting this on hold until I've sorted that out. Sorry, I was testing this stuff the whole day, but apparently still not well enough.

marked as a Work In Progress

changed the description

I understand-- this stuff is hard to fix, and slow to test.

The specific problem with the debuild is that the entire source must be in a self-contained tarball. I can't just upload the entire git repo to Launchpad and build from that -- there's no Internet connection during the build. So everything has to be in the tarball, including all submodules and the files like s_stuff.h that can only be generated in a wc of the git repo. This makes the build fail in slightly different ways.

Well, at least the new build script now tells me exactly where it fails and why. :-)

This shouldn't be hard to fix. BTW, we then also have a way to create a static, self-contained snapshot tarball of the source, so that we can add a make dist target to the toplevel Makefile.

Thinking about it some more, maybe I could use a shallow clone for the snapshot tarball, with the submodules all checked out already. I'll have to give that a try.

added 8 commits

• ff9e9a09 - Make things work when building in a static snapshot of the repo (not git).
• df7f0838 - Add a dist target to the toplevel Makefile.
• f7238f54 - Remove patch to packages Makefile from debuild, it's not needed any more.
• f82d778a - Cosmetic and comment changes
• 5097a10c - Give some feedback about where the build products are when skipping Debian…
• 2c74c1f8 - Fix up realclean target so that it also removes files in .gitignore.
• 71d0ab90 - Add a comment about the new toplevel Makefile to the build instructions.
• cd89104a - debuild: update debhelper version and remove obsolete patches

Compare with previous version

added 1 commit

• f027f4a5 - Formatting glitch

Compare with previous version